From david at ar.media.kyoto-u.ac.jp  Sat Dec  1 03:26:50 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 01 Dec 2007 17:26:50 +0900
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <4750981C.9060409@mspacek.mm.st>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>
	<4750981C.9060409@mspacek.mm.st>
Message-ID: <47511ACA.9040501@ar.media.kyoto-u.ac.jp>

Martin Spacek wrote:
> Kurt Smith wrote:
>  > You might try numpy.memmap -- others have had success with it for
>  > large files (32 bit should be able to handle a 1.3 GB file, AFAIK).
>
> Yeah, I looked into numpy.memmap. Two issues with that. I need to 
> eliminate as much disk access as possible while my app is running. I'm 
> displaying stimuli on a screen at 200Hz, so I have up to 5ms for each 
> movie frame to load before it's too late and it drops a frame. I'm sort 
> of faking a realtime OS on windows by setting the process priority 
> really high. Disk access in the middle of that causes frames to drop. So 
> I need to load the whole file into physical RAM, although it need not be 
> contiguous. memmap doesn't do that, it loads on the fly as you index 
> into the array, which drops frames, so that doesn't work for me.
If you want to do it 'properly', it will be difficult, specially in 
python, specially on windows. This looks really similar to the problem 
of direct to disk recording, that is you record audio signals from the 
soundcard into the hard-drive (think recording a concert), and the 
proper design, at least on linux and mac os X, is to have several 
threads, one for the IO, one for any computation you may want to do 
which do not block on any condition, etc... and use special OS 
facilities (FIFO scheduling, lock pages into physical ram, etc...) as 
well as some special construct (lock-free ring buffers). This design 
works relatively well for musical applications, where the data has the 
same order of magnitude than what you are talking about, and the same 
kind of latency order (a few ms).

This may be overkill for your application, though.
>
> The 2nd problem I had with memmap was that I was getting a WindowsError 
> related to memory:
>
>  >>> data = np.memmap(1.3GBfname, dtype=np.uint8, mode='r')
>
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "C:\bin\Python25\Lib\site-packages\numpy\core\memmap.py", line 
> 67, in __new__
>      mm = mmap.mmap(fid.fileno(), bytes, access=acc)
> WindowsError: [Error 8] Not enough storage is available to process this 
> command
>
>
> This was for the same 1.3GB file. This is different from previous memory 
> errors I mentioned. I don't get this on ubuntu. I can memmap a file up 
> to 2GB on ubuntu no problem, but any larger than that and I get this:
>
>  >>> data = np.memmap(2.1GBfname, dtype=np.uint8, mode='r')
>
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "/usr/lib/python2.5/site-packages/numpy/core/memmap.py", line 
> 67, in __new__
>      mm = mmap.mmap(fid.fileno(), bytes, access=acc)
> OverflowError: cannot fit 'long' into an index-sized integer
>
> The OverflowError is on the bytes argument. If I try doing the mmap.mmap 
> directly in Python, I get the same error. So I guess it's due to me 
> running 32bit ubuntu.
Yes. 32 bits means several things in this context: you have 32 bits for 
the virtual address space, but part of it is reserved for the kernel: 
this is configurable on linux, and there is a switch for this on 
windows. By default, on windows, it is split into half: 2 Gb for the 
kernel, 2 Gb for userspace. On linux, it depends on the distribution(on 
ubuntu, it seems that the default is 3G for user space, 1 G for kernel 
space, by reading the build config). I think part of the problem (memory 
error) is related to this, at least on windows.

But the error you get above is easier to understand: an integer is 32 
bits, but since it is signed, you cannot address more than 2^31 
different locations with an integer. That's why with standard (ansi C 
stlib) functions related to files, you cannot access more than 2Gb; you 
need special api for that. In your case, because you cannot code your 
value with a signed 32 bits integer, you get this error, I guess 
(index-sized integer means signed integer, I guess). But even if it 
succeeded, you would be caught by the above problem (if you only have 2 
Gb user space for virtual adressing, I don't think you can do a mmap 
with a size which is more than that, since the whole mapping is done at 
once; I am not so knowledgeable about OS, though, so I may be totally 
wrong on this).

cheers,

David


From haase at msg.ucsf.edu  Sat Dec  1 05:24:50 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Sat, 1 Dec 2007 11:24:50 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <4750981C.9060409@mspacek.mm.st>
References: <474FCE2D.2000908@mspacek.mm.st>
	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>
	<4750981C.9060409@mspacek.mm.st>
Message-ID: <bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>

On Dec 1, 2007 12:09 AM, Martin Spacek <numpy at mspacek.mm.st> wrote:
> Kurt Smith wrote:
>  > You might try numpy.memmap -- others have had success with it for
>  > large files (32 bit should be able to handle a 1.3 GB file, AFAIK).
>
> Yeah, I looked into numpy.memmap. Two issues with that. I need to
> eliminate as much disk access as possible while my app is running. I'm
> displaying stimuli on a screen at 200Hz, so I have up to 5ms for each
> movie frame to load before it's too late and it drops a frame. I'm sort
> of faking a realtime OS on windows by setting the process priority
> really high. Disk access in the middle of that causes frames to drop. So
> I need to load the whole file into physical RAM, although it need not be
> contiguous. memmap doesn't do that, it loads on the fly as you index
> into the array, which drops frames, so that doesn't work for me.
>
> The 2nd problem I had with memmap was that I was getting a WindowsError
> related to memory:
>
>  >>> data = np.memmap(1.3GBfname, dtype=np.uint8, mode='r')
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "C:\bin\Python25\Lib\site-packages\numpy\core\memmap.py", line
> 67, in __new__
>     mm = mmap.mmap(fid.fileno(), bytes, access=acc)
> WindowsError: [Error 8] Not enough storage is available to process this
> command
>
>
> This was for the same 1.3GB file. This is different from previous memory
> errors I mentioned. I don't get this on ubuntu. I can memmap a file up
> to 2GB on ubuntu no problem, but any larger than that and I get this:
>
>  >>> data = np.memmap(2.1GBfname, dtype=np.uint8, mode='r')
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python2.5/site-packages/numpy/core/memmap.py", line
> 67, in __new__
>     mm = mmap.mmap(fid.fileno(), bytes, access=acc)
> OverflowError: cannot fit 'long' into an index-sized integer
>
> The OverflowError is on the bytes argument. If I try doing the mmap.mmap
> directly in Python, I get the same error. So I guess it's due to me
> running 32bit ubuntu.
>
Hi,
reading this thread I have two comments.
a) *Displaying* at 200Hz probably makes little sense, since humans
would only see about max. of 30Hz (aka video frame rate).
Consequently you would want to separate your data frame rate, that (as
I understand) you want to save data to disk and - asynchrounously -
"display as many frames as you can" (I have used pyOpenGL for this
with great satisfaction)
b) To my knowledge, any OS Linux, Windows an OSX can max. allocate
about 1GB of data - assuming you have a 32 bit machine.
The actual numbers I measured varied  from about 700MB to maybe 1.3GB.
In other words, you would be right at the limit.
(For 64bit, you would have to make sure ALL parts are 64bit, e.g. The
python version must be >=2.5, python must have been compiled using a
64-bit compiler *and* the windows version (XP-64))
This holds true the same for physical ram allocation and for memmap
allocation.
My solution to this was to "wait" for the 64bit .... not tested yet ;-)

Cheers,
Sebastian Haase


From ivilata at carabos.com  Sat Dec  1 06:57:57 2007
From: ivilata at carabos.com (Ivan Vilata i Balaguer)
Date: Sat, 1 Dec 2007 12:57:57 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <20071130181938.GA6080@tardis.terramar.selidor.net>
References: <474FCE2D.2000908@mspacek.mm.st>
	<20071130181938.GA6080@tardis.terramar.selidor.net>
Message-ID: <20071201115757.GA12618@tardis.terramar.selidor.net>

Ivan Vilata i Balaguer (el 2007-11-30 a les 19:19:38 +0100) va dir::

> Well, one thing you could do is dump your data into a PyTables_
> ``CArray`` dataset, which you may afterwards access as if its was a
> NumPy array to get slices which are actually NumPy arrays.  PyTables
> datasets have no problem in working with datasets exceeding memory size.
>[...]

I've put together the simple script I've attached which dumps a binary
file into a PyTables' ``CArray`` or loads it to measure the time taken
to load each frame.  I've run it on my laptop, which has a not very fast
4200 RPM laptop hard disk, and I've reached average times of 16 ms per
frame, after dropping caches with::

    # sync && echo 1 > /proc/sys/vm/drop_caches

This I've done with the standard chunkshape and no compression.  Your
data may lean itself very well to bigger chunkshapes and compression,
which should lower access times even further.  Since (as David pointed
out) 200 Hz may be a little exaggerated for human eye, loading
individual frames from disk may prove more than enough for your problem.

HTH,

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       C?rabos Coop. V.  V  V   Enjoy Data
	                          ""
-------------- next part --------------
A non-text attachment was scrubbed...
Name: frames.py
Type: text/x-python
Size: 2664 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071201/3240371a/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 307 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071201/3240371a/attachment.sig>

From grh at mur.at  Sat Dec  1 11:21:25 2007
From: grh at mur.at (Georg Holzmann)
Date: Sat, 01 Dec 2007 17:21:25 +0100
Subject: [Numpy-discussion] swig numpy2carray converters
In-Reply-To: <9AC12CB0-D618-4322-A740-CD5C1EAD5AC8@sandia.gov>
References: <473DFCC2.1050704@mur.at>
	<47421469.3050408@noaa.gov>	<474296D8.9000405@mur.at>	<AE20F876-B9EF-4914-9C87-895B28DA7C24@sandia.gov>	<4742EE16.8030307@mur.at>	<8710782A-E850-405C-8517-07F8F025E696@sandia.gov>	<474345DF.8040202@mur.at>	<91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov>	<474465EE.20305@mur.at>	<538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov>	<474B4E34.80900@noaa.gov>
	<9AC12CB0-D618-4322-A740-CD5C1EAD5AC8@sandia.gov>
Message-ID: <47518A05.9070709@mur.at>

Hallo!

> * A new ARGOUTVIEW suite of typemaps is provided that allows your  
> wrapped function
>    to provide a pointer to internal data and that returns a numpy  
> array encapsulating
>    it.

Thanks for integrating it !


> * New typemaps are provided that correctly handle FORTRAN ordered 2D  
> and 3D arrays.

I have some problem with your FORTRAN implementation.
The problem is how you set the flags in numpy.i "int 
require_fortran(PyArrayObject* ary)" (~ line 402).

You do it like this:
ary->flags = ary->flags | NPY_F_CONTIGUOUS;
which does not work (at least on my computer) - I still get usual 
C-ordered arrays returned.

However, it does work if you set the flags like this:
ary->flags = NPY_FARRAY;


> Tests for the ARGOUTVIEW and FORTRAN ordered arrays have also been  
> added, and the documentation (doc/numpy_swig.*) has been updated to  
> reflect all of these changes.

A small typo: in the docs you also write about 1D FORTRAN ARGOUTVIEW 
wrappers:

( DATA_TYPE** ARGOUTVIEW_FARRAY1, DIM_TYPE* DIM1 )
( DIM_TYPE* DIM1, DATA_TYPE** ARGOUTVIEW_FARRAY1 )

which of course do not exist.


LG
Georg


From meine at informatik.uni-hamburg.de  Sat Dec  1 12:44:08 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Sat, 1 Dec 2007 18:44:08 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <4750981C.9060409@mspacek.mm.st>
References: <474FCE2D.2000908@mspacek.mm.st>
	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>
	<4750981C.9060409@mspacek.mm.st>
Message-ID: <200712011844.13168.meine@informatik.uni-hamburg.de>

On Samstag 01 Dezember 2007, Martin Spacek wrote:
> Kurt Smith wrote:
>  > You might try numpy.memmap -- others have had success with it for
>  > large files (32 bit should be able to handle a 1.3 GB file, AFAIK).
>
> Yeah, I looked into numpy.memmap. Two issues with that. I need to
> eliminate as much disk access as possible while my app is running. I'm
> displaying stimuli on a screen at 200Hz, so I have up to 5ms for each
> movie frame to load before it's too late and it drops a frame. I'm sort
> of faking a realtime OS on windows by setting the process priority
> really high. Disk access in the middle of that causes frames to drop. So
> I need to load the whole file into physical RAM, although it need not be
> contiguous. memmap doesn't do that, it loads on the fly as you index
> into the array, which drops frames, so that doesn't work for me.

Sounds as if using memmap and then copying each frame into a separate 
in-memory ndarray could help?

Ciao, /  /                                                    .o.
     /--/                                                     ..o
    /  / ANS                                                  ooo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071201/d3cdb96c/attachment.sig>

From meine at informatik.uni-hamburg.de  Sat Dec  1 12:49:04 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Sat, 1 Dec 2007 18:49:04 +0100
Subject: [Numpy-discussion] display numpy array as image (Giorgio F.
	Gilestro)
In-Reply-To: <1196452128.6047.374.camel@glup.physics.ucf.edu>
References: <1196452128.6047.374.camel@glup.physics.ucf.edu>
Message-ID: <200712011849.04512.meine@informatik.uni-hamburg.de>

On Freitag 30 November 2007, Joe Harrington wrote:
> I was misinformed about the status of numdisplay's pages.  The package
> is available as both part of stsci_python and independently, and its
> (up-to-date) home page is here:
>
> http://stsdas.stsci.edu/numdisplay/

I had a look at ds9/numdisplay, and as a summary, I found a nice viewer for 
scalar images with colormap support, and x/y projections.

What I missed though is the display of RGB images.  It looks as if ds9 was 
capable of doing so (I could add "frames" of RGB type), but I did not find a 
way to feed them with data.  numdisplay.display(myarray) seems to only 
support 2D-arrays.

Ciao, /  /                                                    .o.
     /--/                                                     ..o
    /  / ANS                                                  ooo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071201/15192673/attachment.sig>

From wfspotz at sandia.gov  Sat Dec  1 13:13:03 2007
From: wfspotz at sandia.gov (Bill Spotz)
Date: Sat, 1 Dec 2007 11:13:03 -0700
Subject: [Numpy-discussion] [Swig-user]  swig numpy2carray converters
In-Reply-To: <47518A05.9070709@mur.at>
References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov>
	<474296D8.9000405@mur.at>
	<AE20F876-B9EF-4914-9C87-895B28DA7C24@sandia.gov>
	<4742EE16.8030307@mur.at>
	<8710782A-E850-405C-8517-07F8F025E696@sandia.gov>
	<474345DF.8040202@mur.at>
	<91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov>
	<474465EE.20305@mur.at>
	<538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov>
	<474B4E34.80900@noaa.gov>
	<9AC12CB0-D618-4322-A740-CD5C1EAD5AC8@sandia.gov>
	<47518A05.9070709@mur.at>
Message-ID: <2951E6C7-9A85-4BC1-B989-F69D12306A9A@sandia.gov>

These corrections have been committed.

Thanks.

On Dec 1, 2007, at 9:21 AM, Georg Holzmann wrote:

>> * A new ARGOUTVIEW suite of typemaps is provided that allows your
>> wrapped function
>>    to provide a pointer to internal data and that returns a numpy
>> array encapsulating
>>    it.
>
> Thanks for integrating it !
>
>> * New typemaps are provided that correctly handle FORTRAN ordered 2D
>> and 3D arrays.
>
> I have some problem with your FORTRAN implementation.
> The problem is how you set the flags in numpy.i "int
> require_fortran(PyArrayObject* ary)" (~ line 402).
>
> You do it like this:
> ary->flags = ary->flags | NPY_F_CONTIGUOUS;
> which does not work (at least on my computer) - I still get usual
> C-ordered arrays returned.
>
> However, it does work if you set the flags like this:
> ary->flags = NPY_FARRAY;
>
>> Tests for the ARGOUTVIEW and FORTRAN ordered arrays have also been
>> added, and the documentation (doc/numpy_swig.*) has been updated to
>> reflect all of these changes.
>
> A small typo: in the docs you also write about 1D FORTRAN ARGOUTVIEW
> wrappers:
>
> ( DATA_TYPE** ARGOUTVIEW_FARRAY1, DIM_TYPE* DIM1 )
> ( DIM_TYPE* DIM1, DATA_TYPE** ARGOUTVIEW_FARRAY1 )
>
> which of course do not exist.

** Bill Spotz                                              **
** Sandia National Laboratories  Voice: (505)845-0170      **
** P.O. Box 5800                 Fax:   (505)284-0154      **
** Albuquerque, NM 87185-0370    Email: wfspotz at sandia.gov **






From ondrej at certik.cz  Sun Dec  2 19:52:57 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 3 Dec 2007 01:52:57 +0100
Subject: [Numpy-discussion] python-numpy debian package and f2py
Message-ID: <85b5c3130712021652w61fd4141wdc89d938fb1c56c2@mail.gmail.com>

Hi,

I am a comaintainer of the python-scipy package in Debian and now it
seems to be in quite a good shape. However, the python-numpy package
is quite a mess, so as it usually goes in opensource, I got fedup and
I tried to clean it. But I noticed, that f2py was moved from external
package into numpy, however
the versions mishmatch:

The newest (deprecated) python-f2py package in Debian has a version
2.45.241+1926, so I assume this was the version of f2py, before
merging
with numpy. However, the f2py in numpy says when executing:

Version:     2_3816
numpy Version: 1.0.3

so I assume the version of f2py in numpy is 2_3816? So has the
versioning scheme of f2py changed? Another question - since both numpy
and f2py
is now built from the same source, doesn't f2py simply has the same
version as numpy, i.e. 1.0.3?  Note: I know there is a newer numpy
release, but that's
not the point now.

I am asking because we probably will have to remove the old
python-f2py package and build a new one from the sources of numpy,
etc., and it will
take some time until this happens (ftpmasters need to remove the old
package from the archive, then the new binary package needs to go to
the
NEW queue for approval etc.), so I would like to make sure I
understand the versioning and the future plans with numpy and f2py,
before starting
the transition in Debian.

Actually, does it even make sense to create a python-f2py package? It
seems so (to me), it's a separate program. But since you decided to
merge it
with numpy, what are your thoughts about it?

Thanks a lot,
Ondrej


From numpy at mspacek.mm.st  Sun Dec  2 20:22:49 2007
From: numpy at mspacek.mm.st (Martin Spacek)
Date: Sun, 02 Dec 2007 17:22:49 -0800
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>	<4750981C.9060409@mspacek.mm.st>
	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
Message-ID: <47535A69.6000801@mspacek.mm.st>

Sebastian Haase wrote:
> reading this thread I have two comments.
> a) *Displaying* at 200Hz probably makes little sense, since humans
> would only see about max. of 30Hz (aka video frame rate).
> Consequently you would want to separate your data frame rate, that (as
> I understand) you want to save data to disk and - asynchrounously -
> "display as many frames as you can" (I have used pyOpenGL for this
> with great satisfaction)

Hi Sebastian,

Although 30Hz looks pretty good, if you watch a 60fps movie, you can
easily tell the difference. It's much smoother. Try recording AVIs on a
point and shoot digital camera, if you have one that can do both 30fps
and 60fps (like my fairly old Canon SD200).

And that's just perception. We're doing neurophysiology, recording from
neurons in the visual cortex, which can phase lock to CRT screen rasters
up to 100Hz. This is an artifact we don't want to deal with, so we use a
200Hz monitor. I need to be certain of exactly what's on the monitor on
every refresh, ie every 5ms, so I run python (with Andrew Straw's
package VisionEgg) as a "realtime" priority process in windows on a dual
core computer, which lets me reliably update the video frame buffer in
time for the next refresh, without having to worry about windows
multitasking butting in and stealing CPU cycles for the next 15-20ms.
Python runs on one core in "realtime", windows does its junk on the
other core. Right now, every 3rd video refresh (ie every 15ms, which is
66.7 Hz, close to the original 60fps the movie was recorded at) I update
with a new movie frame. That update needs to happen in less than 5ms,
every time. If there's any disk access involved during the update, it
inevitably exceeds that time limit, so I have to have it all in RAM
before playback begins. Having a second I/O thread running on the second
core would be great though.

--
Martin


From listservs at mac.com  Sun Dec  2 23:25:54 2007
From: listservs at mac.com (Chris)
Date: Mon, 3 Dec 2007 04:25:54 +0000 (UTC)
Subject: [Numpy-discussion] random.hypergeometric bug
Message-ID: <loom.20071203T042303-472@post.gmane.org>

There appears to be a bug in numpy's hypergeometric
random number generator. Here is an example -- if I 
generate 1000 hg samples with 4 draws from a space
with 30 successes and 10 failures:

In [39]: x = hg(30, 10, 4, 1000)

I should get a mean value of:

In [40]: 4*30./40
Out[40]: 3.0

But the sample mean is way to small:

In [41]: mean(x)
Out[41]: 0.996





From gael.varoquaux at normalesup.org  Mon Dec  3 01:27:23 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 3 Dec 2007 07:27:23 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <47535A69.6000801@mspacek.mm.st>
References: <474FCE2D.2000908@mspacek.mm.st>
	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>
	<4750981C.9060409@mspacek.mm.st>
	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
	<47535A69.6000801@mspacek.mm.st>
Message-ID: <20071203062723.GB10287@clipper.ens.fr>

On Sun, Dec 02, 2007 at 05:22:49PM -0800, Martin Spacek wrote:
> so I run python (with Andrew Straw's
> package VisionEgg) as a "realtime" priority process in windows on a dual
> core computer, which lets me reliably update the video frame buffer in
> time for the next refresh, without having to worry about windows
> multitasking butting in and stealing CPU cycles for the next 15-20ms.

Very interesting. Have you made measurements to see how many times you
lost one of your cycles. I made these kind of measurements on Linux using
the real-time clock with C and it was very interesting (
http://www.gael-varoquaux.info/computers/real-time ). I want to redo them
with Python, as I except to have similar results with Python. It would be
interesting to see how Windows fits in the picture (I know nothing about
Windows, so I really can't make measurements on Windows).

Cheers,

Ga?l 


From robert.kern at gmail.com  Mon Dec  3 01:43:59 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 03 Dec 2007 00:43:59 -0600
Subject: [Numpy-discussion] random.hypergeometric bug
In-Reply-To: <loom.20071203T042303-472@post.gmane.org>
References: <loom.20071203T042303-472@post.gmane.org>
Message-ID: <4753A5AF.7080908@gmail.com>

Chris wrote:
> There appears to be a bug in numpy's hypergeometric
> random number generator. Here is an example -- if I 
> generate 1000 hg samples with 4 draws from a space
> with 30 successes and 10 failures:
> 
> In [39]: x = hg(30, 10, 4, 1000)
> 
> I should get a mean value of:
> 
> In [40]: 4*30./40
> Out[40]: 3.0
> 
> But the sample mean is way to small:
> 
> In [41]: mean(x)
> Out[41]: 0.996

Fixed in r4527. My original source for the algorithm was incorrect, it seems.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From pearu at cens.ioc.ee  Mon Dec  3 03:20:45 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Mon, 3 Dec 2007 10:20:45 +0200 (EET)
Subject: [Numpy-discussion] python-numpy debian package and f2py
In-Reply-To: <85b5c3130712021652w61fd4141wdc89d938fb1c56c2@mail.gmail.com>
References: <85b5c3130712021652w61fd4141wdc89d938fb1c56c2@mail.gmail.com>
Message-ID: <50976.85.166.31.187.1196670045.squirrel@cens.ioc.ee>

Ho,

f2py is part of numpy and therefore there is no need to
have python-f2py package. The version number of f2py
(2_3816, for instance), indicates which SVN commit changed
the f2py package and has local usage only.

python-f2py package could be retained only for backward
compatibilty for users who are importing f2py2e package
and for users who need Numeric and numarray support in f2py.
Otherwise there is no need for it.

Note that if debian is going to ship Numeric then it should
also ship python-f2py. But this means that one cannot
install numpy as the will be a conflict of f2py scripts.

Regards,
Pearu

On Mon, December 3, 2007 2:52 am, Ondrej Certik wrote:
> Hi,
>
> I am a comaintainer of the python-scipy package in Debian and now it
> seems to be in quite a good shape. However, the python-numpy package
> is quite a mess, so as it usually goes in opensource, I got fedup and
> I tried to clean it. But I noticed, that f2py was moved from external
> package into numpy, however
> the versions mishmatch:
>
> The newest (deprecated) python-f2py package in Debian has a version
> 2.45.241+1926, so I assume this was the version of f2py, before
> merging
> with numpy. However, the f2py in numpy says when executing:
>
> Version:     2_3816
> numpy Version: 1.0.3
>
> so I assume the version of f2py in numpy is 2_3816? So has the
> versioning scheme of f2py changed? Another question - since both numpy
> and f2py
> is now built from the same source, doesn't f2py simply has the same
> version as numpy, i.e. 1.0.3?  Note: I know there is a newer numpy
> release, but that's
> not the point now.
>
> I am asking because we probably will have to remove the old
> python-f2py package and build a new one from the sources of numpy,
> etc., and it will
> take some time until this happens (ftpmasters need to remove the old
> package from the archive, then the new binary package needs to go to
> the
> NEW queue for approval etc.), so I would like to make sure I
> understand the versioning and the future plans with numpy and f2py,
> before starting
> the transition in Debian.
>
> Actually, does it even make sense to create a python-f2py package? It
> seems so (to me), it's a separate program. But since you decided to
> merge it
> with numpy, what are your thoughts about it?
>
> Thanks a lot,
> Ondrej
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>




From pearu at cens.ioc.ee  Mon Dec  3 05:22:51 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Mon, 3 Dec 2007 12:22:51 +0200 (EET)
Subject: [Numpy-discussion] python-numpy debian package and f2py
In-Reply-To: <50976.85.166.31.187.1196670045.squirrel@cens.ioc.ee>
References: <85b5c3130712021652w61fd4141wdc89d938fb1c56c2@mail.gmail.com>
	<50976.85.166.31.187.1196670045.squirrel@cens.ioc.ee>
Message-ID: <57110.85.166.31.187.1196677371.squirrel@cens.ioc.ee>


Some additional notes:

- note that f2py in numpy and in python-f2py are *different*
tools and therefore python-f2py should not be considered
depreciated more than python-numeric is. So, python-f2py
should stay provided python-numeric is there. It might be
appropiate to rename python-f2py to python-f2py2e.

- the conflict between f2py scripts can be easily resolved
by renaming f2py script in python-f2py package to, say,
f2py2e (that would be an simple change in f2py2e/setup.py file).
The f2py script in numpy has --2d-numeric switch
that when used would import f2py2e package instead of numpy.f2py.

Pearu

On Mon, December 3, 2007 10:20 am, Pearu Peterson wrote:
> Ho,
>
> f2py is part of numpy and therefore there is no need to
> have python-f2py package. The version number of f2py
> (2_3816, for instance), indicates which SVN commit changed
> the f2py package and has local usage only.
>
> python-f2py package could be retained only for backward
> compatibilty for users who are importing f2py2e package
> and for users who need Numeric and numarray support in f2py.
> Otherwise there is no need for it.
>
> Note that if debian is going to ship Numeric then it should
> also ship python-f2py. But this means that one cannot
> install numpy as the will be a conflict of f2py scripts.
>
> Regards,
> Pearu
>
> On Mon, December 3, 2007 2:52 am, Ondrej Certik wrote:
>> Hi,
>>
>> I am a comaintainer of the python-scipy package in Debian and now it
>> seems to be in quite a good shape. However, the python-numpy package
>> is quite a mess, so as it usually goes in opensource, I got fedup and
>> I tried to clean it. But I noticed, that f2py was moved from external
>> package into numpy, however
>> the versions mishmatch:
>>
>> The newest (deprecated) python-f2py package in Debian has a version
>> 2.45.241+1926, so I assume this was the version of f2py, before
>> merging
>> with numpy. However, the f2py in numpy says when executing:
>>
>> Version:     2_3816
>> numpy Version: 1.0.3
>>
>> so I assume the version of f2py in numpy is 2_3816? So has the
>> versioning scheme of f2py changed? Another question - since both numpy
>> and f2py
>> is now built from the same source, doesn't f2py simply has the same
>> version as numpy, i.e. 1.0.3?  Note: I know there is a newer numpy
>> release, but that's
>> not the point now.
>>
>> I am asking because we probably will have to remove the old
>> python-f2py package and build a new one from the sources of numpy,
>> etc., and it will
>> take some time until this happens (ftpmasters need to remove the old
>> package from the archive, then the new binary package needs to go to
>> the
>> NEW queue for approval etc.), so I would like to make sure I
>> understand the versioning and the future plans with numpy and f2py,
>> before starting
>> the transition in Debian.
>>
>> Actually, does it even make sense to create a python-f2py package? It
>> seems so (to me), it's a separate program. But since you decided to
>> merge it
>> with numpy, what are your thoughts about it?
>>
>> Thanks a lot,
>> Ondrej
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>




From ondrej at certik.cz  Mon Dec  3 06:16:09 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 3 Dec 2007 12:16:09 +0100
Subject: [Numpy-discussion] python-numpy debian package and f2py
In-Reply-To: <57110.85.166.31.187.1196677371.squirrel@cens.ioc.ee>
References: <85b5c3130712021652w61fd4141wdc89d938fb1c56c2@mail.gmail.com>
	<50976.85.166.31.187.1196670045.squirrel@cens.ioc.ee>
	<57110.85.166.31.187.1196677371.squirrel@cens.ioc.ee>
Message-ID: <85b5c3130712030316o67d8453cu193ee06cf83b6237@mail.gmail.com>

Hi Pearu,

On Dec 3, 2007 11:22 AM, Pearu Peterson <pearu at cens.ioc.ee> wrote:
>
> Some additional notes:
>
> - note that f2py in numpy and in python-f2py are *different*
> tools and therefore python-f2py should not be considered
> depreciated more than python-numeric is. So, python-f2py
> should stay provided python-numeric is there. It might be
> appropiate to rename python-f2py to python-f2py2e.

Thanks for your reply. Yes, I didn't realize Debian still ships
Numerics. Now I think Numerics should stay (still there
are people using it) and also the (old) python-f2py should stay,
because of course it is needed.

And the new f2py will just be part of the python-numpy package. I
created a wiki that explains all of it:

http://wiki.debian.org/python-numpy

Feel free to edit it.

> - the conflict between f2py scripts can be easily resolved
> by renaming f2py script in python-f2py package to, say,
> f2py2e (that would be an simple change in f2py2e/setup.py file).
> The f2py script in numpy has --2d-numeric switch
> that when used would import f2py2e package instead of numpy.f2py.

Actually, we'll just conflict with the python-numpy package with the
python-f2py package and that's it. This is actually how it is now,
but I thought that it is more appropriate to have a separate binary
package for f2py, but now I don't think so.

Thanks,
Ondrej


From faltet at carabos.com  Mon Dec  3 08:40:27 2007
From: faltet at carabos.com (Francesc Altet)
Date: Mon, 3 Dec 2007 14:40:27 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <47535A69.6000801@mspacek.mm.st>
References: <474FCE2D.2000908@mspacek.mm.st>
	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
	<47535A69.6000801@mspacek.mm.st>
Message-ID: <200712031440.28785.faltet@carabos.com>

A Monday 03 December 2007, Martin Spacek escrigu?:
> Sebastian Haase wrote:
> > reading this thread I have two comments.
> > a) *Displaying* at 200Hz probably makes little sense, since humans
> > would only see about max. of 30Hz (aka video frame rate).
> > Consequently you would want to separate your data frame rate, that
> > (as I understand) you want to save data to disk and -
> > asynchrounously - "display as many frames as you can" (I have used
> > pyOpenGL for this with great satisfaction)
>
> Hi Sebastian,
>
> Although 30Hz looks pretty good, if you watch a 60fps movie, you can
> easily tell the difference. It's much smoother. Try recording AVIs on
> a point and shoot digital camera, if you have one that can do both
> 30fps and 60fps (like my fairly old Canon SD200).
>
> And that's just perception. We're doing neurophysiology, recording
> from neurons in the visual cortex, which can phase lock to CRT screen
> rasters up to 100Hz. This is an artifact we don't want to deal with,
> so we use a 200Hz monitor. I need to be certain of exactly what's on
> the monitor on every refresh, ie every 5ms, so I run python (with
> Andrew Straw's package VisionEgg) as a "realtime" priority process in
> windows on a dual core computer, which lets me reliably update the
> video frame buffer in time for the next refresh, without having to
> worry about windows multitasking butting in and stealing CPU cycles
> for the next 15-20ms. Python runs on one core in "realtime", windows
> does its junk on the other core. Right now, every 3rd video refresh
> (ie every 15ms, which is 66.7 Hz, close to the original 60fps the
> movie was recorded at) I update with a new movie frame. That update
> needs to happen in less than 5ms, every time. If there's any disk
> access involved during the update, it inevitably exceeds that time
> limit, so I have to have it all in RAM before playback begins. Having
> a second I/O thread running on the second core would be great though.

Perhaps something that can surely improve your timings is first performing a read of your data file(s) while throwing the data as you are reading it.  This serves only to load the file entirely (if you have memory enough, but this seems your case) in OS page cache.  Then, the second time that your code has to read the data, the OS only have to retrieve it from its cache (i.e. in memory) rather than from disk.

You can do this with whatever technique you want, but if you are after reading from a single container and memmap is giving you headaches in 32-bit platforms, you might try PyTables because it allows 64-bit disk addressing transparently, even on 32-bit machines.

HTH,

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071203/bfb46dc6/attachment.html>

From stefan at sun.ac.za  Mon Dec  3 09:46:58 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Mon, 3 Dec 2007 16:46:58 +0200
Subject: [Numpy-discussion] display numpy array as image
In-Reply-To: <CDBFC89C-DBE6-4C51-AF03-6FBD23895F80@stanford.edu>
References: <1196388219.6047.215.camel@glup.physics.ucf.edu>
	<CDBFC89C-DBE6-4C51-AF03-6FBD23895F80@stanford.edu>
Message-ID: <20071203144658.GJ13844@mentat.za.net>

Hi Zach

Attached is some code for removing radial distortion from images.  It
shows how to draw lines based on user input using matplotlib.  It is
not suited for a big application, but useful for demonstrations.

Try it on

  http://mentat.za.net/results/window.jpg

Regards
St?fan

On Thu, Nov 29, 2007 at 11:59:05PM -0500, Zachary Pincus wrote:
> Thanks for the suggestions, everyone! All very informative and most  
> helpful.
> 
> For what it's worth, here's my application: I'm building a tool for  
> image processing which needs some manual input in a few places (e.g.  
> user draws a few lines). The images are greyscale images with 12-14  
> bits of dynamic range (from a microscope), so I need to have some  
> basic brightness/contrast/gamma controls, as well as allowing basic  
> drawing on the image to get the needed user input. It looks like GL  
> or wx will be best suited here, I think? (I presume that python/numpy/ 
> [GL|wx] can keep up with things like dragging a slider to change  
> brightness/contrast/other LUT changes, as long as I code reasonably.)
> 
> Anyhow, thanks for all the input,
> 
> Zach
-------------- next part --------------
A non-text attachment was scrubbed...
Name: radial.py
Type: text/x-python
Size: 6683 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071203/0fbfd725/attachment.py>

From zpincus at stanford.edu  Mon Dec  3 13:00:03 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Mon, 3 Dec 2007 13:00:03 -0500
Subject: [Numpy-discussion] display numpy array as image
In-Reply-To: <20071203144658.GJ13844@mentat.za.net>
References: <1196388219.6047.215.camel@glup.physics.ucf.edu>
	<CDBFC89C-DBE6-4C51-AF03-6FBD23895F80@stanford.edu>
	<20071203144658.GJ13844@mentat.za.net>
Message-ID: <E8042132-D5CD-4C9F-9E5C-EADA66F92B1F@stanford.edu>

Hi St?fan,

Thanks -- I hadn't realized matplotlib's user-interaction abilities  
were that sophisticated! I'll definitely give that route a shot.

Zach


On Dec 3, 2007, at 9:46 AM, Stefan van der Walt wrote:

> Hi Zach
>
> Attached is some code for removing radial distortion from images.  It
> shows how to draw lines based on user input using matplotlib.  It is
> not suited for a big application, but useful for demonstrations.
>
> Try it on
>
>   http://mentat.za.net/results/window.jpg
>
> Regards
> St?fan
>
> On Thu, Nov 29, 2007 at 11:59:05PM -0500, Zachary Pincus wrote:
>> Thanks for the suggestions, everyone! All very informative and most
>> helpful.
>>
>> For what it's worth, here's my application: I'm building a tool for
>> image processing which needs some manual input in a few places (e.g.
>> user draws a few lines). The images are greyscale images with 12-14
>> bits of dynamic range (from a microscope), so I need to have some
>> basic brightness/contrast/gamma controls, as well as allowing basic
>> drawing on the image to get the needed user input. It looks like GL
>> or wx will be best suited here, I think? (I presume that python/ 
>> numpy/
>> [GL|wx] can keep up with things like dragging a slider to change
>> brightness/contrast/other LUT changes, as long as I code reasonably.)
>>
>> Anyhow, thanks for all the input,
>>
>> Zach<radial.py>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From jdh2358 at gmail.com  Mon Dec  3 13:48:11 2007
From: jdh2358 at gmail.com (John Hunter)
Date: Mon, 3 Dec 2007 12:48:11 -0600
Subject: [Numpy-discussion] display numpy array as image
In-Reply-To: <E8042132-D5CD-4C9F-9E5C-EADA66F92B1F@stanford.edu>
References: <1196388219.6047.215.camel@glup.physics.ucf.edu>
	<CDBFC89C-DBE6-4C51-AF03-6FBD23895F80@stanford.edu>
	<20071203144658.GJ13844@mentat.za.net>
	<E8042132-D5CD-4C9F-9E5C-EADA66F92B1F@stanford.edu>
Message-ID: <88e473830712031048h5b02aed8w5b99dc549b2d18b8@mail.gmail.com>

On Dec 3, 2007 12:00 PM, Zachary Pincus <zpincus at stanford.edu> wrote:

> Thanks -- I hadn't realized matplotlib's user-interaction abilities
> were that sophisticated! I'll definitely give that route a shot.

Here is another example which will help you understand how to do
interaction.  You can drag the vertices of the polygon around, click
'd' to delete a vertex, 'i' to insert a new vertex.  Because it uses
mouse clicks, mouse drags and key presses, it introduces many of the
common things you want to do when building in interaction.  All of
matplotlib's cross GUI panning, zooming, zoom-to-rect, etc are built
using matplotlib's internal event model, so it is pretty stable and
robust.   Paul has recently done some nice work on the picking
infrastructure, with support for hover highlighting and more.  Maybe
Paul you an post an example ?

The example below is also attached to avoid possible line breaks, etc...

"""
This is an example to show how to build cross-GUI applications using
matplotlib event handling to interact with objects on the canvas

"""
from matplotlib.artist import Artist
from matplotlib.patches import Polygon, CirclePolygon
from numpy import sqrt, nonzero, equal, asarray, dot, amin
from matplotlib.mlab import dist_point_to_segment


class PolygonInteractor:
    """
    An polygon editor.

    Key-bindings

      't' toggle vertex markers on and off.  When vertex markers are on,
          you can move them, delete them

      'd' delete the vertex under point

      'i' insert a vertex at point.  You must be within epsilon of the
          line connecting two existing vertices

    """

    showverts = True
    epsilon = 5  # max pixel distance to count as a vertex hit

    def __init__(self, ax, poly):
        if poly.figure is None:
            raise RuntimeError('You must first add the polygon to a
figure or canvas before defining the interactor')
        self.ax = ax
        canvas = poly.figure.canvas
        self.poly = poly

        x, y = zip(*self.poly.xy)
        self.line = Line2D(x,y,marker='o', markerfacecolor='r', animated=True)
        #self._update_line(poly)

        cid = self.poly.add_callback(self.poly_changed)
        self._ind = None # the active vert

        canvas.mpl_connect('draw_event', self.draw_callback)
        canvas.mpl_connect('button_press_event', self.button_press_callback)
        canvas.mpl_connect('key_press_event', self.key_press_callback)
        canvas.mpl_connect('button_release_event', self.button_release_callback)
        canvas.mpl_connect('motion_notify_event', self.motion_notify_callback)
        self.canvas = canvas


    def draw_callback(self, event):
        self.background = self.canvas.copy_from_bbox(self.ax.bbox)
        self.ax.draw_artist(self.poly)
        self.ax.draw_artist(self.line)
        self.canvas.blit(self.ax.bbox)

    def poly_changed(self, poly):
        'this method is called whenever the polygon object is called'
        # only copy the artist props to the line (except visibility)
        vis = self.line.get_visible()
        Artist.update_from(self.line, poly)
        self.line.set_visible(vis)  # don't use the poly visibility state


    def get_ind_under_point(self, event):
        'get the index of the vertex under point if within epsilon tolerance'
        x, y = zip(*self.poly.xy)

        # display coords
        xt, yt = self.poly.get_transform().numerix_x_y(x, y)
        d = sqrt((xt-event.x)**2 + (yt-event.y)**2)
        indseq = nonzero(equal(d, amin(d)))[0]
        ind = indseq[0]

        if d[ind]>=self.epsilon:
            ind = None

        return ind

    def button_press_callback(self, event):
        'whenever a mouse button is pressed'
        if not self.showverts: return
        if event.inaxes==None: return
        if event.button != 1: return
        self._ind = self.get_ind_under_point(event)

    def button_release_callback(self, event):
        'whenever a mouse button is released'
        if not self.showverts: return
        if event.button != 1: return
        self._ind = None

    def key_press_callback(self, event):
        'whenever a key is pressed'
        if not event.inaxes: return
        if event.key=='t':
            self.showverts = not self.showverts
            self.line.set_visible(self.showverts)
            if not self.showverts: self._ind = None
        elif event.key=='d':
            ind = self.get_ind_under_point(event)
            if ind is not None:
                self.poly.xy = [tup for i,tup in
enumerate(self.poly.xy) if i!=ind]
                self.line.set_data(zip(*self.poly.xy))
        elif event.key=='i':
            xys = self.poly.get_transform().seq_xy_tups(self.poly.xy)
            p = event.x, event.y # display coords
            for i in range(len(xys)-1):
                s0 = xys[i]
                s1 = xys[i+1]
                d = dist_point_to_segment(p, s0, s1)
                if d<=self.epsilon:
                    self.poly.xy.insert(i+1, (event.xdata, event.ydata))
                    self.line.set_data(zip(*self.poly.xy))
                    break


        self.canvas.draw()

    def motion_notify_callback(self, event):
        'on mouse movement'
        if not self.showverts: return
        if self._ind is None: return
        if event.inaxes is None: return
        if event.button != 1: return
        x,y = event.xdata, event.ydata

        self.poly.xy[self._ind] = x,y
        self.line.set_data(zip(*self.poly.xy))

        self.canvas.restore_region(self.background)
        self.ax.draw_artist(self.poly)
        self.ax.draw_artist(self.line)
        self.canvas.blit(self.ax.bbox)



from pylab import *





fig = figure()
theta = arange(0, 2*pi, 0.1)
r = 1.5

xs = r*npy.cos(theta)
ys = r*npy.sin(theta)

poly = Polygon(zip(xs, ys,), animated=True)




ax = subplot(111)
ax.add_patch(poly)
p = PolygonInteractor( ax, poly)

ax.add_line(p.line)
ax.set_title('Click and drag a point to move it')
ax.set_xlim((-2,2))
ax.set_ylim((-2,2))
show()
-------------- next part --------------
A non-text attachment was scrubbed...
Name: poly_editor.py
Type: text/x-python
Size: 5092 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071203/d5333882/attachment.py>

From numpy at mspacek.mm.st  Mon Dec  3 15:44:31 2007
From: numpy at mspacek.mm.st (Martin Spacek)
Date: Mon, 03 Dec 2007 12:44:31 -0800
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <200712031440.28785.faltet@carabos.com>
References: <474FCE2D.2000908@mspacek.mm.st>	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>	<47535A69.6000801@mspacek.mm.st>
	<200712031440.28785.faltet@carabos.com>
Message-ID: <47546AAF.4080806@mspacek.mm.st>

Francesc Altet wrote:
> Perhaps something that can surely improve your timings is first 
> performing a read of your data file(s) while throwing the data as you 
> are reading it. This serves only to load the file entirely (if you have 
> memory enough, but this seems your case) in OS page cache. Then, the 
> second time that your code has to read the data, the OS only have to 
> retrieve it from its cache (i.e. in memory) rather than from disk.

I think I tried that, loading the whole file into memory, throwing it 
away, then trying to load on the fly from "disk" (which would now 
hopefully be done more optimally the 2nd time around) while displaying 
the movie, but I still got update times > 5ms. The file's just too big 
to get any improvement by sort of preloading this way.

> You can do this with whatever technique you want, but if you are after 
> reading from a single container and memmap is giving you headaches in 
> 32-bit platforms, you might try PyTables because it allows 64-bit disk 
> addressing transparently, even on 32-bit machines.

PyTables sounds interesting, I might take a look. Thanks.

Martin


From numpy at mspacek.mm.st  Mon Dec  3 20:20:57 2007
From: numpy at mspacek.mm.st (Martin Spacek)
Date: Mon, 03 Dec 2007 17:20:57 -0800
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <20071203062723.GB10287@clipper.ens.fr>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>	<4750981C.9060409@mspacek.mm.st>	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>	<47535A69.6000801@mspacek.mm.st>
	<20071203062723.GB10287@clipper.ens.fr>
Message-ID: <4754AB79.7080004@mspacek.mm.st>

Gael Varoquaux wrote:
> Very interesting. Have you made measurements to see how many times you
> lost one of your cycles. I made these kind of measurements on Linux using
> the real-time clock with C and it was very interesting (
> http://www.gael-varoquaux.info/computers/real-time ). I want to redo them
> with Python, as I except to have similar results with Python. It would be
> interesting to see how Windows fits in the picture (I know nothing about
> Windows, so I really can't make measurements on Windows).

Neat, thanks for that, I'll have a look. I'm very slowly transitioning 
my computing "life" over to linux, but I've been told by Andrew Straw (I 
think http://visionegg.org might have some details, see the mailing 
list) that it's harder to get close to a real-time OS with linux (while 
running high level stuff like opengl and python) than it is in windows. 
I hope that's changed, or is changing. I'd love to switch over to 64-bit 
linux. As far as windows is considered, I'd like 32bit winxp to be my 
last iteration.

Martin


From david at ar.media.kyoto-u.ac.jp  Tue Dec  4 00:13:53 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 04 Dec 2007 14:13:53 +0900
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <4754AB79.7080004@mspacek.mm.st>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>	<4750981C.9060409@mspacek.mm.st>	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>	<47535A69.6000801@mspacek.mm.st>	<20071203062723.GB10287@clipper.ens.fr>
	<4754AB79.7080004@mspacek.mm.st>
Message-ID: <4754E211.2010903@ar.media.kyoto-u.ac.jp>

Martin Spacek wrote:
> Gael Varoquaux wrote:
>> Very interesting. Have you made measurements to see how many times you
>> lost one of your cycles. I made these kind of measurements on Linux using
>> the real-time clock with C and it was very interesting (
>> http://www.gael-varoquaux.info/computers/real-time ). I want to redo them
>> with Python, as I except to have similar results with Python. It would be
>> interesting to see how Windows fits in the picture (I know nothing about
>> Windows, so I really can't make measurements on Windows).
>
> Neat, thanks for that, I'll have a look. I'm very slowly transitioning 
> my computing "life" over to linux, but I've been told by Andrew Straw (I 
> think http://visionegg.org might have some details, see the mailing 
> list) that it's harder to get close to a real-time OS with linux (while 
> running high level stuff like opengl and python) than it is in windows.
My impression is that is is more like the contrary; linux implements 
many  posix facilities for more 'real-time' behaviour: it implements a 
FIFO scheduler, you have mlock facilities to avoid paging, etc... and of 
course, you can control your environment much more easily (one buggy 
driver can kill the whole thing as far as latency is concerned, for 
example). I did not find those info you are talking about on visionegg  ?

Now, for python, this is a different matter. In you need to do things in 
real-time, setting a high priority is not enough, and python has several 
characteristics which make it less than suitable for real-time (heavy 
usage of memory allocation, garbage collector, etc...). I guess that 
when you do things every few ms, with enough memory (every ms gives you 
millions of cycle on modern machines), you can hope that is it not too 
much of a problem (at least for memory allocation; I could not find in 
Andrew's slides whether he disabled the GC). But I doubt you can do much 
better.

I wonder if python can be compiled with a real time memory allocator, 
and even whether it makes sense at all (I am thinking about something 
like TLSF: http://rtportal.upv.es/rtmalloc/).
>  
> I hope that's changed, or is changing. I'd love to switch over to 64-bit 
> linux. As far as windows is considered, I'd like 32bit winxp to be my 
> last iteration.
With recent kernels, you can get really good latency if you do it right 
(around 1-2 ms worst case under high load, including high IO pressure). 
I know nothing about video programming, but I would guess that as far as 
the kernel is concerned, this does not change much. I have not tried 
them myself, but ubuntu studio has its own kernel with 'real-time' 
patched (voluntary preempt from Ingo, for example), and is available 
both for 32 and 64 bits architectures. One problem I can think of for 
video is that if you need binary-only drivers: those are generally 
pretty bad as far as low latency is concerned (nvidia drivers always 
cause some kind of problems with low latency and 'real-time' kernels).

http://ubuntustudio.org/

David


From gael.varoquaux at normalesup.org  Tue Dec  4 04:36:02 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 4 Dec 2007 10:36:02 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <4754E211.2010903@ar.media.kyoto-u.ac.jp>
References: <474FCE2D.2000908@mspacek.mm.st>
	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>
	<4750981C.9060409@mspacek.mm.st>
	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
	<47535A69.6000801@mspacek.mm.st>
	<20071203062723.GB10287@clipper.ens.fr>
	<4754AB79.7080004@mspacek.mm.st>
	<4754E211.2010903@ar.media.kyoto-u.ac.jp>
Message-ID: <20071204093602.GD21476@clipper.ens.fr>

On Tue, Dec 04, 2007 at 02:13:53PM +0900, David Cournapeau wrote:
> With recent kernels, you can get really good latency if you do it right 
> (around 1-2 ms worst case under high load, including high IO pressure). 

As you can see on my page, I indeed measured less than 1ms latency on
Linux under load with kernel more than a year old. These things have
gotten much better recently and with a premptible kernel you should be
able to get 1ms easily. Going below 0.5ms without using a realtime OS (ie
a realtime kernel, under linux) is really pushing it.

Cheers,

Ga?l


From david at ar.media.kyoto-u.ac.jp  Tue Dec  4 05:05:26 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 04 Dec 2007 19:05:26 +0900
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <20071204093602.GD21476@clipper.ens.fr>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>	<4750981C.9060409@mspacek.mm.st>	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>	<47535A69.6000801@mspacek.mm.st>	<20071203062723.GB10287@clipper.ens.fr>	<4754AB79.7080004@mspacek.mm.st>	<4754E211.2010903@ar.media.kyoto-u.ac.jp>
	<20071204093602.GD21476@clipper.ens.fr>
Message-ID: <47552666.3050601@ar.media.kyoto-u.ac.jp>

Gael Varoquaux wrote:
> On Tue, Dec 04, 2007 at 02:13:53PM +0900, David Cournapeau wrote:
>   
>> With recent kernels, you can get really good latency if you do it right 
>> (around 1-2 ms worst case under high load, including high IO pressure). 
>>     
>
> As you can see on my page, I indeed measured less than 1ms latency on
> Linux under load with kernel more than a year old. These things have
> gotten much better recently and with a premptible kernel you should be
> able to get 1ms easily. Going below 0.5ms without using a realtime OS (ie
> a realtime kernel, under linux) is really pushing it.
>   
Yes, 1ms is possible for quite a long time; the problem was how to get 
there (kernel patches, special permissions, etc... Many of those 
problems are now gone). I've read that you could get around 0.2 ms and 
even below (worst case) with the last kernels + RT preempt (that is you 
still use linux, and not rtlinux). Below 1 ms does not make much sense 
for audio applications, so I don't know much below this range :)

But I am really curious if you can get those numbers with python, 
because of malloc, the gc and co. I mean for example, 0.5 ms latency for 
a 1 Ghz CPU means that you get something like a 500 000 CPU cycles, and 
I can imagine a cycle of garbage collection taking that many cycles, 
without even considering pages of virtual memory which are swapped (in 
this case, we are talking millions of cycles).

cheers,

David


From strawman at astraw.com  Tue Dec  4 05:28:54 2007
From: strawman at astraw.com (Andrew Straw)
Date: Tue, 04 Dec 2007 02:28:54 -0800
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <20071204093602.GD21476@clipper.ens.fr>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>	<4750981C.9060409@mspacek.mm.st>	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>	<47535A69.6000801@mspacek.mm.st>	<20071203062723.GB10287@clipper.ens.fr>	<4754AB79.7080004@mspacek.mm.st>	<4754E211.2010903@ar.media.kyoto-u.ac.jp>
	<20071204093602.GD21476@clipper.ens.fr>
Message-ID: <47552BE6.9010705@astraw.com>

Hi all,

I haven't done any serious testing in the past couple years, but for 
this particular task -- drawing frames using OpenGL without ever 
skipping a video update -- it is my impression that as of a few Ubuntu 
releases ago (Edgy?) Windows still beat linux.

Just now, I have investigated on 2.6.22-14-generic x86_64 as pacakged by 
Ubuntu 7.10, and I didn't skip a frame out of 1500 at 60 Hz. That's not 
much testing, but it is certainly better performance than I've seen in 
the recent past, so I'll certainly be doing some more testing soon. Oh, 
how I'd love to never be forced to use Windows again.

Leaving my computer displaying moving images overnight, (and tomorrow at 
lab on a 200 Hz display),
Andrew

Gael Varoquaux wrote:
> On Tue, Dec 04, 2007 at 02:13:53PM +0900, David Cournapeau wrote:
>   
>> With recent kernels, you can get really good latency if you do it right 
>> (around 1-2 ms worst case under high load, including high IO pressure). 
>>     
>
> As you can see on my page, I indeed measured less than 1ms latency on
> Linux under load with kernel more than a year old. These things have
> gotten much better recently and with a premptible kernel you should be
> able to get 1ms easily. Going below 0.5ms without using a realtime OS (ie
> a realtime kernel, under linux) is really pushing it.
>
> Cheers,
>
> Ga?l
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From david at ar.media.kyoto-u.ac.jp  Tue Dec  4 05:38:05 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 04 Dec 2007 19:38:05 +0900
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <47552BE6.9010705@astraw.com>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>	<4750981C.9060409@mspacek.mm.st>	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>	<47535A69.6000801@mspacek.mm.st>	<20071203062723.GB10287@clipper.ens.fr>	<4754AB79.7080004@mspacek.mm.st>	<4754E211.2010903@ar.media.kyoto-u.ac.jp>	<20071204093602.GD21476@clipper.ens.fr>
	<47552BE6.9010705@astraw.com>
Message-ID: <47552E0D.6030101@ar.media.kyoto-u.ac.jp>

Andrew Straw wrote:
> Hi all,
>
> I haven't done any serious testing in the past couple years, but for 
> this particular task -- drawing frames using OpenGL without ever 
> skipping a video update -- it is my impression that as of a few Ubuntu 
> releases ago (Edgy?) Windows still beat linux.
>   
The problem is that this is the kind of things which is really 
distribution (because of kernel patch) dependent.
> Just now, I have investigated on 2.6.22-14-generic x86_64 as pacakged by 
> Ubuntu 7.10, and I didn't skip a frame out of 1500 at 60 Hz. That's not 
> much testing, but it is certainly better performance than I've seen in 
> the recent past, so I'll certainly be doing some more testing soon. Oh, 
> how I'd love to never be forced to use Windows again.
>   
You should try the rt kernel: https://wiki.ubuntu.com/RealTime/Gutsy. 
This does make a huge difference.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Tue Dec  4 06:13:56 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 04 Dec 2007 20:13:56 +0900
Subject: [Numpy-discussion] Missing file in SVN repository (test_ufunc)
Message-ID: <47553674.4010204@ar.media.kyoto-u.ac.jp>

Hi,

    Since revision 4528, it is not possible to run the tests for 
numpy.core, because of a missing file, test_ufunc.py. Since the change 
in the test was done in r4528 whose log refers " increase of code 
coverage", I guess the file was not added to the repository,

    cheers,

    David


From david at ar.media.kyoto-u.ac.jp  Tue Dec  4 06:21:39 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 04 Dec 2007 20:21:39 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
Message-ID: <47553843.7020709@ar.media.kyoto-u.ac.jp>

Hi,

    I've just reached a first usable scipy.scons branch, so that scipy 
can be built entirely with scons (assuming you build numpy with scons 
too). You can get it from 
http://svn.scipy.org/svn/scipy/branches/scipy.scons. To build it, you 
just need to use numpy.scons branch instead of the trunk, and use 
setupscons.py instead of setup.py. Again, I would be happy to hear about 
failures, success (please report a ticket in this case), etc...

Some of the most interesting things I can think of which work with scons:
    - you can control fortran and C flags from the command line: CFLAGS 
and FFLAGS won't override necessary flags, only optimization flags, so 
you can easily play with warning, optimization flags. For example:

CFLAGS='-W -Wall -Wextra -DDEBUG' FFLAGS='-DDEBUG -W -Wall -Wextra' 
python setupscons build

for debugging will work. No need to care about -fPIC and co, all this is 
handled automatically.
    - dependencies are handled correctly thanks to scons: for example, 
if you change a library (e.g. by using MKL=None to disable mkl), only 
link step will be redone.

platforms known to work
-----------------------

    - linux with gcc/g77 or gcc/gfortran (both atlas and mkl 9 were tested).
    - linux with intel compilers (intel and gnu compilers can also be 
mixed, AFAIK).
    - solaris with sun compilers with sunperf, only tested on indiana.

Notable non working things:
---------------------------
 
    - using netlib BLAS and LAPACK is not supported (only optimized ones 
are available: sunperf, atlas, mkl, and vecLib/Accelerate).
    - parallel build does NOT work (AFAICS, this is because f2py which 
do some things which are not thread-safe, but I have not yet found the 
exact problem).
    - I have not yet implemented umfpack checker, and as such umfpack 
cannot be built yet
    - I have not yet tweaked fortran compiler configurations for 
optimizations except for gnu compilers
    - c++ compilers configurations are not handled either.

cheers,

David


From stefan at sun.ac.za  Tue Dec  4 08:11:41 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Tue, 4 Dec 2007 15:11:41 +0200
Subject: [Numpy-discussion] Missing file in SVN repository (test_ufunc)
In-Reply-To: <47553674.4010204@ar.media.kyoto-u.ac.jp>
References: <47553674.4010204@ar.media.kyoto-u.ac.jp>
Message-ID: <20071204131141.GA8346@mentat.za.net>

On Tue, Dec 04, 2007 at 08:13:56PM +0900, David Cournapeau wrote:
>     Since revision 4528, it is not possible to run the tests for 
> numpy.core, because of a missing file, test_ufunc.py. Since the change 
> in the test was done in r4528 whose log refers " increase of code 
> coverage", I guess the file was not added to the repository,

Sorry about that.  I added the file in r4492.

Cheers
St?fan


From david.huard at gmail.com  Tue Dec  4 09:20:25 2007
From: david.huard at gmail.com (David Huard)
Date: Tue, 4 Dec 2007 09:20:25 -0500
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <47553843.7020709@ar.media.kyoto-u.ac.jp>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
Message-ID: <91cf711d0712040620g3a33dcaftcfa34d3f2f640c26@mail.gmail.com>

David,

I tried building the scons numpy and scipy. Numpy apparently built fine
since all tests pass. For scipy, I am having problem during the build.
Something seems to be wrong with libdfftpack.a

I'm attaching the terminal output.

Ubuntu 7.10, Xeon 64.

HTH,

David

This is great stuff by the way, consider me a scons convert.

2007/12/4, David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>
> Hi,
>
>     I've just reached a first usable scipy.scons branch, so that scipy
> can be built entirely with scons (assuming you build numpy with scons
> too). You can get it from
> http://svn.scipy.org/svn/scipy/branches/scipy.scons. To build it, you
> just need to use numpy.scons branch instead of the trunk, and use
> setupscons.py instead of setup.py. Again, I would be happy to hear about
> failures, success (please report a ticket in this case), etc...
>
> Some of the most interesting things I can think of which work with scons:
>     - you can control fortran and C flags from the command line: CFLAGS
> and FFLAGS won't override necessary flags, only optimization flags, so
> you can easily play with warning, optimization flags. For example:
>
> CFLAGS='-W -Wall -Wextra -DDEBUG' FFLAGS='-DDEBUG -W -Wall -Wextra'
> python setupscons build
>
> for debugging will work. No need to care about -fPIC and co, all this is
> handled automatically.
>     - dependencies are handled correctly thanks to scons: for example,
> if you change a library (e.g. by using MKL=None to disable mkl), only
> link step will be redone.
>
> platforms known to work
> -----------------------
>
>     - linux with gcc/g77 or gcc/gfortran (both atlas and mkl 9 were
> tested).
>     - linux with intel compilers (intel and gnu compilers can also be
> mixed, AFAIK).
>     - solaris with sun compilers with sunperf, only tested on indiana.
>
> Notable non working things:
> ---------------------------
>
>     - using netlib BLAS and LAPACK is not supported (only optimized ones
> are available: sunperf, atlas, mkl, and vecLib/Accelerate).
>     - parallel build does NOT work (AFAICS, this is because f2py which
> do some things which are not thread-safe, but I have not yet found the
> exact problem).
>     - I have not yet implemented umfpack checker, and as such umfpack
> cannot be built yet
>     - I have not yet tweaked fortran compiler configurations for
> optimizations except for gnu compilers
>     - c++ compilers configurations are not handled either.
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071204/b44621d8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log
Type: text/x-log
Size: 18227 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071204/b44621d8/attachment.bin>

From matthieu.brucher at gmail.com  Tue Dec  4 09:23:57 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 4 Dec 2007 15:23:57 +0100
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <91cf711d0712040620g3a33dcaftcfa34d3f2f640c26@mail.gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<91cf711d0712040620g3a33dcaftcfa34d3f2f640c26@mail.gmail.com>
Message-ID: <e76aa17f0712040623v7128b220xe32378cf93b9eda0@mail.gmail.com>

>
> This is great stuff by the way, consider me a scons convert.
>

I'm already convinced for some time. Now, we just need an egg Builder :|

Matthieu
-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071204/6bfd56c9/attachment.html>

From cournape at gmail.com  Tue Dec  4 10:52:17 2007
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 5 Dec 2007 00:52:17 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <91cf711d0712040620g3a33dcaftcfa34d3f2f640c26@mail.gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<91cf711d0712040620g3a33dcaftcfa34d3f2f640c26@mail.gmail.com>
Message-ID: <5b8d13220712040752i5832882egc15682e5fa0a8020@mail.gmail.com>

On Dec 4, 2007 11:20 PM, David Huard <david.huard at gmail.com> wrote:
> David,
>
> I tried building the scons numpy and scipy. Numpy apparently built fine
> since all tests pass. For scipy, I am having problem during the build.
> Something seems to be wrong with libdfftpack.a
>
Grrrr, this is because of a scons bug. I temporarily fixed it using a
quick hack in rev 4551 (the only clean way is to  fix those upstream,
but scons developers are not always quick to response to patches and
suggestions, unfortunately)

David


From cournape at gmail.com  Tue Dec  4 10:55:05 2007
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 5 Dec 2007 00:55:05 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <e76aa17f0712040623v7128b220xe32378cf93b9eda0@mail.gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<91cf711d0712040620g3a33dcaftcfa34d3f2f640c26@mail.gmail.com>
	<e76aa17f0712040623v7128b220xe32378cf93b9eda0@mail.gmail.com>
Message-ID: <5b8d13220712040755h3a2a1964h99686bde20f03f08@mail.gmail.com>

On Dec 4, 2007 11:23 PM, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>
> > This is great stuff by the way, consider me a scons convert.
> >
>
> I'm already convinced for some time. Now, we just need an egg Builder :|
No we don't, since we are still using distutils: scons really just
handles compiled extensions, but everything else is still done by
distutils, hence can be handled by setuptools. I have not tested it,
but since I try to keep the interaction with numpy.distutils as
minimal as possible, it should be quick to fix any issues regarding
eggs, if any.

David


From tim.hochberg at ieee.org  Tue Dec  4 11:07:53 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Tue, 4 Dec 2007 09:07:53 -0700
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <47552666.3050601@ar.media.kyoto-u.ac.jp>
References: <474FCE2D.2000908@mspacek.mm.st>
	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>
	<4750981C.9060409@mspacek.mm.st>
	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
	<47535A69.6000801@mspacek.mm.st>
	<20071203062723.GB10287@clipper.ens.fr>
	<4754AB79.7080004@mspacek.mm.st>
	<4754E211.2010903@ar.media.kyoto-u.ac.jp>
	<20071204093602.GD21476@clipper.ens.fr>
	<47552666.3050601@ar.media.kyoto-u.ac.jp>
Message-ID: <e4412d6b0712040807ga3c4ca3xebfe9201710d921b@mail.gmail.com>

On Dec 4, 2007 3:05 AM, David Cournapeau <david at ar.media.kyoto-u.ac.jp>
wrote:

> Gael Varoquaux wrote:
> > On Tue, Dec 04, 2007 at 02:13:53PM +0900, David Cournapeau wrote:
> >
> >> With recent kernels, you can get really good latency if you do it right
> >> (around 1-2 ms worst case under high load, including high IO pressure).
> >>
> >
> > As you can see on my page, I indeed measured less than 1ms latency on
> > Linux under load with kernel more than a year old. These things have
> > gotten much better recently and with a premptible kernel you should be
> > able to get 1ms easily. Going below 0.5ms without using a realtime OS
> (ie
> > a realtime kernel, under linux) is really pushing it.
> >
> Yes, 1ms is possible for quite a long time; the problem was how to get
> there (kernel patches, special permissions, etc... Many of those
> problems are now gone). I've read that you could get around 0.2 ms and
> even below (worst case) with the last kernels + RT preempt (that is you
> still use linux, and not rtlinux). Below 1 ms does not make much sense
> for audio applications, so I don't know much below this range :)
>
> But I am really curious if you can get those numbers with python,
> because of malloc, the gc and co. I mean for example, 0.5 ms latency for
> a 1 Ghz CPU means that you get something like a 500 000 CPU cycles, and
> I can imagine a cycle of garbage collection taking that many cycles,
> without even considering pages of virtual memory which are swapped (in
> this case, we are talking millions of cycles).


If the garbage collector is causing a slowdown, it is possible to turn it
off. Then you have to be careful to break cycles manually. Non cyclic
garbage will get picked up by reference counting, so you can ignore that.
Figuring out references in the context of numpy might be a little tricky
given that views imply references, but it's probably not impossible.

-tim




>
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071204/d6f479d3/attachment.html>

From robert.kern at gmail.com  Tue Dec  4 14:27:54 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 04 Dec 2007 13:27:54 -0600
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <47553843.7020709@ar.media.kyoto-u.ac.jp>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
Message-ID: <4755AA3A.30802@gmail.com>

David Cournapeau wrote:
> Some of the most interesting things I can think of which work with scons:
>     - you can control fortran and C flags from the command line: CFLAGS 
> and FFLAGS won't override necessary flags, only optimization flags, so 
> you can easily play with warning, optimization flags. For example:
> 
> CFLAGS='-W -Wall -Wextra -DDEBUG' FFLAGS='-DDEBUG -W -Wall -Wextra' 
> python setupscons build
> 
> for debugging will work. No need to care about -fPIC and co, all this is 
> handled automatically.

Can I override the flags which are handled automatically without modifying
numpy? I just spent much of last night trying to get Intel Fortran on OS X
working, and I had to dive into numpy.distutils.fcompiler.intel a lot. This is
mildly acceptable, if irritating, for a numpy developer, but not acceptable for
even a sophisticated user. Even if we were to keep our knowledge-base of Fortran
compiler flags immediately up-to-date with every release of every Fortran
compiler we follow, people will still be stuck with older versions of numpy.

numpy.distutils' behavior of using LDFLAGS, for example, to completely replace
the flags instead of extending them mitigated this, somewhat. It allowed someone
to work around the stale flags in numpy.distutils in order to get something
built. This is a hack, and it causes confusion when this isn't the desired
behavior, but it worked.

But can we do something better with scons? One option which would work both with
scons and the current numpy.distutils is to provide something like
LDFLAGS_NO_REALLY which replaces the flags and let LDFLAGS just extend the
flags. That doesn't help the ultimate problem, but it makes the workaround more
user-friendly. Another option is to have our Fortran compiler "knowledge-base"
separable from the rest of the package. scons could try to import them from,
say, numpy_fcompilers first and then look inside numpy.distutils if
numpy_fcompilers is not found. That way, a user could download a fresh
"knowledge-base" into their source tree (and possibly tweak it) without the
burden of installing a new numpy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From fperez.net at gmail.com  Tue Dec  4 14:36:06 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Dec 2007 12:36:06 -0700
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <4755AA3A.30802@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<4755AA3A.30802@gmail.com>
Message-ID: <db6b5ecc0712041136x73e7ab11wccdbdd833cf027e8@mail.gmail.com>

On Dec 4, 2007 12:27 PM, Robert Kern <robert.kern at gmail.com> wrote:

> user-friendly. Another option is to have our Fortran compiler "knowledge-base"
> separable from the rest of the package. scons could try to import them from,
> say, numpy_fcompilers first and then look inside numpy.distutils if
> numpy_fcompilers is not found. That way, a user could download a fresh
> "knowledge-base" into their source tree (and possibly tweak it) without the
> burden of installing a new numpy.

Is this something that really needs to be a code package?  Why can't
this knowledge (or at least the easily overridable part of it) be
packaged in one or more .conf/.ini plaintext files?  In that way,
users could easily grab new data files or tweak the builtin ones, and
at build time say

setup.py install  --compiler_conf=~/my_tweaked.conf

Is that impossible/unreasonable for some reason?

Cheers,

f


From robert.kern at gmail.com  Tue Dec  4 15:24:24 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 04 Dec 2007 14:24:24 -0600
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <db6b5ecc0712041136x73e7ab11wccdbdd833cf027e8@mail.gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>	<4755AA3A.30802@gmail.com>
	<db6b5ecc0712041136x73e7ab11wccdbdd833cf027e8@mail.gmail.com>
Message-ID: <4755B778.9020007@gmail.com>

Fernando Perez wrote:
> On Dec 4, 2007 12:27 PM, Robert Kern <robert.kern at gmail.com> wrote:
> 
>> user-friendly. Another option is to have our Fortran compiler "knowledge-base"
>> separable from the rest of the package. scons could try to import them from,
>> say, numpy_fcompilers first and then look inside numpy.distutils if
>> numpy_fcompilers is not found. That way, a user could download a fresh
>> "knowledge-base" into their source tree (and possibly tweak it) without the
>> burden of installing a new numpy.
> 
> Is this something that really needs to be a code package?  Why can't
> this knowledge (or at least the easily overridable part of it) be
> packaged in one or more .conf/.ini plaintext files?  In that way,
> users could easily grab new data files or tweak the builtin ones, and
> at build time say
> 
> setup.py install  --compiler_conf=~/my_tweaked.conf
> 
> Is that impossible/unreasonable for some reason?

It's not impossible, but there are at least a couple of places where it might be
unreasonable. For example, look at the get_flags_arch() for Intel compilers:

    def get_flags_arch(self):
        v = self.get_version()
        opt = []
        if cpu.has_fdiv_bug():
            opt.append('-fdiv_check')
        if cpu.has_f00f_bug():
            opt.append('-0f_check')
        if cpu.is_PentiumPro() or cpu.is_PentiumII() or cpu.is_PentiumIII():
            opt.extend(['-tpp6'])
        elif cpu.is_PentiumM():
            opt.extend(['-tpp7','-xB'])
        elif cpu.is_Pentium():
            opt.append('-tpp5')
        elif cpu.is_PentiumIV() or cpu.is_Xeon():
            opt.extend(['-tpp7','-xW'])
        if v and v <= '7.1':
            if cpu.has_mmx() and (cpu.is_PentiumII() or cpu.is_PentiumIII()):
                opt.append('-xM')
        elif v and v >= '8.0':
            if cpu.is_PentiumIII():
                opt.append('-xK')
                if cpu.has_sse3():
                    opt.extend(['-xP'])
            elif cpu.is_PentiumIV():
                opt.append('-xW')
                if cpu.has_sse2():
                    opt.append('-xN')
            elif cpu.is_PentiumM():
                opt.extend(['-xB'])
            if (cpu.is_Xeon() or cpu.is_Core2() or cpu.is_Core2Extreme()) and
cpu.getNCPUs()==2:
                opt.extend(['-xT'])
            if cpu.has_sse3() and (cpu.is_PentiumIV() or cpu.is_CoreDuo() or
cpu.is_CoreSolo()):
                opt.extend(['-xP'])

        if cpu.has_sse2():
            opt.append('-arch SSE2')
        elif cpu.has_sse():
            opt.append('-arch SSE')
        return opt


Expressing that without code could be hairy.

That said, using configuration files as override mechanisms for each of the
get_flags_*() methods would be feasible especially if there were a script to
dump the current flag set to the configuration file.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From fperez.net at gmail.com  Tue Dec  4 16:40:35 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Dec 2007 14:40:35 -0700
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <4755B778.9020007@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<4755AA3A.30802@gmail.com>
	<db6b5ecc0712041136x73e7ab11wccdbdd833cf027e8@mail.gmail.com>
	<4755B778.9020007@gmail.com>
Message-ID: <db6b5ecc0712041340i4e12414cp98088a82168c34ff@mail.gmail.com>

On Dec 4, 2007 1:24 PM, Robert Kern <robert.kern at gmail.com> wrote:
>
> Fernando Perez wrote:

> > Is this something that really needs to be a code package?  Why can't
> > this knowledge (or at least the easily overridable part of it) be
> > packaged in one or more .conf/.ini plaintext files?  In that way,
> > users could easily grab new data files or tweak the builtin ones, and
> > at build time say
> >
> > setup.py install  --compiler_conf=~/my_tweaked.conf
> >
> > Is that impossible/unreasonable for some reason?
>
> It's not impossible, but there are at least a couple of places where it might be
> unreasonable. For example, look at the get_flags_arch() for Intel compilers:

[...]

I see.  How about an alternate approach: exposing a simple api and
allowing users to declare a *python* file to execfile() at load time
looking for the config?

Something like:

setup.py install  --compiler_conf=~/my_tweaked_config.py

where the config file would be (sketch, not real code here):

def make_flags(compiler, etc...):
  flags = []
  ....
 return flags

There could be a simple API for what functions the config file (down
to their names and signatures) can declare, and if any of them are
declared, they get called and their output is used.  They get fed the
default state of the same variables, so that they can choose to modify
or outright replace them based on the user's need.

The config code would then

user_ns = {}
execfile(user_config_filename,user_ns)
for name,val in user_ns.items():
  if name in approved_functions and callable(val):
    flags[name] = val(*approved_functions[name].default_args)



What say you?

f


From robert.kern at gmail.com  Tue Dec  4 17:12:24 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 04 Dec 2007 16:12:24 -0600
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <db6b5ecc0712041340i4e12414cp98088a82168c34ff@mail.gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>	<4755AA3A.30802@gmail.com>	<db6b5ecc0712041136x73e7ab11wccdbdd833cf027e8@mail.gmail.com>	<4755B778.9020007@gmail.com>
	<db6b5ecc0712041340i4e12414cp98088a82168c34ff@mail.gmail.com>
Message-ID: <4755D0C8.4020504@gmail.com>

Fernando Perez wrote:
> On Dec 4, 2007 1:24 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> Fernando Perez wrote:
> 
>>> Is this something that really needs to be a code package?  Why can't
>>> this knowledge (or at least the easily overridable part of it) be
>>> packaged in one or more .conf/.ini plaintext files?  In that way,
>>> users could easily grab new data files or tweak the builtin ones, and
>>> at build time say
>>>
>>> setup.py install  --compiler_conf=~/my_tweaked.conf
>>>
>>> Is that impossible/unreasonable for some reason?
>> It's not impossible, but there are at least a couple of places where it might be
>> unreasonable. For example, look at the get_flags_arch() for Intel compilers:
> 
> [...]
> 
> I see.  How about an alternate approach: exposing a simple api and
> allowing users to declare a *python* file to execfile() at load time
> looking for the config?
> 
> Something like:
> 
> setup.py install  --compiler_conf=~/my_tweaked_config.py
> 
> where the config file would be (sketch, not real code here):
> 
> def make_flags(compiler, etc...):
>   flags = []
>   ....
>  return flags
> 
> There could be a simple API for what functions the config file (down
> to their names and signatures) can declare, and if any of them are
> declared, they get called and their output is used.  They get fed the
> default state of the same variables, so that they can choose to modify
> or outright replace them based on the user's need.
> 
> The config code would then
> 
> user_ns = {}
> execfile(user_config_filename,user_ns)
> for name,val in user_ns.items():
>   if name in approved_functions and callable(val):
>     flags[name] = val(*approved_functions[name].default_args)
> 
> 
> 
> What say you?

Well, like I said, for tweaking, a simple data file works better than code.
There's no need to do all of those "if" tests since I know what platform I'm on.
We should definitely have a simple data file that we can read flags from. It's
just the general case that requires code.

One thing in favor of numpy_fcompilers is that we can ship the updates to the
general case more frequently. This means that other packages using Fortran (but
not tied to a particular platform) can ship the updated code instead of telling
all of their users to read their Fortran compiler manuals or ask on the mailing
list for the correct settings to work around our old defects.

Tweaking should be the province of the developer and the desperate. We have to
be able to tweak, but we should spend more time on preventing the need for tweaking.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Tue Dec  4 19:01:12 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 04 Dec 2007 18:01:12 -0600
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <47553843.7020709@ar.media.kyoto-u.ac.jp>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
Message-ID: <4755EA48.9010302@gmail.com>

David Cournapeau wrote:

>     - I have not yet tweaked fortran compiler configurations for 
> optimizations except for gnu compilers

Can you give us a brief overview about how to do this? For example, the Intel
Fortran compiler's SHLINKFLAGS in scons-local/.../SCons/Tool/ifort.py are
incorrect for version 10 on OS X. Would I copy that file to scons/tool/ and make
my edits there? Do I then add 'ifort' to the list in scons/core/default.py?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Tue Dec  4 21:55:46 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 05 Dec 2007 11:55:46 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <4755AA3A.30802@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<4755AA3A.30802@gmail.com>
Message-ID: <47561332.3090200@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> David Cournapeau wrote:
>> Some of the most interesting things I can think of which work with scons:
>>     - you can control fortran and C flags from the command line: CFLAGS 
>> and FFLAGS won't override necessary flags, only optimization flags, so 
>> you can easily play with warning, optimization flags. For example:
>>
>> CFLAGS='-W -Wall -Wextra -DDEBUG' FFLAGS='-DDEBUG -W -Wall -Wextra' 
>> python setupscons build
>>
>> for debugging will work. No need to care about -fPIC and co, all this is 
>> handled automatically.
>
> Can I override the flags which are handled automatically without modifying
> numpy? I just spent much of last night trying to get Intel Fortran on OS X
> working, and I had to dive into numpy.distutils.fcompiler.intel a lot. This is
> mildly acceptable, if irritating, for a numpy developer, but not acceptable for
> even a sophisticated user. Even if we were to keep our knowledge-base of Fortran
> compiler flags immediately up-to-date with every release of every Fortran
> compiler we follow, people will still be stuck with older versions of numpy.
>
> numpy.distutils' behavior of using LDFLAGS, for example, to completely replace
> the flags instead of extending them mitigated this, somewhat. It allowed someone
> to work around the stale flags in numpy.distutils in order to get something
> built. This is a hack, and it causes confusion when this isn't the desired
> behavior, but it worked.
>
> But can we do something better with scons? One option which would work both with
> scons and the current numpy.distutils is to provide something like
> LDFLAGS_NO_REALLY which replaces the flags and let LDFLAGS just extend the
> flags. That doesn't help the ultimate problem, but it makes the workaround more
> user-friendly. Another option is to have our Fortran compiler "knowledge-base"
> separable from the rest of the package. scons could try to import them from,
> say, numpy_fcompilers first and then look inside numpy.distutils if
> numpy_fcompilers is not found. That way, a user could download a fresh
> "knowledge-base" into their source tree (and possibly tweak it) without the
> burden of installing a new numpy.
>
First, let me briefly outline the basic flow of scons (for every 
package, I call scons; since they are totally independant, we just 
consider one package):
    - the distutils scons command find the compilers, their path, and 
set up the build directory, and then call scons with those parameters at 
the command line (scons runs in its own process). This is done in 
numpy\distutils\command\scons.py
    - I create an environment using the parameters given by distutils: I 
initialize scons CC, FORTRAN and CXX tools. Those tools define several 
flags, like -fPIC for shared objects, etc... this is done in 
numpy\distutils\scons\core\numpyenv.py (_GetNumpyEnvironment).
    - I customize the flags depending on the compiler used: this is done 
in numpy\distutils\scons\core\default.py. This is where the optimization 
flags, warning flags are set. I then merge those flags with scons 
environments, and those are the ones the user get (in SConstruct) in 
GetNumpyEnvironment.

So you really have two places where flags are set:
    - optimization and warning are set in the default.py. We could use a 
file to override those quite easily.
    - flags like -fPIC are defined at the tool level. This is done 
inside scons; for cases where it does not work, I use my own modified 
tools (in numpy\distutils\scons\tools: any tool in this directory will 
be picked up first, before the scons ones).

So to go back to your problem: if I understand correctly, what is needed 
is to update the scons tools. Since those are kept at one place, I think 
it would be safe to update them independently. But I don't understand 
exactly how this could be installed in the source tree without 
reinstalling numpy ? I think this is better than completely overriding 
the compilation flags, personally. But if you really want this 
possibility, I can add it, too, without too much trouble.

Normally, the way I define and use compilation flags should be flexible 
enough to enable several approaches. I implemented the ones I most 
needed, but other are welcome.

David



From david at ar.media.kyoto-u.ac.jp  Tue Dec  4 22:00:39 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 05 Dec 2007 12:00:39 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <4755D0C8.4020504@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>	<4755AA3A.30802@gmail.com>	<db6b5ecc0712041136x73e7ab11wccdbdd833cf027e8@mail.gmail.com>	<4755B778.9020007@gmail.com>	<db6b5ecc0712041340i4e12414cp98088a82168c34ff@mail.gmail.com>
	<4755D0C8.4020504@gmail.com>
Message-ID: <47561457.2080908@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
>
> Tweaking should be the province of the developer and the desperate. We have to
> be able to tweak, but we should spend more time on preventing the need for tweaking.
>   
Yes, I can only agree 100 % with you. This is maybe *the* reason why I 
started this whole thing. Distutils "architecture" (I am talking about 
distutils in the python stdlib) makes tweaking totally impossible for 
the mere mortal because it modifies everything everywhere.

We should be able to build out of the box without tweaking in more 
situations than now; tweaking should be possible, but not at the expense 
of the out-of-the-box experience.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Tue Dec  4 22:19:31 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 05 Dec 2007 12:19:31 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <4755EA48.9010302@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<4755EA48.9010302@gmail.com>
Message-ID: <475618C3.8090901@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> David Cournapeau wrote:
>
>>     - I have not yet tweaked fortran compiler configurations for 
>> optimizations except for gnu compilers
>
> Can you give us a brief overview about how to do this? For example, the Intel
> Fortran compiler's SHLINKFLAGS in scons-local/.../SCons/Tool/ifort.py are
> incorrect for version 10 on OS X. Would I copy that file to scons/tool/ and make
> my edits there? Do I then add 'ifort' to the list in scons/core/default.py?
>
The basic rule is: if the code cannot run without a flag, the flag 
should be put in a tool, or at worse (but really if you have no choice) 
in numpyenv.py. If the flag is optimization, warning, etc... then it 
should be put into default.py. Basically, tools are not always 
up-to-date in scons, perticularly for fortran. So I provided a way to 
override the tools: as you noticed, you can put tools in 
.../scons/tools/, those will be picked up first. This is independent 
from adding ifort in scons/core/default.py.

For Mac OS X, you may be bitten by  -undefined dynamic_lookup. This is 
my fault: this flag is added at the wrong place, I put it temporarily in 
the python extension builder, but this is not where it should be put. 
Depending on its meaning, I can put it at the right place: does it give 
the traditional unix semantic of enabling unresolved symbols instead of 
the default one, which is similar to windows (even for shared code, 
every symbol must be resolved) ?

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Tue Dec  4 22:21:50 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 05 Dec 2007 12:21:50 +0900
Subject: [Numpy-discussion] Missing file in SVN repository (test_ufunc)
In-Reply-To: <20071204131141.GA8346@mentat.za.net>
References: <47553674.4010204@ar.media.kyoto-u.ac.jp>
	<20071204131141.GA8346@mentat.za.net>
Message-ID: <4756194E.7030402@ar.media.kyoto-u.ac.jp>

Stefan van der Walt wrote:
> On Tue, Dec 04, 2007 at 08:13:56PM +0900, David Cournapeau wrote:
>   
>>     Since revision 4528, it is not possible to run the tests for 
>> numpy.core, because of a missing file, test_ufunc.py. Since the change 
>> in the test was done in r4528 whose log refers " increase of code 
>> coverage", I guess the file was not added to the repository,
>>     
>
> Sorry about that.  I added the file in r4492.
>   
Thanks !

David


From robert.kern at gmail.com  Tue Dec  4 23:14:42 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 04 Dec 2007 22:14:42 -0600
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <47561332.3090200@ar.media.kyoto-u.ac.jp>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>	<4755AA3A.30802@gmail.com>
	<47561332.3090200@ar.media.kyoto-u.ac.jp>
Message-ID: <475625B2.1010806@gmail.com>

David Cournapeau wrote:

> So to go back to your problem: if I understand correctly, what is needed 
> is to update the scons tools. Since those are kept at one place, I think 
> it would be safe to update them independently. But I don't understand 
> exactly how this could be installed in the source tree without 
> reinstalling numpy ? I think this is better than completely overriding 
> the compilation flags, personally. But if you really want this 
> possibility, I can add it, too, without too much trouble.

I don't think you could install it into an already installed numpy package. My
suggestion is to keep the implementations of the tools inside the numpy package
as they are now *except* that we look for another package first before going
inside numpy.distutils.scons.tools . I called it "numpy_fcompilers" though I now
suspect "numpy_scons_tools" might be more appropriate. If the package
numpy_scons_tools doesn't exist, the implementations inside
numpy.distutils.scons.tools are used. A variation on this would be to provide an
--option to the scons command to provide a "package path" to look for tools. E.g.

  python setup.py scons
--tool-path=my_local_fcompilers,site_fcompilers,numpy.distutils.scons.tools

This, too, is a workaround for the less-than-desirable situation of having
numpy's sizable build infrastructure bundled with the numpy package itself. If
this stuff were an entirely separate package focused on providing this
scons-based build infrastructure, then we wouldn't have a problem. We could just
update it on its own release schedule. People would probably be more willing to
use the development versions of it, too, instead of having to also buy into the
development version of their array package as well.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Tue Dec  4 23:19:27 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 04 Dec 2007 22:19:27 -0600
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <475618C3.8090901@ar.media.kyoto-u.ac.jp>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>	<4755EA48.9010302@gmail.com>
	<475618C3.8090901@ar.media.kyoto-u.ac.jp>
Message-ID: <475626CF.8040404@gmail.com>

David Cournapeau wrote:
> Robert Kern wrote:
>> David Cournapeau wrote:
>>
>>>     - I have not yet tweaked fortran compiler configurations for 
>>> optimizations except for gnu compilers
>> Can you give us a brief overview about how to do this? For example, the Intel
>> Fortran compiler's SHLINKFLAGS in scons-local/.../SCons/Tool/ifort.py are
>> incorrect for version 10 on OS X. Would I copy that file to scons/tool/ and make
>> my edits there? Do I then add 'ifort' to the list in scons/core/default.py?
>>
> The basic rule is: if the code cannot run without a flag, the flag 
> should be put in a tool, or at worse (but really if you have no choice) 
> in numpyenv.py. If the flag is optimization, warning, etc... then it 
> should be put into default.py. Basically, tools are not always 
> up-to-date in scons, perticularly for fortran. So I provided a way to 
> override the tools: as you noticed, you can put tools in 
> .../scons/tools/, those will be picked up first. This is independent 
> from adding ifort in scons/core/default.py.

Right. In this case, "-shared" needs to be "-dynamiclib" on OS X, so this should
go into the tool.

> For Mac OS X, you may be bitten by  -undefined dynamic_lookup. This is 
> my fault: this flag is added at the wrong place, I put it temporarily in 
> the python extension builder, but this is not where it should be put. 
> Depending on its meaning, I can put it at the right place: does it give 
> the traditional unix semantic of enabling unresolved symbols instead of 
> the default one, which is similar to windows (even for shared code, 
> every symbol must be resolved) ?

That's the basic idea. Rigorously, it's probably a bit more involved when you
start considering two-level namespaces and framework.

One thing to note is that this option is only valid for GNU compilers. Linking
with ifort, I need to use -Wl,-undefined,dynamic_lookup .

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From cournape at gmail.com  Wed Dec  5 00:38:58 2007
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 5 Dec 2007 14:38:58 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <475626CF.8040404@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<4755EA48.9010302@gmail.com> <475618C3.8090901@ar.media.kyoto-u.ac.jp>
	<475626CF.8040404@gmail.com>
Message-ID: <5b8d13220712042138p7459b0bp24d2e64c6dab9d5c@mail.gmail.com>

On Dec 5, 2007 1:19 PM, Robert Kern <robert.kern at gmail.com> wrote:
> David Cournapeau wrote:
> > Robert Kern wrote:
> >> David Cournapeau wrote:
> >>
> >>>     - I have not yet tweaked fortran compiler configurations for
> >>> optimizations except for gnu compilers
> >> Can you give us a brief overview about how to do this? For example, the Intel
> >> Fortran compiler's SHLINKFLAGS in scons-local/.../SCons/Tool/ifort.py are
> >> incorrect for version 10 on OS X. Would I copy that file to scons/tool/ and make
> >> my edits there? Do I then add 'ifort' to the list in scons/core/default.py?
> >>
> > The basic rule is: if the code cannot run without a flag, the flag
> > should be put in a tool, or at worse (but really if you have no choice)
> > in numpyenv.py. If the flag is optimization, warning, etc... then it
> > should be put into default.py. Basically, tools are not always
> > up-to-date in scons, perticularly for fortran. So I provided a way to
> > override the tools: as you noticed, you can put tools in
> > .../scons/tools/, those will be picked up first. This is independent
> > from adding ifort in scons/core/default.py.
>
> Right. In this case, "-shared" needs to be "-dynamiclib" on OS X, so this should
> go into the tool.
That's strange: -shared should not be used at all on mac os X. Either
-bundle or -dynamiclib should be used (this is in applelink tool, so
this is independant from the compiler used, normally).

But I may have done something wrong, because I don't know much about
mac os X idiosyncraties on this: basically, what's the difference
between -dynamiclib and -bundle ?

When I build python extension, I used the module scons builder, which
is the same than shared library except on mac os X (on mac os X,
shared libraries use -dynamiclib, modules use -bundle). I must confess
that I used the thing which worked in thise case, without knowing
exactly what i was doing.

>
> That's the basic idea. Rigorously, it's probably a bit more involved when you
> start considering two-level namespaces and framework.
>
scons handles frameworks, but I feel that it is gcc specific.

> One thing to note is that this option is only valid for GNU compilers. Linking
> with ifort, I need to use -Wl,-undefined,dynamic_lookup .
Can't we just add a linker flag instead of using it from the compiler
? We still use the apple linker with ifort/icc, no ?

To sum it up: I think that implementation-wise, scons on mac os X has
many gcc-only idiosyncraties; fortunately, once we know exactly what
the flags should be in which cases, this is only a matter of changing
the intel tools (for intel compilers on mac os X). OTOH, API-wise,
there is no gcc idiodyncraties, which is the thing which matters in
the mid-term: scons tools abstraction is quite good IMHO, and you
don't have to fear breaking unrelated things like you do with
distutils.

All this should be modified to be sent upstream to scons, also.

cheers,

David


From cournape at gmail.com  Wed Dec  5 00:47:20 2007
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 5 Dec 2007 14:47:20 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <475625B2.1010806@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<4755AA3A.30802@gmail.com> <47561332.3090200@ar.media.kyoto-u.ac.jp>
	<475625B2.1010806@gmail.com>
Message-ID: <5b8d13220712042147x72c2fcb3j9bce2887afe37604@mail.gmail.com>

On Dec 5, 2007 1:14 PM, Robert Kern <robert.kern at gmail.com> wrote:
> David Cournapeau wrote:
>
> > So to go back to your problem: if I understand correctly, what is needed
> > is to update the scons tools. Since those are kept at one place, I think
> > it would be safe to update them independently. But I don't understand
> > exactly how this could be installed in the source tree without
> > reinstalling numpy ? I think this is better than completely overriding
> > the compilation flags, personally. But if you really want this
> > possibility, I can add it, too, without too much trouble.
>
> I don't think you could install it into an already installed numpy package. My
> suggestion is to keep the implementations of the tools inside the numpy package
> as they are now *except* that we look for another package first before going
> inside numpy.distutils.scons.tools . I called it "numpy_fcompilers" though I now
> suspect "numpy_scons_tools" might be more appropriate. If the package
> numpy_scons_tools doesn't exist, the implementations inside
> numpy.distutils.scons.tools are used. A variation on this would be to provide an
> --option to the scons command to provide a "package path" to look for tools. E.g.
>
>   python setup.py scons
> --tool-path=my_local_fcompilers,site_fcompilers,numpy.distutils.scons.tools
>
Ok, I was confused by numpy_fcompilers, I thought you were talking
about the numpy.distutils.fcompiler module.

A priori, I don't think it would cause any trouble to add this option.
Internally, I already use a list of directories to look for paths, so
this should be a matter of pre-pending the new directory to this list.

> This, too, is a workaround for the less-than-desirable situation of having
> numpy's sizable build infrastructure bundled with the numpy package itself. If
> this stuff were an entirely separate package focused on providing this
> scons-based build infrastructure, then we wouldn't have a problem. We could just
> update it on its own release schedule. People would probably be more willing to
> use the development versions of it, too, instead of having to also buy into the
> development version of their array package as well.

The only problem I see is that this increases the chance of losing
synchronization. I don't know if this is significant. IMHO, the only
real solution would be to fix distutils (how many people want a shared
library builder in python community, for example ?), but well, it is
not gonna happen in a foreseeable future, unfortunately

cheers,

David


From robert.kern at gmail.com  Wed Dec  5 01:07:02 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 05 Dec 2007 00:07:02 -0600
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <5b8d13220712042147x72c2fcb3j9bce2887afe37604@mail.gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>	<4755AA3A.30802@gmail.com>
	<47561332.3090200@ar.media.kyoto-u.ac.jp>	<475625B2.1010806@gmail.com>
	<5b8d13220712042147x72c2fcb3j9bce2887afe37604@mail.gmail.com>
Message-ID: <47564006.1060509@gmail.com>

David Cournapeau wrote:
> On Dec 5, 2007 1:14 PM, Robert Kern <robert.kern at gmail.com> wrote:

>> This, too, is a workaround for the less-than-desirable situation of having
>> numpy's sizable build infrastructure bundled with the numpy package itself. If
>> this stuff were an entirely separate package focused on providing this
>> scons-based build infrastructure, then we wouldn't have a problem. We could just
>> update it on its own release schedule. People would probably be more willing to
>> use the development versions of it, too, instead of having to also buy into the
>> development version of their array package as well.
> 
> The only problem I see is that this increases the chance of losing
> synchronization. I don't know if this is significant.

The problem I see is that numpy-the-array-library and
numpy.distutils-the-build-infrastructure are two related packages with
*over*-synchronized cycles. We aren't going to push out a micro-release of
numpy-the-array-library just because a new version of Intel Fortran comes out
and changes its flags.

> IMHO, the only
> real solution would be to fix distutils (how many people want a shared
> library builder in python community, for example ?), but well, it is
> not gonna happen in a foreseeable future, unfortunately

I don't see how that's relevant to the problem I raised. Supporting Fortran in
the standard library would make the problem even worse. distutils is certainly
not broken because it doesn't support Fortran.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Wed Dec  5 01:11:28 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 05 Dec 2007 00:11:28 -0600
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <5b8d13220712042138p7459b0bp24d2e64c6dab9d5c@mail.gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>	<4755EA48.9010302@gmail.com>
	<475618C3.8090901@ar.media.kyoto-u.ac.jp>	<475626CF.8040404@gmail.com>
	<5b8d13220712042138p7459b0bp24d2e64c6dab9d5c@mail.gmail.com>
Message-ID: <47564110.2000700@gmail.com>

David Cournapeau wrote:
> On Dec 5, 2007 1:19 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> David Cournapeau wrote:
>>> Robert Kern wrote:
>>>> David Cournapeau wrote:
>>>>
>>>>>     - I have not yet tweaked fortran compiler configurations for
>>>>> optimizations except for gnu compilers
>>>> Can you give us a brief overview about how to do this? For example, the Intel
>>>> Fortran compiler's SHLINKFLAGS in scons-local/.../SCons/Tool/ifort.py are
>>>> incorrect for version 10 on OS X. Would I copy that file to scons/tool/ and make
>>>> my edits there? Do I then add 'ifort' to the list in scons/core/default.py?
>>>>
>>> The basic rule is: if the code cannot run without a flag, the flag
>>> should be put in a tool, or at worse (but really if you have no choice)
>>> in numpyenv.py. If the flag is optimization, warning, etc... then it
>>> should be put into default.py. Basically, tools are not always
>>> up-to-date in scons, perticularly for fortran. So I provided a way to
>>> override the tools: as you noticed, you can put tools in
>>> .../scons/tools/, those will be picked up first. This is independent
>>> from adding ifort in scons/core/default.py.
>> Right. In this case, "-shared" needs to be "-dynamiclib" on OS X, so this should
>> go into the tool.
> That's strange: -shared should not be used at all on mac os X. Either
> -bundle or -dynamiclib should be used (this is in applelink tool, so
> this is independant from the compiler used, normally).

I was only reading code; I haven't tested building Fortran extensions, yet.
However, using a generic link tool would be the wrong thing to do for most
Fortran extensions, I think. Where does it get the correct Fortran runtime
libraries from? Some Fortran compilers really like to be the linker when mixing
languages.

> But I may have done something wrong, because I don't know much about
> mac os X idiosyncraties on this: basically, what's the difference
> between -dynamiclib and -bundle ?
> 
> When I build python extension, I used the module scons builder, which
> is the same than shared library except on mac os X (on mac os X,
> shared libraries use -dynamiclib, modules use -bundle). I must confess
> that I used the thing which worked in thise case, without knowing
> exactly what i was doing.

ifort only supports -dynamiclib. For the regular linker, -bundle is correct for
building Python extensions; I may have to rethink about using ifort to link,
then. Basically, a bundle can be dynamically loaded while dylibs can't, so
Python uses bundles for extension modules.

  http://www.finkproject.org/doc/porting/shared.php

What confuses me is that I successfully built some Fortran modules last night
using numpy.distutils and ifort -dynamiclib. Hmm.

>> One thing to note is that this option is only valid for GNU compilers. Linking
>> with ifort, I need to use -Wl,-undefined,dynamic_lookup .
> Can't we just add a linker flag instead of using it from the compiler
> ? We still use the apple linker with ifort/icc, no ?

I don't know. We'd have to locate all of the Fortran runtime libraries and add
them. How do we do that? Or is that already done?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From cournape at gmail.com  Wed Dec  5 01:19:33 2007
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 5 Dec 2007 15:19:33 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <47564006.1060509@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<4755AA3A.30802@gmail.com> <47561332.3090200@ar.media.kyoto-u.ac.jp>
	<475625B2.1010806@gmail.com>
	<5b8d13220712042147x72c2fcb3j9bce2887afe37604@mail.gmail.com>
	<47564006.1060509@gmail.com>
Message-ID: <5b8d13220712042219m70a69fd5xfec820d051f9ee15@mail.gmail.com>

On Dec 5, 2007 3:07 PM, Robert Kern <robert.kern at gmail.com> wrote:
>
>
> I don't see how that's relevant to the problem I raised. Supporting Fortran in
> the standard library would make the problem even worse. distutils is certainly
> not broken because it doesn't support Fortran.
>
Fortran support is indeed certainly not in the scope of distutils. I
was just answering to the general problem that we have a huge build
infrastructure, not to the particular fortran problem. Having an
infrastructure for adding easily new tools, that can be distributed
separately, is something that distutils severely lacks IMHO.

David


From robert.kern at gmail.com  Wed Dec  5 01:28:58 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 05 Dec 2007 00:28:58 -0600
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
 with scons
In-Reply-To: <5b8d13220712042219m70a69fd5xfec820d051f9ee15@mail.gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>	<4755AA3A.30802@gmail.com>
	<47561332.3090200@ar.media.kyoto-u.ac.jp>	<475625B2.1010806@gmail.com>	<5b8d13220712042147x72c2fcb3j9bce2887afe37604@mail.gmail.com>	<47564006.1060509@gmail.com>
	<5b8d13220712042219m70a69fd5xfec820d051f9ee15@mail.gmail.com>
Message-ID: <4756452A.9080009@gmail.com>

David Cournapeau wrote:
> On Dec 5, 2007 3:07 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> I don't see how that's relevant to the problem I raised. Supporting Fortran in
>> the standard library would make the problem even worse. distutils is certainly
>> not broken because it doesn't support Fortran.
>>
> Fortran support is indeed certainly not in the scope of distutils. I
> was just answering to the general problem that we have a huge build
> infrastructure, not to the particular fortran problem. Having an
> infrastructure for adding easily new tools, that can be distributed
> separately, is something that distutils severely lacks IMHO.

Ah, yes. I agree.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From cournape at gmail.com  Wed Dec  5 01:29:33 2007
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 5 Dec 2007 15:29:33 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <47564110.2000700@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<4755EA48.9010302@gmail.com> <475618C3.8090901@ar.media.kyoto-u.ac.jp>
	<475626CF.8040404@gmail.com>
	<5b8d13220712042138p7459b0bp24d2e64c6dab9d5c@mail.gmail.com>
	<47564110.2000700@gmail.com>
Message-ID: <5b8d13220712042229m60fba569s71c2b5d3e4a82072@mail.gmail.com>

On Dec 5, 2007 3:11 PM, Robert Kern <robert.kern at gmail.com> wrote:

>
> I was only reading code; I haven't tested building Fortran extensions, yet.
> However, using a generic link tool would be the wrong thing to do for most
> Fortran extensions, I think. Where does it get the correct Fortran runtime
> libraries from? Some Fortran compilers really like to be the linker when mixing
> languages.

Yes. SCons does not know how to do that, so I did it the "autoconf"
way: I implemented a mini library to get those linker flags, so that I
can still link C and Fortran code with gcc, not with gfortran or g77
(for example). The relevant code (+ tests) is in scons/fortran.py,
scons/fortran_scons.py + scons/tests. I have tested the parsing part
with g77, gfortran, ifort and sunfort.
>
> > But I may have done something wrong, because I don't know much about
> > mac os X idiosyncraties on this: basically, what's the difference
> > between -dynamiclib and -bundle ?
> >
> > When I build python extension, I used the module scons builder, which
> > is the same than shared library except on mac os X (on mac os X,
> > shared libraries use -dynamiclib, modules use -bundle). I must confess
> > that I used the thing which worked in thise case, without knowing
> > exactly what i was doing.
>
> ifort only supports -dynamiclib. For the regular linker, -bundle is correct for
> building Python extensions; I may have to rethink about using ifort to link,
> then. Basically, a bundle can be dynamically loaded while dylibs can't, so
> Python uses bundles for extension modules.
>
>   http://www.finkproject.org/doc/porting/shared.php

Ah, that rings a bell, I remember now.

>
> What confuses me is that I successfully built some Fortran modules last night
> using numpy.distutils and ifort -dynamiclib. Hmm.
>
> >> One thing to note is that this option is only valid for GNU compilers. Linking
> >> with ifort, I need to use -Wl,-undefined,dynamic_lookup .
> > Can't we just add a linker flag instead of using it from the compiler
> > ? We still use the apple linker with ifort/icc, no ?
>
> I don't know. We'd have to locate all of the Fortran runtime libraries and add
> them. How do we do that? Or is that already done?

Yes, this is already done. To see how it works concretely (from the
package developer POW), you could take a look there, for example:

http://projects.scipy.org/scipy/scipy/browser/branches/scipy.scons/scipy/integrate/SConstruct

You use CheckF77Clib during the configuration stage, and if
successfull, this put all the relevant link flags into
env['F77_LDFLAGS']. This is more "hackish" than defining fortran
runtime and co in the tools, but also more robust. What I like with
this approach is that it is testable (you can have trivial unit tests
to test those checkers, without the need for the tested tools to be
present).

cheers,

David


From kwgoodman at gmail.com  Wed Dec  5 13:30:44 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 5 Dec 2007 10:30:44 -0800
Subject: [Numpy-discussion] Unexpected conversion from matrix to array
Message-ID: <f4f93d420712051030sb1ff5dfsab7273aae9ef891f@mail.gmail.com>

>> import numpy.matlib as M
>> x = M.asmatrix(['a', 'b', 'c'])
>> x == 'a'
array([[ True, False, False]], dtype=bool)   #  <---- I expected a matrix

>> x = M.asmatrix([1, 2, 3])
>> x == 1
matrix([[ True, False, False]], dtype=bool)  #  <---- This looks good
>>
>> M.__version__
'1.0.5.dev4445'


From fperez.net at gmail.com  Wed Dec  5 13:38:34 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 5 Dec 2007 11:38:34 -0700
Subject: [Numpy-discussion] A quick f2py question
Message-ID: <db6b5ecc0712051038s6527eb7dt9644a577ede70389@mail.gmail.com>

Hi all,

I have a quick question on f2py.  I have a fortran lib I've wrapped
for a while and was updating one routine today, when I noticed a
message I'm curious about.  The .pyf file contains this signature:

        subroutine
createblocks(nnod,ll,nscale,nterms,pp,qq,aoffset,iflag,rintphi,rnorm)
! in :mwrep:createblocks.f
            integer intent(in) :: nnod
            integer intent(in) :: ll
	    integer intent(in) :: nscale
	    integer intent(hide),depend(pp) :: nterms = len(pp)
            real*8 dimension(nterms),intent(in) :: pp
            real*8 dimension(nterms),intent(in) :: qq
            real*8 dimension(nterms),intent(in) :: aoffset
            integer intent(in) :: iflag
            real*8
dimension(nterms*nnod*nnod),intent(out),depend(nterms,nnod) :: rintphi
	    real*8 dimension(nterms),intent(out),depend(nterms):: rnorm
        end subroutine createblocks

And I see this message in the build:

In: mwrep.pyf:mwrep:unknown_interface:createblocks
_get_depend_dict: no dependence info for 'len'

The build does actually continue, and in the end, this routine seems
to be correctly wrapped.  But I still worry that it may be getting the
right allocations by chance/accident.  Is this a real error message,
or just an internal warning from an intermediate pass?  Or has the
call for len() changed recently in f2py? (this code was originally
wrapped years ago, now I'm just doing minor updates).

Thanks for any info...

Cheers,

f


From pearu at cens.ioc.ee  Wed Dec  5 15:05:58 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Wed, 5 Dec 2007 22:05:58 +0200 (EET)
Subject: [Numpy-discussion] A quick f2py question
In-Reply-To: <db6b5ecc0712051038s6527eb7dt9644a577ede70389@mail.gmail.com>
References: <db6b5ecc0712051038s6527eb7dt9644a577ede70389@mail.gmail.com>
Message-ID: <55477.85.166.31.187.1196885158.squirrel@cens.ioc.ee>

On Wed, December 5, 2007 8:38 pm, Fernando Perez wrote:
...
> And I see this message in the build:
>
> In: mwrep.pyf:mwrep:unknown_interface:createblocks
> _get_depend_dict: no dependence info for 'len'

This is due to a typo introduced in r4511 and is now fixed in
r4553. This bug should not affect resulting extension module.

Thanks for the issue report,
Pearu



From fperez.net at gmail.com  Wed Dec  5 15:23:30 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 5 Dec 2007 13:23:30 -0700
Subject: [Numpy-discussion] A quick f2py question
In-Reply-To: <55477.85.166.31.187.1196885158.squirrel@cens.ioc.ee>
References: <db6b5ecc0712051038s6527eb7dt9644a577ede70389@mail.gmail.com>
	<55477.85.166.31.187.1196885158.squirrel@cens.ioc.ee>
Message-ID: <db6b5ecc0712051223l2ca6a2fawebe4b8740ea3b865@mail.gmail.com>

On Dec 5, 2007 1:05 PM, Pearu Peterson <pearu at cens.ioc.ee> wrote:
> On Wed, December 5, 2007 8:38 pm, Fernando Perez wrote:
> ...
> > And I see this message in the build:
> >
> > In: mwrep.pyf:mwrep:unknown_interface:createblocks
> > _get_depend_dict: no dependence info for 'len'
>
> This is due to a typo introduced in r4511 and is now fixed in
> r4553. This bug should not affect resulting extension module.
>
> Thanks for the issue report,

Great, many thanks for the clarification and quick action.

Cheers,

f


From fperez.net at gmail.com  Wed Dec  5 16:54:28 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 5 Dec 2007 14:54:28 -0700
Subject: [Numpy-discussion] site down...
Message-ID: <db6b5ecc0712051354t7a157e84u24dc35e88ea1f13e@mail.gmail.com>

The whole scipy.org site seems down.  Is it just on my end?

Cheers,
f


From robert.kern at gmail.com  Wed Dec  5 17:05:49 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 05 Dec 2007 16:05:49 -0600
Subject: [Numpy-discussion] site down...
In-Reply-To: <db6b5ecc0712051354t7a157e84u24dc35e88ea1f13e@mail.gmail.com>
References: <db6b5ecc0712051354t7a157e84u24dc35e88ea1f13e@mail.gmail.com>
Message-ID: <475720BD.1080404@gmail.com>

Fernando Perez wrote:
> The whole scipy.org site seems down.  Is it just on my end?

No. Our new IT guy, Ryan Earl <jre at enthought.com>, is on the case.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From steve at shrogers.com  Wed Dec  5 17:04:51 2007
From: steve at shrogers.com (Steven H. Rogers)
Date: Wed, 05 Dec 2007 15:04:51 -0700
Subject: [Numpy-discussion] site down...
In-Reply-To: <db6b5ecc0712051354t7a157e84u24dc35e88ea1f13e@mail.gmail.com>
References: <db6b5ecc0712051354t7a157e84u24dc35e88ea1f13e@mail.gmail.com>
Message-ID: <47572083.9090102@shrogers.com>

Fernando Perez wrote:
> The whole scipy.org site seems down.  Is it just on my end?
>   
Works for me, though it seems pretty slow.

# Steve


From robert.kern at gmail.com  Wed Dec  5 17:15:20 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 05 Dec 2007 16:15:20 -0600
Subject: [Numpy-discussion] site down...
In-Reply-To: <47572083.9090102@shrogers.com>
References: <db6b5ecc0712051354t7a157e84u24dc35e88ea1f13e@mail.gmail.com>
	<47572083.9090102@shrogers.com>
Message-ID: <475722F8.6040400@gmail.com>

Steven H. Rogers wrote:
> Fernando Perez wrote:
>> The whole scipy.org site seems down.  Is it just on my end?
>>   
> Works for me, though it seems pretty slow.

The system has been restarted.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From aisaac at american.edu  Wed Dec  5 20:38:23 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 5 Dec 2007 20:38:23 -0500
Subject: [Numpy-discussion] multinomial question
Message-ID: <Mahogany-0.67.0-1812-20071205-203823.00@american.edu>

I would think that
    multinomial(1,prob,size=ntrials).sum(axis=0)
would be equivalent to
    multinomial(ntrials,prob)
but the first gives a surprising result.  (See below.)
Explanation?

Thank you,
Alan Isaac


>>> ntrials = 100000
>>> prob = N.arange(100,dtype=N.float32)/4950
>>> multinomial(1,prob,size=ntrials).sum(axis=0)
array([   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
          0,    0,    0,    0,    0,    0,    0,    0,  990, 1058,  996,
       1102, 1088, 1137, 1144, 1150, 1196, 1198, 1272, 1273, 1268, 1265,
       1380, 1336, 1371, 1405, 1348, 1420, 1515, 1506, 1571, 1499, 1556,
       1517, 1603, 1691, 1696, 1763, 1622, 1716, 1722, 1785, 1866, 1799,
       1918, 1818, 1868, 1938, 2010, 1916, 1950, 1983, 2062, 2079, 2224,
       2165, 2136, 2156, 2215, 2118, 2309, 2389, 2377, 2423, 2328, 2325,
       2469])
>>> multinomial(ntrials,prob)
array([   0,   27,   33,   55,  104,  104,  116,  153,  166,  181,  189,
        199,  244,  262,  259,  303,  330,  343,  373,  360,  371,  437,
        423,  470,  460,  550,  551,  497,  517,  593,  623,  623,  648,
        660,  638,  718,  713,  754,  784,  831,  804,  868,  902,  851,
        918,  932,  945,  972,  966, 1025, 1005, 1038, 1075, 1046, 1121,
       1069, 1121, 1152, 1209, 1148, 1196, 1261, 1288, 1304, 1250, 1324,
       1348, 1430, 1370, 1419, 1388, 1364, 1473, 1414, 1511, 1523, 1583,
       1574, 1575, 1575, 1613, 1559, 1665, 1666, 1712, 1728, 1715, 1709,
       1846, 1774, 1819, 1869, 1886, 1963, 1837, 1906, 1983, 1867, 1968,
       1916])
>>>





From robert.kern at gmail.com  Wed Dec  5 21:34:41 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 05 Dec 2007 20:34:41 -0600
Subject: [Numpy-discussion] multinomial question
In-Reply-To: <Mahogany-0.67.0-1812-20071205-203823.00@american.edu>
References: <Mahogany-0.67.0-1812-20071205-203823.00@american.edu>
Message-ID: <47575FC1.5030401@gmail.com>

Alan G Isaac wrote:
> I would think that
>     multinomial(1,prob,size=ntrials).sum(axis=0)
> would be equivalent to
>     multinomial(ntrials,prob)
> but the first gives a surprising result.  (See below.)
> Explanation?

A bug in rk_binomial_inversion(). Unfortunately, this looks like a logical bug
in the sources I was deriving this code from. The "safety bound" on the search
inversion search cuts out too early. Now that I re-examine it, it looks the
bound (whichever of the multiple choice of bounds one could use) could always be
legitimately exceeded, so there shouldn't be a bound at all. I'll have to dive
deeper to figure out what is going on. This makes me grumpy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Dec  6 00:19:32 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 05 Dec 2007 23:19:32 -0600
Subject: [Numpy-discussion] multinomial question
In-Reply-To: <Mahogany-0.67.0-1812-20071205-203823.00@american.edu>
References: <Mahogany-0.67.0-1812-20071205-203823.00@american.edu>
Message-ID: <47578664.1040209@gmail.com>

Alan G Isaac wrote:
> I would think that
>     multinomial(1,prob,size=ntrials).sum(axis=0)
> would be equivalent to
>     multinomial(ntrials,prob)
> but the first gives a surprising result.  (See below.)
> Explanation?

Pretty much anyone who derives their binomial distribution algorithm from
http://www.unc.edu/~gfish/fcmc.html is also wrong.

SVN now has a bound such that CDF(bound) is within 1e-16 (or so) of 1.0.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From wright at esrf.fr  Thu Dec  6 06:39:00 2007
From: wright at esrf.fr (Jon Wright)
Date: Thu, 06 Dec 2007 12:39:00 +0100
Subject: [Numpy-discussion] GPU blas with numpy?
In-Reply-To: <47578664.1040209@gmail.com>
References: <Mahogany-0.67.0-1812-20071205-203823.00@american.edu>
	<47578664.1040209@gmail.com>
Message-ID: <4757DF54.1090009@esrf.fr>

Hello,

Does anyone have any experience with numpy and using a GPU? eg:

http://developer.nvidia.com/object/cuda.html

...they seem to have a GPU based blas available.

Thanks for any info,

Jon



From aisaac at american.edu  Thu Dec  6 08:19:40 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 6 Dec 2007 08:19:40 -0500
Subject: [Numpy-discussion] multinomial question
In-Reply-To: <47578664.1040209@gmail.com>
References: <Mahogany-0.67.0-1812-20071205-203823.00@american.edu><47578664.1040209@gmail.com>
Message-ID: <Mahogany-0.67.0-1776-20071206-081940.00@american.edu>

> Alan G Isaac wrote:
>> I would think that 
>>     multinomial(1,prob,size=ntrials).sum(axis=0)
>> would be equivalent to 
>>     multinomial(ntrials,prob)
>> but the first gives a surprising result.  (See below.)
>> Explanation? 


On Wed, 05 Dec 2007, Robert Kern apparently wrote:
> Pretty much anyone who derives their binomial distribution algorithm from 
> http://www.unc.edu/~gfish/fcmc.html is also wrong. 
> SVN now has a bound such that CDF(bound) is within 1e-16 (or so) of 1.0. 


Thanks!
Alan





From arnar.flatberg at gmail.com  Thu Dec  6 09:29:12 2007
From: arnar.flatberg at gmail.com (Arnar Flatberg)
Date: Thu, 6 Dec 2007 15:29:12 +0100
Subject: [Numpy-discussion] GPU blas with numpy?
In-Reply-To: <4757DF54.1090009@esrf.fr>
References: <Mahogany-0.67.0-1812-20071205-203823.00@american.edu>
	<47578664.1040209@gmail.com> <4757DF54.1090009@esrf.fr>
Message-ID: <5d3194020712060629t5920f5yd7969788f2f7a493@mail.gmail.com>

Hi Jon

Have you looked at pystream?
http://code.google.com/p/pystream/

Arnar

On Dec 6, 2007 12:39 PM, Jon Wright <wright at esrf.fr> wrote:
> Hello,
>
> Does anyone have any experience with numpy and using a GPU? eg:
>
> http://developer.nvidia.com/object/cuda.html
>
> ...they seem to have a GPU based blas available.
>
> Thanks for any info,
>
> Jon
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From tjhnson at gmail.com  Thu Dec  6 18:15:30 2007
From: tjhnson at gmail.com (Tom Johnson)
Date: Thu, 6 Dec 2007 15:15:30 -0800
Subject: [Numpy-discussion] Max on Complex Array
Message-ID: <b02c60870712061515k1af45862sbc6d2194731a051a@mail.gmail.com>

What is the meaning of numpy's maximum on a complex-valued array?


From robert.kern at gmail.com  Thu Dec  6 18:23:37 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 06 Dec 2007 17:23:37 -0600
Subject: [Numpy-discussion] Max on Complex Array
In-Reply-To: <b02c60870712061515k1af45862sbc6d2194731a051a@mail.gmail.com>
References: <b02c60870712061515k1af45862sbc6d2194731a051a@mail.gmail.com>
Message-ID: <47588479.1080006@gmail.com>

Tom Johnson wrote:
> What is the meaning of numpy's maximum on a complex-valued array?

We impose a lexicographical ordering on the complex space. Complex numbers are
compared first by their real component and then, if the real components are
equal, by their imaginary component.


In [1]: from numpy import *

In [19]: c = random.randint(0, 5, 10) + random.random(10)*1j

In [20]: c
Out[20]:
array([ 0.+0.68448275j,  1.+0.97849291j,  3.+0.22114928j,  4.+0.65409519j,
        3.+0.91550523j,  4.+0.50667105j,  1.+0.34576644j,  4.+0.97286048j,
        1.+0.07268317j,  0.+0.52885086j])

In [21]: c.sort()

In [22]: c
Out[22]:
array([ 0.+0.52885086j,  0.+0.68448275j,  1.+0.07268317j,  1.+0.34576644j,
        1.+0.97849291j,  3.+0.22114928j,  3.+0.91550523j,  4.+0.50667105j,
        4.+0.65409519j,  4.+0.97286048j])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tjhnson at gmail.com  Thu Dec  6 18:36:35 2007
From: tjhnson at gmail.com (Tom Johnson)
Date: Thu, 6 Dec 2007 15:36:35 -0800
Subject: [Numpy-discussion] Max on Complex Array
In-Reply-To: <47588479.1080006@gmail.com>
References: <b02c60870712061515k1af45862sbc6d2194731a051a@mail.gmail.com>
	<47588479.1080006@gmail.com>
Message-ID: <b02c60870712061536n74970999x71b218ed6c15bbe7@mail.gmail.com>

On Dec 6, 2007 3:23 PM, Robert Kern <robert.kern at gmail.com> wrote:
> Tom Johnson wrote:
> > What is the meaning of numpy's maximum on a complex-valued array?
>
> We impose a lexicographical ordering on the complex space.

Good to know, but is this documented somewhere.  a.max? does not mention this.


From cournape at gmail.com  Fri Dec  7 04:12:15 2007
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 7 Dec 2007 18:12:15 +0900
Subject: [Numpy-discussion] scipy.scons branch: building numpy and scipy
	with scons
In-Reply-To: <475625B2.1010806@gmail.com>
References: <47553843.7020709@ar.media.kyoto-u.ac.jp>
	<4755AA3A.30802@gmail.com> <47561332.3090200@ar.media.kyoto-u.ac.jp>
	<475625B2.1010806@gmail.com>
Message-ID: <5b8d13220712070112t40d39202o6a4ed814a98c322@mail.gmail.com>

On Dec 5, 2007 1:14 PM, Robert Kern <robert.kern at gmail.com> wrote:
> David Cournapeau wrote:
>
> > So to go back to your problem: if I understand correctly, what is needed
> > is to update the scons tools. Since those are kept at one place, I think
> > it would be safe to update them independently. But I don't understand
> > exactly how this could be installed in the source tree without
> > reinstalling numpy ? I think this is better than completely overriding
> > the compilation flags, personally. But if you really want this
> > possibility, I can add it, too, without too much trouble.
>
> I don't think you could install it into an already installed numpy package. My
> suggestion is to keep the implementations of the tools inside the numpy package
> as they are now *except* that we look for another package first before going
> inside numpy.distutils.scons.tools . I called it "numpy_fcompilers" though I now
> suspect "numpy_scons_tools" might be more appropriate. If the package
> numpy_scons_tools doesn't exist, the implementations inside
> numpy.distutils.scons.tools are used. A variation on this would be to provide an
> --option to the scons command to provide a "package path" to look for tools. E.g.
>
>   python setup.py scons
> --tool-path=my_local_fcompilers,site_fcompilers,numpy.distutils.scons.tools
>
Done in 4558 (I named the option --scons-tool-path). The problem I can
foresee with this, though, is that if a custom tool is buggy, scons
will fail, and it is not always obvious why.

cheers,

David


From renato.serodio at gmail.com  Fri Dec  7 10:44:20 2007
From: renato.serodio at gmail.com (Renato Serodio)
Date: Fri, 7 Dec 2007 16:44:20 +0100
Subject: [Numpy-discussion] data transit
Message-ID: <4f4101bf0712070744i1e6cbd0ex2c189f8c6ab1bc20@mail.gmail.com>

Hello all,

I'm developing a custom computational application which I chose to
base in Numpy. Already quite in love with Python, and with proprietary
things making me increasingly sick (through forced exposure to stupid
errors I can't correct), Numpy was the way to go.

Now, it is in times like this that I regret not going for graduate
studies in computation - I'm a bit locked in the old paradigms that my
[fortran] generation learn.

Since my application is only vaguely required to be 'generic', I had
to dive into the wonderful world of computer science - a previous post
in this group led to some very interesting solutions for the
application, which, while doing nothing, is capable of doing
everything :)

A bit of context: the application is supposed to process telemetry,
outputing some chart, alarm, etc. Raw data is obtained through
plugin-like objects, which provide a uniform interface to distinct
sources. The processing routines are objects as well, but operate on
data as if they were functions (sort of sin(x)). This way, I don't
need to define anything other than the interfaces - the core remains
flexible.

I came to a problem, though, while trying to define some structure for
data transit. At first I imagined I could keep both raw data and
results inside the same object; unfortunately, if I want to use these
results in a second stage, my flexibility is rather impaired.

Then I thought about getting raw data into an object, passing that to
the processing core, and finally storing its output in another object.
While this has the advantage of clearing raw data out of memory as
soon as I finish chewing it, I seem to lose the relation between raw
and result data sets - which I have to maintain somewhere else.

Yet another issue crops up, in relation to very large data sets. If
there's not enough memory to cope with the data set, one either relies
on swapping or changes the algorithm - and in this case having
'inteligent' data objects allows good, textbook encapsulation.

My question is thus, does anyone have experience or could point to
literature/code where related problems are addressed? I understand
that my application may be suffering from excessive 'generality', but
certainly this problem has surfaced elsewhere.

Looking forward to your answers,

Renato


From rolf.wester at ilt.fraunhofer.de  Fri Dec  7 10:49:26 2007
From: rolf.wester at ilt.fraunhofer.de (Rolf Wester)
Date: Fri, 07 Dec 2007 16:49:26 +0100
Subject: [Numpy-discussion] fft inplace?
Message-ID: <47596B86.4030500@ilt.fraunhofer.de>

Hi,

I have huge complex arrays that have to be Fourier transformed. Is it 
possible to to that inplace with numpy. As far as I can remember this 
was possible with either Numeric or older versions of numpy, but I 
couldn't find any hint concerning this in the numpy manual or in the 
python sources.

Regards

Rolf



From aisaac at american.edu  Fri Dec  7 12:48:30 2007
From: aisaac at american.edu (Alan Isaac)
Date: Fri, 7 Dec 2007 12:48:30 -0500
Subject: [Numpy-discussion] data transit
In-Reply-To: <4f4101bf0712070744i1e6cbd0ex2c189f8c6ab1bc20@mail.gmail.com>
References: <4f4101bf0712070744i1e6cbd0ex2c189f8c6ab1bc20@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1192-20071207-124830.00@CASC64043305.american.edu>

It sounds like you want a new class X that does three things:
knows where the data is and how to access  it,
knows how the data are to be processed and can do this when asked,
is able to provide a "results" object when requested.
The results object can store who made it and with what 
processing module, thus indirectly linking to the data and techniques
(which the X instance may or may not store, as is convenient).

fwiw,
Alan Isaac





From nadavh at visionsense.com  Sat Dec  8 10:11:57 2007
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sat, 8 Dec 2007 17:11:57 +0200
Subject: [Numpy-discussion] data transit
Message-ID: <710F2847B0018641891D9A21602763600B6E51@ex3.envision.co.il>

Did you look at pytables (http://www.pytables.org/moin)?

   Nadav.

-----Original Message-----
From: numpy-discussion-bounces at scipy.org on behalf of Renato Serodio
Sent: Fri 07-Dec-07 17:44
To: Discussion of Numerical Python
Subject: [Numpy-discussion] data transit
 
Hello all,

I'm developing a custom computational application which I chose to
base in Numpy. Already quite in love with Python, and with proprietary
things making me increasingly sick (through forced exposure to stupid
errors I can't correct), Numpy was the way to go.

Now, it is in times like this that I regret not going for graduate
studies in computation - I'm a bit locked in the old paradigms that my
[fortran] generation learn.

Since my application is only vaguely required to be 'generic', I had
to dive into the wonderful world of computer science - a previous post
in this group led to some very interesting solutions for the
application, which, while doing nothing, is capable of doing
everything :)

A bit of context: the application is supposed to process telemetry,
outputing some chart, alarm, etc. Raw data is obtained through
plugin-like objects, which provide a uniform interface to distinct
sources. The processing routines are objects as well, but operate on
data as if they were functions (sort of sin(x)). This way, I don't
need to define anything other than the interfaces - the core remains
flexible.

I came to a problem, though, while trying to define some structure for
data transit. At first I imagined I could keep both raw data and
results inside the same object; unfortunately, if I want to use these
results in a second stage, my flexibility is rather impaired.

Then I thought about getting raw data into an object, passing that to
the processing core, and finally storing its output in another object.
While this has the advantage of clearing raw data out of memory as
soon as I finish chewing it, I seem to lose the relation between raw
and result data sets - which I have to maintain somewhere else.

Yet another issue crops up, in relation to very large data sets. If
there's not enough memory to cope with the data set, one either relies
on swapping or changes the algorithm - and in this case having
'inteligent' data objects allows good, textbook encapsulation.

My question is thus, does anyone have experience or could point to
literature/code where related problems are addressed? I understand
that my application may be suffering from excessive 'generality', but
certainly this problem has surfaced elsewhere.

Looking forward to your answers,

Renato
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From jean_lucregnier at yahoo.fr  Mon Dec 10 05:06:09 2007
From: jean_lucregnier at yahoo.fr (=?iso-8859-1?q?Jean-Luc=20R=E9gnier?=)
Date: Mon, 10 Dec 2007 11:06:09 +0100 (CET)
Subject: [Numpy-discussion] "NaN"
Message-ID: <623512.43679.qm@web27812.mail.ukl.yahoo.com>

Hello,
  I switched from numarray to numpy and I have now some "NaN" 
  in my matrix. What that means ?
  None a numeric ? 
  regards
   
   


Jean-Luc REGNIER 
ACR Mimarlik Ltd. Sti 
Savas Cad. 26/B Sirinyali 
ANTALYA, TURKEY 
Tel. & Fax: 0090-(0).242.316.08.09 
GSM: 0090-0.532.303.36.21 
http://www.acrmim.com
   
   

             
---------------------------------
 Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071210/4162a690/attachment.html>

From matthieu.brucher at gmail.com  Mon Dec 10 05:23:37 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 10 Dec 2007 11:23:37 +0100
Subject: [Numpy-discussion] "NaN"
In-Reply-To: <623512.43679.qm@web27812.mail.ukl.yahoo.com>
References: <623512.43679.qm@web27812.mail.ukl.yahoo.com>
Message-ID: <e76aa17f0712100223y2aac89d3gfab3b7de3804d074@mail.gmail.com>

It's "Not A Number". It can occur when you have a division by zero, a
difference between two infinite numbers, ...

Matthieu

2007/12/10, Jean-Luc R?gnier <jean_lucregnier at yahoo.fr>:
>
> Hello,
> I switched from numarray to numpy and I have now some "NaN"
> in my matrix. What that means ?
> None a numeric ?
> regards
>
>
>
>
> Jean-Luc REGNIER
> ACR Mimarlik Ltd. Sti
> Savas Cad. 26/B Sirinyali
> ANTALYA, TURKEY
> Tel. & Fax: 0090-(0).242.316.08.09
> GSM: 0090-0.532.303.36.21
> http://www.acrmim.com
>
>
> ------------------------------
> Ne gardez plus qu'une seule adresse mail ! Copiez vos mails<http://fr.rd.yahoo.com/mail/mail_taglines/trueswitch/*http://www.trueswitch.com/yahoo-fr/>vers Yahoo! Mail
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071210/6bc0fe61/attachment.html>

From matthieu.brucher at gmail.com  Mon Dec 10 05:38:20 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 10 Dec 2007 11:38:20 +0100
Subject: [Numpy-discussion] Epsilon for each dtype
Message-ID: <e76aa17f0712100238g16b729d0td174baca32720d5e@mail.gmail.com>

Hi,

Is there somewhere a equivalent to std::numerical_limits<>::epsilon, that
is, the greatest value such that 1. + epsilon is numerically equal to 1. ?
I saw something that could be related in oldnumeric, but nothing in numpy
itself.

Matthieu
-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071210/ebf0a89d/attachment.html>

From silva at lma.cnrs-mrs.fr  Mon Dec 10 06:16:39 2007
From: silva at lma.cnrs-mrs.fr (Fabrice Silva)
Date: Mon, 10 Dec 2007 12:16:39 +0100
Subject: [Numpy-discussion] Epsilon for each dtype
In-Reply-To: <e76aa17f0712100238g16b729d0td174baca32720d5e@mail.gmail.com>
References: <e76aa17f0712100238g16b729d0td174baca32720d5e@mail.gmail.com>
Message-ID: <1197285399.3472.4.camel@localhost>

Le lundi 10 d?cembre 2007 ? 11:38 +0100, Matthieu Brucher a ?crit :
> Hi,
> Is there somewhere a equivalent to std::numerical_limits<>::epsilon,
> that is, the greatest value such that 1. + epsilon is numerically
> equal to 1. ?
> I saw something that could be related in oldnumeric, but nothing in
> numpy itself. 

if X is a numpy object:
numpy.finfo(type(X)).eps gives the epsilon of the type of X.
You may look at other attributes of finfo too..

-- 
Fabrice Silva <silva at lma.cnrs-mrs.fr>
LMA UPR CNRS 7051



From matthieu.brucher at gmail.com  Mon Dec 10 06:50:13 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 10 Dec 2007 12:50:13 +0100
Subject: [Numpy-discussion] Epsilon for each dtype
In-Reply-To: <1197285399.3472.4.camel@localhost>
References: <e76aa17f0712100238g16b729d0td174baca32720d5e@mail.gmail.com>
	<1197285399.3472.4.camel@localhost>
Message-ID: <e76aa17f0712100350wa219f34t96bac808f831965a@mail.gmail.com>

Excellent, that was what I was looking for, thank you.

Matthieu

2007/12/10, Fabrice Silva <silva at lma.cnrs-mrs.fr>:
>
> Le lundi 10 d?cembre 2007 ? 11:38 +0100, Matthieu Brucher a ?crit :
> > Hi,
> > Is there somewhere a equivalent to std::numerical_limits<>::epsilon,
> > that is, the greatest value such that 1. + epsilon is numerically
> > equal to 1. ?
> > I saw something that could be related in oldnumeric, but nothing in
> > numpy itself.
>
> if X is a numpy object:
> numpy.finfo(type(X)).eps gives the epsilon of the type of X.
> You may look at other attributes of finfo too..
>
> --
> Fabrice Silva <silva at lma.cnrs-mrs.fr>
> LMA UPR CNRS 7051
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071210/97ba5042/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Mon Dec 10 06:48:05 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 10 Dec 2007 20:48:05 +0900
Subject: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4
	for windows ?
Message-ID: <475D2775.2080504@ar.media.kyoto-u.ac.jp>

Hi,

    Several people reported problems with numpy 1.0.4 (See #627 and 
#628, but also other problems mentionned on the ML, which I cannot 
find). They were all solved, as far as I know, by a binary I produced 
(simply using mingw + netlib BLAS/LAPACK,  no ATLAS). Maybe it would be 
good to use those instead ? (I can recompile them if there is a special 
thing to do to build them)

    cheers,

    David


From lxander.m at gmail.com  Mon Dec 10 08:59:43 2007
From: lxander.m at gmail.com (Alexander Michael)
Date: Mon, 10 Dec 2007 08:59:43 -0500
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
Message-ID: <525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>

On Dec 10, 2007 6:48 AM, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
>     Several people reported problems with numpy 1.0.4 (See #627 and
> #628, but also other problems mentionned on the ML, which I cannot
> find). They were all solved, as far as I know, by a binary I produced
> (simply using mingw + netlib BLAS/LAPACK,  no ATLAS). Maybe it would be
> good to use those instead ? (I can recompile them if there is a special
> thing to do to build them)

Do I understand correctly that you are suggesting removing ATLAS from
the Windows distribution? Wouldn't this make numpy very slow? I know
on RHEL5 I see a very large improvement between the basic BLAS/LAPACK
and ATLAS. Perhaps we should make an alternative Windows binary
available without ATLAS just for those having problems with ATLAS?


From meine at informatik.uni-hamburg.de  Mon Dec 10 09:21:56 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Mon, 10 Dec 2007 15:21:56 +0100
Subject: [Numpy-discussion] ndarray.clip only with lower or upper values?
Message-ID: <200712101521.56989.meine@informatik.uni-hamburg.de>

Hi again,

I noticed that clip() needs two parameters, but wouldn't it be nice and 
straightforward to just pass min= or max= as keyword arg?

In [2]: a = arange(10)

In [3]: a.clip(min = 2, max = 5)
Out[3]: array([2, 2, 2, 3, 4, 5, 5, 5, 5, 5])

In [4]: a.clip(min = 2)
---------------------------------------------------------------------------
exceptions.TypeError                                 Traceback (most recent 
call last)

/home/meine/<ipython console>

TypeError: function takes at least 2 arguments (1 given)

(I could simulate that by passing max = maximum_value_of(a.dtype), if that 
existed, see my other mail.)

Ciao, /  /
     /--/
    /  / ANS


From meine at informatik.uni-hamburg.de  Mon Dec 10 09:18:34 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Mon, 10 Dec 2007 15:18:34 +0100
Subject: [Numpy-discussion] Minimum and maximum values of numpy datatypes?
Message-ID: <200712101518.35218.meine@informatik.uni-hamburg.de>

Hi!

Is there a way to query the minimum and maximum values of the numpy datatypes?

E.g. numpy.uint8.max == 255, numpy.uint8.min == 0 (these attributes exist, but 
they are functions, obviously for technical reasons).

Ciao, /  /
     /--/
    /  / ANS


From jean_lucregnier at yahoo.fr  Mon Dec 10 09:44:22 2007
From: jean_lucregnier at yahoo.fr (=?iso-8859-1?q?Jean-Luc=20R=E9gnier?=)
Date: Mon, 10 Dec 2007 15:44:22 +0100 (CET)
Subject: [Numpy-discussion] RE : Re:  "NaN"
In-Reply-To: <e76aa17f0712100223y2aac89d3gfab3b7de3804d074@mail.gmail.com>
Message-ID: <880466.4503.qm@web27811.mail.ukl.yahoo.com>

many thanks for answering Matthieu 
  Actually my problem concerned the old command "matrixmultiply" vs "dot".
  I solved it now.
  However I have a question regarding Tkinter.
  I am doing a small 3D engine using Tkinter, Pmw and, numpy.
  I am basically plotting the result from the matrix calculation on
  the Tkinter canevas and it appears to be relatively slow depending
  on the processor. I did a test on two different PC.
  Is there someone here who has experienced such problem ? It looks like the Tkinter canevas needs time to react with my bindings
  Vpython for instance is really fast whatever the PC. Should I use Wxpython
  instead of Tkinter ?
  Best regards
  Jean-Luc
   
  It's "Not A Number". It can occur when you have a division by zero, a difference between two infinite numbers, ...

Matthieu

             
---------------------------------
 Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071210/d21b4f56/attachment.html>

From renato.serodio at gmail.com  Mon Dec 10 09:44:04 2007
From: renato.serodio at gmail.com (Renato Serodio)
Date: Mon, 10 Dec 2007 15:44:04 +0100
Subject: [Numpy-discussion] data transit
In-Reply-To: <Mahogany-0.67.0-1192-20071207-124830.00@CASC64043305.american.edu>
References: <4f4101bf0712070744i1e6cbd0ex2c189f8c6ab1bc20@mail.gmail.com>
	<Mahogany-0.67.0-1192-20071207-124830.00@CASC64043305.american.edu>
Message-ID: <4f4101bf0712100644i4e268212kd1909c04a2424a49@mail.gmail.com>

Hello there,

indeed, the tasks you described correspond to what I'm seeking to
implement. The thing is, for the sake of encapsulation (and laziness
in the programming sense), I'm keeping responsibilities well defined
in several objects. I guess this type of coding is pretty much
ordinary for an OO person - it's just me having trouble with the
philosophy.

So, upon much thought, it breaks down like this:
- Crunchers: lowest level objects that encapsulate stuff I do with
Numpy/Scipy functions, on Numpy objects. Say, get data from arguments,
unbias the data, zero-stuff, fft the set, etc. They are meant to be
written as needed.

- DataContainers: abstraction layer to data sources (DB, files, etc)
and to other data objects still in memory. Data returned by Crunchers
is stored inside - in practice, piped here by an Analysis object. So
far, I see no need for nesting DCs inside other DCs.

- Analysis: these are the glue between Crunchers, DataContainers and
the user (batch, GUI, CLI). An Analysis is instanciated by the user,
and directs both data flow into DCs, as well as out of them. While
each Analysis has one and only one 'results' attribute, which points
to some place within a DataContainer, I imagine Analysis made by
concatenating several Analysis - just call Analysis.result() to access
data at a certain stage of processing.

Well, so it is. Hopefully this setup will lend a good degree of
flexibility to my application - the crunchers are hard to develop,
since I haven't seen the data yet.

Nadav: I had looked into pytables while investigating low-level
interfaces that would have to be supported. It's a lot below what I
was looking for - my DataContainers do obtain their nature from other
classes which are responsible for talking to DBs, files and the like -
but it is the design of these containers that's hard to conceive!

Cheers,

Renato



On Dec 7, 2007 6:48 PM, Alan Isaac <aisaac at american.edu> wrote:
> It sounds like you want a new class X that does three things:
> knows where the data is and how to access  it,
> knows how the data are to be processed and can do this when asked,
> is able to provide a "results" object when requested.
> The results object can store who made it and with what
> processing module, thus indirectly linking to the data and techniques
> (which the X instance may or may not store, as is convenient).
>
> fwiw,
> Alan Isaac
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From matthieu.brucher at gmail.com  Mon Dec 10 10:30:18 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 10 Dec 2007 16:30:18 +0100
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
Message-ID: <e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>

2007/12/10, Alexander Michael <lxander.m at gmail.com>:
>
> On Dec 10, 2007 6:48 AM, David Cournapeau <david at ar.media.kyoto-u.ac.jp>
> wrote:
> > Hi,
> >
> >     Several people reported problems with numpy 1.0.4 (See #627 and
> > #628, but also other problems mentionned on the ML, which I cannot
> > find). They were all solved, as far as I know, by a binary I produced
> > (simply using mingw + netlib BLAS/LAPACK,  no ATLAS). Maybe it would be
> > good to use those instead ? (I can recompile them if there is a special
> > thing to do to build them)
>
> Do I understand correctly that you are suggesting removing ATLAS from
> the Windows distribution? Wouldn't this make numpy very slow? I know
> on RHEL5 I see a very large improvement between the basic BLAS/LAPACK
> and ATLAS. Perhaps we should make an alternative Windows binary
> available without ATLAS just for those having problems with ATLAS?


That's why David proposed the netlib version of BLAS/LAPACK and not the
default implementation in numpy.

I would agree with David ;)

Matthieu
-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071210/96c1c27f/attachment.html>

From matthieu.brucher at gmail.com  Mon Dec 10 11:23:07 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 10 Dec 2007 17:23:07 +0100
Subject: [Numpy-discussion] Minimum and maximum values of numpy
	datatypes?
In-Reply-To: <200712101518.35218.meine@informatik.uni-hamburg.de>
References: <200712101518.35218.meine@informatik.uni-hamburg.de>
Message-ID: <e76aa17f0712100823s6ff46d17w3a983d26bc55dc6f@mail.gmail.com>

I had the same problem sooner today, someone told me the answer : use
numpy.info object ;)

Matthieu

2007/12/10, Hans Meine <meine at informatik.uni-hamburg.de>:
>
> Hi!
>
> Is there a way to query the minimum and maximum values of the numpy
> datatypes?
>
> E.g. numpy.uint8.max == 255, numpy.uint8.min == 0 (these attributes exist,
> but
> they are functions, obviously for technical reasons).
>
> Ciao, /  /
>      /--/
>     /  / ANS
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071210/4de7cd23/attachment.html>

From oliphant at enthought.com  Mon Dec 10 12:41:06 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 10 Dec 2007 11:41:06 -0600
Subject: [Numpy-discussion] Minimum and maximum values of numpy
	datatypes?
In-Reply-To: <200712101518.35218.meine@informatik.uni-hamburg.de>
References: <200712101518.35218.meine@informatik.uni-hamburg.de>
Message-ID: <475D7A32.7070803@enthought.com>

Hans Meine wrote:
> Hi!
>
> Is there a way to query the minimum and maximum values of the numpy datatypes?
>   

numpy.iinfo  (notice the two i's) (integer information)
numpy.finfo (floating point information)

Example:

numpy.iinfo(numpy.uint8).max
numpy.iinfo(numpy.int16).min

You pass the datatype to the numpy.iinfo() constructor and then get 
attributes on the result..

-Travis



From cookedm at physics.mcmaster.ca  Mon Dec 10 16:50:10 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Mon, 10 Dec 2007 16:50:10 -0500
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
Message-ID: <952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>

On Dec 10, 2007, at 10:30 , Matthieu Brucher wrote:
> 2007/12/10, Alexander Michael <lxander.m at gmail.com>: On Dec 10, 2007  
> 6:48 AM, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> > Hi,
> >
> >     Several people reported problems with numpy 1.0.4 (See #627 and
> > #628, but also other problems mentionned on the ML, which I cannot
> > find). They were all solved, as far as I know, by a binary I  
> produced
> > (simply using mingw + netlib BLAS/LAPACK,  no ATLAS). Maybe it  
> would be
> > good to use those instead ? (I can recompile them if there is a  
> special
> > thing to do to build them)
>
> Do I understand correctly that you are suggesting removing ATLAS from
> the Windows distribution? Wouldn't this make numpy very slow? I know
> on RHEL5 I see a very large improvement between the basic BLAS/LAPACK
> and ATLAS. Perhaps we should make an alternative Windows binary
> available without ATLAS just for those having problems with ATLAS?
> That's why David proposed the netlib version of BLAS/LAPACK and not  
> the default implementation in numpy.
>
> I would agree with David ;)


Our versions of BLAS/LAPACK are f2c'd versions of the netlib 3.0 BLAS/ 
LAPACK (actually, of Debian's version of these -- they include several  
fixes that weren't upstream).

So netlib's versions aren't going to be any faster, really. And  
netlib's BLAS is slow. Now, if there is a BLAS that's easier to  
compile than ATLAS on windows, that'd be improvement.

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From tim.hochberg at ieee.org  Mon Dec 10 17:46:17 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Mon, 10 Dec 2007 15:46:17 -0700
Subject: [Numpy-discussion] ndarray.clip only with lower or upper values?
In-Reply-To: <200712101521.56989.meine@informatik.uni-hamburg.de>
References: <200712101521.56989.meine@informatik.uni-hamburg.de>
Message-ID: <e4412d6b0712101446o6e8c65c9g4de088d4c8a1937d@mail.gmail.com>

On Dec 10, 2007 7:21 AM, Hans Meine <meine at informatik.uni-hamburg.de> wrote:

> Hi again,
>
> I noticed that clip() needs two parameters, but wouldn't it be nice and
> straightforward to just pass min= or max= as keyword arg?
>
> In [2]: a = arange(10)
>
> In [3]: a.clip(min = 2, max = 5)
> Out[3]: array([2, 2, 2, 3, 4, 5, 5, 5, 5, 5])
>
> In [4]: a.clip(min = 2)
>
> ---------------------------------------------------------------------------
> exceptions.TypeError                                 Traceback (most
> recent
> call last)
>
> /home/meine/<ipython console>
>
> TypeError: function takes at least 2 arguments (1 given)
>
> (I could simulate that by passing max = maximum_value_of(a.dtype), if that
> existed, see my other mail.)


Why not just use minimum or maximum as needed instead of overloading clip?


-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071210/e37bccd3/attachment.html>

From cournape at gmail.com  Mon Dec 10 17:56:35 2007
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 11 Dec 2007 07:56:35 +0900
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
Message-ID: <5b8d13220712101456x14ab937bp29ce360333ce8677@mail.gmail.com>

On Dec 10, 2007 10:59 PM, Alexander Michael <lxander.m at gmail.com> wrote:
> On Dec 10, 2007 6:48 AM, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> > Hi,
> >
> >     Several people reported problems with numpy 1.0.4 (See #627 and
> > #628, but also other problems mentionned on the ML, which I cannot
> > find). They were all solved, as far as I know, by a binary I produced
> > (simply using mingw + netlib BLAS/LAPACK,  no ATLAS). Maybe it would be
> > good to use those instead ? (I can recompile them if there is a special
> > thing to do to build them)
>
> Do I understand correctly that you are suggesting removing ATLAS from
> the Windows distribution? Wouldn't this make numpy very slow? I know
> on RHEL5 I see a very large improvement between the basic BLAS/LAPACK
> and ATLAS. Perhaps we should make an alternative Windows binary
> available without ATLAS just for those having problems with ATLAS?
If you care about speed, then you should compile your own atlas
anyway. I don't quite understand the discussion about speed:
BLAS/LAPACK is really slower for relatively big sizes, which many
people do not use. And more important, a non working, crashing, fast
BLAS is much slower than a working one :)

We could propose two versions, but that just makes things more
complicated: on average, people using windows are less inclined to try
to understand why things do not work, and when your numpy crashes, it
is not obvious that ATLAS is involved (we still do not know the exact
problem).

David


From robert.kern at gmail.com  Mon Dec 10 18:41:31 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 10 Dec 2007 17:41:31 -0600
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
Message-ID: <475DCEAB.8080503@gmail.com>

David M. Cooke wrote:
> On Dec 10, 2007, at 10:30 , Matthieu Brucher wrote:
>> 2007/12/10, Alexander Michael <lxander.m at gmail.com>: On Dec 10, 2007  
>> 6:48 AM, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>>> Hi,
>>>
>>>     Several people reported problems with numpy 1.0.4 (See #627 and
>>> #628, but also other problems mentionned on the ML, which I cannot
>>> find). They were all solved, as far as I know, by a binary I  
>> produced
>>> (simply using mingw + netlib BLAS/LAPACK,  no ATLAS). Maybe it  
>> would be
>>> good to use those instead ? (I can recompile them if there is a  
>> special
>>> thing to do to build them)
>> Do I understand correctly that you are suggesting removing ATLAS from
>> the Windows distribution? Wouldn't this make numpy very slow? I know
>> on RHEL5 I see a very large improvement between the basic BLAS/LAPACK
>> and ATLAS. Perhaps we should make an alternative Windows binary
>> available without ATLAS just for those having problems with ATLAS?
>> That's why David proposed the netlib version of BLAS/LAPACK and not  
>> the default implementation in numpy.
>>
>> I would agree with David ;)
> 
> Our versions of BLAS/LAPACK are f2c'd versions of the netlib 3.0 BLAS/ 
> LAPACK (actually, of Debian's version of these -- they include several  
> fixes that weren't upstream).
> 
> So netlib's versions aren't going to be any faster, really. And  
> netlib's BLAS is slow. Now, if there is a BLAS that's easier to  
> compile than ATLAS on windows, that'd be improvement.

The current situation is untenable. I will gladly accept a slow BLAS for an
official binary that won't segfault anywhere. We can look for a faster BLAS later.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From fperez.net at gmail.com  Mon Dec 10 21:03:57 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 10 Dec 2007 19:03:57 -0700
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <475DCEAB.8080503@gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
Message-ID: <db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>

On Dec 10, 2007 4:41 PM, Robert Kern <robert.kern at gmail.com> wrote:

> The current situation is untenable. I will gladly accept a slow BLAS for an
> official binary that won't segfault anywhere. We can look for a faster BLAS later.

Just to add a note to this: John Hunter and I just finished teaching a
python workshop here in Boulder, and one attendee had a recurring
all-out crash on WinXP.  Eventually John was able to track it to a bad
BLAS call, but the death was an 'illegal instruction'. We then noticed
that this was on an older Pentium III laptop, and I'd be willing to
bet that the problem is an ATLAS compiled with SSE2 support.  The PIII
chip only has plain SSE, not SSE2, and that's the kind of crash I've
seen when  accidentally running code compiled in my office machine (a
P4) on my laptop (a similarly old PIII).

It may very well be that it's OK to ship binaries with ATLAS, but just
to build them without any fancy instruction support (no SSE, SSE2 or
anything else of that kind, just plain x87 code).


Cheers,

f


From strawman at astraw.com  Mon Dec 10 21:59:43 2007
From: strawman at astraw.com (Andrew Straw)
Date: Mon, 10 Dec 2007 18:59:43 -0800
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
Message-ID: <475DFD1F.4000809@astraw.com>

An idea that occurred to me after reading Fernando's email. A function
could be called at numpy import time that specifically checks for the
instruction set on the CPU running and makes sure that is completely
covers the instruction set available through all the various calls,
including to BLAS. If this kind of thing were added, numpy could fail
with a loud warning rather than dying with mysterious errors later on.
The trouble would seem that you can switch your BLAS shared library
without re-compiling numpy, so numpy would have to do a run-time query
of ATLAS, etc. for compilation issues. Which is likely
library-dependent, and furthermore, not having looked into BLAS
implementations, I'm not sure that (m)any of them provide such
information. Do they? Is this idea technically possible?

-Andrew

Fernando Perez wrote:
> On Dec 10, 2007 4:41 PM, Robert Kern <robert.kern at gmail.com> wrote:
> 
>> The current situation is untenable. I will gladly accept a slow BLAS for an
>> official binary that won't segfault anywhere. We can look for a faster BLAS later.
> 
> Just to add a note to this: John Hunter and I just finished teaching a
> python workshop here in Boulder, and one attendee had a recurring
> all-out crash on WinXP.  Eventually John was able to track it to a bad
> BLAS call, but the death was an 'illegal instruction'. We then noticed
> that this was on an older Pentium III laptop, and I'd be willing to
> bet that the problem is an ATLAS compiled with SSE2 support.  The PIII
> chip only has plain SSE, not SSE2, and that's the kind of crash I've
> seen when  accidentally running code compiled in my office machine (a
> P4) on my laptop (a similarly old PIII).
> 
> It may very well be that it's OK to ship binaries with ATLAS, but just
> to build them without any fancy instruction support (no SSE, SSE2 or
> anything else of that kind, just plain x87 code).
> 
> 
> Cheers,
> 
> f
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From oliphant at enthought.com  Mon Dec 10 22:28:58 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 10 Dec 2007 21:28:58 -0600
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
Message-ID: <475E03FA.9010608@enthought.com>

Fernando Perez wrote:
> On Dec 10, 2007 4:41 PM, Robert Kern <robert.kern at gmail.com> wrote:
>
>   
>> The current situation is untenable. I will gladly accept a slow BLAS for an
>> official binary that won't segfault anywhere. We can look for a faster BLAS later.
>>     
>
> Just to add a note to this: John Hunter and I just finished teaching a
> python workshop here in Boulder, and one attendee had a recurring
> all-out crash on WinXP.  Eventually John was able to track it to a bad
> BLAS call, but the death was an 'illegal instruction'. We then noticed
> that this was on an older Pentium III laptop, and I'd be willing to
> bet that the problem is an ATLAS compiled with SSE2 support.  The PIII
> chip only has plain SSE, not SSE2, and that's the kind of crash I've
> seen when  accidentally running code compiled in my office machine (a
> P4) on my laptop (a similarly old PIII).
>
> It may very well be that it's OK to ship binaries with ATLAS, but just
> to build them without any fancy instruction support (no SSE, SSE2 or
> anything else of that kind, just plain x87 code).
>   

I think this is what the default should be (but plain SSE allowed).  
However, since I have moved, the machine I was using to build "official" 
binaries has switched and that is probably at the core of the problem.

Also,  I've tried to build ATLAS 3.8.0 without SSE without success (when 
I'm on a machine that has it).

It would be useful to track which binaries are giving people problems as 
I built the most recent ones on a VM against an old version of ATLAS 
(3.6.0) that has been compiled on windows for a long time.

I'm happy to upload a better binary of NumPy (if I can figure out which 
one is giving people grief and how to create a decent one).

-Travis O.



From strawman at astraw.com  Mon Dec 10 22:46:53 2007
From: strawman at astraw.com (Andrew Straw)
Date: Mon, 10 Dec 2007 19:46:53 -0800
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <475E03FA.9010608@enthought.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>	<475DCEAB.8080503@gmail.com>	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475E03FA.9010608@enthought.com>
Message-ID: <475E082D.1010107@astraw.com>

According to the QEMU website, QEMU does not (yet) emulate SSE on x86
target, so a Windows installation on a QEMU virtual machine may be a
good way to build binaries free of these issues.
http://fabrice.bellard.free.fr/qemu/qemu-tech.html

-Andrew

Travis E. Oliphant wrote:
> Fernando Perez wrote:
>> On Dec 10, 2007 4:41 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>>   
>>> The current situation is untenable. I will gladly accept a slow BLAS for an
>>> official binary that won't segfault anywhere. We can look for a faster BLAS later.
>>>     
>> Just to add a note to this: John Hunter and I just finished teaching a
>> python workshop here in Boulder, and one attendee had a recurring
>> all-out crash on WinXP.  Eventually John was able to track it to a bad
>> BLAS call, but the death was an 'illegal instruction'. We then noticed
>> that this was on an older Pentium III laptop, and I'd be willing to
>> bet that the problem is an ATLAS compiled with SSE2 support.  The PIII
>> chip only has plain SSE, not SSE2, and that's the kind of crash I've
>> seen when  accidentally running code compiled in my office machine (a
>> P4) on my laptop (a similarly old PIII).
>>
>> It may very well be that it's OK to ship binaries with ATLAS, but just
>> to build them without any fancy instruction support (no SSE, SSE2 or
>> anything else of that kind, just plain x87 code).
>>   
> 
> I think this is what the default should be (but plain SSE allowed).  
> However, since I have moved, the machine I was using to build "official" 
> binaries has switched and that is probably at the core of the problem.
> 
> Also,  I've tried to build ATLAS 3.8.0 without SSE without success (when 
> I'm on a machine that has it).
> 
> It would be useful to track which binaries are giving people problems as 
> I built the most recent ones on a VM against an old version of ATLAS 
> (3.6.0) that has been compiled on windows for a long time.
> 
> I'm happy to upload a better binary of NumPy (if I can figure out which 
> one is giving people grief and how to create a decent one).
> 
> -Travis O.
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From aisaac at american.edu  Mon Dec 10 23:54:22 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 10 Dec 2007 23:54:22 -0500
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <475E03FA.9010608@enthought.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>	<475DCEAB.8080503@gmail.com><db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com><475E03FA.9010608@enthought.com>
Message-ID: <Mahogany-0.67.0-1392-20071210-235422.00@american.edu>

This may be a naive question, but just to be sure...

If troubles building without SSE2 support on an SSE2
processor are the problem, withould the problem be addressed
by purchasing an old PIII like
<URL:http://cgi.ebay.com/Dell-OptiPlex-GX110-Pentium-III-1GHz-40GB-256MB-DVD-XP_W0QQitemZ130180707038QQihZ003QQcategoryZ140070QQcmdZViewItem>
or
<URL:http://cgi.ebay.com/Dell-Precision-210-Pentium-III-Dual-500MHz-512MB-30GB_W0QQitemZ130181576949QQihZ003QQcategoryZ51225QQcmdZViewItem>

If so I'd be happy to contribute part of the purchase price,
and I assume others would too.

What's more, I *have* an old PIII at home.  (Doesn't 
everybody?)  Unfortunately, I have almost no experience with 
compiled languages.  However if it would be useful, I'd be 
happy to try to build on my home machine (after this 
Friday).  I would have to ask a lot of questions...

Cheers,
Alan Isaac





From Chris.Barker at noaa.gov  Tue Dec 11 00:28:05 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 10 Dec 2007 21:28:05 -0800
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <475DFD1F.4000809@astraw.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475DFD1F.4000809@astraw.com>
Message-ID: <475E1FE5.4020808@noaa.gov>

Andrew Straw wrote:
> A function
> could be called at numpy import time that specifically checks for the
> instruction set on the CPU running 

Even better would be a run-time selection of the "best" version. I've 
often fantasized about an ATLAS that could do this.

I think the Intel MKL has this feature (though maybe only for Intel 
processors). The MKL runtime is re-distributable, but somehow I doubt 
that we could have one person buy one copy and distribute binaries to 
the entire numpy-using world --- but does anyone know?

http://www.intel.com/cd/software/products/asmo-na/eng/346084.htm

and

http://www.intel.com/cd/software/products/asmo-na/eng/266854.htm#copies

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From cournape at gmail.com  Tue Dec 11 00:55:13 2007
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 11 Dec 2007 14:55:13 +0900
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
Message-ID: <5b8d13220712102155k5bbe571fl41986d9a34dcce6b@mail.gmail.com>

On Dec 11, 2007 11:03 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Dec 10, 2007 4:41 PM, Robert Kern <robert.kern at gmail.com> wrote:
>
> > The current situation is untenable. I will gladly accept a slow BLAS for an
> > official binary that won't segfault anywhere. We can look for a faster BLAS later.
>
> Just to add a note to this: John Hunter and I just finished teaching a
> python workshop here in Boulder, and one attendee had a recurring
> all-out crash on WinXP.  Eventually John was able to track it to a bad
> BLAS call, but the death was an 'illegal instruction'. We then noticed
> that this was on an older Pentium III laptop, and I'd be willing to
> bet that the problem is an ATLAS compiled with SSE2 support.  The PIII
> chip only has plain SSE, not SSE2, and that's the kind of crash I've
> seen when  accidentally running code compiled in my office machine (a
> P4) on my laptop (a similarly old PIII).
>
> It may very well be that it's OK to ship binaries with ATLAS, but just
> to build them without any fancy instruction support (no SSE, SSE2 or
> anything else of that kind, just plain x87 code).
>
The problem is that this is non trivial, and unsupported. I tried to
do it, asked the main author of ATLAS, but building an ATLAS which
does not use any instruction set present on the CPU used for build is
too difficult, and not a priority for the main author of ATLAS. The
build system of ATLAS being extremely complex (code is generated by C
programs themselves compiled on the fly by other C softwares), I gave
up on the idea of doing it by myself.

But anyway, honestly, this is kind of stupid: ATLAS needs to be
compiled with the same CPU it is run for for good performances. For
example, if the L1 cache is of different size, ATLAS performances are
already not so good. So if you are not event using SSE/SSE2...

David


From cournape at gmail.com  Tue Dec 11 01:00:21 2007
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 11 Dec 2007 15:00:21 +0900
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <475DFD1F.4000809@astraw.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475DFD1F.4000809@astraw.com>
Message-ID: <5b8d13220712102200w6513fa68y92e2628928650789@mail.gmail.com>

On Dec 11, 2007 11:59 AM, Andrew Straw <strawman at astraw.com> wrote:
> An idea that occurred to me after reading Fernando's email. A function
> could be called at numpy import time that specifically checks for the
> instruction set on the CPU running and makes sure that is completely
> covers the instruction set available through all the various calls,
> including to BLAS. If this kind of thing were added, numpy could fail
> with a loud warning rather than dying with mysterious errors later on.
> The trouble would seem that you can switch your BLAS shared library
> without re-compiling numpy, so numpy would have to do a run-time query
> of ATLAS, etc. for compilation issues. Which is likely
> library-dependent, and furthermore, not having looked into BLAS
> implementations, I'm not sure that (m)any of them provide such
> information. Do they? Is this idea technically possible?

It is possible, and has been done: that's how matlab did it when it
used ATLAS. Now, it is not easy, and would require some changes
Basically, the solution I would see would be to have a wrapper
library, to which every call would be done, and the wrapper library
could "reroute" the calls to the right, dynamically loaded library.
This requires several things which are not theoretically difficult,
but a pain to do right (cross platform library loader, etc...)

David


From cournape at gmail.com  Tue Dec 11 01:04:09 2007
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 11 Dec 2007 15:04:09 +0900
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <475E082D.1010107@astraw.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475E03FA.9010608@enthought.com> <475E082D.1010107@astraw.com>
Message-ID: <5b8d13220712102204w7b81aa3by6e308882b91619ea@mail.gmail.com>

On Dec 11, 2007 12:46 PM, Andrew Straw <strawman at astraw.com> wrote:
> According to the QEMU website, QEMU does not (yet) emulate SSE on x86
> target, so a Windows installation on a QEMU virtual machine may be a
> good way to build binaries free of these issues.
> http://fabrice.bellard.free.fr/qemu/qemu-tech.html
I tried this, this does not work (it actually emulates SSE). I went
further, and managed to disable SSE support in qemu...

But again, what's the point:  it takes ages to compile (qemu without
the hardware accelerator is slow, like ten times slower), and you will
end up with a really bad atlas, since atlas optimizaton is entirely
based on runtime timers, which do not make sense anymore.

I mean, really, what's the point of doing all this compared to using
blas/lapack from netlib ? In practice, is it really slower ? For what
? I know I don't care so much, and I am a heavy user of numpy.

David


From fperez.net at gmail.com  Tue Dec 11 01:16:37 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 10 Dec 2007 23:16:37 -0700
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <5b8d13220712102204w7b81aa3by6e308882b91619ea@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475E03FA.9010608@enthought.com> <475E082D.1010107@astraw.com>
	<5b8d13220712102204w7b81aa3by6e308882b91619ea@mail.gmail.com>
Message-ID: <db6b5ecc0712102216m185ec0c3i6e226934bf5a6d18@mail.gmail.com>

On Dec 10, 2007 11:04 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Dec 11, 2007 12:46 PM, Andrew Straw <strawman at astraw.com> wrote:
> > According to the QEMU website, QEMU does not (yet) emulate SSE on x86
> > target, so a Windows installation on a QEMU virtual machine may be a
> > good way to build binaries free of these issues.
> > http://fabrice.bellard.free.fr/qemu/qemu-tech.html
> I tried this, this does not work (it actually emulates SSE). I went
> further, and managed to disable SSE support in qemu...
>
> But again, what's the point:  it takes ages to compile (qemu without
> the hardware accelerator is slow, like ten times slower), and you will
> end up with a really bad atlas, since atlas optimizaton is entirely
> based on runtime timers, which do not make sense anymore.
>
> I mean, really, what's the point of doing all this compared to using
> blas/lapack from netlib ? In practice, is it really slower ? For what
> ? I know I don't care so much, and I am a heavy user of numpy.

For certain cases the difference can be pretty dramatic, but I think
there's a simple, reasonable solution that is likely to work: ship TWO
binaries of Numpy/Scipy each time:

1. {numpy,scipy}-reference: built with the reference blas from netlib,
no atlas, period.

2. {}-atlas: built with whatever the developers have at the time,
which will likely mean these days a core 2 duo with SSE2 support.
What hardware it was built on should be indicated, so people can at
least know this fact.

Just indicate that:

- The atlas version is likely faster, but fully unsupported and likely
to crash older platforms, no refunds.

- If you *really* care about performance, you should build Atlas
yourself or be 100% sure that you're using an Atlas built on the same
chip you're using, so the build-time timing and blocking choices are
actually meaningful.

That sounds like a reasonable bit of extra work (a lot easier than
building a run-time dynamic atlas) with a true payoff in terms of
stability.  No?

Cheers,

f


From meine at informatik.uni-hamburg.de  Tue Dec 11 04:32:05 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Tue, 11 Dec 2007 10:32:05 +0100
Subject: [Numpy-discussion] ndarray.clip only with lower or upper values?
In-Reply-To: <e4412d6b0712101446o6e8c65c9g4de088d4c8a1937d@mail.gmail.com>
References: <200712101521.56989.meine@informatik.uni-hamburg.de>
	<e4412d6b0712101446o6e8c65c9g4de088d4c8a1937d@mail.gmail.com>
Message-ID: <200712111032.05435.meine@informatik.uni-hamburg.de>

Am Montag, 10. Dezember 2007 23:46:17 schrieb Timothy Hochberg:
> > TypeError: function takes at least 2 arguments (1 given)
> >
> > (I could simulate that by passing max = maximum_value_of(a.dtype), if
> > that existed, see my other mail.)
>
> Why not just use minimum or maximum as needed instead of overloading clip?

You mean one of the following?
  a.clip(min = 10, max = numpy.finfo(a.dtype).max)
  a.clip(min = 10, max = numpy.iinfo(a.dtype).max)

Three reasons:
- this is not very concise
- it is less efficient than specialized clip variants
  (not much / too important though)
- I would have to discriminate integral and floating point types

How is the latter done in numpy?  Is there a good reason not to have 
numpy.rangetraits(sometype) with min/max as in iinfo/finfo?
Should I use isinstance(mytype, int)?

-- 
Ciao, /  /
     /--/
    /  / ANS


From meine at informatik.uni-hamburg.de  Tue Dec 11 04:43:20 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Tue, 11 Dec 2007 10:43:20 +0100
Subject: [Numpy-discussion] Minimum and maximum values of numpy
	datatypes?
In-Reply-To: <e76aa17f0712100823s6ff46d17w3a983d26bc55dc6f@mail.gmail.com>
References: <200712101518.35218.meine@informatik.uni-hamburg.de>
	<e76aa17f0712100823s6ff46d17w3a983d26bc55dc6f@mail.gmail.com>
Message-ID: <200712111043.21037.meine@informatik.uni-hamburg.de>

Am Montag, 10. Dezember 2007 17:23:07 schrieb Matthieu Brucher:
> I had the same problem sooner today, someone told me the answer : use
> numpy.info object ;)

I saw this shortly after posting (what a coincidence), and I planned to reply 
to myself, but my mail did not make it to the list very quickly.. (nearly two 
hours!)

Thanks, that's indeed what I was looking for.

-- 
Ciao, /  /
     /--/
    /  / ANS


From cournape at gmail.com  Tue Dec 11 05:31:56 2007
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 11 Dec 2007 19:31:56 +0900
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <db6b5ecc0712102216m185ec0c3i6e226934bf5a6d18@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475E03FA.9010608@enthought.com> <475E082D.1010107@astraw.com>
	<5b8d13220712102204w7b81aa3by6e308882b91619ea@mail.gmail.com>
	<db6b5ecc0712102216m185ec0c3i6e226934bf5a6d18@mail.gmail.com>
Message-ID: <5b8d13220712110231w2723325eh29c3a709aa9d9c3d@mail.gmail.com>

On Dec 11, 2007 3:16 PM, Fernando Perez <fperez.net at gmail.com> wrote:
>
> On Dec 10, 2007 11:04 PM, David Cournapeau <cournape at gmail.com> wrote:
> > On Dec 11, 2007 12:46 PM, Andrew Straw <strawman at astraw.com> wrote:
> > > According to the QEMU website, QEMU does not (yet) emulate SSE on x86
> > > target, so a Windows installation on a QEMU virtual machine may be a
> > > good way to build binaries free of these issues.
> > > http://fabrice.bellard.free.fr/qemu/qemu-tech.html
> > I tried this, this does not work (it actually emulates SSE). I went
> > further, and managed to disable SSE support in qemu...
> >
> > But again, what's the point:  it takes ages to compile (qemu without
> > the hardware accelerator is slow, like ten times slower), and you will
> > end up with a really bad atlas, since atlas optimizaton is entirely
> > based on runtime timers, which do not make sense anymore.
> >
> > I mean, really, what's the point of doing all this compared to using
> > blas/lapack from netlib ? In practice, is it really slower ? For what
> > ? I know I don't care so much, and I am a heavy user of numpy.
>
> For certain cases the difference can be pretty dramatic,

Is it if you use it on a CPU which is really different than the one
used for the compilation ? For example, the default atlas on debian
(built on pentium 2, no sse) is not really much faster than netlib,
IMHO.

> ship TWO
> binaries of Numpy/Scipy each time:
>

I think that in the short term, that's the only solution. The netlib
one being the default, the atlas one being the optional (with
CoreDuoSSE2 something in the name). It should work by default
everywhere.

David


From fullung at gmail.com  Tue Dec 11 06:47:48 2007
From: fullung at gmail.com (Albert Strasheim)
Date: Tue, 11 Dec 2007 13:47:48 +0200
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp><525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com><e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com><952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca><475DCEAB.8080503@gmail.com><db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com><475DFD1F.4000809@astraw.com>
	<475E1FE5.4020808@noaa.gov>
Message-ID: <008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>

Hello

I think this idea is the way to go (maybe along with an ACML build, but my 
limited testing seemed to indicate that MKL works on AMD CPUs).

In fact, I apparently proposed it about a year ago:

https://svn.enthought.com/enthought/ticket/899

No takers so far...

Cheers,

Albert

P.S. NumPy on Windows and Linux built with MKL works like a charm for me.

----- Original Message ----- 
From: "Christopher Barker" <Chris.Barker at noaa.gov>
To: "Discussion of Numerical Python" <numpy-discussion at scipy.org>
Sent: Tuesday, December 11, 2007 7:28 AM
Subject: Re: [Numpy-discussion] Changing the distributed binary for numpy 
1.0.4 for windows ?


> Andrew Straw wrote:
>> A function
>> could be called at numpy import time that specifically checks for the
>> instruction set on the CPU running
>
> Even better would be a run-time selection of the "best" version. I've
> often fantasized about an ATLAS that could do this.
>
> I think the Intel MKL has this feature (though maybe only for Intel
> processors). The MKL runtime is re-distributable, but somehow I doubt
> that we could have one person buy one copy and distribute binaries to
> the entire numpy-using world --- but does anyone know?
>
> http://www.intel.com/cd/software/products/asmo-na/eng/346084.htm
>
> and
>
> http://www.intel.com/cd/software/products/asmo-na/eng/266854.htm#copies
>
> -Chris



From matthieu.brucher at gmail.com  Tue Dec 11 07:01:18 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 11 Dec 2007 13:01:18 +0100
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475DFD1F.4000809@astraw.com> <475E1FE5.4020808@noaa.gov>
	<008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
Message-ID: <e76aa17f0712110401x532ce6eyfd659ebd5750bc60@mail.gmail.com>

2007/12/11, Albert Strasheim <fullung at gmail.com>:
>
> Hello
>
> I think this idea is the way to go (maybe along with an ACML build, but my
> limited testing seemed to indicate that MKL works on AMD CPUs).


I'm trying to build numpy with ACML, but ACML misses the CBLAS interface and
thus the compilation fails. When Scons will be used, it should be doable.

Matthieu
-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071211/b400fdd4/attachment.html>

From cournape at gmail.com  Tue Dec 11 07:15:35 2007
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 11 Dec 2007 21:15:35 +0900
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475DFD1F.4000809@astraw.com> <475E1FE5.4020808@noaa.gov>
	<008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
Message-ID: <5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>

On Dec 11, 2007 8:47 PM, Albert Strasheim <fullung at gmail.com> wrote:
> Hello
>
> I think this idea is the way to go (maybe along with an ACML build, but my
> limited testing seemed to indicate that MKL works on AMD CPUs).
>
I am personally totally against it. It is one thing to support
proprietary software, that's quite another to build our official
binaries against it. I consider myself far from any kind of open
source zealot, but that would be crossing a line I would much prefer
avoiding to cross.

David


From tim.hochberg at ieee.org  Tue Dec 11 10:59:45 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Tue, 11 Dec 2007 08:59:45 -0700
Subject: [Numpy-discussion] ndarray.clip only with lower or upper values?
In-Reply-To: <200712111032.05435.meine@informatik.uni-hamburg.de>
References: <200712101521.56989.meine@informatik.uni-hamburg.de>
	<e4412d6b0712101446o6e8c65c9g4de088d4c8a1937d@mail.gmail.com>
	<200712111032.05435.meine@informatik.uni-hamburg.de>
Message-ID: <e4412d6b0712110759r30d9e7edoe7cee4feb0631233@mail.gmail.com>

On Dec 11, 2007 2:32 AM, Hans Meine <meine at informatik.uni-hamburg.de> wrote:

> Am Montag, 10. Dezember 2007 23:46:17 schrieb Timothy Hochberg:
> > > TypeError: function takes at least 2 arguments (1 given)
> > >
> > > (I could simulate that by passing max = maximum_value_of(a.dtype), if
> > > that existed, see my other mail.)
> >
> > Why not just use minimum or maximum as needed instead of overloading
> clip?
>
> You mean one of the following?
>  a.clip(min = 10, max = numpy.finfo(a.dtype).max)
>  a.clip(min = 10, max = numpy.iinfo(a.dtype).max)


No. I mean:

  numpy.maximum(a, 10)

To correspond to the above example.



>
> Three reasons:
> - this is not very concise
> - it is less efficient than specialized clip variants
>  (not much / too important though)
> - I would have to discriminate integral and floating point types
>
> How is the latter done in numpy?  Is there a good reason not to have
> numpy.rangetraits(sometype) with min/max as in iinfo/finfo?
> Should I use isinstance(mytype, int)?
>
> --
> Ciao, /  /
>     /--/
>    /  / ANS
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071211/57c736a4/attachment.html>

From subscriber100 at rjs.org  Tue Dec 11 11:24:40 2007
From: subscriber100 at rjs.org (Ray Schumacher)
Date: Tue, 11 Dec 2007 08:24:40 -0800
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <mailman.9792.1197369124.2301.numpy-discussion@scipy.org>
References: <mailman.9792.1197369124.2301.numpy-discussion@scipy.org>
Message-ID: <6.2.3.4.2.20071211081156.035c5460@rjs.org>

At 02:32 AM 12/11/2007, you wrote:
>If so I'd be happy to contribute part of the purchase price,
>and I assume others would too.
>
>What's more, I *have* an old PIII at home.

The main company I consult for is set to buy the Intel compiler and 
FFT lib for Windows, for the express purpose of compiling Python, 
numpy, and the fastest FFT for each CPU, for new products.
I develop on both an old dual PIII and a new Core Duo, and there is 
also every other flavor in the shop. As I read it, the binaries 
should be distributable.
The catch is, I have never compiled Python or numpy - I had seen a 
poster who offered help. When the company actually purchases the 
product I'd be glad to do it on 2-3 targets if someone can assist 
with the parameters. We have one consultant here who has done it on Linux.


Ray Schumacher
Congitive Vision
8580 Production Ave., Suite B
San Diego, CA 92121
858.578.2778
http://cognitivevision.com/  


-- 
No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.503 / Virus Database: 269.17.0/1180 - Release Date: 12/10/2007 2:51 PM




From Chris.Barker at noaa.gov  Tue Dec 11 12:58:33 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 11 Dec 2007 09:58:33 -0800
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475DFD1F.4000809@astraw.com> <475E1FE5.4020808@noaa.gov>
	<008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
	<5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>
Message-ID: <475ECFC9.8060202@noaa.gov>

David Cournapeau wrote:
>> I think this idea is the way to go (maybe along with an ACML build, but my
>> limited testing seemed to indicate that MKL works on AMD CPUs).
>>
> I am personally totally against it. It is one thing to support
> proprietary software, that's quite another to build our official
> binaries against it. I consider myself far from any kind of open
> source zealot, but that would be crossing a line I would much prefer
> avoiding to cross.

Interesting -- I DO consider myself a kind of Open Source Zealot -- and 
this doesn't bother me a bit.

It would bother me a LOT if numpy could only be built against this lib, 
and not an Open Source one -- but I don't see this as any different than 
providing a binary built with the Microsoft compiler.

The way it "SHOULD" be done is the OS vendor should provide these libs 
-- Apple does, and most of the Linux Distros do (with varying success). 
If MS could just cut a deal with Intel and AMD, we could count on all 
versions of Windows having an optimized lib-- oh well, one can dream!

I'm not sure the licensing really makes it possible though. Numpy isn't 
exactly an application, but rather a development tool, so I'm not sure 
how Intel would feel about it being distributed. Also, it looks like 
they require each "developer" to have license, rather than only the 
person building the final binary -- so having the one person building 
the final distro may not be kosher. IANAL.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Tue Dec 11 13:04:50 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 11 Dec 2007 10:04:50 -0800
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <db6b5ecc0712102216m185ec0c3i6e226934bf5a6d18@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475E03FA.9010608@enthought.com> <475E082D.1010107@astraw.com>
	<5b8d13220712102204w7b81aa3by6e308882b91619ea@mail.gmail.com>
	<db6b5ecc0712102216m185ec0c3i6e226934bf5a6d18@mail.gmail.com>
Message-ID: <475ED142.1080902@noaa.gov>

Fernando Perez wrote:
> a simple, reasonable solution that is likely to work: ship TWO
> binaries of Numpy/Scipy each time:
> 
> 1. {numpy,scipy}-reference: built with the reference blas from netlib,
> no atlas, period.
> 
> 2. {}-atlas: built with whatever the developers have at the time,
> which will likely mean these days a core 2 duo with SSE2 support.
> What hardware it was built on should be indicated, so people can at
> least know this fact.

I disagree -- having an atlas version that only works on recent hardware 
is just asking for complaints -- I think the ONLY way to go is for the 
"standard" binary to be universal. Instructions should be provided for 
building other versions, and if third parties want to distribute 
processor-dependent versions, then great, but that's an extra.

By the way, I've always been confused by static linking of lapack/atlas 
-- it seems to me that this kind of thing is on of the best uses of 
dynamic linking -- the main binary is processor dependent, and it is 
linked, at runtime, with the host's processor specific lib. -- could we 
do it that way:

The standard distro includes a universal dynamic lib.

Folks build processor-specific libs that can be dropped in to replace 
the universal one if someone so desires.

Then it's a "small" step to include a runtime tool that selects the best 
lib it can find on the system.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From fperez.net at gmail.com  Tue Dec 11 13:43:23 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 11 Dec 2007 11:43:23 -0700
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <475ED142.1080902@noaa.gov>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475E03FA.9010608@enthought.com> <475E082D.1010107@astraw.com>
	<5b8d13220712102204w7b81aa3by6e308882b91619ea@mail.gmail.com>
	<db6b5ecc0712102216m185ec0c3i6e226934bf5a6d18@mail.gmail.com>
	<475ED142.1080902@noaa.gov>
Message-ID: <db6b5ecc0712111043g2cb909c1o302b48e0158ba045@mail.gmail.com>

On Dec 11, 2007 11:04 AM, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> Fernando Perez wrote:
> > a simple, reasonable solution that is likely to work: ship TWO
> > binaries of Numpy/Scipy each time:
> >
> > 1. {numpy,scipy}-reference: built with the reference blas from netlib,
> > no atlas, period.
> >
> > 2. {}-atlas: built with whatever the developers have at the time,
> > which will likely mean these days a core 2 duo with SSE2 support.
> > What hardware it was built on should be indicated, so people can at
> > least know this fact.
>
> I disagree -- having an atlas version that only works on recent hardware
> is just asking for complaints -- I think the ONLY way to go is for the
> "standard" binary to be universal. Instructions should be provided for
> building other versions, and if third parties want to distribute
> processor-dependent versions, then great, but that's an extra.

Well, what I had in mind was not something that would run *only* on
recent hardware.  A typical Atlas build on a recent box will run with
anything with SSE2, which means boxes as far back as the Pentium4, I
think.  Or is Atlas using anything that's core2-duo-specific in its
build?  At the workshop, the only problem John and I noticed was for
the Pentium III user, and there were plenty of old-looking boxes
around that were most certainly not Core2.

Cheers,

f


From meine at informatik.uni-hamburg.de  Tue Dec 11 14:52:03 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Tue, 11 Dec 2007 20:52:03 +0100
Subject: [Numpy-discussion] ndarray.clip only with lower or upper values?
In-Reply-To: <e4412d6b0712110759r30d9e7edoe7cee4feb0631233@mail.gmail.com>
References: <200712101521.56989.meine@informatik.uni-hamburg.de>
	<200712111032.05435.meine@informatik.uni-hamburg.de>
	<e4412d6b0712110759r30d9e7edoe7cee4feb0631233@mail.gmail.com>
Message-ID: <200712112052.11572.meine@informatik.uni-hamburg.de>

On Dienstag 11 Dezember 2007, Timothy Hochberg wrote:
> > You mean one of the following?
> >  a.clip(min = 10, max = numpy.finfo(a.dtype).max)
> >  a.clip(min = 10, max = numpy.iinfo(a.dtype).max)
>
> No. I mean:
>
>   numpy.maximum(a, 10)
>
> To correspond to the above example.

Great, thanks for the hints.  That's good enough; could be pythonic to let 
clip() forward calls to minimum/maximum if only one bound is given though.
I had a look at the code, but I am not used enough to the Python/C API, let 
alone to numpy's internals, to *quickly* hack this.

Ciao, /  /                                                    .o.
     /--/                                                     ..o
    /  / ANS                                                  ooo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071211/748befff/attachment.sig>

From oliphant at enthought.com  Tue Dec 11 15:58:05 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Tue, 11 Dec 2007 14:58:05 -0600
Subject: [Numpy-discussion] ndarray.clip only with lower or upper values?
In-Reply-To: <200712112052.11572.meine@informatik.uni-hamburg.de>
References: <200712101521.56989.meine@informatik.uni-hamburg.de>	<200712111032.05435.meine@informatik.uni-hamburg.de>	<e4412d6b0712110759r30d9e7edoe7cee4feb0631233@mail.gmail.com>
	<200712112052.11572.meine@informatik.uni-hamburg.de>
Message-ID: <475EF9DD.4050502@enthought.com>

Hans Meine wrote:
> On Dienstag 11 Dezember 2007, Timothy Hochberg wrote:
>   
>>> You mean one of the following?
>>>  a.clip(min = 10, max = numpy.finfo(a.dtype).max)
>>>  a.clip(min = 10, max = numpy.iinfo(a.dtype).max)
>>>       
>> No. I mean:
>>
>>   numpy.maximum(a, 10)
>>
>> To correspond to the above example.
>>     
>
> Great, thanks for the hints.  That's good enough; could be pythonic to let 
> clip() forward calls to minimum/maximum if only one bound is given though.
> I had a look at the code, but I am not used enough to the Python/C API, let 
> alone to numpy's internals, to *quickly* hack this.
>   

It is done, now in SVN.

-Travis




From cournape at gmail.com  Tue Dec 11 19:22:44 2007
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 12 Dec 2007 09:22:44 +0900
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <475ECFC9.8060202@noaa.gov>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475DFD1F.4000809@astraw.com> <475E1FE5.4020808@noaa.gov>
	<008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
	<5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>
	<475ECFC9.8060202@noaa.gov>
Message-ID: <5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com>

On Dec 12, 2007 2:58 AM, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> David Cournapeau wrote:
> >> I think this idea is the way to go (maybe along with an ACML build, but my
> >> limited testing seemed to indicate that MKL works on AMD CPUs).
> >>
> > I am personally totally against it. It is one thing to support
> > proprietary software, that's quite another to build our official
> > binaries against it. I consider myself far from any kind of open
> > source zealot, but that would be crossing a line I would much prefer
> > avoiding to cross.
>
> Interesting -- I DO consider myself a kind of Open Source Zealot -- and
> this doesn't bother me a bit.
>
> It would bother me a LOT if numpy could only be built against this lib,
> and not an Open Source one -- but I don't see this as any different than
> providing a binary built with the Microsoft compiler.
>
For me it is: when using a MS compiler, you are not forcing people to
use a non open source product (except maybe the C runtime). What will
happen if we offer binaries using MKL ? The ATLAS will not be tested
anymore on windows, it forces every developer to use the MKL to
support it.... At least, now, with atlas problems, I can reproduce the
problems. With the MKL, not so much.

cheers,

David


From cournape at gmail.com  Tue Dec 11 19:41:14 2007
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 12 Dec 2007 09:41:14 +0900
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <475ED142.1080902@noaa.gov>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>
	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475E03FA.9010608@enthought.com> <475E082D.1010107@astraw.com>
	<5b8d13220712102204w7b81aa3by6e308882b91619ea@mail.gmail.com>
	<db6b5ecc0712102216m185ec0c3i6e226934bf5a6d18@mail.gmail.com>
	<475ED142.1080902@noaa.gov>
Message-ID: <5b8d13220712111641g7bd5848cmcc005161a7f5e304@mail.gmail.com>

On Dec 12, 2007 3:04 AM, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> Fernando Perez wrote:
> > a simple, reasonable solution that is likely to work: ship TWO
> > binaries of Numpy/Scipy each time:
> >
> > 1. {numpy,scipy}-reference: built with the reference blas from netlib,
> > no atlas, period.
> >
> > 2. {}-atlas: built with whatever the developers have at the time,
> > which will likely mean these days a core 2 duo with SSE2 support.
> > What hardware it was built on should be indicated, so people can at
> > least know this fact.
>
> I disagree -- having an atlas version that only works on recent hardware
> is just asking for complaints -- I think the ONLY way to go is for the
> "standard" binary to be universal. Instructions should be provided for
> building other versions, and if third parties want to distribute
> processor-dependent versions, then great, but that's an extra.
>

But that's the problem: it is next to impossible to build ATLAS which
works on any processor ! At least, it is not supported by ATLAS
developers, and the way it suggested did not work for me.

> By the way, I've always been confused by static linking of lapack/atlas
> -- it seems to me that this kind of thing is on of the best uses of
> dynamic linking -- the main binary is processor dependent, and it is
> linked, at runtime, with the host's processor specific lib. -- could we
> do it that way:
>

I believe that lapack/blas are dynamically linked by default. But for
a portable solution, I don't see any other solution than  dynamic
loading. The BLAS/LAPACK library would be like a plug-in, loaded at
runtime. But this requires some work, and in perticular, doing it in a
cross platform way is not trivial

> The standard distro includes a universal dynamic lib.
>
> Folks build processor-specific libs that can be dropped in to replace
> the universal one if someone so desires.

Asking the users to do is  just asking for new a set of problems,
IMHO. I used this approach for the rpms on ashigabou repository (they
are compiled against netlib blas/lapack, but you can change the used
libraries using runtime library path), but that's not portable. What
is needed is a true runtime system in numpy which can detect the
processor (not too difficult, although doing it for all compilers is
not trivial either), load the right library (difficult to do in a
cross platform way), and use it accordingly (not too difficult, but
requires some work).

David


From robert.kern at gmail.com  Tue Dec 11 19:48:10 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 11 Dec 2007 18:48:10 -0600
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>	<e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com>	<952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca>	<475DCEAB.8080503@gmail.com>	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>	<475DFD1F.4000809@astraw.com>
	<475E1FE5.4020808@noaa.gov>	<008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>	<5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>	<475ECFC9.8060202@noaa.gov>
	<5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com>
Message-ID: <475F2FCA.5070605@gmail.com>

David Cournapeau wrote:
> On Dec 12, 2007 2:58 AM, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>> David Cournapeau wrote:
>>>> I think this idea is the way to go (maybe along with an ACML build, but my
>>>> limited testing seemed to indicate that MKL works on AMD CPUs).
>>>>
>>> I am personally totally against it. It is one thing to support
>>> proprietary software, that's quite another to build our official
>>> binaries against it. I consider myself far from any kind of open
>>> source zealot, but that would be crossing a line I would much prefer
>>> avoiding to cross.
>> Interesting -- I DO consider myself a kind of Open Source Zealot -- and
>> this doesn't bother me a bit.
>>
>> It would bother me a LOT if numpy could only be built against this lib,
>> and not an Open Source one -- but I don't see this as any different than
>> providing a binary built with the Microsoft compiler.
>>
> For me it is: when using a MS compiler, you are not forcing people to
> use a non open source product (except maybe the C runtime). What will
> happen if we offer binaries using MKL ? The ATLAS will not be tested
> anymore on windows, it forces every developer to use the MKL to
> support it.... At least, now, with atlas problems, I can reproduce the
> problems. With the MKL, not so much.

I agree. The official-official Win32 binaries (numpy-<version>-py<pyversion>.msi
and numpy-<version>-py<pyversion>-win32.egg on the SourceForge donwload page)
should be unencumbered. Other versions can be on the download page, too, but
they should be named differently, like numpy-mkl-<version>-... .

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From ryanlists at gmail.com  Tue Dec 11 20:45:20 2007
From: ryanlists at gmail.com (Ryan Krauss)
Date: Tue, 11 Dec 2007 19:45:20 -0600
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <475F2FCA.5070605@gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<475DCEAB.8080503@gmail.com>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475DFD1F.4000809@astraw.com> <475E1FE5.4020808@noaa.gov>
	<008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
	<5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>
	<475ECFC9.8060202@noaa.gov>
	<5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com>
	<475F2FCA.5070605@gmail.com>
Message-ID: <c5b438120712111745i3b4fccecsf4f22b05bf70c43b@mail.gmail.com>

Near as I can tell, this is still unresolved for people with non-sse2
machines.  Is that right?

I have a student trying to get started with such a machine.  Numpy is
causing Python to crash.  What is the easiest solution?  Does he need
to build numpy from source on that machine (I actually still have
access to one and could do it)?

Is it just Numpy or also Scipy?

Here is his responses to me:

Laptop - Ok
Windows XP Professional, Service Pack 2
AMD Athlon 64 3400+  (ClawHammer)
1.67 GHz, 768 MB of RAM
Chipset:  SiS 755/755FX
Southbridge:  SiS LPC Bridge
Instructions:  MMX (+), 3DNow! (+), SSE, SSE2, x86-64

Machine 1 - Crashes
Windows XP Professional, Service Pack 2
AMD Athlon XP 2000+  (Thoroughbred)
1.67 GHz, 768 MB of RAM
ASUS A7V8X-X motherboard
Chipset:  VIA KT400 (VT8377)
Southbridge:  VIA VT8235
Instructions:  MMX (+), 3DNow! (+), SSE

Machine 2 - Crashes
Windows XP Professional, Service Pack 2
AMD Athlon XP 2600+  (Barton)
1.92 GHz, 2.0 GB of RAM
ASUS A7V880 motherboard
Chipset:  VIA KT880
Southbridge:  VIA VT8237
Instructions:  MMX (+), 3DNow! (+), SSE

I ran the following statements on both machines which caused it to crash:

import numpy
numpy.test()

Here is the output:

    Numpy is installed in C:\Python25\lib\site-packages\numpy
    Numpy version 1.0.4
    Python version 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC
v.1310 32 bit (Int
    el)]
      Found 10/10 tests for numpy.core.defmatrix
      Found 36/36 tests for numpy.core.ma
      Found 223/223 tests for numpy.core.multiarray
      Found 65/65 tests for numpy.core.numeric
      Found 31/31 tests for numpy.core.numerictypes
      Found 12/12 tests for numpy.core.records
      Found 6/6 tests for numpy.core.scalarmath
      Found 14/14 tests for numpy.core.umath
      Found 4/4 tests for numpy.ctypeslib
      Found 5/5 tests for numpy.distutils.misc_util
      Found 1/1 tests for numpy.fft.fftpack
      Found 3/3 tests for numpy.fft.helper
      Found 9/9 tests for numpy.lib.arraysetops
      Found 46/46 tests for numpy.lib.function_base
      Found 5/5 tests for numpy.lib.getlimits
      Found 4/4 tests for numpy.lib.index_tricks
      Found 3/3 tests for numpy.lib.polynomial
      Found 49/49 tests for numpy.lib.shape_base
      Found 15/15 tests for numpy.lib.twodim_base
      Found 43/43 tests for numpy.lib.type_check
      Found 1/1 tests for numpy.lib.ufunclike
      Found 40/40 tests for numpy.linalg
      Found 2/2 tests for numpy.random
      Found 0/0 tests for __main__
    ................................................................................
    ................................................................................
    ................................................................................
    ................................................................................
    .....................................

Sounds like the problem is the fact that my desktop computers do not
support SSE2 instructions which are in the latest numpy binaries.
This also explains why it works fine on the laptop which does support
SSE2.

On Dec 11, 2007 6:48 PM, Robert Kern <robert.kern at gmail.com> wrote:
> David Cournapeau wrote:
> > On Dec 12, 2007 2:58 AM, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> >> David Cournapeau wrote:
> >>>> I think this idea is the way to go (maybe along with an ACML build, but my
> >>>> limited testing seemed to indicate that MKL works on AMD CPUs).
> >>>>
> >>> I am personally totally against it. It is one thing to support
> >>> proprietary software, that's quite another to build our official
> >>> binaries against it. I consider myself far from any kind of open
> >>> source zealot, but that would be crossing a line I would much prefer
> >>> avoiding to cross.
> >> Interesting -- I DO consider myself a kind of Open Source Zealot -- and
> >> this doesn't bother me a bit.
> >>
> >> It would bother me a LOT if numpy could only be built against this lib,
> >> and not an Open Source one -- but I don't see this as any different than
> >> providing a binary built with the Microsoft compiler.
> >>
> > For me it is: when using a MS compiler, you are not forcing people to
> > use a non open source product (except maybe the C runtime). What will
> > happen if we offer binaries using MKL ? The ATLAS will not be tested
> > anymore on windows, it forces every developer to use the MKL to
> > support it.... At least, now, with atlas problems, I can reproduce the
> > problems. With the MKL, not so much.
>
> I agree. The official-official Win32 binaries (numpy-<version>-py<pyversion>.msi
> and numpy-<version>-py<pyversion>-win32.egg on the SourceForge donwload page)
> should be unencumbered. Other versions can be on the download page, too, but
> they should be named differently, like numpy-mkl-<version>-... .
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless enigma
>  that is made terrible by our own mad attempt to interpret it as though it had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
>
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From fperez.net at gmail.com  Tue Dec 11 20:58:19 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 11 Dec 2007 18:58:19 -0700
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <c5b438120712111745i3b4fccecsf4f22b05bf70c43b@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com>
	<475DFD1F.4000809@astraw.com> <475E1FE5.4020808@noaa.gov>
	<008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
	<5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>
	<475ECFC9.8060202@noaa.gov>
	<5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com>
	<475F2FCA.5070605@gmail.com>
	<c5b438120712111745i3b4fccecsf4f22b05bf70c43b@mail.gmail.com>
Message-ID: <db6b5ecc0712111758w668b4be7ica9598813625242@mail.gmail.com>

On Dec 11, 2007 6:45 PM, Ryan Krauss <ryanlists at gmail.com> wrote:
> Near as I can tell, this is still unresolved for people with non-sse2
> machines.  Is that right?

Yup.  Your more detailed testing seems to confirm the hunch I had at
the weekend workshop that SSE2 is the culprit.  Thanks for the info.

It would be really great if we could somehow resolve this quickly.  I
have a WinXP install under Linux, but it seems to see my CPU as an
Athlon X2, so that won't work.  But I also have an old laptop with a
dual-boot XP that's a PIII (no SSE2, and probably the oldest
reasonable hardware we can expect to support).

If someone can provide me with simple instructions on what needs to be
done to build numpy on that thing, without buying any software, I can
volunteer to run a build script on every release that's PIII-safe.

I know nothing about developing on XP, but I'm willing to install the
necessary tools (if free) to at least get this built, because this is
really an untenable situation.  Auto-crashing isn't really a selling
point for software, even under Windows :)

Cheers,

f


From aisaac at american.edu  Tue Dec 11 21:05:25 2007
From: aisaac at american.edu (Alan Isaac)
Date: Tue, 11 Dec 2007 21:05:25 -0500
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <c5b438120712111745i3b4fccecsf4f22b05bf70c43b@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp><475DCEAB.8080503@gmail.com><db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com><475DFD1F.4000809@astraw.com>
	<475E1FE5.4020808@noaa.gov><008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za><5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com><475ECFC9.8060202@noaa.gov><5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com><475F2FCA.5070605@gmail.com><c5b438120712111745i3b4fccecsf4f22b05bf70c43b@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-2960-20071211-210525.00@CASC64043305.american.edu>

On Tue, 11 Dec 2007, Ryan Krauss wrote:
> I have a student trying to get started with such 
> a machine.  Numpy is causing Python to crash.  What is the 
> easiest solution?

Use 1.0.3.1

> Is it just Numpy or also Scipy? 

It is also SciPy.  Stuff that relies
only on NumPy wil work, plus anything
that is pure Python.

Cheers,
Alan Isaac





From fullung at gmail.com  Wed Dec 12 01:25:43 2007
From: fullung at gmail.com (Albert Strasheim)
Date: Wed, 12 Dec 2007 08:25:43 +0200
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp><525f23e80712100559q2be9eeb1s4d7164663539fbe2@mail.gmail.com><e76aa17f0712100730h4bb9f61dta2ff86a197ace193@mail.gmail.com><952962D8-18E4-4F54-A0F7-6405964E0EBD@physics.mcmaster.ca><475DCEAB.8080503@gmail.com><db6b5ecc0712101803n4129554ak57e1d10e2dfca396@mail.gmail.com><475DFD1F.4000809@astraw.com>
	<475E1FE5.4020808@noaa.gov><008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za><5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>
	<475ECFC9.8060202@noaa.gov>
Message-ID: <00ee01c83c87$d2bf40e0$0a83a8c0@sun.ac.za>

Hello all,

> I'm not sure the licensing really makes it possible though. Numpy isn't
> exactly an application, but rather a development tool, so I'm not sure
> how Intel would feel about it being distributed. Also, it looks like
> they require each "developer" to have license, rather than only the
> person building the final binary -- so having the one person building
> the final distro may not be kosher. IANAL.

It comes down to who is allowed to have the link libraries and who isn't. I 
doubt whether Intel's license agreement distinguishes between "normal" 
programs and "development tools".

If you're a developer, you need the link libraries (.lib files) to link your 
program against Intel MKL. According to Intel's redist.txt, you are not 
allowed to redistribute these files. Without these files, you can't link a 
new program against the Intel MKL DLLs (generally speaking).

You are allowed to redistribute the DLLs (as listed in redist.txt), without 
having to pay any further royalties. This means that you give any user the 
files they need to run a program you have linked against Intel MKL.

So as I see it, one NumPy developer would need to pay for Intel MKL.

Cheers,

Albert



From millman at berkeley.edu  Wed Dec 12 07:20:35 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 12 Dec 2007 04:20:35 -0800
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <db6b5ecc0712111758w668b4be7ica9598813625242@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<475DFD1F.4000809@astraw.com> <475E1FE5.4020808@noaa.gov>
	<008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
	<5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>
	<475ECFC9.8060202@noaa.gov>
	<5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com>
	<475F2FCA.5070605@gmail.com>
	<c5b438120712111745i3b4fccecsf4f22b05bf70c43b@mail.gmail.com>
	<db6b5ecc0712111758w668b4be7ica9598813625242@mail.gmail.com>
Message-ID: <c7009a550712120420g268a20dbo49004964ec394afd@mail.gmail.com>

On Dec 11, 2007 5:58 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Dec 11, 2007 6:45 PM, Ryan Krauss <ryanlists at gmail.com> wrote:
> > Near as I can tell, this is still unresolved for people with non-sse2
> > machines.  Is that right?
>
> Yup.  Your more detailed testing seems to confirm the hunch I had at
> the weekend workshop that SSE2 is the culprit.  Thanks for the info.
>
> It would be really great if we could somehow resolve this quickly.  I
> have a WinXP install under Linux, but it seems to see my CPU as an
> Athlon X2, so that won't work.  But I also have an old laptop with a
> dual-boot XP that's a PIII (no SSE2, and probably the oldest
> reasonable hardware we can expect to support).

I will put new binaries on the sourceforge site this weekend for both
NumPy 1.0.4 and SciPy 0.6.0.  I should be able to find an old PIII
WinXP machine around somewhere, which I will devote to building the
official non-SSE2 releases from here on out.

Cheers,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From sorendyrsting at gmail.com  Wed Dec 12 09:29:57 2007
From: sorendyrsting at gmail.com (=?ISO-8859-1?Q?S=F8ren_Dyrsting?=)
Date: Wed, 12 Dec 2007 15:29:57 +0100
Subject: [Numpy-discussion] numpy large arrays?
Message-ID: <dc0925740712120629r1bdfd6fchd141c765231017b6@mail.gmail.com>

Hi all

I need to perform computations involving large arrays. A lot of rows and no
more than e.g. 34 columns. My first choice is python/numpy because I'm
already used to code in matlab.

However I'm experiencing memory problems even though there is still 500 MB
available (2 GB total). I have cooked down my code to following meaningless
code snip. This code share some of the same structure and calls as my real
program and shows the same behaviour.

********************************************************
import numpy as N
import scipy as S

def stress():
    x = S.randn(200000,80)
    for i in range(8):
        print "%(0)d" % {"0": i}
        s = N.dot(x.T, x)
        sd = N.array([s.diagonal()])
        r = N.dot(N.ones((N.size(x,0),1),'d'), sd)
        x = x + r
        x = x / 1.01

********************************************************


To different symptoms depending how big x are:
1) the program becomes extremely slow after a few iterations.
2) if the size of x is increased a little the program fails with the message
"MemoryError" for example at line 'x = x + r', but different places in the
code depending on the matrice size and which computer I'm testing. This
might also occur after several iterations, not just during the first pass.

I'm using Windows XP, ActivePython 2.5.1.1, NumPy 1.0.4, SciPy  0.6.0.

- Is there an error under the hood in NumPy?
- Am I balancing on the edge of the performance of Python/NumPy and should
consider other environments. Fortran, C, BLAS, LAPACK e.t.c.
- Am I misusing NumPy? Changing coding style will be a good workaround and
even perform on larger datasets without errors?

Thanks in advance
/S?ren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071212/cd3e93d0/attachment.html>

From fperez.net at gmail.com  Wed Dec 12 11:10:06 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 12 Dec 2007 09:10:06 -0700
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <c7009a550712120420g268a20dbo49004964ec394afd@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<475E1FE5.4020808@noaa.gov> <008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
	<5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>
	<475ECFC9.8060202@noaa.gov>
	<5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com>
	<475F2FCA.5070605@gmail.com>
	<c5b438120712111745i3b4fccecsf4f22b05bf70c43b@mail.gmail.com>
	<db6b5ecc0712111758w668b4be7ica9598813625242@mail.gmail.com>
	<c7009a550712120420g268a20dbo49004964ec394afd@mail.gmail.com>
Message-ID: <db6b5ecc0712120810qe9723ma14f292a070b5951@mail.gmail.com>

> I will put new binaries on the sourceforge site this weekend for both
> NumPy 1.0.4 and SciPy 0.6.0.  I should be able to find an old PIII
> WinXP machine around somewhere, which I will devote to building the
> official non-SSE2 releases from here on out.

Great, many thanks!

Keep in mind that as Ryan's tests showed, Athlon XP chips also seem to
lack the SSE2 set, and those are a bit newer than the PIII, so they
might be a bit easier to find.  The Athlon 64 and Athlon X2 do have
SSE2, but apparently the XP doesn't.  I'm sure there's a wikipedia
page somewhere with a list of all the chips where SSE2 first
appeared...

Cheers,

f


From oliphant at enthought.com  Wed Dec 12 13:36:02 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 12 Dec 2007 12:36:02 -0600
Subject: [Numpy-discussion] Changing the distributed binary for numpy
 1.0.4 for windows ?
In-Reply-To: <db6b5ecc0712120810qe9723ma14f292a070b5951@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>	<475E1FE5.4020808@noaa.gov>
	<008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>	<5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>	<475ECFC9.8060202@noaa.gov>	<5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com>	<475F2FCA.5070605@gmail.com>	<c5b438120712111745i3b4fccecsf4f22b05bf70c43b@mail.gmail.com>	<db6b5ecc0712111758w668b4be7ica9598813625242@mail.gmail.com>	<c7009a550712120420g268a20dbo49004964ec394afd@mail.gmail.com>
	<db6b5ecc0712120810qe9723ma14f292a070b5951@mail.gmail.com>
Message-ID: <47602A12.8090601@enthought.com>

Fernando Perez wrote:
>> I will put new binaries on the sourceforge site this weekend for both
>> NumPy 1.0.4 and SciPy 0.6.0.  I should be able to find an old PIII
>> WinXP machine around somewhere, which I will devote to building the
>> official non-SSE2 releases from here on out.
>>     
>
> Great, many thanks!
>
> Keep in mind that as Ryan's tests showed, Athlon XP chips also seem to
> lack the SSE2 set, and those are a bit newer than the PIII, so they
> might be a bit easier to find.  The Athlon 64 and Athlon X2 do have
> SSE2, but apparently the XP doesn't.  I'm sure there's a wikipedia
> page somewhere with a list of all the chips where SSE2 first
> appeared...
>   

I suspect the problem is that I was using an Athlon chip without SSE2 to 
build binaries in the past.  That machine is now being used by my kids 
and my environment on it has been "messed-up" 

There may also be a problem with using the Microsoft compiler to build 
NumPy (and then calling out to ATLAS which was built with a different 
compiler).  Issues like this have happened in the past.   And, the most 
recent binaries of NumPy were built with the Microsoft compiler.

-Travis O.





From meesters at uni-mainz.de  Wed Dec 12 13:39:49 2007
From: meesters at uni-mainz.de (Christian Meesters)
Date: Wed, 12 Dec 2007 19:39:49 +0100
Subject: [Numpy-discussion] resize in old Numeric
Message-ID: <1197484789.6890.21.camel@meesters.biologie.uni-mainz.de>

Hi,

For compatibility reasons (work with Biopython) I would like to use to
Numeric in some code of mine. (Of course, I could make a little detour
converting into numpy.array, but first I wonder whether somebody might
know a solution for the problem with Numeric.)

I need to flatten a simple 3x3 array and try to do that with resize.
First time works, next time it doesn't. Here is an example:

In [1]: from Numeric import *

In [2]: a = array([[ 0.50622966, -0.54764389, -0.66619644],
[ 0.44338279, -0.4973092, 0.7457228 ],[-0.73969131, -0.67288704,
-0.00893311]])

In [3]: a.resize((9,))
Out[3]: 
array([ 0.50622966, -0.54764389, -0.66619644,  0.44338279, -0.4973092 ,
0.7457228 ,
            -0.73969131, -0.67288704, -0.00893311])

In [4]: a.resize((9,))
---------------------------------------------------------------------------
<type 'exceptions.ValueError'>            Traceback (most recent call
last)

/home/cm/Documents/projects/SAXS/Eury/MC_rigid/<ipython console> in
<module>()

<type 'exceptions.ValueError'>: cannot resize an array that has been
referenced or is referencing
  another array in this way.  Use the resize function.

Unfortunately, when applying resize on the actual array I have, I get a
similar, yet slightly different error:

Traceback (most recent call last):
  File "sym_get.py", line 37, in <module>
    s.inquire_relation()
  File "/usr/lib/python2.5/site-packages/mc_saxs/mc_rigid.py", line 915,
in inquire_relation
    print rotation, rotation.resize((9,))
ValueError: resize only works on contiguous arrays

The arrays 'a' and 'rotation' are identical.

Any ideas, what's going on or how to solve this?
Numeric version is: 24.2

TIA
Christian



From stefan at sun.ac.za  Wed Dec 12 13:57:43 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 12 Dec 2007 20:57:43 +0200
Subject: [Numpy-discussion] numpy large arrays?
In-Reply-To: <dc0925740712120629r1bdfd6fchd141c765231017b6@mail.gmail.com>
References: <dc0925740712120629r1bdfd6fchd141c765231017b6@mail.gmail.com>
Message-ID: <20071212185743.GF26469@mentat.za.net>

On Wed, Dec 12, 2007 at 03:29:57PM +0100, S?ren Dyrsting wrote:
> I need to perform computations involving large arrays. A lot of rows and no
> more than e.g. 34 columns. My first choice is python/numpy because I'm already
> used to code in matlab.
> 
> However I'm experiencing memory problems even though there is still 500 MB
> available (2 GB total). I have cooked down my code to following meaningless
> code snip. This code share some of the same structure and calls as my real
> program and shows the same behaviour.

I would guess that this is due to memory fragmentation.  Have you
tried the same experiment under Linux?  This article details some of
the problems you may encounter under Windows:

http://www.ittvis.com/services/techtip.asp?ttid=3346

Regards
St?fan


From tim.hochberg at ieee.org  Wed Dec 12 14:21:56 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 12 Dec 2007 12:21:56 -0700
Subject: [Numpy-discussion] resize in old Numeric
In-Reply-To: <1197484789.6890.21.camel@meesters.biologie.uni-mainz.de>
References: <1197484789.6890.21.camel@meesters.biologie.uni-mainz.de>
Message-ID: <e4412d6b0712121121h5531807g8fd938dfe5e7767f@mail.gmail.com>

On Dec 12, 2007 11:39 AM, Christian Meesters <meesters at uni-mainz.de> wrote:

> Hi,
>
> For compatibility reasons (work with Biopython) I would like to use to
> Numeric in some code of mine. (Of course, I could make a little detour
> converting into numpy.array, but first I wonder whether somebody might
> know a solution for the problem with Numeric.)
>
> I need to flatten a simple 3x3 array and try to do that with resize.
> First time works, next time it doesn't. Here is an example:


The first thing that I would try is switching from resize to reshape. In
general, you shouldn't be using resize except for certain odd applications
as it can potentially result in reallocating the data. I'm pretty sure what
you want is reshape.




>
>
> In [1]: from Numeric import *
>
> In [2]: a = array([[ 0.50622966, -0.54764389, -0.66619644],
> [ 0.44338279, -0.4973092, 0.7457228 ],[-0.73969131, -0.67288704,
> -0.00893311]])
>
> In [3]: a.resize((9,))
> Out[3]:
> array([ 0.50622966, -0.54764389, -0.66619644,  0.44338279, -0.4973092 ,
> 0.7457228 ,
>            -0.73969131, -0.67288704, -0.00893311])
>
> In [4]: a.resize((9,))
>
> ---------------------------------------------------------------------------
> <type 'exceptions.ValueError'>            Traceback (most recent call
> last)
>
> /home/cm/Documents/projects/SAXS/Eury/MC_rigid/<ipython console> in
> <module>()
>
> <type 'exceptions.ValueError'>: cannot resize an array that has been
> referenced or is referencing
>  another array in this way.  Use the resize function.
>
> Unfortunately, when applying resize on the actual array I have, I get a
> similar, yet slightly different error:
>
> Traceback (most recent call last):
>  File "sym_get.py", line 37, in <module>
>    s.inquire_relation()
>  File "/usr/lib/python2.5/site-packages/mc_saxs/mc_rigid.py", line 915,
> in inquire_relation
>    print rotation, rotation.resize((9,))
> ValueError: resize only works on contiguous arrays
>
> The arrays 'a' and 'rotation' are identical.
>
> Any ideas, what's going on or how to solve this?
> Numeric version is: 24.2
>
> TIA
> Christian
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071212/c109afac/attachment.html>

From stefan at sun.ac.za  Wed Dec 12 14:35:15 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 12 Dec 2007 21:35:15 +0200
Subject: [Numpy-discussion] resize in old Numeric
In-Reply-To: <1197484789.6890.21.camel@meesters.biologie.uni-mainz.de>
References: <1197484789.6890.21.camel@meesters.biologie.uni-mainz.de>
Message-ID: <20071212193515.GA14894@mentat.za.net>

Hi Christiaan

On Wed, Dec 12, 2007 at 07:39:49PM +0100, Christian Meesters wrote:
> I need to flatten a simple 3x3 array and try to do that with resize.
> First time works, next time it doesn't. Here is an example:
> 
> In [1]: from Numeric import *
> 
> In [2]: a = array([[ 0.50622966, -0.54764389, -0.66619644],
> [ 0.44338279, -0.4973092, 0.7457228 ],[-0.73969131, -0.67288704,
> -0.00893311]])
> 
> In [3]: a.resize((9,))
> Out[3]: 
> array([ 0.50622966, -0.54764389, -0.66619644,  0.44338279, -0.4973092 ,
> 0.7457228 ,
>             -0.73969131, -0.67288704, -0.00893311])
> 
> In [4]: a.resize((9,))
> ---------------------------------------------------------------------------
> <type 'exceptions.ValueError'>            Traceback (most recent call
> last)

You'll notice that this doesn't happen when you run those commands as
a script.  IPython remembers the answer to the last command (as '_'
and in the output history), hence the reference that Numeric complains
about.

> Unfortunately, when applying resize on the actual array I have, I get a
> similar, yet slightly different error:
> 
> Traceback (most recent call last):
>   File "sym_get.py", line 37, in <module>
>     s.inquire_relation()
>   File "/usr/lib/python2.5/site-packages/mc_saxs/mc_rigid.py", line 915,
> in inquire_relation
>     print rotation, rotation.resize((9,))
> ValueError: resize only works on contiguous arrays
> 
> The arrays 'a' and 'rotation' are identical.

They may have identical values without having identical layout in
memory.  Try a.ascontiguousarray().

Cheers
St?fan


From oliphant at enthought.com  Wed Dec 12 14:37:57 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 12 Dec 2007 13:37:57 -0600
Subject: [Numpy-discussion] resize in old Numeric
In-Reply-To: <1197484789.6890.21.camel@meesters.biologie.uni-mainz.de>
References: <1197484789.6890.21.camel@meesters.biologie.uni-mainz.de>
Message-ID: <47603895.9050502@enthought.com>

Christian Meesters wrote:
> Hi,
>
> For compatibility reasons (work with Biopython) I would like to use to
> Numeric in some code of mine. (Of course, I could make a little detour
> converting into numpy.array, but first I wonder whether somebody might
> know a solution for the problem with Numeric.)
>
> I need to flatten a simple 3x3 array and try to do that with resize.
> First time works, next time it doesn't. Here is an example:
>
> In [1]: from Numeric import *
>
> In [2]: a = array([[ 0.50622966, -0.54764389, -0.66619644],
> [ 0.44338279, -0.4973092, 0.7457228 ],[-0.73969131, -0.67288704,
> -0.00893311]])
>
> In [3]: a.resize((9,))
> Out[3]: 
> array([ 0.50622966, -0.54764389, -0.66619644,  0.44338279, -0.4973092 ,
> 0.7457228 ,
>             -0.73969131, -0.67288704, -0.00893311])
>
> In [4]: a.resize((9,))
> ---------------------------------------------------------------------------
> <type 'exceptions.ValueError'>            Traceback (most recent call
> last)
>
> /home/cm/Documents/projects/SAXS/Eury/MC_rigid/<ipython console> in
> <module>()
>
> <type 'exceptions.ValueError'>: cannot resize an array that has been
> referenced or is referencing
>   another array in this way.  Use the resize function.
>
> Unfortunately, when applying resize on the actual array I have, I get a
> similar, yet slightly different error:
>   

The problem is that Numeric has no way of knowing whether a reference 
held is safe or not.  In other-words, is the reference because of a view 
into a part of the array or a simple name binding (like having the '_' 
variable in the interpreter hold on to the old reference).

A reallocation of the memory while other views are expecting to be able 
to walk over the old pointer is a recipe for problems and is the reason 
that Numeric has this restriction.

NumPy has the same restriction but allows you to 'over-ride' it if you 
know what you are doing.

To fix the problem, you need to get rid of all other references to the 
array that you want to resize.

-Travis O.



From tim.hochberg at ieee.org  Wed Dec 12 14:40:24 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 12 Dec 2007 12:40:24 -0700
Subject: [Numpy-discussion] numpy large arrays?
In-Reply-To: <dc0925740712120629r1bdfd6fchd141c765231017b6@mail.gmail.com>
References: <dc0925740712120629r1bdfd6fchd141c765231017b6@mail.gmail.com>
Message-ID: <e4412d6b0712121140y1a5a5fa3s15e2ae906d290082@mail.gmail.com>

On Dec 12, 2007 7:29 AM, S?ren Dyrsting <sorendyrsting at gmail.com> wrote:

> Hi all
>
> I need to perform computations involving large arrays. A lot of rows and
> no more than e.g. 34 columns. My first choice is python/numpy because I'm
> already used to code in matlab.
>
> However I'm experiencing memory problems even though there is still 500 MB
> available (2 GB total). I have cooked down my code to following meaningless
> code snip. This code share some of the same structure and calls as my real
> program and shows the same behaviour.
>
> ********************************************************
> import numpy as N
> import scipy as S
>
> def stress():
>     x = S.randn(200000,80)
>     for i in range(8):
>         print "%(0)d" % {"0": i}
>         s = N.dot(x.T, x)
>         sd = N.array([s.diagonal()])
>         r = N.dot(N.ones((N.size(x,0),1),'d'), sd)
>         x = x + r
>         x = x / 1.01
>
> ********************************************************
>
>
> To different symptoms depending how big x are:
> 1) the program becomes extremely slow after a few iterations.


This appears to be because you are overflowing your floating point
variables. Once your data has INFs in it, it will tend to run much slower.

>
> 2) if the size of x is increased a little the program fails with the
> message "MemoryError" for example at line 'x = x + r', but different places
> in the code depending on the matrice size and which computer I'm testing.
> This might also occur after several iterations, not just during the first
> pass.


Why it would occur after several iterations I'm not sure. It's possible that
there are some cycles that it takes a while for the garbage collector to get
to and in the meantime you are chewing through all of your memory. Their are
a couple different things you could try to address that, but before you do
that, you need to clean up your algorithm and right it in idiomatic numpy. I
realize that you said the above code is meaningless, but I'm going to assume
that it's indicative of how your numpy code is written. That can be
rewritten as:

    def stress2(x):
        for i in range(8):
            print i
            x += (x**2).sum(axis=0)
            x /= 1.01
        return x.sum()


Not only is the above about sixty times faster, it's considerably clearer as
well. FWIW, on my box, which has a very similar setup to yours, neither
version throws a memory error.




>
> I'm using Windows XP, ActivePython 2.5.1.1, NumPy 1.0.4, SciPy  0.6.0.
>
> - Is there an error under the hood in NumPy?


Probably not in this case.


>
> - Am I balancing on the edge of the performance of Python/NumPy and should
> consider other environments. Fortran, C, BLAS, LAPACK e.t.c.


Maybe, but try cleaning things up first.


>
> - Am I misusing NumPy? Changing coding style will be a good workaround and
> even perform on larger datasets without errors?


Your code is doing a lot of extra work and creating a lot of temporaries.
I'd clean it up before I did anything else.



>
>
> Thanks in advance
> /S?ren
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071212/31f51121/attachment.html>

From Glen.Mabey at swri.org  Wed Dec 12 15:46:34 2007
From: Glen.Mabey at swri.org (Glen W. Mabey)
Date: Wed, 12 Dec 2007 14:46:34 -0600
Subject: [Numpy-discussion] linalg.lapack_lite has undefined symbols
Message-ID: <20071212204634.GA11730@bams.ccf.swri.edu>

Hello,

I am converting my build scripts to use the Intel Compiler Suite, using
today's svn code, on RHEL 4.0.

Building numpy succeeds, but importing it fails:

  ImportError: /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: dgesdd_

And sure enough:

  [gmabey at hydra Diamondback]$ ldd /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so
          libpython2.5.so.1.0 => /usr/local/lib/libpython2.5.so.1.0 (0x0000002a9592a000)
          libimf.so => /usr/local/stow/intel_cce-10.1.008/lib/libimf.so (0x0000002a95c4e000)
          libsvml.so => /usr/local/stow/intel_cce-10.1.008/lib/libsvml.so (0x0000002a95fb0000)
          libm.so.6 => /lib64/tls/libm.so.6 (0x0000002a96138000)
          libintlc.so.5 => /usr/local/stow/intel_cce-10.1.008/lib/libintlc.so.5 (0x0000002a962be000)
          libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000002a963f7000)
          libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a96505000)
          libdl.so.2 => /lib64/libdl.so.2 (0x0000002a9673a000)
          libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a9683d000)
          libutil.so.1 => /lib64/libutil.so.1 (0x0000002a96953000)
          /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)

  [gmabey at hydra Diamondback]$ nm /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so | grep " U " | head
          U dgeev_
          U dgelsd_
          U dgeqrf_
          U dgesdd_
          U dorgqr_
          U dsyevd_
          U exit@@GLIBC_2.2.5
          U floor
          U for_stop_core
          U for_write_seq_fmt

I'm using ATLAS 3.8.0 also compiled with ICS, but the undefined symbol
is in lapack_lite.so .

Any suggestions on where to go from here?

Thank you,
Glen Mabey


From oliphant at enthought.com  Wed Dec 12 18:55:00 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 12 Dec 2007 17:55:00 -0600
Subject: [Numpy-discussion] [SciPy-user] maskedarray
In-Reply-To: <200712120924.49482.pgmdevlist@gmail.com>
References: <b02c60870712111248i7cd6a33bu3b8c1458244cf214@mail.gmail.com>
	<200712120924.49482.pgmdevlist@gmail.com>
Message-ID: <476074D4.4080003@enthought.com>

Pierre GM wrote:
> On Tuesday 11 December 2007 15:48:01 Tom Johnson wrote:
>   
>> What is the status regarding 'maskedarray'?  When will this become
>> part (replace ma) of the standard distribution?  Also, what is the
>> recommended way to import sandbox modules so that code changes are
>> minimized when (if) the package becomes part of the standard
>> distribution.
>>     
>
>
> Mmh, no answer from our leaders ? Well, for now maskedarray lives in the scipy 
> sandbox, but it probably could be transferred to the numpy sandbox. There's 
> still no particular decision I'm aware of about putting it the official 
> release. More feedback is probably needed from users.
>   

I apologize for not chiming in here.  It has been difficult to keep up 
with all the lists on which I should ostensibly know what is being 
discussed as I have changed jobs and am settling in to new responsibilites.

It seems to me that it is time to move the scipy.sandbox.maskedarray 
implementation over to numpy.ma.

I'm a little concerned about doing this before 1.1, but if the API has 
not changed, then it should be fine.   If there is no opposition, then 
it seems that this can move forward before 1.0.5 which I would like to 
see in January.

For the future 1.1 release (late 2008), I would like to see (in no 
particular priority order):

1) The MA made a C-subclass
2) The matrix object made a C-subclass (for speed).
3) The sparse matrix structure (but not all the algorithms) brought over 
into NumPy with the indexing matching the matrix indexing.
4) Weave brought over into NumPy

The additions should be created in such a way that they can be disabled 
should so that perhaps a numpy-lite could be produced as well.   Or, 
perhaps better separate packages like numpy-devel, numpy-compat, 
numpy-sparse could be created. 

I'm looking forward to some good discussions over the weekend.

-Travis O.




From aisaac at american.edu  Wed Dec 12 22:27:48 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 12 Dec 2007 22:27:48 -0500
Subject: [Numpy-discussion] [SciPy-user] maskedarray
In-Reply-To: <476074D4.4080003@enthought.com>
References: <b02c60870712111248i7cd6a33bu3b8c1458244cf214@mail.gmail.com><200712120924.49482.pgmdevlist@gmail.com><476074D4.4080003@enthought.com>
Message-ID: <Mahogany-0.67.0-272-20071212-222748.00@american.edu>

On Wed, 12 Dec 2007, "Travis E. Oliphant" apparently wrote:
> 2) The matrix object made a C-subclass (for speed). 

This will probably be the last chance for such a change,
so I again hope that consideration will be given to *one*
change in the matrix object:
iteration over a matrix should return arrays
(instead of matrices).

So if A is a matrix,
A[1] should be an array,
but A[1,:] should be a matrix.

Obviously this is an argument from design rather
than from functionality.  Current behavior is not "natural".

E.g., it makes it awkward to iterate over all elements in
the "natural" way, which I claim is::

    for row in A:
        for element in row:
            print element

This example is just meant to illustrate in a simple way 
what is "odd" about the current behavior.  (It is not meant 
to be an "argument" nor to suggest that the current absence 
simple ways to do the same thing---e.g., using A.A.)
Whenever I am working with matrices, I notice some aspect of 
this "oddity", and it is annoying when so much else is quite 
aesthetic.

Cheers,
Alan Isaac

PS For those who do not know, here is an example of the 
current behavior.  (The following prints 2 matrices instead 
of 4 elements.)

>>> A = N.mat('1 2;3 4')
>>> for row in A:
...     for element in row:
...         print element
...
[[1 2]]
[[3 4]]
>>>





From sorendyrsting at gmail.com  Thu Dec 13 07:59:19 2007
From: sorendyrsting at gmail.com (=?ISO-8859-1?Q?S=F8ren_Dyrsting?=)
Date: Thu, 13 Dec 2007 13:59:19 +0100
Subject: [Numpy-discussion] numpy large arrays?
Message-ID: <dc0925740712130459i62bcd188lf42b28bbeedae566@mail.gmail.com>

Thanks guys.

Stefan:
> I would guess that this is due to memory fragmentation.

Timothy:
> Your code is doing a lot of extra work and creating a lot of temporaries.
> I'd clean it up before I did anything else.

You suggestions turned out to be two sides of the same problem since
the data amounts in the task is too close to the ceiling of memory.

The approach:

"First make it work.
Then optimize"

apparently isn't suitable here.

Thanks
/S?ren


From david.huard at gmail.com  Thu Dec 13 09:32:53 2007
From: david.huard at gmail.com (David Huard)
Date: Thu, 13 Dec 2007 09:32:53 -0500
Subject: [Numpy-discussion] Propose modification to binary_repr
Message-ID: <91cf711d0712130632sebb58cbr94d45b949549a59b@mail.gmail.com>

Hi,

The current behavior of numpy's binary_repr is the following:

>>> binary_repr(1,width=2)
'01'

>>> binary_repr(0,width=2)
'0'

This seems inconsistent and I'd suggest always padding with zeros to make
sure that the return string always has length=width.

Objections ?


David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071213/10ed19b4/attachment.html>

From meesters at uni-mainz.de  Thu Dec 13 10:05:07 2007
From: meesters at uni-mainz.de (Christian Meesters)
Date: Thu, 13 Dec 2007 16:05:07 +0100
Subject: [Numpy-discussion] resize in old Numeric
In-Reply-To: <20071212193515.GA14894@mentat.za.net>
References: <1197484789.6890.21.camel@meesters.biologie.uni-mainz.de>
	<20071212193515.GA14894@mentat.za.net>
Message-ID: <1197558307.9577.1.camel@meesters.biologie.uni-mainz.de>

Thank you all for your valuable input. Learned something 'bout Numeric
again. And my problem is solved ;-).

Thanks
Christian



From oliphant at enthought.com  Thu Dec 13 11:17:55 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 13 Dec 2007 10:17:55 -0600
Subject: [Numpy-discussion] Propose modification to binary_repr
In-Reply-To: <91cf711d0712130632sebb58cbr94d45b949549a59b@mail.gmail.com>
References: <91cf711d0712130632sebb58cbr94d45b949549a59b@mail.gmail.com>
Message-ID: <47615B33.20104@enthought.com>

David Huard wrote:
> Hi,
>
> The current behavior of numpy's binary_repr is the following:
>
> >>> binary_repr(1,width=2)
> '01'
>
> >>> binary_repr(0,width=2)
> '0'
>
> This seems inconsistent and I'd suggest always padding with zeros to 
> make sure that the return string always has length=width.
>
> Objections ?
>
This sounds like a good idea.   I don't think it would break anything, 
but it would be good to test.

-Travis O.

> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From tonyyu at MIT.EDU  Thu Dec 13 14:00:32 2007
From: tonyyu at MIT.EDU (Tony S Yu)
Date: Thu, 13 Dec 2007 14:00:32 -0500
Subject: [Numpy-discussion] Traceback on divide by zero error
Message-ID: <56EA2DD1-A2CB-46A9-AC22-5F06905AB000@mit.edu>

Hello,

This is something that's been bothering for awhile. When numpy raises  
the following divide by zero error:
 >>> Warning: divide by zero encountered in double_scalars
is there a way to get a Traceback on where that warning occurred.

Thanks,
Tony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071213/6b52d87d/attachment.html>

From robert.kern at gmail.com  Thu Dec 13 14:21:14 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 13 Dec 2007 13:21:14 -0600
Subject: [Numpy-discussion] Traceback on divide by zero error
In-Reply-To: <56EA2DD1-A2CB-46A9-AC22-5F06905AB000@mit.edu>
References: <56EA2DD1-A2CB-46A9-AC22-5F06905AB000@mit.edu>
Message-ID: <4761862A.6090803@gmail.com>

Tony S Yu wrote:
> Hello,
> 
> This is something that's been bothering for awhile. When numpy raises
> the following divide by zero error:
>>>> Warning: divide by zero encountered in double_scalars
> is there a way to get a Traceback on where that warning occurred.

In [1]: from numpy import *

In [2]: seterr?
Type:             function
Base Class:       <type 'function'>
Namespace:        Interactive
File:             /Users/rkern/svn/numpy/numpy/core/numeric.py
Definition:       seterr(all=None, divide=None, over=None, under=None, invalid=None)
Docstring:
    Set how floating-point errors are handled.

    Valid values for each type of error are the strings
    "ignore", "warn", "raise", and "call". Returns the old settings.
    If 'all' is specified, values that are not otherwise specified
    will be set to 'all', otherwise they will retain their old
    values.

    Note that operations on integer scalar types (such as int16) are
    handled like floating point, and are affected by these settings.

    Example:

    >>> seterr(over='raise') # doctest: +SKIP
    {'over': 'ignore', 'divide': 'ignore', 'invalid': 'ignore', 'under': 'ignore'}

    >>> seterr(all='warn', over='raise') # doctest: +SKIP
    {'over': 'raise', 'divide': 'ignore', 'invalid': 'ignore', 'under': 'ignore'}

    >>> int16(32000) * int16(3) # doctest: +SKIP
    Traceback (most recent call last):
          File "<stdin>", line 1, in ?
    FloatingPointError: overflow encountered in short_scalars
    >>> seterr(all='ignore') # doctest: +SKIP
    {'over': 'ignore', 'divide': 'ignore', 'invalid': 'ignore', 'under': 'ignore'}

In [3]: seterr(divide='raise')
Out[3]: {'divide': 'print', 'invalid': 'print', 'over': 'print', 'under': 'ignore'}

In [4]: ones(10) / zeros(10)
---------------------------------------------------------------------------
FloatingPointError                        Traceback (most recent call last)

/Users/rkern/svn/mpl-toolkits/<ipython console> in <module>()

FloatingPointError: divide by zero encountered in divide

In [5]: seterr(all='ignore')
Out[5]: {'divide': 'raise', 'invalid': 'print', 'over': 'print', 'under': 'ignore'}

In [6]: ones(10) / zeros(10)
Out[6]: array([ Inf,  Inf,  Inf,  Inf,  Inf,  Inf,  Inf,  Inf,  Inf,  Inf])


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From david.huard at gmail.com  Thu Dec 13 14:33:01 2007
From: david.huard at gmail.com (David Huard)
Date: Thu, 13 Dec 2007 14:33:01 -0500
Subject: [Numpy-discussion] Propose modification to binary_repr
In-Reply-To: <47615B33.20104@enthought.com>
References: <91cf711d0712130632sebb58cbr94d45b949549a59b@mail.gmail.com>
	<47615B33.20104@enthought.com>
Message-ID: <91cf711d0712131133w3e706ac8g8a7737020603feb8@mail.gmail.com>

Change done.

All tests pass.

Should I document the change somewhere ? Although it's a small change, I'm
guessing it could be very annoying to debug for someone depending on the
previous behavior.


2007/12/13, Travis E. Oliphant <oliphant at enthought.com>:
>
> David Huard wrote:
> > Hi,
> >
> > The current behavior of numpy's binary_repr is the following:
> >
> > >>> binary_repr(1,width=2)
> > '01'
> >
> > >>> binary_repr(0,width=2)
> > '0'
> >
> > This seems inconsistent and I'd suggest always padding with zeros to
> > make sure that the return string always has length=width.
> >
> > Objections ?
> >
> This sounds like a good idea.   I don't think it would break anything,
> but it would be good to test.
>
> -Travis O.
>
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071213/c455fb33/attachment.html>

From tonyyu at MIT.EDU  Thu Dec 13 14:41:31 2007
From: tonyyu at MIT.EDU (Tony S Yu)
Date: Thu, 13 Dec 2007 14:41:31 -0500
Subject: [Numpy-discussion] Traceback on divide by zero error
In-Reply-To: <4761862A.6090803@gmail.com>
References: <56EA2DD1-A2CB-46A9-AC22-5F06905AB000@mit.edu>
	<4761862A.6090803@gmail.com>
Message-ID: <05E48F93-0384-401D-A372-11EA3D4E0E98@mit.edu>


On Dec 13, 2007, at 2:21 PM, Robert Kern wrote:

> Tony S Yu wrote:
>> Hello,
>>
>> This is something that's been bothering for awhile. When numpy raises
>> the following divide by zero error:
>>>>> Warning: divide by zero encountered in double_scalars
>> is there a way to get a Traceback on where that warning occurred.
>
> In [1]: from numpy import *
> In [3]: seterr(divide='raise')

seterr(divide='warn') is exactly what I was looking for.

Thanks!
-Tony


From stefan at sun.ac.za  Thu Dec 13 16:07:26 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 13 Dec 2007 23:07:26 +0200
Subject: [Numpy-discussion] Propose modification to binary_repr
In-Reply-To: <91cf711d0712131133w3e706ac8g8a7737020603feb8@mail.gmail.com>
References: <91cf711d0712130632sebb58cbr94d45b949549a59b@mail.gmail.com>
	<47615B33.20104@enthought.com>
	<91cf711d0712131133w3e706ac8g8a7737020603feb8@mail.gmail.com>
Message-ID: <20071213210726.GG31296@mentat.za.net>

On Thu, Dec 13, 2007 at 02:33:01PM -0500, David Huard wrote:
> Change done.
> 
> All tests pass.

Now's a good time to fix that :)

Cheers
St?fan


From jonathan.taylor at utoronto.ca  Thu Dec 13 18:33:47 2007
From: jonathan.taylor at utoronto.ca (Jonathan Taylor)
Date: Thu, 13 Dec 2007 18:33:47 -0500
Subject: [Numpy-discussion] Faster array version of ndindex
Message-ID: <463e11f90712131533u17e5a976gb1e4b31ab61107a7@mail.gmail.com>

I was needing an array representation of ndindex since ndindex only
gives an iterator but array(list(ndindex)) takes too long.  There is
prob some obvious way to do this I am missing but if not feel free to
include this code which is much faster.

In [252]: time a=np.array(list(np.ndindex(10,10,10,10,10,10)))
CPU times: user 11.61 s, sys: 0.09 s, total: 11.70 s
Wall time: 11.82

In [253]: time a=ndtuples(10,10,10,10,10,10)
CPU times: user 0.32 s, sys: 0.21 s, total: 0.53 s
Wall time: 0.60

def ndtuples(*dims):
    """Fast implementation of array(list(ndindex(*dims)))."""

    # Need a list because we will go through it in reverse popping
    # off the size of the last dimension.
    dims = list(dims)

    # N will keep track of the current length of the indices.
    N = dims.pop()

    # At the beginning the current list of indices just ranges over the
    # last dimension.
    cur = np.arange(N)
    cur = cur[:,np.newaxis]

    while dims != []:

        d = dims.pop()

        # This repeats the current set of indices d times.
        # e.g. [0,1,2] -> [0,1,2,0,1,2,...,0,1,2]
        cur = np.kron(np.ones((d,1)),cur)

        # This ranges over the new dimension and 'stretches' it by N.
        # e.g. [0,1,2] -> [0,0,...,0,1,1,...,1,2,2,...,2]
        front = np.arange(d).repeat(N)[:,np.newaxis]

        # This puts these two together.
        cur = np.column_stack((front,cur))
        N *= d

    return cur


From dmitrey.kroshko at scipy.org  Fri Dec 14 03:24:50 2007
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Fri, 14 Dec 2007 10:24:50 +0200
Subject: [Numpy-discussion] matrix multiplication A x, x has some zeros
Message-ID: <47623DD2.4010103@scipy.org>

Hi all,
I have to get Ax, A  is n x n matrix, x is vector of length n.
Some coords of x are zeros, so I want to economy time/cputime somehow 
w/o connecting sparse module from scipy.
What's the easiest way to do so?

Thx, D.


From haase at msg.ucsf.edu  Fri Dec 14 04:31:56 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Fri, 14 Dec 2007 10:31:56 +0100
Subject: [Numpy-discussion] Faster array version of ndindex
In-Reply-To: <463e11f90712131533u17e5a976gb1e4b31ab61107a7@mail.gmail.com>
References: <463e11f90712131533u17e5a976gb1e4b31ab61107a7@mail.gmail.com>
Message-ID: <bc657ead0712140131s411deb6ej549a9efcb72417a7@mail.gmail.com>

Do you know about
N.fromiter()
?

-Sebastian Haase


On Dec 14, 2007 12:33 AM, Jonathan Taylor <jonathan.taylor at utoronto.ca> wrote:
> I was needing an array representation of ndindex since ndindex only
> gives an iterator but array(list(ndindex)) takes too long.  There is
> prob some obvious way to do this I am missing but if not feel free to
> include this code which is much faster.
>
> In [252]: time a=np.array(list(np.ndindex(10,10,10,10,10,10)))
> CPU times: user 11.61 s, sys: 0.09 s, total: 11.70 s
> Wall time: 11.82
>
> In [253]: time a=ndtuples(10,10,10,10,10,10)
> CPU times: user 0.32 s, sys: 0.21 s, total: 0.53 s
> Wall time: 0.60
>
> def ndtuples(*dims):
>    """Fast implementation of array(list(ndindex(*dims)))."""
>
>    # Need a list because we will go through it in reverse popping
>    # off the size of the last dimension.
>    dims = list(dims)
>
>    # N will keep track of the current length of the indices.
>    N = dims.pop()
>
>    # At the beginning the current list of indices just ranges over the
>    # last dimension.
>    cur = np.arange(N)
>    cur = cur[:,np.newaxis]
>
>    while dims != []:
>
>        d = dims.pop()
>
>        # This repeats the current set of indices d times.
>        # e.g. [0,1,2] -> [0,1,2,0,1,2,...,0,1,2]
>        cur = np.kron(np.ones((d,1)),cur)
>
>        # This ranges over the new dimension and 'stretches' it by N.
>        # e.g. [0,1,2] -> [0,0,...,0,1,1,...,1,2,2,...,2]
>        front = np.arange(d).repeat(N)[:,np.newaxis]
>
>        # This puts these two together.
>        cur = np.column_stack((front,cur))
>        N *= d
>
>    return cur
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From charlesr.harris at gmail.com  Fri Dec 14 04:37:31 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 14 Dec 2007 02:37:31 -0700
Subject: [Numpy-discussion] matrix multiplication A x, x has some zeros
In-Reply-To: <47623DD2.4010103@scipy.org>
References: <47623DD2.4010103@scipy.org>
Message-ID: <e06186140712140137g467bc48bn42dc599827ceea84@mail.gmail.com>

How big is n?

On Dec 14, 2007 1:24 AM, dmitrey <dmitrey.kroshko at scipy.org> wrote:

> Hi all,
> I have to get Ax, A  is n x n matrix, x is vector of length n.
> Some coords of x are zeros, so I want to economy time/cputime somehow
> w/o connecting sparse module from scipy.
> What's the easiest way to do so?
>
> Thx, D.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071214/6674e69e/attachment.html>

From dmitrey.kroshko at scipy.org  Fri Dec 14 04:44:04 2007
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Fri, 14 Dec 2007 11:44:04 +0200
Subject: [Numpy-discussion] matrix multiplication A x, x has some zeros
In-Reply-To: <e06186140712140137g467bc48bn42dc599827ceea84@mail.gmail.com>
References: <47623DD2.4010103@scipy.org>
	<e06186140712140137g467bc48bn42dc599827ceea84@mail.gmail.com>
Message-ID: <47625064.1080909@scipy.org>

I guess it doesn't matter, but typical n are 1...1000.
However, I need to call the operation hundreds or thousands times (while 
running NLP solver ralg, so 4..5 * nIter times).
Number of zeros can be 0...n-1

Charles R Harris wrote:
> How big is n?
>
> On Dec 14, 2007 1:24 AM, dmitrey <dmitrey.kroshko at scipy.org 
> <mailto:dmitrey.kroshko at scipy.org>> wrote:
>
>     Hi all,
>     I have to get Ax, A  is n x n matrix, x is vector of length n.
>     Some coords of x are zeros, so I want to economy time/cputime somehow
>     w/o connecting sparse module from scipy.
>     What's the easiest way to do so?
>
>     Thx, D.
>     _______________________________________________
>     Numpy-discussion mailing list
>     Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>     http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From aisaac at american.edu  Fri Dec 14 08:28:47 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 14 Dec 2007 08:28:47 -0500
Subject: [Numpy-discussion] matrix multiplication A x, x has some zeros
In-Reply-To: <47625064.1080909@scipy.org>
References: <47623DD2.4010103@scipy.org><e06186140712140137g467bc48bn42dc599827ceea84@mail.gmail.com><47625064.1080909@scipy.org>
Message-ID: <Mahogany-0.67.0-300-20071214-082847.00@american.edu>

On Fri, 14 Dec 2007, dmitrey apparently wrote:
> I guess it doesn't matter, but typical n are 1...1000. 
> However, I need to call the operation hundreds or thousands times (while 
> running NLP solver ralg, so 4..5 * nIter times). 
> Number of zeros can be 0...n-1 

Do both A and x change every iteration?

Anyway, if x is sparse I think you'll get some
benefit by doing
idx=N.ravel(x)!=0
A[:,idx]*x[idx]

hth,
Alan Isaac






From stefan at sun.ac.za  Fri Dec 14 08:32:53 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 14 Dec 2007 15:32:53 +0200
Subject: [Numpy-discussion] Faster array version of ndindex
In-Reply-To: <bc657ead0712140131s411deb6ej549a9efcb72417a7@mail.gmail.com>
References: <463e11f90712131533u17e5a976gb1e4b31ab61107a7@mail.gmail.com>
	<bc657ead0712140131s411deb6ej549a9efcb72417a7@mail.gmail.com>
Message-ID: <20071214133253.GB7167@mentat.za.net>

Hi Sebastian

N.fromiter only works on 1D arrays.  I thought the following may work,
but it doesn't:

 np.fromiter(np.ndindex(10,10,10),N.dtype((int,3)))

This kind of loop is probably best implemented in C, although I think
Jonathan's version is rather clever.

Regards
St?fan

On Fri, Dec 14, 2007 at 10:31:56AM +0100, Sebastian Haase wrote:
> Do you know about
> N.fromiter()
> ?
> 
> -Sebastian Haase
> 
> 
> On Dec 14, 2007 12:33 AM, Jonathan Taylor <jonathan.taylor at utoronto.ca> wrote:
> > I was needing an array representation of ndindex since ndindex only
> > gives an iterator but array(list(ndindex)) takes too long.  There is
> > prob some obvious way to do this I am missing but if not feel free to
> > include this code which is much faster.
> >
> > In [252]: time a=np.array(list(np.ndindex(10,10,10,10,10,10)))
> > CPU times: user 11.61 s, sys: 0.09 s, total: 11.70 s
> > Wall time: 11.82
> >
> > In [253]: time a=ndtuples(10,10,10,10,10,10)
> > CPU times: user 0.32 s, sys: 0.21 s, total: 0.53 s
> > Wall time: 0.60
> >
> > def ndtuples(*dims):
> >    """Fast implementation of array(list(ndindex(*dims)))."""
> >
> >    # Need a list because we will go through it in reverse popping
> >    # off the size of the last dimension.
> >    dims = list(dims)
> >
> >    # N will keep track of the current length of the indices.
> >    N = dims.pop()
> >
> >    # At the beginning the current list of indices just ranges over the
> >    # last dimension.
> >    cur = np.arange(N)
> >    cur = cur[:,np.newaxis]
> >
> >    while dims != []:
> >
> >        d = dims.pop()
> >
> >        # This repeats the current set of indices d times.
> >        # e.g. [0,1,2] -> [0,1,2,0,1,2,...,0,1,2]
> >        cur = np.kron(np.ones((d,1)),cur)
> >
> >        # This ranges over the new dimension and 'stretches' it by N.
> >        # e.g. [0,1,2] -> [0,0,...,0,1,1,...,1,2,2,...,2]
> >        front = np.arange(d).repeat(N)[:,np.newaxis]
> >
> >        # This puts these two together.
> >        cur = np.column_stack((front,cur))
> >        N *= d
> >
> >    return cur


From bsouthey at gmail.com  Fri Dec 14 09:33:08 2007
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 14 Dec 2007 08:33:08 -0600
Subject: [Numpy-discussion] matrix multiplication A x, x has some zeros
In-Reply-To: <47625064.1080909@scipy.org>
References: <47623DD2.4010103@scipy.org>
	<e06186140712140137g467bc48bn42dc599827ceea84@mail.gmail.com>
	<47625064.1080909@scipy.org>
Message-ID: <bbcd77d00712140633j64a4f882v6c423a4419c61f05@mail.gmail.com>

Hi,
How sparse are A, x and Ax?

You probably will not find any time or operation efficiency unless A
and x are relatively sparse. With the small size of n, any efficiency
will typically be small in any case.

As you indicate that this has to be repeated multiple times, perhaps
you should look at your overall algorithm for efficiencies.

Bruce

On Dec 14, 2007 3:44 AM, dmitrey <dmitrey.kroshko at scipy.org> wrote:
> I guess it doesn't matter, but typical n are 1...1000.
> However, I need to call the operation hundreds or thousands times (while
> running NLP solver ralg, so 4..5 * nIter times).
> Number of zeros can be 0...n-1
>
> Charles R Harris wrote:
> > How big is n?
> >
> > On Dec 14, 2007 1:24 AM, dmitrey <dmitrey.kroshko at scipy.org
> > <mailto:dmitrey.kroshko at scipy.org>> wrote:
> >
> >     Hi all,
> >     I have to get Ax, A  is n x n matrix, x is vector of length n.
> >     Some coords of x are zeros, so I want to economy time/cputime somehow
> >     w/o connecting sparse module from scipy.
> >     What's the easiest way to do so?
> >
> >     Thx, D.
> >     _______________________________________________
> >     Numpy-discussion mailing list
> >     Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
> >     http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> > ------------------------------------------------------------------------
>
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From cimrman3 at ntc.zcu.cz  Fri Dec 14 10:00:42 2007
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Fri, 14 Dec 2007 16:00:42 +0100
Subject: [Numpy-discussion] ANN: SFE-00.35.01
Message-ID: <47629A9A.8030708@ntc.zcu.cz>

Let me announce SFE-00.35.01, bringing per term integration - now each 
term can use its own quadrature points. This is a major change at the 
heart of the code - some parts may not work as all terms were not 
migrated yet to the new framework. All test examples work, though, as 
well as acoustic band gaps. See http://ui505p06-mbs.ntc.zcu.cz/sfe .

SFE is a finite element analysis software written almost entirely in 
Python. The code is released under BSD license.

best regards,
r.


From Chris.Barker at noaa.gov  Fri Dec 14 14:33:37 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 14 Dec 2007 11:33:37 -0800
Subject: [Numpy-discussion] RAdian <--> degres conversion
Message-ID: <4762DA91.1060808@noaa.gov>

HI all,

Someone on the wxPython list just pointed out that the math module now 
includes includes angle-conversion utilities:

 >>> degrees.__doc__
degrees(x) -> converts angle x from radians to degrees
 >>> radians.__doc__
radians(x) -> converts angle x from degrees to radians

Not a big deal, but handy. As I generally like to think if numpy as a 
superset of the math module, perhaps is should include these too.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From dmitrey.kroshko at scipy.org  Fri Dec 14 15:29:59 2007
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Fri, 14 Dec 2007 22:29:59 +0200
Subject: [Numpy-discussion] matrix multiplication A x, x has some zeros
In-Reply-To: <Mahogany-0.67.0-300-20071214-082847.00@american.edu>
References: <47623DD2.4010103@scipy.org><e06186140712140137g467bc48bn42dc599827ceea84@mail.gmail.com><47625064.1080909@scipy.org>
	<Mahogany-0.67.0-300-20071214-082847.00@american.edu>
Message-ID: <4762E7C7.1080105@scipy.org>

Thank you for the tip, however, I expect <A,x> being vector of length n
A and x change every iter. A is not sparse.

Alan G Isaac wrote:
> On Fri, 14 Dec 2007, dmitrey apparently wrote:
>   
>> I guess it doesn't matter, but typical n are 1...1000. 
>> However, I need to call the operation hundreds or thousands times (while 
>> running NLP solver ralg, so 4..5 * nIter times). 
>> Number of zeros can be 0...n-1 
>>     
>
> Do both A and x change every iteration?
>
> Anyway, if x is sparse I think you'll get some
> benefit by doing
> idx=N.ravel(x)!=0
> A[:,idx]*x[idx]
>
> hth,
> Alan Isaac
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>   



From cookedm at physics.mcmaster.ca  Fri Dec 14 17:20:57 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Fri, 14 Dec 2007 17:20:57 -0500
Subject: [Numpy-discussion] RAdian <--> degres conversion
In-Reply-To: <4762DA91.1060808@noaa.gov>
References: <4762DA91.1060808@noaa.gov>
Message-ID: <E73679FA-0BC6-43CB-B55E-70A1580DFCE0@physics.mcmaster.ca>

On Dec 14, 2007, at 14:33 , Christopher Barker wrote:

> HI all,
>
> Someone on the wxPython list just pointed out that the math module now
> includes includes angle-conversion utilities:
>
>>>> degrees.__doc__
> degrees(x) -> converts angle x from radians to degrees
>>>> radians.__doc__
> radians(x) -> converts angle x from degrees to radians
>
> Not a big deal, but handy. As I generally like to think if numpy as a
> superset of the math module, perhaps is should include these too.


Done.

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From aisaac at american.edu  Fri Dec 14 18:53:46 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 14 Dec 2007 18:53:46 -0500
Subject: [Numpy-discussion] matrix multiplication A x, x has some zeros
In-Reply-To: <4762E7C7.1080105@scipy.org>
References: <47623DD2.4010103@scipy.org><e06186140712140137g467bc48bn42dc599827ceea84@mail.gmail.com><47625064.1080909@scipy.org><Mahogany-0.67.0-300-20071214-082847.00@american.edu><4762E7C7.1080105@scipy.org>
Message-ID: <Mahogany-0.67.0-312-20071214-185346.00@american.edu>

>> idx=N.ravel(x)!=0
>> A[:,idx]*x[idx] 


On Fri, 14 Dec 2007, dmitrey apparently wrote:
> I expect <A,x> being vector of length n A and x change 
> every iter. A is not sparse. 

Still, whenever x has a lot of zeros,
this should have a substantial payoff.
That seems the only exploitable information
you have offered in the problem.

Cheers,
Alan





From neilcrighton at gmail.com  Sun Dec 16 04:50:48 2007
From: neilcrighton at gmail.com (Neil Crighton)
Date: Sun, 16 Dec 2007 09:50:48 +0000
Subject: [Numpy-discussion] RAdian <--> degres conversion
Message-ID: <63751c30712160150s17e8b7a3g6e9f4da3215454c3@mail.gmail.com>

Do we really need these functions in numpy?  I mean it's just
multiplying/dividing the value by pi/180 (who knows why they're in the
math module..). Numpy doesn't have asin, acos, or atan either (they're
arcsin, arcos and arctan) so it isn't a superset of the math module.

I would like there to be fewer functions in numpy, not more.

> > Someone on the wxPython list just pointed out that the math module now
> > includes includes angle-conversion utilities:
> >
> >>>> degrees.__doc__
> > degrees(x) -> converts angle x from radians to degrees
> >>>> radians.__doc__
> > radians(x) -> converts angle x from degrees to radians
> >
> > Not a big deal, but handy. As I generally like to think if numpy as a
> > superset of the math module, perhaps is should include these too.
>
>
> Done.
>


From meine at informatik.uni-hamburg.de  Sun Dec 16 07:30:56 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Sun, 16 Dec 2007 13:30:56 +0100
Subject: [Numpy-discussion] RAdian <--> degres conversion
In-Reply-To: <63751c30712160150s17e8b7a3g6e9f4da3215454c3@mail.gmail.com>
References: <63751c30712160150s17e8b7a3g6e9f4da3215454c3@mail.gmail.com>
Message-ID: <200712161331.02778.meine@informatik.uni-hamburg.de>

On Sonntag 16 Dezember 2007, Neil Crighton wrote:
> Do we really need these functions in numpy?  I mean it's just
> multiplying/dividing the value by pi/180 (who knows why they're in the
> math module..).

I like them in math ("explicit is better than implicit"*), but I don't want to 
comment on whether they should be in numpy.

(*: It's similar with math.hypot, which I have got to know and appreciate 
nowadays.)

Ciao, /  /                                                    .o.
     /--/                                                     ..o
    /  / ANS                                                  ooo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071216/5c09f8b4/attachment.sig>

From peridot.faceted at gmail.com  Sun Dec 16 08:18:13 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 16 Dec 2007 08:18:13 -0500
Subject: [Numpy-discussion] RAdian <--> degres conversion
In-Reply-To: <200712161331.02778.meine@informatik.uni-hamburg.de>
References: <63751c30712160150s17e8b7a3g6e9f4da3215454c3@mail.gmail.com>
	<200712161331.02778.meine@informatik.uni-hamburg.de>
Message-ID: <ce557a360712160518l343883b8vd2ce00a36db0302e@mail.gmail.com>

On 16/12/2007, Hans Meine <meine at informatik.uni-hamburg.de> wrote:

> (*: It's similar with math.hypot, which I have got to know and appreciate
> nowadays.)

I'd like to point out that math.hypot is a nontrivial function which
is easy to get wrong:

In [6]: x=1e200; y=1e200;

In [7]: math.hypot(x,y)
Out[7]: 1.414213562373095e+200

In [8]: math.sqrt(x**2+y**2)
---------------------------------------------------------------------------
<type 'exceptions.OverflowError'>         Traceback (most recent call last)

/home/peridot/<ipython console> in <module>()

<type 'exceptions.OverflowError'>: (34, 'Numerical result out of range')


Anne


From brussel13 at yahoo.com  Sun Dec 16 14:10:41 2007
From: brussel13 at yahoo.com (Ross Harder)
Date: Sun, 16 Dec 2007 11:10:41 -0800 (PST)
Subject: [Numpy-discussion] where construct
In-Reply-To: <47615B33.20104@enthought.com>
Message-ID: <599862.50765.qm@web55902.mail.re3.yahoo.com>


What's the correct way to do something like this?

a=array( (0,1,1,0) )
b=array( (4,3,2,1) )
c=array( (1,2,3,4) )

where( (a<1 or b<3), b,c)

Python throws a ValueError
I would expect to get an array that looks like
[4,2,2,1] I think


Thanks,
Ross


      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs


From lbolla at gmail.com  Sun Dec 16 14:22:13 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Sun, 16 Dec 2007 20:22:13 +0100
Subject: [Numpy-discussion] where construct
In-Reply-To: <599862.50765.qm@web55902.mail.re3.yahoo.com>
References: <47615B33.20104@enthought.com>
	<599862.50765.qm@web55902.mail.re3.yahoo.com>
Message-ID: <80c99e790712161122j55a4e18cl866bd66fbab9b5b1@mail.gmail.com>

use '+' instead of 'or' for bool arrays.

In [8]: numpy.where((a<1) + (b<3), b, c)
Out[8]: array([4, 2, 2, 1])

hth,
L.

On Dec 16, 2007 8:10 PM, Ross Harder <brussel13 at yahoo.com> wrote:

>
> What's the correct way to do something like this?
>
> a=array( (0,1,1,0) )
> b=array( (4,3,2,1) )
> c=array( (1,2,3,4) )
>
> where( (a<1 or b<3), b,c)
>
> Python throws a ValueError
> I would expect to get an array that looks like
> [4,2,2,1] I think
>
>
> Thanks,
> Ross
>
>
>
>  ____________________________________________________________________________________
> Never miss a thing.  Make Yahoo your home page.
> http://www.yahoo.com/r/hs
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071216/e5b89e48/attachment.html>

From strawman at astraw.com  Sun Dec 16 14:38:56 2007
From: strawman at astraw.com (Andrew Straw)
Date: Sun, 16 Dec 2007 11:38:56 -0800
Subject: [Numpy-discussion] where construct
In-Reply-To: <599862.50765.qm@web55902.mail.re3.yahoo.com>
References: <599862.50765.qm@web55902.mail.re3.yahoo.com>
Message-ID: <47657ED0.5040804@astraw.com>

"or" is logical or. You want "|" which is bitwise/elementwise or. Also, 
watch the order of operations -- | has higher precedence than <.

Thus, you want

where( (a<1) | (b<3), b,c)

Ross Harder wrote:
> What's the correct way to do something like this?
>
> a=array( (0,1,1,0) )
> b=array( (4,3,2,1) )
> c=array( (1,2,3,4) )
>
> where( (a<1 or b<3), b,c)
>
> Python throws a ValueError
> I would expect to get an array that looks like
> [4,2,2,1] I think
>
>
> Thanks,
> Ross
>
>
>       ____________________________________________________________________________________
> Never miss a thing.  Make Yahoo your home page. 
> http://www.yahoo.com/r/hs
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From e.howick at irl.cri.nz  Sun Dec 16 21:58:29 2007
From: e.howick at irl.cri.nz (elfnor)
Date: Mon, 17 Dec 2007 02:58:29 +0000 (UTC)
Subject: [Numpy-discussion] assign a variable to each column of an array
Message-ID: <loom.20071217T025538-729@post.gmane.org>

Is there a more concise way of assigning a variable to each column of an array?

This works

x,y,z = X[:,0],X[:,1],X[:,2]

but seems clumsy.



From tim.hochberg at ieee.org  Sun Dec 16 22:20:28 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Sun, 16 Dec 2007 20:20:28 -0700
Subject: [Numpy-discussion] assign a variable to each column of an array
In-Reply-To: <loom.20071217T025538-729@post.gmane.org>
References: <loom.20071217T025538-729@post.gmane.org>
Message-ID: <e4412d6b0712161920n161a33aahc02e65fe53be4e8e@mail.gmail.com>

On Dec 16, 2007 7:58 PM, elfnor <e.howick at irl.cri.nz> wrote:

> Is there a more concise way of assigning a variable to each column of an
> array?
>
> This works
>
> x,y,z = X[:,0],X[:,1],X[:,2]
>
> but seems clumsy.


You could try:
  x, y, z = X.T
Although some people might think that's a little obscure...




-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071216/67eaef11/attachment.html>

From millman at berkeley.edu  Mon Dec 17 01:06:43 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 16 Dec 2007 22:06:43 -0800
Subject: [Numpy-discussion] Changes to the NumPy's developer trac site
Message-ID: <c7009a550712162206w3dba4978t3548d34e4afa4456@mail.gmail.com>

Hello,

It looks like the spammers have figured out how to use our trac site
again.  I locked down the trac site a bit to make it more difficult
for them.  Basically, I removed some of the default permissions that
all users get.  I don't think this will effect many people; but if you
have a trac acount and aren't able to do something because you don't
have permission just send me an email.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From ondrej at certik.cz  Mon Dec 17 10:51:19 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 17 Dec 2007 16:51:19 +0100
Subject: [Numpy-discussion] segfault with atlas3-sse2, works with atlas3-sse
Message-ID: <85b5c3130712170751n52f26317h4917ce152a26b032@mail.gmail.com>

Hi,

when compiled on Debian, numpy segfaults when used with ATLAS sse2,
but works when used against ATLAS sse. More information here:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=448530

I have no idea, where the problem is. Do you have any ideas, what we should try?

Thanks a lot,
Ondrej


From lists at cheimes.de  Mon Dec 17 19:03:40 2007
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 18 Dec 2007 01:03:40 +0100
Subject: [Numpy-discussion] Float and math module enhancements for Python
	2.6 and 3.0
Message-ID: <fk72ot$a1h$1@ger.gmane.org>

Hello!

The Python core developers are currently working on several improvements
for floats and the math module. Since you are "power users" I like to
get your opinion and suggestions on several patches:

http://bugs.python.org/issue1534 -- sys.float_info [done]
http://bugs.python.org/issue1580 -- shorter repr of floats
http://bugs.python.org/issue1635 -- platform independent representation
and creation of +/-inf and nan
http://bugs.python.org/issue1640 -- additional functions for the math module

new: sys.maxsize, gone in 3.0: sys.maxint

Christian



From david at ar.media.kyoto-u.ac.jp  Tue Dec 18 02:23:27 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 18 Dec 2007 16:23:27 +0900
Subject: [Numpy-discussion] [OT] Which version of svn does svn.scipy.org run
	?
Message-ID: <4767756F.20507@ar.media.kyoto-u.ac.jp>

Hi,

    Is it possible to know which version of subversion is used by the 
svn.scipy.org server ? I am trying to use svnsync on it, without any 
success, and wanted to know if this was coming from svn.scipy.org or 
from something else ?

    cheers,

    David


From cournape at gmail.com  Tue Dec 18 05:21:32 2007
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 18 Dec 2007 19:21:32 +0900
Subject: [Numpy-discussion] Towards using scons in numpy: 1st step
Message-ID: <5b8d13220712180221r2da5e9d9t645eeec21914d51e@mail.gmail.com>

Hi,

as discussed with some other numpy developers, in particular Travis, I
started to prepare my work related to scons for step-by-step merging
into the trunk. The first step is done, and is in cleanconfig_rtm
branch (rtm for ready to merge).

This branch basically:
 - avoid introducing new SIZEOF_* like symbols, which are usually used
by autotools. Python itself defines some of those, but should not.
 - avoid #ifdef/#endif in config.h files: normally, configuration
headers should only contains define. This also makes their generation
more straightforward, and more compatible with standard build tools
(autotools/scons). This is the main goal of the branch
 - split configuration defines into two files: one private (config.h),
which is not visible by any other packages compiled against numpy
headers, and one public (numpyconfig.h), which replaces the config.h.

If there is no problem, I would like to merge this one as quickly as
possible: since it changes numpy.core public header, it may
potentially break things, and I would prefer detecting those breakages
now.

cheers,

David


From millman at berkeley.edu  Tue Dec 18 13:59:43 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 18 Dec 2007 10:59:43 -0800
Subject: [Numpy-discussion] Faster array version of ndindex
In-Reply-To: <20071214133253.GB7167@mentat.za.net>
References: <463e11f90712131533u17e5a976gb1e4b31ab61107a7@mail.gmail.com>
	<bc657ead0712140131s411deb6ej549a9efcb72417a7@mail.gmail.com>
	<20071214133253.GB7167@mentat.za.net>
Message-ID: <c7009a550712181059y7f9c9189l8a3290a213623f42@mail.gmail.com>

Hey Jonathan,

Thanks for providing this.  I created a ticket for now:
http://scipy.org/scipy/numpy/ticket/636

I will take some time to add this functionality to NumPy before the
end of the year.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Tue Dec 18 14:05:37 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 18 Dec 2007 11:05:37 -0800
Subject: [Numpy-discussion] Changing the distributed binary for numpy
	1.0.4 for windows ?
In-Reply-To: <c7009a550712120420g268a20dbo49004964ec394afd@mail.gmail.com>
References: <475D2775.2080504@ar.media.kyoto-u.ac.jp>
	<475E1FE5.4020808@noaa.gov> <008d01c83beb$a716ecf0$0a83a8c0@sun.ac.za>
	<5b8d13220712110415t4bfbcefbw11d140b194c54ad9@mail.gmail.com>
	<475ECFC9.8060202@noaa.gov>
	<5b8d13220712111622g4e997d5neb3992ae4d0b1ce9@mail.gmail.com>
	<475F2FCA.5070605@gmail.com>
	<c5b438120712111745i3b4fccecsf4f22b05bf70c43b@mail.gmail.com>
	<db6b5ecc0712111758w668b4be7ica9598813625242@mail.gmail.com>
	<c7009a550712120420g268a20dbo49004964ec394afd@mail.gmail.com>
Message-ID: <c7009a550712181105x179d22b8y4438d165442ea698@mail.gmail.com>

On Dec 12, 2007 4:20 AM, Jarrod Millman <millman at berkeley.edu> wrote:
> I will put new binaries on the sourceforge site this weekend for both
> NumPy 1.0.4 and SciPy 0.6.0.  I should be able to find an old PIII
> WinXP machine around somewhere, which I will devote to building the
> official non-SSE2 releases from here on out.

Hey,

I just wanted to update everyone that the new binaries are still
coming.  They are taking longer to build than expected.  I will make
an announcement as soon as the new binaries are available.

Sorry,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From sdb at cloud9.net  Tue Dec 18 14:07:34 2007
From: sdb at cloud9.net (Stuart Brorson)
Date: Tue, 18 Dec 2007 14:07:34 -0500 (EST)
Subject: [Numpy-discussion] Compiler change for Windows version between
	1.0.3.1 and 1.0.4?
Message-ID: <Pine.BSF.4.64.0712181358590.57479@earl-grey.cloud9.net>

Hi --

I have found a bug in numpy-1.0.4.  The bug is a crash when doing
linalg.inv() on a Windows machine.   The machine is an old Pentium 3
machine.  The error report indicates an invalid op code.  Details
about this bug are in the numpy tracker under ticket 635.

Numpy-1.0.3.1 doesn't crash when running the same code.

I installed the pre-built binaries available from numpy's SF site.
Since the binaries were built by you, I can only suspect that the
problem involves using a different compiler to build 1.0.4.  I suspect
the new version was built with a compiler which emitted op codes the
old P3 chip couldn't handle.

Can the developers confirm or deny my hypothesis?

Thanks,

Stuart Brorson
Interactive Supercomputing, inc.
135 Beaver Street | Waltham | MA | 02452 | USA
http://www.interactivesupercomputing.com/


From oliphant at enthought.com  Wed Dec 19 13:52:35 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 19 Dec 2007 12:52:35 -0600
Subject: [Numpy-discussion] SciPy Sprint results
Message-ID: <47696873.1060008@enthought.com>


Hi all,

We had a great Sprint at Berkeley over last weekend.  Jarrod deserves a 
huge hand for organizing it and Fernando should be also congradulated 
for making the Sprint a productive communication session with a lot of 
different people. 

Going forward, there will be a relatively informal SciPy board whose 
purpose is to keep SciPy (and NumPy) moving forward.   Currently, this 
board consists of  (alphabetically)

Eric Jones
Robert Kern
Jarrod Millman
Travis Oliphant

Our goal is to clean up SciPy and get it ready for 1.0 release over the 
next year or so (which will need lots of help from the community).   If 
anybody else is interested in serving on this board, just send me email. 

As part of this goal, we will be having regular "sprints" as well 
virtual "bug-days" and "doc-days" where people who want to participate 
using IRC can join in and coordinate efforts.   There will be at least 
one bug-day or doc-day every month over the next year (on the last 
Friday of the month).   The first one is a "doc-day" which will be held 
Friday on December 28, 2007 (getting started on New Year's resolutions 
early).   This doc-day will be virtual where anyone with an internet 
connection can join in on the scipy channel on  irc.freenode.net.

At least one board member will be available at each "doc-day" or 
"bug-day" (even if we have to recruit board members to make it happen :-) )

The recent Sprint was very helpful.  Jarrod is putting together some 
material from the Sprint.  But, I wanted to provide over-view 
information for those who may be interested in what happend.

Summary:

A lot of great discussion took place (and some fine actual coding by a 
few) which resulted in the following plans:

Schedule
------------
  * NumPy 1.0.5 in mid January
  * SciPy 0.7.0 in mid March to April
  * NumPy 1.1 by August 2008 (may slip a bit depending on what is wanted 
to be included)

The plans below are for NumPy 1.0.5 and SciPy 0.7.0 unless otherwise noted.

IO
----
  * scipy.io will be gutted and what functionality remains will be 
placed in numpy.io.
  * scipy.io will be a place for file readers and writers for various 
data formats (data, audio, video, images, matlab, excel, etc.)
  * NumPy will get a standard binary file format (.npy/.npz) for 
arrays/groups_of_arrays.
  * NumPy will be trying to incorporate some of matplotlib's csv2rec and 
rec2csv functionality.
  * Pickling arrays will be discouraged (you will still be able to do 
it, we will just try to not make it seem that it is the "default" way to 
save arrays).

Testing
---------
  * scipy unit-testing will be "nose-compliant" and therefore nose will 
be required to run the SciPy tests. 
  * NumPy will still use the current testing framework but will support 
SciPy's desire to be nose-compliant.  NumPy 1.1 tests may move to just 
being "nose-compliant"
  * The goal is to make tests easier for contributors to write.

Weave
---------
  * weave will not move into NumPy yet, but possibly at NumPy 1.1, there 
could be a separate package containing all the "wrapping" support code 
for NumPy in a more unified fashion (if somebody is interested in this, 
it is a great time to jump in).

Sandbox
-----------
  * the scipy sandbox is disappearing (except for user playgrounds) and 
useful code in it will be placed in other areas.

Python versions
--------------------
  * SciPy 0.7.0 will require Python 2.4 (we can now use decorators for 
SciPy).
  * NumPy will still be useful with Python 2.3 until at least 1.1

Other discussions
----------------------
  * numpy-scons will be a separate package for now for building 
extensions with scons (we need experience to figure out what to do with 
it).
  * fixes to repr for numpy float scalars were put in place
  * Thanks to Rob Falck scipy.optimize grew slsqp  (sequential 
least-squares programming) method (allows for equality and inequality 
constraints).   The code by Dieter Kraft was wrapped.
  * We will be working to coordinate efforts with William Stein (of SAGE 
fame) in the future.  Sage developers will be coming to Austin at the 
end of February to do some cooperative sprinting.
  * Brian Granger is working on a parallel version of NumPy that is very 
interesting.


Deprecation approaches
-------------------------------

Functions in SciPy that are disappearing will be "deprecated" with 
appendages to the docstring to explain how to do it differently.  The 
deprecation will issue a warning when the function is run.  In the next 
version, the function will disappear.   Once SciPy hits 1.0, the 
deprecation paradigm will be a bit more conservative.


A lot of fabulous things are happening with SciPy.  It is an exciting 
time to be a part of it.   There are a lot of ways to jump in and 
participate so feel free.   If there is something you think needs 
addressing, then please share it.   We may have a simple PEP process in 
the future, but for now the rule is basically "working code."


Best regards,

-Travis O.




From oliphant at enthought.com  Wed Dec 19 20:21:46 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 19 Dec 2007 19:21:46 -0600
Subject: [Numpy-discussion] Compiler change for Windows version between
 1.0.3.1 and 1.0.4?
In-Reply-To: <Pine.BSF.4.64.0712181358590.57479@earl-grey.cloud9.net>
References: <Pine.BSF.4.64.0712181358590.57479@earl-grey.cloud9.net>
Message-ID: <4769C3AA.4060501@enthought.com>

Stuart Brorson wrote:
> Hi --
>
> I have found a bug in numpy-1.0.4.  The bug is a crash when doing
> linalg.inv() on a Windows machine.   The machine is an old Pentium 3
> machine.  The error report indicates an invalid op code.  Details
> about this bug are in the numpy tracker under ticket 635.
>
> Numpy-1.0.3.1 doesn't crash when running the same code.
>
> I installed the pre-built binaries available from numpy's SF site.
> Since the binaries were built by you, I can only suspect that the
> problem involves using a different compiler to build 1.0.4.  I suspect
> the new version was built with a compiler which emitted op codes the
> old P3 chip couldn't handle.
>
> Can the developers confirm or deny my hypothesis?
>   

You are correct, and the problem is in the process of being rectified by 
building on a different machine.   There are several Trac tickets 
already posted due to this problem.

-Travis O.




From oliphant at enthought.com  Wed Dec 19 20:26:35 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 19 Dec 2007 19:26:35 -0600
Subject: [Numpy-discussion] Mailing list was not sending out new mails for
	awhile.
Message-ID: <4769C4CB.7080609@enthought.com>


Hi all,

The postfix service on the server hosting several of the mailing lists 
was down since Monday.  Mails to the list were preserved and archived 
but were not being distributed to subscribers.   We restarted the 
postfix service and messages should now be going out.

Apologies for the problem.

-Travis O.




From oliphant at enthought.com  Wed Dec 19 20:29:17 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 19 Dec 2007 19:29:17 -0600
Subject: [Numpy-discussion] Towards using scons in numpy: 1st step
In-Reply-To: <5b8d13220712180221r2da5e9d9t645eeec21914d51e@mail.gmail.com>
References: <5b8d13220712180221r2da5e9d9t645eeec21914d51e@mail.gmail.com>
Message-ID: <4769C56D.3000809@enthought.com>

David Cournapeau wrote:
> Hi,
>
> as discussed with some other numpy developers, in particular Travis, I
> started to prepare my work related to scons for step-by-step merging
> into the trunk. The first step is done, and is in cleanconfig_rtm
> branch (rtm for ready to merge).
>
> This branch basically:
>  - avoid introducing new SIZEOF_* like symbols, which are usually used
> by autotools. Python itself defines some of those, but should not.
>  - avoid #ifdef/#endif in config.h files: normally, configuration
> headers should only contains define. This also makes their generation
> more straightforward, and more compatible with standard build tools
> (autotools/scons). This is the main goal of the branch
>   
How did you get around using #ifdef etc?  This concerns me to just 
change it, unless I'm convinced there is no problem.

>  - split configuration defines into two files: one private (config.h),
> which is not visible by any other packages compiled against numpy
> headers, and one public (numpyconfig.h), which replaces the config.h.
>   
I think this should be fine.  
> If there is no problem, I would like to merge this one as quickly as
> possible: since it changes numpy.core public header, it may
> potentially break things, and I would prefer detecting those breakages
> now.
>   
I'd like more reassurances about the #ifdef change,  but other than 
that, it sounds fine.

-Travis

> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   



From ryanlists at gmail.com  Wed Dec 19 20:35:35 2007
From: ryanlists at gmail.com (Ryan Krauss)
Date: Wed, 19 Dec 2007 19:35:35 -0600
Subject: [Numpy-discussion] Compiler change for Windows version between
	1.0.3.1 and 1.0.4?
In-Reply-To: <Pine.BSF.4.64.0712181358590.57479@earl-grey.cloud9.net>
References: <Pine.BSF.4.64.0712181358590.57479@earl-grey.cloud9.net>
Message-ID: <c5b438120712191735h438556c4ica62d7418307f416@mail.gmail.com>

Sounds like exactly the problem Jarrod Millman is working to solve.
The new binary uses SSE2 options which P3's and older Athlons don't
support.  It sounds like Jarrod has found a machine he can use to
create the new binaries and solve the problem.

On Dec 18, 2007 1:07 PM, Stuart Brorson <sdb at cloud9.net> wrote:
> Hi --
>
> I have found a bug in numpy-1.0.4.  The bug is a crash when doing
> linalg.inv() on a Windows machine.   The machine is an old Pentium 3
> machine.  The error report indicates an invalid op code.  Details
> about this bug are in the numpy tracker under ticket 635.
>
> Numpy-1.0.3.1 doesn't crash when running the same code.
>
> I installed the pre-built binaries available from numpy's SF site.
> Since the binaries were built by you, I can only suspect that the
> problem involves using a different compiler to build 1.0.4.  I suspect
> the new version was built with a compiler which emitted op codes the
> old P3 chip couldn't handle.
>
> Can the developers confirm or deny my hypothesis?
>
> Thanks,
>
> Stuart Brorson
> Interactive Supercomputing, inc.
> 135 Beaver Street | Waltham | MA | 02452 | USA
> http://www.interactivesupercomputing.com/
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From numpy at mspacek.mm.st  Wed Dec 19 21:22:57 2007
From: numpy at mspacek.mm.st (Martin Spacek)
Date: Wed, 19 Dec 2007 18:22:57 -0800
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>	<4750981C.9060409@mspacek.mm.st>
	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
Message-ID: <4769D201.1090907@mspacek.mm.st>

Sebastian Haase wrote:
> b) To my knowledge, any OS Linux, Windows an OSX can max. allocate
> about 1GB of data - assuming you have a 32 bit machine.
> The actual numbers I measured varied  from about 700MB to maybe 1.3GB.
> In other words, you would be right at the limit.
> (For 64bit, you would have to make sure ALL parts are 64bit, e.g. The
> python version must be >=2.5, python must have been compiled using a
> 64-bit compiler *and* the windows version (XP-64))
> This holds true the same for physical ram allocation and for memmap
> allocation.
> My solution to this was to "wait" for the 64bit .... not tested yet ;-)

By the way, I installed 64-bit linux (ubuntu 7.10) on the same machine, 
and now numpy.memmap works like a charm. Slicing around a 15 GB file is fun!

--
Martin


From matthew.brett at gmail.com  Thu Dec 20 00:12:00 2007
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 19 Dec 2007 21:12:00 -0800
Subject: [Numpy-discussion] segfault with atlas3-sse2,
	works with atlas3-sse
In-Reply-To: <85b5c3130712170751n52f26317h4917ce152a26b032@mail.gmail.com>
References: <85b5c3130712170751n52f26317h4917ce152a26b032@mail.gmail.com>
Message-ID: <1e2af89e0712192112r4b56625u254429cba8bdebca@mail.gmail.com>

Hi,

> when compiled on Debian, numpy segfaults when used with ATLAS sse2,
> but works when used against ATLAS sse. More information here:

What is the machine on which you are getting the segfault?  Is it
possible you are trying to run SSE2 instructions on a machine without
SSE2?

Best,

Matthew


From meine at informatik.uni-hamburg.de  Thu Dec 20 07:42:53 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Thu, 20 Dec 2007 13:42:53 +0100
Subject: [Numpy-discussion] where construct
In-Reply-To: <599862.50765.qm@web55902.mail.re3.yahoo.com>
References: <599862.50765.qm@web55902.mail.re3.yahoo.com>
Message-ID: <200712201342.53451.meine@informatik.uni-hamburg.de>

Am Sonntag, 16. Dezember 2007 20:10:41 schrieb Ross Harder:
> What's the correct way to do something like this?
>
> a=array( (0,1,1,0) )
> b=array( (4,3,2,1) )
> c=array( (1,2,3,4) )
>
> where( (a<1 or b<3), b,c)

Now + and | have been proposed to you, but it looks to me as if the "correct 
way" would be logical_or.  All solutions give the same result, but logical_or 
better expresses what you're trying to do:

In [9]: print where( (logical_or(a<1, b<3)), b,c)
[4 2 2 1]

(Think of the Zen.)

Ciao, /  /
     /--/
    /  / ANS


From oliphant at enthought.com  Thu Dec 20 11:10:44 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 20 Dec 2007 10:10:44 -0600
Subject: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
In-Reply-To: <20071220070728.GD7721@mentat.za.net>
References: <47696873.1060008@enthought.com>
	<20071220070728.GD7721@mentat.za.net>
Message-ID: <476A9404.2030009@enthought.com>

Stefan van der Walt wrote:
> Hi Travis,
>
> During the sprint I also merged Pierre's MaskedArray code into the
> maskedarray branch.  That is nearly done, with only a few unit tests
> still failing -- ones brought over from the old numpy.ma.
>
> This is mainly due to some changes in the API, for example put and
> putmask now behave like their ndarray counterparts.  Pierre, would you
> remind us of any other such changes, and why they were made?
>
> What is the road forward with this code?  We will probably only merge
> API changes into 1.4.  Do we use svnmerge to keep the branch up to
> date until then?
>   

I'd like to move forward with it sooner (for 1.0.5) if the API changes 
are not drastic.  Although ideally 0 API changes would be desireable, 
I'm not sure if that is feasible.   Are put and putmask the only changes 
in the API.  What are the rest of them?

-Travis



From ondrej at certik.cz  Thu Dec 20 11:32:06 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 20 Dec 2007 17:32:06 +0100
Subject: [Numpy-discussion] segfault with atlas3-sse2,
	works with atlas3-sse
In-Reply-To: <1e2af89e0712192112r4b56625u254429cba8bdebca@mail.gmail.com>
References: <85b5c3130712170751n52f26317h4917ce152a26b032@mail.gmail.com>
	<1e2af89e0712192112r4b56625u254429cba8bdebca@mail.gmail.com>
Message-ID: <85b5c3130712200832k4d2b1c35pb372289833e04521@mail.gmail.com>

> > when compiled on Debian, numpy segfaults when used with ATLAS sse2,
> > but works when used against ATLAS sse. More information here:
>
> What is the machine on which you are getting the segfault?  Is it

I don't know which machine the reporter of this bug in Debian uses, but I use
Intel Core Duo.

> possible you are trying to run SSE2 instructions on a machine without
> SSE2?

the /proc/cpuinfo says:

flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc
pni monitor est tm2 xtpr

so imho I do have SSE2.

Ondrej


From kwgoodman at gmail.com  Thu Dec 20 12:10:52 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 20 Dec 2007 09:10:52 -0800
Subject: [Numpy-discussion] Indexing into a matrix
Message-ID: <f4f93d420712200910n17376308t5634d6267bbf8df4@mail.gmail.com>

>> x = M.rand(3,1)  <-------- 3x1
>> x[M.isfinite(x)]
matrix([[ 0.36541551,  0.6305087 ,  0.66054899]])   <---------- Should
this be 3x1?

My workaround is awkward:

>> x[M.where(M.isfinite(x).A)[0]]
matrix([[ 0.36541551],
        [ 0.6305087 ],
        [ 0.66054899]])


From Bruce_Sherwood at ncsu.edu  Thu Dec 20 12:19:20 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Thu, 20 Dec 2007 12:19:20 -0500
Subject: [Numpy-discussion] Numpy or Boost problem
Message-ID: <476AA418.4080108@ncsu.edu>

I'm not sure whether this is a Numpy problem or a Boost problem, so I'm 
posting to both communities.

In old Numeric, type(sqrt(5.5)) was float, but in numpy, type(sqrt(5.5)) 
is numpy.float64. This leads to a big performance hit in calculations in 
a beta version of VPython, using the VPython 3D "vector" class, compared 
with the old version that used Numeric (VPython is a 3D graphics module 
for Python; see vpython.org).

Operator overloading of the VPython vector class works fine for 
vector*sqrt(5.5) but not for sqrt(5.5)*vector. The following free 
function catches 5.5*vector but fails to catch sqrt(5.5)*vector, whose 
type ends up as numpy.ndarray instead of the desired vector, with 
concomitant slow conversions in later vector calculations:

inline vector
operator*( const double& s, const vector& v)
{ return vector( s*v.x, s*v.y, s*v.z); } 

I've thrashed around on this, including trying to add this:

inline vector
operator*( const npy_float64& s, const vector& v)
{ return vector( s*v.x, s*v.y, s*v.z); }

But the compiler correctly complains that this is in conflict with the 
version of double*vector, since in fact npy_float64 is actually double.

It's interesting and presumably meaningful to the knowledgeable (not me) 
that vector*sqrt(5.5) yields a vector, even though the overloading 
speaks of double, not a specifically numpy name:

inline vector
operator*( const double s) const throw()
{ return vector( s*x, s*y, s*z); }

VPython uses Boost, and the glue concerning vectors includes the following:

py::class_<vector>("vector", py::init< py::optional<double, double, 
double> >())
       .def( self * double())
       .def( double() * self)

As far as I can understand from the Boost Python documentation, this is 
the proper way to specify the left-hand and right-hand overloadings. But 
do I have to add something like .def( npy_float64() * self)? Help would 
be much appreciated.

Bruce Sherwood
 



From pgmdevlist at gmail.com  Thu Dec 20 12:54:48 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 20 Dec 2007 12:54:48 -0500
Subject: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
In-Reply-To: <476A9404.2030009@enthought.com>
References: <47696873.1060008@enthought.com>
	<20071220070728.GD7721@mentat.za.net>
	<476A9404.2030009@enthought.com>
Message-ID: <200712201254.48284.pgmdevlist@gmail.com>

All,

> I'd like to move forward with it sooner (for 1.0.5) if the API changes
> are not drastic.  Although ideally 0 API changes would be desireable,
> I'm not sure if that is feasible.   Are put and putmask the only changes
> in the API.  What are the rest of them?

* put, putmask, take should behave like the numpy equivalent.
* fill_value is now a property, not a method.
* cumsum(cumprod) works as if the _data array was filled with 0 (1). The mask 
is preserved, but not updated. (the output of numpy.core.ma has nomask).
* bool(x) raises a ValueError, as it does for ndarrays.

I'm afraid there still might be a couple I forget about, but that should be 
it. I tried to limit the changes as much as possible, but there were some 
instances where the original numpy.core.ma functions were confusing (at least 
I found them confusing), such as the fill_value (makes more sense to have it 
as an attribute), or putmask.


From Chris.Barker at noaa.gov  Thu Dec 20 13:21:29 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 20 Dec 2007 10:21:29 -0800
Subject: [Numpy-discussion] where construct
In-Reply-To: <200712201342.53451.meine@informatik.uni-hamburg.de>
References: <599862.50765.qm@web55902.mail.re3.yahoo.com>
	<200712201342.53451.meine@informatik.uni-hamburg.de>
Message-ID: <476AB2A9.2010409@noaa.gov>

Hans Meine wrote:
>> where( (a<1 or b<3), b,c)
> 
> Now + and | have been proposed to you, but it looks to me as if the "correct 
> way" would be logical_or.  All solutions give the same result, but logical_or 
> better expresses what you're trying to do:
> 
> In [9]: print where( (logical_or(a<1, b<3)), b,c)
> [4 2 2 1]
> (Think of the Zen.)

I'm not sure the Zen answers this one for us. It's really a matter of 
taste, and many of us prefer infix notation for this kind of thing. I think:

where( ( (a<1) | (b<3) ), b,c)

reads better than:

where( (logical_or(a<1, b<3)), b,c)

The only reason we need to use "|" (which means logical_or for boolean 
arrays) rather than "or" is that we can't overload the python "or" or 
"and" because they are designed to short-circuit. If you want to be 
really explicit, you could do:

where( (logical_or(less(a,1), less(b,3))), b,c)

or, since I like to use namespaces (also see the Zen):

import numpy as N
N.where( (N.logical_or(N.less(a,1), N.less(b,3))), b,c)

which are even harder to read.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Thu Dec 20 13:22:22 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 20 Dec 2007 10:22:22 -0800
Subject: [Numpy-discussion] SciPy Sprint results
In-Reply-To: <47696873.1060008@enthought.com>
References: <47696873.1060008@enthought.com>
Message-ID: <476AB2DE.3040001@noaa.gov>

Travis E. Oliphant wrote:

> Currently, this 
> board consists of  (alphabetically)
> 
> Eric Jones
> Robert Kern
> Jarrod Millman
> Travis Oliphant

Excellent team -- thanks guys!

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From pav at iki.fi  Thu Dec 20 14:27:16 2007
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 20 Dec 2007 21:27:16 +0200
Subject: [Numpy-discussion] segfault with atlas3-sse2,
	works with	atlas3-sse
In-Reply-To: <85b5c3130712200832k4d2b1c35pb372289833e04521@mail.gmail.com>
References: <85b5c3130712170751n52f26317h4917ce152a26b032@mail.gmail.com>
	<1e2af89e0712192112r4b56625u254429cba8bdebca@mail.gmail.com>
	<85b5c3130712200832k4d2b1c35pb372289833e04521@mail.gmail.com>
Message-ID: <1198178836.8408.1.camel@localhost.localdomain>

to, 2007-12-20 kello 17:32 +0100, Ondrej Certik kirjoitti:
> > > when compiled on Debian, numpy segfaults when used with ATLAS sse2,
> > > but works when used against ATLAS sse. More information here:
> >
> > What is the machine on which you are getting the segfault?  Is it
> 
> I don't know which machine the reporter of this bug in Debian uses, but I use
> Intel Core Duo.
> 
> > possible you are trying to run SSE2 instructions on a machine without
> > SSE2?
> 
> the /proc/cpuinfo says:
> 
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc
> pni monitor est tm2 xtpr
> 
> so imho I do have SSE2.

If it's not wrong instructions for the processor type, this reminds me
of http://scipy.org/scipy/numpy/ticket/551 which also seemed to be
SSE2-specific on Debian/Ubuntu in my tests.

-- 
Pauli Virtanen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digitaalisesti allekirjoitettu viestin osa
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071220/7186ff5e/attachment.sig>

From seb.haase at gmx.net  Thu Dec 20 15:21:00 2007
From: seb.haase at gmx.net (Sebastian Haase)
Date: Thu, 20 Dec 2007 21:21:00 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <4769D201.1090907@mspacek.mm.st>
References: <474FCE2D.2000908@mspacek.mm.st>
	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>
	<4750981C.9060409@mspacek.mm.st>
	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
	<4769D201.1090907@mspacek.mm.st>
Message-ID: <bc657ead0712201221y5c449523w7f5ede372f4c171d@mail.gmail.com>

On Dec 20, 2007 3:22 AM, Martin Spacek <numpy at mspacek.mm.st> wrote:
> Sebastian Haase wrote:
> > b) To my knowledge, any OS Linux, Windows an OSX can max. allocate
> > about 1GB of data - assuming you have a 32 bit machine.
> > The actual numbers I measured varied  from about 700MB to maybe 1.3GB.
> > In other words, you would be right at the limit.
> > (For 64bit, you would have to make sure ALL parts are 64bit, e.g. The
> > python version must be >=2.5, python must have been compiled using a
> > 64-bit compiler *and* the windows version (XP-64))
> > This holds true the same for physical ram allocation and for memmap
> > allocation.
> > My solution to this was to "wait" for the 64bit .... not tested yet ;-)
>
> By the way, I installed 64-bit linux (ubuntu 7.10) on the same machine,
> and now numpy.memmap works like a charm. Slicing around a 15 GB file is fun!
>
Thanks for the feedback !
Did you get the kind of speed you need and/or the speed you were hoping for ?

-Sebastian


From haase at msg.ucsf.edu  Thu Dec 20 15:21:55 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Thu, 20 Dec 2007 21:21:55 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <4769D201.1090907@mspacek.mm.st>
References: <474FCE2D.2000908@mspacek.mm.st>
	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>
	<4750981C.9060409@mspacek.mm.st>
	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
	<4769D201.1090907@mspacek.mm.st>
Message-ID: <bc657ead0712201221n6155a3f1r1b48d804384f29e6@mail.gmail.com>

On Dec 20, 2007 3:22 AM, Martin Spacek <numpy at mspacek.mm.st> wrote:
> Sebastian Haase wrote:
> > b) To my knowledge, any OS Linux, Windows an OSX can max. allocate
> > about 1GB of data - assuming you have a 32 bit machine.
> > The actual numbers I measured varied  from about 700MB to maybe 1.3GB.
> > In other words, you would be right at the limit.
> > (For 64bit, you would have to make sure ALL parts are 64bit, e.g. The
> > python version must be >=2.5, python must have been compiled using a
> > 64-bit compiler *and* the windows version (XP-64))
> > This holds true the same for physical ram allocation and for memmap
> > allocation.
> > My solution to this was to "wait" for the 64bit .... not tested yet ;-)
>
> By the way, I installed 64-bit linux (ubuntu 7.10) on the same machine,
> and now numpy.memmap works like a charm. Slicing around a 15 GB file is fun!
>

Thanks for the feedback !
Did you get the kind of speed you need and/or the speed you were hoping for ?

-Sebastian


From meine at informatik.uni-hamburg.de  Thu Dec 20 16:12:42 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Thu, 20 Dec 2007 22:12:42 +0100
Subject: [Numpy-discussion] where construct
In-Reply-To: <476AB2A9.2010409@noaa.gov>
References: <599862.50765.qm@web55902.mail.re3.yahoo.com>
	<200712201342.53451.meine@informatik.uni-hamburg.de>
	<476AB2A9.2010409@noaa.gov>
Message-ID: <200712202212.42256.meine@informatik.uni-hamburg.de>

On Donnerstag 20 Dezember 2007, Christopher Barker wrote:
> > In [9]: print where( (logical_or(a<1, b<3)), b,c)
> > [4 2 2 1]
> > (Think of the Zen.)
>
> I'm not sure the Zen answers this one for us.

As you have guessed correctly, I was thinking of "explicit is better than 
implicit".

> It's really a matter of taste, and many of us prefer infix notation for this
> kind of thing.

You're right, "where( ( (a<1) | (b<3) ), b,c)" *does* look better.
I should have written "for completeness, I want to mention that logical or is 
called logical_or in numpy", and for boolean array, "|" seems to be "the 
correct way".

Ciao, /  /                                                    .o.
     /--/                                                     ..o
    /  / ANS                                                  ooo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071220/36bc9ae1/attachment.sig>

From millman at berkeley.edu  Thu Dec 20 17:15:54 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 20 Dec 2007 14:15:54 -0800
Subject: [Numpy-discussion] new P3 binaries for NumPy 1.0.4 and SciPy 0.6.0
Message-ID: <c7009a550712201415r29ee07edg9703d1b77e47c2c6@mail.gmail.com>

Hey,

If you are having problems with NumPy and SciPy on Pentium III
machines running Windows, please try the newly released binaries:

numpy-1.0.4.win32-p3-py2.3.exe
numpy-1.0.4.win32-p3-py2.4.exe
numpy-1.0.4.win32-p3-py2.5.exe
numpy-1.0.4.win32-p3-py2.5.msi

scipy-0.6.0.win32-p3-py2.3.exe
scipy-0.6.0.win32-p3-py2.4.exe
scipy-0.6.0.win32-p3-py2.5.exe
scipy-0.6.0.win32-p3-py2.5.msi

Enjoy,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From numpy at mspacek.mm.st  Thu Dec 20 18:11:07 2007
From: numpy at mspacek.mm.st (Martin Spacek)
Date: Thu, 20 Dec 2007 15:11:07 -0800
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <bc657ead0712201221y5c449523w7f5ede372f4c171d@mail.gmail.com>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>	<4750981C.9060409@mspacek.mm.st>	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>	<4769D201.1090907@mspacek.mm.st>
	<bc657ead0712201221y5c449523w7f5ede372f4c171d@mail.gmail.com>
Message-ID: <476AF68B.5000105@mspacek.mm.st>

>> By the way, I installed 64-bit linux (ubuntu 7.10) on the same machine,
>> and now numpy.memmap works like a charm. Slicing around a 15 GB file is fun!
>>
> Thanks for the feedback !
> Did you get the kind of speed you need and/or the speed you were hoping for ?

Nope. Like I wrote earlier, it seems there isn't time for disk access in 
my main loop, which is what memmap is all about. I resolved this by 
loading the whole file into memory as a python list of 2D arrays, 
instead of one huge contiguous 3D array. That got me an extra 100 to 200 
MB of physical memory to work with (about 1.4GB out of 2GB total) on 
win32, which is all I needed.

Martin


From oliphant at enthought.com  Thu Dec 20 18:24:44 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 20 Dec 2007 17:24:44 -0600
Subject: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
In-Reply-To: <200712201254.48284.pgmdevlist@gmail.com>
References: <47696873.1060008@enthought.com>	<20071220070728.GD7721@mentat.za.net>	<476A9404.2030009@enthought.com>
	<200712201254.48284.pgmdevlist@gmail.com>
Message-ID: <476AF9BC.8070705@enthought.com>

Pierre GM wrote:
> All,
>
>   
>> I'd like to move forward with it sooner (for 1.0.5) if the API changes
>> are not drastic.  Although ideally 0 API changes would be desireable,
>> I'm not sure if that is feasible.   Are put and putmask the only changes
>> in the API.  What are the rest of them?
>>     
>
>   
These do not seem too bad.
> * put, putmask, take should behave like the numpy equivalent.
>   
This does not sound like a problem.  Am I missing something?
> * fill_value is now a property, not a method.
>   
I can see this is a good choice, but it is a backward compatibility 
issue that we should document.
> * cumsum(cumprod) works as if the _data array was filled with 0 (1). The mask 
> is preserved, but not updated. (the output of numpy.core.ma has nomask).
>   
I don't understand what you mean here.    So, the mask effectively 
removes those elements from the sum(product) computation?  What does it 
mean that the mask is not updated?
> * bool(x) raises a ValueError, as it does for ndarrays.
>   
What does bool(x) raise for numpy.core.ma.

If we can document exactly what the compatibility issues are (and it 
looks like we are almost there), we should move forward.

-Travis
 



From pgmdevlist at gmail.com  Thu Dec 20 18:52:38 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 20 Dec 2007 18:52:38 -0500
Subject: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
In-Reply-To: <476AF9BC.8070705@enthought.com>
References: <47696873.1060008@enthought.com>
	<200712201254.48284.pgmdevlist@gmail.com>
	<476AF9BC.8070705@enthought.com>
Message-ID: <200712201852.38920.pgmdevlist@gmail.com>

> > * cumsum(cumprod) works as if the _data array was filled with 0 (1). The
> > mask is preserved, but not updated. (the output of numpy.core.ma has
> > nomask).
>
> I don't understand what you mean here.    So, the mask effectively
> removes those elements from the sum(product) computation?  What does it
> mean that the mask is not updated?

Quick example:
>>> x= masked_array([1,2,3],mask=[0,1,0])
>>> x
masked_array(data = [1 -- 3],
      mask = [False  True False],
      fill_value=999999)
>>> x.cumsum()
masked_array(data = [1 -- 4],
      mask = [False  True False],
      fill_value=999999)

So, cumsum works as if the masked values were 0. The mask stays the same as it 
was initially (that's what I meant by "mask not updated").  An alternative 
would be to set the mask to True for all the values past the first masked: 
with our example, we would have:
masked_array(data = [1 --  --],
      mask = [False  True True],
      fill_value=999999)
I prefer the first version (the one that is currently implemented).




> > * bool(x) raises a ValueError, as it does for ndarrays.
>
> What does bool(x) raise for numpy.core.ma.

True

> If we can document exactly what the compatibility issues are (and it
> looks like we are almost there), we should move forward.

OK, I'll take care of that this week-end. Stefan, feel free to beat me to 
it...


From cournape at gmail.com  Thu Dec 20 20:04:06 2007
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 21 Dec 2007 10:04:06 +0900
Subject: [Numpy-discussion] Towards using scons in numpy: 1st step
In-Reply-To: <4769C56D.3000809@enthought.com>
References: <5b8d13220712180221r2da5e9d9t645eeec21914d51e@mail.gmail.com>
	<4769C56D.3000809@enthought.com>
Message-ID: <5b8d13220712201704v1e4d7acboee36c05c7d46f70a@mail.gmail.com>

On Dec 20, 2007 10:29 AM, Travis E. Oliphant <oliphant at enthought.com> wrote:
> David Cournapeau wrote:
> > Hi,
> >
> > as discussed with some other numpy developers, in particular Travis, I
> > started to prepare my work related to scons for step-by-step merging
> > into the trunk. The first step is done, and is in cleanconfig_rtm
> > branch (rtm for ready to merge).
> >
> > This branch basically:
> >  - avoid introducing new SIZEOF_* like symbols, which are usually used
> > by autotools. Python itself defines some of those, but should not.
> >  - avoid #ifdef/#endif in config.h files: normally, configuration
> > headers should only contains define. This also makes their generation
> > more straightforward, and more compatible with standard build tools
> > (autotools/scons). This is the main goal of the branch
> >
> How did you get around using #ifdef etc?  This concerns me to just
> change it, unless I'm convinced there is no problem.
I don't get around them, I just do not use them inside generated
files. For example, NPY_ALLOW_THREADS is defined in ndarrayobject.h,
now, depending on some values defined in the config headers. Having
#ifdef inside generated files is a bit against the spirit of config
headers, and it makes also more complicated to generate them through
standard tools (neither scons nor autoheader supports this, for
example).

But this is definitely a change which should be reviewed by someone,
because it can break things if done wrong. I tried to be as careful as
possible, but since there are several generators stages involved for
config.h, it is always difficult to be 100 % sure that I have not done
any mistake.

David


From devnew at gmail.com  Fri Dec 21 02:37:44 2007
From: devnew at gmail.com (devnew at gmail.com)
Date: Thu, 20 Dec 2007 23:37:44 -0800 (PST)
Subject: [Numpy-discussion] setting items in a matrix
Message-ID: <d7ad8c21-b98c-4d04-872a-c7c6ae6b8436@i12g2000prf.googlegroups.com>

hi
i am a beginner with numpy and python,so pardon me if this doubt seems
silly
i want to create a matrix with say 3 rows and 5 columns..and then set
the values of each item in it .for this i did something like below

myarray=zeros((3,5))
#then set the items
for row in range(3):
    for col in range(5):
        myarray[row][col]=999999.9999


mymatrix=matrix(myarray)

is this the way to do the matrix creation and value setting?  is the
use of  zeros() unnecessary?  i  am in the early learning stage so
your reply wd help me much

dn


From devnew at gmail.com  Fri Dec 21 02:40:01 2007
From: devnew at gmail.com (devnew at gmail.com)
Date: Thu, 20 Dec 2007 23:40:01 -0800 (PST)
Subject: [Numpy-discussion] setting items in a matrix
Message-ID: <9380affd-b743-4d65-9d94-b234cf4304a3@b1g2000pra.googlegroups.com>

hi
i am a beginner with numpy and python,so pardon me if this doubt seems
silly
i want to create a matrix with say 3 rows and 5 columns..and then set
the values of each item in it .for this i did something like below

myarray=zeros((3,5))
#then set the items
for row in range(3):
    for col in range(5):
        myarray[row][col]=999999.9999


mymatrix=matrix(myarray)

is this the way to do the matrix creation and value setting?  is the
use of  zeros() unnecessary?  i  am in the early learning stage so
your reply wd help me much

dn


From matthieu.brucher at gmail.com  Fri Dec 21 03:02:31 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 21 Dec 2007 09:02:31 +0100
Subject: [Numpy-discussion] setting items in a matrix
In-Reply-To: <9380affd-b743-4d65-9d94-b234cf4304a3@b1g2000pra.googlegroups.com>
References: <9380affd-b743-4d65-9d94-b234cf4304a3@b1g2000pra.googlegroups.com>
Message-ID: <e76aa17f0712210002t15ad4038xcf05cd79fc3fdd05@mail.gmail.com>

Hi,

You can use ones as well if the array (not matrix) has the same values, or
the array function to create an array from a sequence, or matrix for matrix
and a sequence of sequences

a = n.ones((3,5)) * 99999
b = n.array((1, 2, 3), (4, 5, 6), (6, 7, 8))
c = n.matrix((1, 2, 3), (4, 5, 6), (6, 7, 8))

Matthieu

2007/12/21, devnew at gmail.com <devnew at gmail.com>:
>
> hi
> i am a beginner with numpy and python,so pardon me if this doubt seems
> silly
> i want to create a matrix with say 3 rows and 5 columns..and then set
> the values of each item in it .for this i did something like below
>
> myarray=zeros((3,5))
> #then set the items
> for row in range(3):
>     for col in range(5):
>         myarray[row][col]=999999.9999
>
>
> mymatrix=matrix(myarray)
>
> is this the way to do the matrix creation and value setting?  is the
> use of  zeros() unnecessary?  i  am in the early learning stage so
> your reply wd help me much
>
> dn
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071221/3b9ca4d7/attachment.html>

From ondrej at certik.cz  Fri Dec 21 03:11:42 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 21 Dec 2007 09:11:42 +0100
Subject: [Numpy-discussion] Fwd: SSE2 segfault
Message-ID: <85b5c3130712210011n6e628fabq9c5270db5cc0be32@mail.gmail.com>

Hi,

forwarding more information about the problem + backtrace.

Ondrej


---------- Forwarded message ----------
From: Jan Medlock <medlock at turboshower.net>
Date: Dec 21, 2007 4:17 AM
Subject: Re: Bug#448530: bug reproduced
To: Ondrej Certik <ondrej at certik.cz>
Cc: 448530 at bugs.debian.org, Kumar Appaiah <akumar at ee.iitm.ac.in>


On Thu, Dec 20, 2007 at 08:01:01PM +0100, Ondrej Certik wrote:
> Paul Metcalfe has sent me this explanation:
>
> I think it's a packaging problem with linear algebra.
>
> ...

Hi Ondrej,
I'm still here.  I saw the bug in the NumPy bug tracker that
suggests the problem is specific to Debian & Ubuntu and Paul
Metcalfe's details about linking.

Linking against ATLAS instead of BLAS & LAPACK may indeed ultimately
be the problem and I'm afraid I don't understand the details.  That
said, I believe the problem is SSE2 specific.  The machine that I'm
having the problem with is an Intel Pentium4 with SSE2, but I also
have an AMD K7 machine that has 3DNow but not SSE2.  The AMD machine
does not exhibit the bug with the 3DNow version of ATLAS.  (Of course
neither do with the SSE or base versions of ATLAS.)

I haven't participated in the thread on the NumPy mailing list because
I don't see a way to reply to the thread since I wasn't subscribed at
the time...

Attached is a gdb backtrace.  Please let me know how I can make it
more informative, if necessary.

Thanks again for tracking down this bug,
Jan
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: backtrace
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071221/0086e411/attachment.ksh>

From lbolla at gmail.com  Fri Dec 21 03:29:18 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Fri, 21 Dec 2007 09:29:18 +0100
Subject: [Numpy-discussion] setting items in a matrix
In-Reply-To: <d7ad8c21-b98c-4d04-872a-c7c6ae6b8436@i12g2000prf.googlegroups.com>
References: <d7ad8c21-b98c-4d04-872a-c7c6ae6b8436@i12g2000prf.googlegroups.com>
Message-ID: <80c99e790712210029h2083ad09ja719373da640f24@mail.gmail.com>

or, you can either use fill.

In [53]: M = numpy.matrix(numpy.zeros((3,5)))
In [55]: M.fill(999)
In [56]: M
Out[56]:
matrix([[ 999.,  999.,  999.,  999.,  999.],
        [ 999.,  999.,  999.,  999.,  999.],
        [ 999.,  999.,  999.,  999.,  999.]])

L.

On 12/21/07, devnew at gmail.com <devnew at gmail.com> wrote:
>
> hi
> i am a beginner with numpy and python,so pardon me if this doubt seems
> silly
> i want to create a matrix with say 3 rows and 5 columns..and then set
> the values of each item in it .for this i did something like below
>
> myarray=zeros((3,5))
> #then set the items
> for row in range(3):
>    for col in range(5):
>        myarray[row][col]=999999.9999
>
>
> mymatrix=matrix(myarray)
>
> is this the way to do the matrix creation and value setting?  is the
> use of  zeros() unnecessary?  i  am in the early learning stage so
> your reply wd help me much
>
> dn
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071221/033a1fba/attachment.html>

From charlesr.harris at gmail.com  Fri Dec 21 03:40:26 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 21 Dec 2007 01:40:26 -0700
Subject: [Numpy-discussion] segfault with atlas3-sse2,
	works with atlas3-sse
In-Reply-To: <85b5c3130712200832k4d2b1c35pb372289833e04521@mail.gmail.com>
References: <85b5c3130712170751n52f26317h4917ce152a26b032@mail.gmail.com>
	<1e2af89e0712192112r4b56625u254429cba8bdebca@mail.gmail.com>
	<85b5c3130712200832k4d2b1c35pb372289833e04521@mail.gmail.com>
Message-ID: <e06186140712210040g33b9c8d7gd292b1fb22fc28db@mail.gmail.com>

On Dec 20, 2007 9:32 AM, Ondrej Certik <ondrej at certik.cz> wrote:

> > > when compiled on Debian, numpy segfaults when used with ATLAS sse2,
> > > but works when used against ATLAS sse. More information here:
> >
> > What is the machine on which you are getting the segfault?  Is it
>
> I don't know which machine the reporter of this bug in Debian uses, but I
> use
> Intel Core Duo.
>
> > possible you are trying to run SSE2 instructions on a machine without
> > SSE2?
>
> the /proc/cpuinfo says:
>
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov
> pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc
> pni monitor est tm2 xtpr


Are you running 32 bits or 64 bits? The Debian versions of ATLAS used to
encounter illegal instructions on 64 bit Linux on Intel, but ran fine on 32
bits. You might try compiling your own ATLAS.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071221/b0fc536f/attachment.html>

From stefan at sun.ac.za  Fri Dec 21 03:43:28 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 21 Dec 2007 10:43:28 +0200
Subject: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
In-Reply-To: <476AF9BC.8070705@enthought.com>
References: <47696873.1060008@enthought.com>
	<20071220070728.GD7721@mentat.za.net>
	<476A9404.2030009@enthought.com>
	<200712201254.48284.pgmdevlist@gmail.com>
	<476AF9BC.8070705@enthought.com>
Message-ID: <20071221084328.GB9048@mentat.za.net>

Hi Travis

On Thu, Dec 20, 2007 at 05:24:44PM -0600, Travis E. Oliphant wrote:
> > * bool(x) raises a ValueError, as it does for ndarrays.
> >   
> What does bool(x) raise for numpy.core.ma.

It now behaves the same way as numpy does, raising a ValueError:

In [1]: bool(N.ma.array([0,1]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

/home/stefan/work/scipy/<ipython console> in <module>()

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

In [2]: bool(N.array([0,1]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

/home/stefan/work/scipy/<ipython console> in <module>()

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Regards
St?fan


From haase at msg.ucsf.edu  Fri Dec 21 04:02:28 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Fri, 21 Dec 2007 10:02:28 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <476AF68B.5000105@mspacek.mm.st>
References: <474FCE2D.2000908@mspacek.mm.st>
	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>
	<4750981C.9060409@mspacek.mm.st>
	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>
	<4769D201.1090907@mspacek.mm.st>
	<bc657ead0712201221y5c449523w7f5ede372f4c171d@mail.gmail.com>
	<476AF68B.5000105@mspacek.mm.st>
Message-ID: <bc657ead0712210102r7e385204rc10931c453a60e51@mail.gmail.com>

On Dec 21, 2007 12:11 AM, Martin Spacek <numpy at mspacek.mm.st> wrote:
> >> By the way, I installed 64-bit linux (ubuntu 7.10) on the same machine,
> >> and now numpy.memmap works like a charm. Slicing around a 15 GB file is fun!
> >>
> > Thanks for the feedback !
> > Did you get the kind of speed you need and/or the speed you were hoping for ?
>
> Nope. Like I wrote earlier, it seems there isn't time for disk access in
> my main loop, which is what memmap is all about. I resolved this by
> loading the whole file into memory as a python list of 2D arrays,
> instead of one huge contiguous 3D array. That got me an extra 100 to 200
> MB of physical memory to work with (about 1.4GB out of 2GB total) on
> win32, which is all I needed.
>

Instead of saying "memmap is ALL about disc access" I would rather
like to say that "memap is all about SMART disk access" -- what I mean
is that memmap should run as fast as a normal ndarray if it works on
the cached part of an array.  Maybe there is a way of telling memmap
when and what to cache  and when to sync that cache to the disk.
In other words, memmap should perform just like a in-pysical-memory
array  -- only that it once-in-a-while saves/load to/from the disk.
Or is this just wishful thinking ?
Is there a way of "pre loading" a given part into cache
(pysical-memory) or prevent disc writes at "bad times" ?
How about doing the sync from a different thread ;-)

-Sebastian


From stefan at sun.ac.za  Fri Dec 21 04:23:53 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 21 Dec 2007 11:23:53 +0200
Subject: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
In-Reply-To: <20071221084328.GB9048@mentat.za.net>
References: <47696873.1060008@enthought.com>
	<20071220070728.GD7721@mentat.za.net>
	<476A9404.2030009@enthought.com>
	<200712201254.48284.pgmdevlist@gmail.com>
	<476AF9BC.8070705@enthought.com>
	<20071221084328.GB9048@mentat.za.net>
Message-ID: <20071221092353.GC9048@mentat.za.net>

On Fri, Dec 21, 2007 at 10:43:28AM +0200, Stefan van der Walt wrote:
> On Thu, Dec 20, 2007 at 05:24:44PM -0600, Travis E. Oliphant wrote:
> > > * bool(x) raises a ValueError, as it does for ndarrays.
> > >   
> > What does bool(x) raise for numpy.core.ma.

Sorry, I realise you were talking about the old numpy.core.ma:

z = N.core.ma.masked_array([True,False,True])
bool(z) == True

Regards
St?fan


From stefan at sun.ac.za  Fri Dec 21 04:37:20 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 21 Dec 2007 11:37:20 +0200
Subject: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
In-Reply-To: <200712201852.38920.pgmdevlist@gmail.com>
References: <47696873.1060008@enthought.com>
	<200712201254.48284.pgmdevlist@gmail.com>
	<476AF9BC.8070705@enthought.com>
	<200712201852.38920.pgmdevlist@gmail.com>
Message-ID: <20071221093720.GD9048@mentat.za.net>

On Thu, Dec 20, 2007 at 06:52:38PM -0500, Pierre GM wrote:
> > If we can document exactly what the compatibility issues are (and it
> > looks like we are almost there), we should move forward.
> 
> OK, I'll take care of that this week-end. Stefan, feel free to beat me to 
> it...

A first draft is here:

http://svn.scipy.org/svn/numpy/branches/maskedarray/numpy/ma/API_CHANGES.txt

Regards
St?fan


From ondrej at certik.cz  Fri Dec 21 06:26:28 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 21 Dec 2007 12:26:28 +0100
Subject: [Numpy-discussion] segfault with atlas3-sse2,
	works with atlas3-sse
In-Reply-To: <e06186140712210040g33b9c8d7gd292b1fb22fc28db@mail.gmail.com>
References: <85b5c3130712170751n52f26317h4917ce152a26b032@mail.gmail.com>
	<1e2af89e0712192112r4b56625u254429cba8bdebca@mail.gmail.com>
	<85b5c3130712200832k4d2b1c35pb372289833e04521@mail.gmail.com>
	<e06186140712210040g33b9c8d7gd292b1fb22fc28db@mail.gmail.com>
Message-ID: <85b5c3130712210326n28de875el165aff5505244994@mail.gmail.com>

On Dec 21, 2007 9:40 AM, Charles R Harris <charlesr.harris at gmail.com> wrote:
>
>
>
>
> On Dec 20, 2007 9:32 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> > > > when compiled on Debian, numpy segfaults when used with ATLAS sse2,
> > > > but works when used against ATLAS sse. More information here:
> > >
> > > What is the machine on which you are getting the segfault?  Is it
> >
> > I don't know which machine the reporter of this bug in Debian uses, but I
> use
> > Intel Core Duo.
> >
> > > possible you are trying to run SSE2 instructions on a machine without
> > > SSE2?
> >
> > the /proc/cpuinfo says:
> >
> > flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov
> > pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc
> > pni monitor est tm2 xtpr
>
> Are you running 32 bits or 64 bits? The Debian versions of ATLAS used to
> encounter illegal instructions on 64 bit Linux on Intel, but ran fine on 32
> bits. You might try compiling your own ATLAS.

My processor is imho 64 bits, but I am running 32 bits Debian.

Yes, it's probably Debian specific problem, but the solution is to
find the bug, and fix it, so that the standard Debian packages work.

Ondrej


From david at ar.media.kyoto-u.ac.jp  Fri Dec 21 07:23:49 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 21 Dec 2007 21:23:49 +0900
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <bc657ead0712210102r7e385204rc10931c453a60e51@mail.gmail.com>
References: <474FCE2D.2000908@mspacek.mm.st>	<c266a7930711301000x3cbe51e2p39668dfe31b402ba@mail.gmail.com>	<4750981C.9060409@mspacek.mm.st>	<bc657ead0712010224y5b346d03kd18507b0e4d90b50@mail.gmail.com>	<4769D201.1090907@mspacek.mm.st>	<bc657ead0712201221y5c449523w7f5ede372f4c171d@mail.gmail.com>	<476AF68B.5000105@mspacek.mm.st>
	<bc657ead0712210102r7e385204rc10931c453a60e51@mail.gmail.com>
Message-ID: <476BB055.70305@ar.media.kyoto-u.ac.jp>

Sebastian Haase wrote:
> On Dec 21, 2007 12:11 AM, Martin Spacek <numpy at mspacek.mm.st> wrote:
>   
>>>> By the way, I installed 64-bit linux (ubuntu 7.10) on the same machine,
>>>> and now numpy.memmap works like a charm. Slicing around a 15 GB file is fun!
>>>>
>>>>         
>>> Thanks for the feedback !
>>> Did you get the kind of speed you need and/or the speed you were hoping for ?
>>>       
>> Nope. Like I wrote earlier, it seems there isn't time for disk access in
>> my main loop, which is what memmap is all about. I resolved this by
>> loading the whole file into memory as a python list of 2D arrays,
>> instead of one huge contiguous 3D array. That got me an extra 100 to 200
>> MB of physical memory to work with (about 1.4GB out of 2GB total) on
>> win32, which is all I needed.
>>
>>     
>
> Instead of saying "memmap is ALL about disc access" I would rather
> like to say that "memap is all about SMART disk access" -- what I mean
> is that memmap should run as fast as a normal ndarray if it works on
> the cached part of an array.  Maybe there is a way of telling memmap
> when and what to cache  and when to sync that cache to the disk.
> In other words, memmap should perform just like a in-pysical-memory
> array  -- only that it once-in-a-while saves/load to/from the disk.
> Or is this just wishful thinking ?
> Is there a way of "pre loading" a given part into cache
> (pysical-memory) or prevent disc writes at "bad times" ?
> How about doing the sync from a different thread ;-)
>   
mmap is using the OS IO caches, that's kind of the point of using mmap 
(at least in this case). Instead of doing the caching yourself, the OS 
does it for you, and OS are supposed to be smart about this :)

cheers,

David


From meine at informatik.uni-hamburg.de  Fri Dec 21 08:14:25 2007
From: meine at informatik.uni-hamburg.de (Hans Meine)
Date: Fri, 21 Dec 2007 14:14:25 +0100
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <476BB055.70305@ar.media.kyoto-u.ac.jp>
References: <474FCE2D.2000908@mspacek.mm.st>
	<bc657ead0712210102r7e385204rc10931c453a60e51@mail.gmail.com>
	<476BB055.70305@ar.media.kyoto-u.ac.jp>
Message-ID: <200712211414.26279.meine@informatik.uni-hamburg.de>

Am Freitag, 21. Dezember 2007 13:23:49 schrieb David Cournapeau:
> > Instead of saying "memmap is ALL about disc access" I would rather
> > like to say that "memap is all about SMART disk access" -- what I mean
> > is that memmap should run as fast as a normal ndarray if it works on
> > the cached part of an array.  Maybe there is a way of telling memmap
> > when and what to cache  and when to sync that cache to the disk.
> > In other words, memmap should perform just like a in-pysical-memory
> > array  -- only that it once-in-a-while saves/load to/from the disk.
> > Or is this just wishful thinking ?
> > Is there a way of "pre loading" a given part into cache
> > (pysical-memory) or prevent disc writes at "bad times" ?
> > How about doing the sync from a different thread ;-)
>
> mmap is using the OS IO caches, that's kind of the point of using mmap
> (at least in this case). Instead of doing the caching yourself, the OS
> does it for you, and OS are supposed to be smart about this :)

AFAICS this is what Sebastian wanted to say, but as the OP indicated, 
preloading e.g. by reading the whole array once did not work for him.
Thus, I understand Sebastian's questions as "is it possible to help the OS 
when it is not smart enough?".  Maybe something along the lines of mlock, 
only not quite as aggressive.

Ciao, /  /
     /--/
    /  / ANS


From david at ar.media.kyoto-u.ac.jp  Fri Dec 21 08:45:37 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 21 Dec 2007 22:45:37 +0900
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <200712211414.26279.meine@informatik.uni-hamburg.de>
References: <474FCE2D.2000908@mspacek.mm.st>	<bc657ead0712210102r7e385204rc10931c453a60e51@mail.gmail.com>	<476BB055.70305@ar.media.kyoto-u.ac.jp>
	<200712211414.26279.meine@informatik.uni-hamburg.de>
Message-ID: <476BC381.9030304@ar.media.kyoto-u.ac.jp>

Hans Meine wrote:
> Am Freitag, 21. Dezember 2007 13:23:49 schrieb David Cournapeau:
>   
>>> Instead of saying "memmap is ALL about disc access" I would rather
>>> like to say that "memap is all about SMART disk access" -- what I mean
>>> is that memmap should run as fast as a normal ndarray if it works on
>>> the cached part of an array.  Maybe there is a way of telling memmap
>>> when and what to cache  and when to sync that cache to the disk.
>>> In other words, memmap should perform just like a in-pysical-memory
>>> array  -- only that it once-in-a-while saves/load to/from the disk.
>>> Or is this just wishful thinking ?
>>> Is there a way of "pre loading" a given part into cache
>>> (pysical-memory) or prevent disc writes at "bad times" ?
>>> How about doing the sync from a different thread ;-)
>>>       
>> mmap is using the OS IO caches, that's kind of the point of using mmap
>> (at least in this case). Instead of doing the caching yourself, the OS
>> does it for you, and OS are supposed to be smart about this :)
>>     
>
> AFAICS this is what Sebastian wanted to say, but as the OP indicated, 
> preloading e.g. by reading the whole array once did not work for him.
> Thus, I understand Sebastian's questions as "is it possible to help the OS 
> when it is not smart enough?".  Maybe something along the lines of mlock, 
> only not quite as aggressive.
>   
I don't know exactly why it did not work, but it is not difficult to 
imagine why it could fail (when you read a 2 Gb file, it may not be 
smart on average to put the whole file in the buffer, since everything 
else is kicked out). It all depends on the situation, but there are many 
different things which can influence this behaviour: the IO scheduler, 
how smart the VM is, the FS (on linux, some FS are better than others 
for RT audio dsp, and some options are better left out), etc... On 
Linux, using the deadline IO scheduler can help, for example (that's the 
recommended scheduler for IO intensive musical applications).

But if what you want is to reliable being able to read "in real time" a 
big file which cannot fit in memory, then you need a design where 
something is doing the disk buffering as you want (again, taking the 
example I am somewhat familiar with, in audio processing, you often have 
a IO thread which does the pre-caching, and put the data into mlock'ed 
buffers to another thread, the one which is RT).

cheers,

David


From aisaac at american.edu  Fri Dec 21 09:53:28 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 21 Dec 2007 09:53:28 -0500
Subject: [Numpy-discussion] setting items in a matrix
In-Reply-To: <e76aa17f0712210002t15ad4038xcf05cd79fc3fdd05@mail.gmail.com>
References: <9380affd-b743-4d65-9d94-b234cf4304a3@b1g2000pra.googlegroups.com><e76aa17f0712210002t15ad4038xcf05cd79fc3fdd05@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1280-20071221-095328.00@american.edu>

To learn array basics:
<URL:http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/arrays.html>
<URL:http://www.scipy.org/Numpy_Example_List#head-a8a8874581c2ebfc69a37ab513974a229ff3bfaa>
<URL:http://www.scipy.org/Numpy_Example_List#head-a261b8dd10bda6a5fc268fe4f4171acee2f83968>
<URL:http://homes.esat.kuleuven.be/~python/doc/numpy/array.html>

To learn matrix basics:
<URL:http://www.scipy.org/NumPy_for_Matlab_Users>

hth,
Alan Isaac





From charlesr.harris at gmail.com  Fri Dec 21 11:11:51 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 21 Dec 2007 09:11:51 -0700
Subject: [Numpy-discussion] Loading a > GB file into array
In-Reply-To: <476BC381.9030304@ar.media.kyoto-u.ac.jp>
References: <474FCE2D.2000908@mspacek.mm.st>
	<bc657ead0712210102r7e385204rc10931c453a60e51@mail.gmail.com>
	<476BB055.70305@ar.media.kyoto-u.ac.jp>
	<200712211414.26279.meine@informatik.uni-hamburg.de>
	<476BC381.9030304@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140712210811o11704bc2tf56069db0ae5a3a7@mail.gmail.com>

On Dec 21, 2007 6:45 AM, David Cournapeau <david at ar.media.kyoto-u.ac.jp>
wrote:

> Hans Meine wrote:
> > Am Freitag, 21. Dezember 2007 13:23:49 schrieb David Cournapeau:
> >
> >>> Instead of saying "memmap is ALL about disc access" I would rather
> >>> like to say that "memap is all about SMART disk access" -- what I mean
> >>> is that memmap should run as fast as a normal ndarray if it works on
> >>> the cached part of an array.  Maybe there is a way of telling memmap
> >>> when and what to cache  and when to sync that cache to the disk.
> >>> In other words, memmap should perform just like a in-pysical-memory
> >>> array  -- only that it once-in-a-while saves/load to/from the disk.
> >>> Or is this just wishful thinking ?
> >>> Is there a way of "pre loading" a given part into cache
> >>> (pysical-memory) or prevent disc writes at "bad times" ?
> >>> How about doing the sync from a different thread ;-)
> >>>
> >> mmap is using the OS IO caches, that's kind of the point of using mmap
> >> (at least in this case). Instead of doing the caching yourself, the OS
> >> does it for you, and OS are supposed to be smart about this :)
> >>
> >
> > AFAICS this is what Sebastian wanted to say, but as the OP indicated,
> > preloading e.g. by reading the whole array once did not work for him.
> > Thus, I understand Sebastian's questions as "is it possible to help the
> OS
> > when it is not smart enough?".  Maybe something along the lines of
> mlock,
> > only not quite as aggressive.
> >
> I don't know exactly why it did not work, but it is not difficult to
> imagine why it could fail (when you read a 2 Gb file, it may not be
> smart on average to put the whole file in the buffer, since everything
> else is kicked out). It all depends on the situation, but there are many
> different things which can influence this behaviour: the IO scheduler,
> how smart the VM is, the FS (on linux, some FS are better than others
> for RT audio dsp, and some options are better left out), etc... On
> Linux, using the deadline IO scheduler can help, for example (that's the
> recommended scheduler for IO intensive musical applications).
>
<snip>

>
> But if what you want is to reliable being able to read "in real time" a
> big file which cannot fit in memory, then you need a design where
> something is doing the disk buffering as you want (again, taking the
> example I am somewhat familiar with, in audio processing, you often have
> a IO thread which does the pre-caching, and put the data into mlock'ed
> buffers to another thread, the one which is RT).


IIRC, Martin really wanted something like streaming IO broken up into
smaller frames with previously cached results ideally discarded.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071221/aa0cc43a/attachment.html>

From sdb at cloud9.net  Fri Dec 21 12:00:26 2007
From: sdb at cloud9.net (Stuart Brorson)
Date: Fri, 21 Dec 2007 12:00:26 -0500 (EST)
Subject: [Numpy-discussion] new P3 binaries for NumPy 1.0.4 and SciPy
 0.6.0
In-Reply-To: <c7009a550712201415r29ee07edg9703d1b77e47c2c6@mail.gmail.com>
References: <c7009a550712201415r29ee07edg9703d1b77e47c2c6@mail.gmail.com>
Message-ID: <Pine.BSF.4.64.0712211157330.2374@earl-grey.cloud9.net>

Thanks for your work on this.

I can confirm that the numpy-1.0.4.win32-p3-py2.5.msi binary works as
advertised on my crufty PIII test platform.  numpy.test() passes all
tests.

Cheers,

Stuart Brorson
Interactive Supercomputing, inc.
135 Beaver Street | Waltham | MA | 02452 | USA
http://www.interactivesupercomputing.com/


On Thu, 20 Dec 2007, Jarrod Millman wrote:

> Hey,
>
> If you are having problems with NumPy and SciPy on Pentium III
> machines running Windows, please try the newly released binaries:
>
> numpy-1.0.4.win32-p3-py2.3.exe
> numpy-1.0.4.win32-p3-py2.4.exe
> numpy-1.0.4.win32-p3-py2.5.exe
> numpy-1.0.4.win32-p3-py2.5.msi
>
> scipy-0.6.0.win32-p3-py2.3.exe
> scipy-0.6.0.win32-p3-py2.4.exe
> scipy-0.6.0.win32-p3-py2.5.exe
> scipy-0.6.0.win32-p3-py2.5.msi
>
> Enjoy,
>
> -- 
> Jarrod Millman
> Computational Infrastructure for Research Labs
> 10 Giannini Hall, UC Berkeley
> phone: 510.643.4014
> http://cirl.berkeley.edu/
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From charlesr.harris at gmail.com  Fri Dec 21 12:01:49 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 21 Dec 2007 10:01:49 -0700
Subject: [Numpy-discussion] setting items in a matrix
In-Reply-To: <d7ad8c21-b98c-4d04-872a-c7c6ae6b8436@i12g2000prf.googlegroups.com>
References: <d7ad8c21-b98c-4d04-872a-c7c6ae6b8436@i12g2000prf.googlegroups.com>
Message-ID: <e06186140712210901q72571123t8996d02c58da7ad8@mail.gmail.com>

On Dec 21, 2007 12:37 AM, devnew at gmail.com <devnew at gmail.com> wrote:

> hi
> i am a beginner with numpy and python,so pardon me if this doubt seems
> silly
> i want to create a matrix with say 3 rows and 5 columns..and then set
> the values of each item in it .for this i did something like below
>
> myarray=zeros((3,5))
> #then set the items
> for row in range(3):
>    for col in range(5):
>        myarray[row][col]=999999.9999
>
>
In [1]: a = zeros([3,5])

In [2]: a[...] = 999

In [3]: a[0:2,0:2] = 99

In [4]: a
Out[4]:
array([[  99.,   99.,  999.,  999.,  999.],
       [  99.,   99.,  999.,  999.,  999.],
       [ 999.,  999.,  999.,  999.,  999.]])

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071221/c6e0da0e/attachment.html>

From efiring at hawaii.edu  Fri Dec 21 13:13:13 2007
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 21 Dec 2007 08:13:13 -1000
Subject: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
In-Reply-To: <20071221093720.GD9048@mentat.za.net>
References: <47696873.1060008@enthought.com>
	<200712201254.48284.pgmdevlist@gmail.com>
	<476AF9BC.8070705@enthought.com>
	<200712201852.38920.pgmdevlist@gmail.com>
	<20071221093720.GD9048@mentat.za.net>
Message-ID: <476C0239.1000209@hawaii.edu>

Stefan,

I think the description of the putmask difference is missing the point. 
  The real difference is not in the way the third argument is handled, 
or  its required shape, but in whether the mask is updated or not.

numpy.ma.putmask updates the mask; that is, if it "puts" something into 
the array in a formerly masked location, the mask at that location is 
changed to False.

maskedarray.putmask does *not* update the original mask; it does a 
ndarray putmask operation on only the *unmasked* values.


In [18]:import maskedarray as ma
In [19]:mask1 = [False, True, False]
In [21]:x = ma.array([1,2,3], mask=mask1)
In [22]:mask2 = [False, True, True]
In [23]:ma.putmask(x, mask2, 10)
In [24]:x
Out[24]:
masked_array(data = [1 -- 10],
       mask = [False  True False],
       fill_value=999999)

In [25]:x = numpy.ma.array([1,2,3], mask=mask1)
In [26]:numpy.ma.putmask(x, mask2, 10)
In [27]:x
Out[27]:
array(data =
  [ 1 10 10],
       mask =
  [False False False],
       fill_value=999999)


This is a fundamental difference.  The maskedarray (new) behavior 
perhaps is more consistent with the general pattern of differences 
between masked and unmasked array functions and methods.

Eric

Stefan van der Walt wrote:
> On Thu, Dec 20, 2007 at 06:52:38PM -0500, Pierre GM wrote:
>>> If we can document exactly what the compatibility issues are (and it
>>> looks like we are almost there), we should move forward.
>> OK, I'll take care of that this week-end. Stefan, feel free to beat me to 
>> it...
> 
> A first draft is here:
> 
> http://svn.scipy.org/svn/numpy/branches/maskedarray/numpy/ma/API_CHANGES.txt
> 
> Regards
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From aisaac at american.edu  Fri Dec 21 14:51:12 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 21 Dec 2007 14:51:12 -0500
Subject: [Numpy-discussion] new P3 binaries for NumPy 1.0.4 and SciPy
	0.6.0
In-Reply-To: <c7009a550712201415r29ee07edg9703d1b77e47c2c6@mail.gmail.com>
References: <c7009a550712201415r29ee07edg9703d1b77e47c2c6@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1280-20071221-145112.00@american.edu>

On Thu, 20 Dec 2007, Jarrod Millman apparently wrote:
> If you are having problems with NumPy and SciPy on Pentium III 
> machines running Windows, please try the newly released binaries:

I used the Python 2.5 binaries (.exe) on my home P3 and all 
seems well in use.  I got no failures of numpy.test() but 
some warnings and failures from scipy.test().  The failures
are listed below.

Comment: finding the SciPy binaries is not obvious,
since they do not show here 
<URL:http://www.scipy.org/Download#head-312ad78cdf85a9ca6fa17a266752069d23f785d1>

Thank you!
Alan Isaac




======================================================================
FAIL: check_syevr (scipy.lib.tests.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python25\lib\site-packages\scipy\lib\lapack\tests\esv_tests.py", line 41, in check_syevr
    assert_array_almost_equal(w,exact_w)
  File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 232, in assert_array_almost_equa
l
    header='Arrays are not almost equal')
  File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 217, in assert_array_compare
    assert cond, msg
AssertionError:
Arrays are not almost equal

(mismatch 33.3333333333%)
 x: array([-0.66992444,  0.48769444,  9.18222618], dtype=float32)
 y: array([-0.66992434,  0.48769389,  9.18223045])

======================================================================
FAIL: check_syevr_irange (scipy.lib.tests.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python25\lib\site-packages\scipy\lib\lapack\tests\esv_tests.py", line 66, in check_syevr_
irange
    assert_array_almost_equal(w,exact_w[rslice])
  File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 232, in assert_array_almost_equa
l
    header='Arrays are not almost equal')
  File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 217, in assert_array_compare
    assert cond, msg
AssertionError:
Arrays are not almost equal

(mismatch 33.3333333333%)
 x: array([-0.66992444,  0.48769444,  9.18222618], dtype=float32)
 y: array([-0.66992434,  0.48769389,  9.18223045])

======================================================================
FAIL: check_bicg (scipy.linalg.tests.test_iterative.test_iterative_solvers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python25\lib\site-packages\scipy\linalg\tests\test_iterative.py", line 57, in check_bicg
    assert norm(dot(self.A, x) - self.b) < 5*self.tol
AssertionError

======================================================================
FAIL: check_bicgstab (scipy.linalg.tests.test_iterative.test_iterative_solvers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python25\lib\site-packages\scipy\linalg\tests\test_iterative.py", line 69, in check_bicgs
tab
    assert norm(dot(self.A, x) - self.b) < 5*self.tol
AssertionError

======================================================================
FAIL: check_cgs (scipy.linalg.tests.test_iterative.test_iterative_solvers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python25\lib\site-packages\scipy\linalg\tests\test_iterative.py", line 63, in check_cgs
    assert norm(dot(self.A, x) - self.b) < 5*self.tol
AssertionError

======================================================================
FAIL: test_fromimage (scipy.misc.tests.test_pilutil.test_pilutil)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python25\lib\site-packages\scipy\misc\tests\test_pilutil.py", line 34, in test_fromimage
    assert img.min() >= imin
AssertionError

======================================================================
FAIL: test_imresize (scipy.misc.tests.test_pilutil.test_pilutil)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python25\lib\site-packages\scipy\misc\tests\test_pilutil.py", line 18, in test_imresize
    assert_equal(im1.shape,(11,22))
  File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 137, in assert_equal
    assert_equal(len(actual),len(desired),err_msg,verbose)
  File "C:\Python25\Lib\site-packages\numpy\testing\utils.py", line 145, in assert_equal
    assert desired == actual, msg
AssertionError:
Items are not equal:
 ACTUAL: 0
 DESIRED: 2

----------------------------------------------------------------------
Ran 1719 tests in 73.656s

FAILED (failures=7)
<unittest._TextTestResult run=1719 errors=0 failures=7>




From stefan at sun.ac.za  Sat Dec 22 15:45:34 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Sat, 22 Dec 2007 22:45:34 +0200
Subject: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
In-Reply-To: <476C0239.1000209@hawaii.edu>
References: <47696873.1060008@enthought.com>
	<200712201254.48284.pgmdevlist@gmail.com>
	<476AF9BC.8070705@enthought.com>
	<200712201852.38920.pgmdevlist@gmail.com>
	<20071221093720.GD9048@mentat.za.net> <476C0239.1000209@hawaii.edu>
Message-ID: <20071222204534.GB7803@mentat.za.net>

Hi Eric

On Fri, Dec 21, 2007 at 08:13:13AM -1000, Eric Firing wrote:
> I think the description of the putmask difference is missing the point. 
>   The real difference is not in the way the third argument is handled, 
> or  its required shape, but in whether the mask is updated or not.

This behaviour was changed in the maskedarray branch:

> In [18]:import maskedarray as ma
> In [19]:mask1 = [False, True, False]
> In [21]:x = ma.array([1,2,3], mask=mask1)
> In [22]:mask2 = [False, True, True]
> In [23]:ma.putmask(x, mask2, 10)
> In [24]:x
> Out[24]:
> masked_array(data = [1 -- 10],
>        mask = [False  True False],
>        fill_value=999999)

In [2]: import numpy as N
In [3]: ma = N.ma
In [4]: mask1 = [False, True, False]
In [5]: x = ma.array([1,2,3], mask=mask1)
In [6]: mask2 = [False, True, True]
In [7]: ma.putmask(x, mask2, 10)
In [8]: x
Out[8]: 
masked_array(data = [1 10 10],
      mask = [False False False],
      fill_value=999999)

Regards
St?fan


From ondrej at certik.cz  Sat Dec 22 20:29:07 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 23 Dec 2007 02:29:07 +0100
Subject: [Numpy-discussion] ising model: f2py vs cython comparison
Message-ID: <85b5c3130712221729u542267aauad136ed184e7d9d6@mail.gmail.com>

Hi,

I need to write 2D Ising model simulation into my school, so I wrote
it in Python, then rewrote it in Fortran + f2py, and also Cython:

http://hg.sharesource.org/isingmodel/

And Cython solution is 2x faster than f2py. I understand, that I am
comparing many things - wrappers, my fortran coding skills
vs Cython C code generation and gcc vs gfortran. But still any ideas
how to speed up fortran+f2py solution is very welcomed.

How to play with that - just do this (after installing Mercurial):

$ hg clone http://hg.sharesource.org/isingmodel/
[...]
$ hg up db7dd01cdc26                                 # just to be sure
that we are talking about the same revision / code
$ cd isingmodel
$ make
[...]
$ time python simulate.py
[...]
real	0m2.026s
user	0m1.988s
sys	0m0.020s

This runs Cython code. Then apply this patch to run fortran code instead:

$ hg di
diff -r db7dd01cdc26 simulate.py
--- a/simulate.py	Sun Dec 23 02:23:30 2007 +0100
+++ b/simulate.py	Sun Dec 23 02:24:33 2007 +0100
@@ -31,8 +31,8 @@ def MC(mu = 1, temp = 2, dim = 20, steps
     J=1 #coupling constant
     k=1 #Boltzman constant

-    #from mcising import mc
-    from pyising import mc
+    from mcising import mc
+    #from pyising import mc
     B = D1(A)
     mc(B, dim, steps, temp, H, mu, J, k)
     return D2(B)


And then again:

$ time python simulate.py
[...]
real	0m3.600s
user	0m3.528s
sys	0m0.052s



So it's a lot slower.

Maybe I did some stupid mistake, like doing different amount of steps
in Cython and Fortran, but I tried to check that and if I did
everything all right, it's interesting to find why is fortran+f2py
slower.

Ondrej


From ondrej at certik.cz  Sat Dec 22 20:32:54 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 23 Dec 2007 02:32:54 +0100
Subject: [Numpy-discussion] ising model: f2py vs cython comparison
In-Reply-To: <85b5c3130712221729u542267aauad136ed184e7d9d6@mail.gmail.com>
References: <85b5c3130712221729u542267aauad136ed184e7d9d6@mail.gmail.com>
Message-ID: <85b5c3130712221732y1ad025edp65fcb3ae5e8949cf@mail.gmail.com>

On Dec 23, 2007 2:29 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I need to write 2D Ising model simulation into my school, so I wrote
> it in Python, then rewrote it in Fortran + f2py, and also Cython:
>
> http://hg.sharesource.org/isingmodel/
>
> And Cython solution is 2x faster than f2py. I understand, that I am
> comparing many things - wrappers, my fortran coding skills
> vs Cython C code generation and gcc vs gfortran. But still any ideas
> how to speed up fortran+f2py solution is very welcomed.
>
> How to play with that - just do this (after installing Mercurial):
>
> $ hg clone http://hg.sharesource.org/isingmodel/
> [...]
> $ hg up db7dd01cdc26                                 # just to be sure
> that we are talking about the same revision / code
> $ cd isingmodel

The last two lines should have been interchanged of course:

$ cd isingmodel
$ hg up db7dd01cdc26

Also you may want to edit the Makefile:

all:
	f2py --fcompiler=gnu95 --f77flags="-W -Wall -fdefault-real-8" -c -m
mcising mcising.f
	cython pyising.pyx
	gcc -fPIC -O3 -I/usr/include/python2.4/
-I/usr/share/python-support/python-numpy/numpy/core/include/numpy/ -c
-o pyising.o pyising.c
	gcc -shared pyising.o -o pyising.so


Especially the include path to numpy/*.h files.

Ondrej


From oliphant at enthought.com  Sat Dec 22 20:49:36 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Sat, 22 Dec 2007 19:49:36 -0600
Subject: [Numpy-discussion] ising model: f2py vs cython comparison
In-Reply-To: <85b5c3130712221729u542267aauad136ed184e7d9d6@mail.gmail.com>
References: <85b5c3130712221729u542267aauad136ed184e7d9d6@mail.gmail.com>
Message-ID: <476DBEB0.9050303@enthought.com>

Ondrej Certik wrote:
> Hi,
>
> I need to write 2D Ising model simulation into my school, so I wrote
> it in Python, then rewrote it in Fortran + f2py, and also Cython:
>
> http://hg.sharesource.org/isingmodel/
>
> And Cython solution is 2x faster than f2py. I understand, that I am
> comparing many things - wrappers, my fortran coding skills
> vs Cython C code generation and gcc vs gfortran. But still any ideas
> how to speed up fortran+f2py solution is very welcomed.
>   

My experience with similar kinds of comparisons is that gnu fortran 
compilers are not very good, especially on 2-d problems. Try using a 
different fortran compiler to see if speeds improve.

-Travis O.



From beckers at orn.mpg.de  Sun Dec 23 12:52:38 2007
From: beckers at orn.mpg.de (Gabriel J.L. Beckers)
Date: Sun, 23 Dec 2007 18:52:38 +0100
Subject: [Numpy-discussion] matlab to numpy help
Message-ID: <1198432358.6426.15.camel@gabriel-desktop>

I found a matlab script that I want to translate into numpy, but have
difficulties with understanding indexing in matlab. I haven't used
matlab very much and I was hoping that someone could help with the
following:

It says:

	Uns = ones(1,m);
	
... and then later

	Xijm(:,Uns);


m is a positive integer. The problem is that Xijm should be a
1-dimensional array (I think), so what is Uns doing in the second
statement? Is this some way to expand Xijm into a second dimension? What
would be a numpy equivalent?

Gabriel




From pearu at cens.ioc.ee  Sun Dec 23 05:38:29 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Sun, 23 Dec 2007 12:38:29 +0200 (EET)
Subject: [Numpy-discussion] ising model: f2py vs cython comparison
In-Reply-To: <85b5c3130712221729u542267aauad136ed184e7d9d6@mail.gmail.com>
References: <85b5c3130712221729u542267aauad136ed184e7d9d6@mail.gmail.com>
Message-ID: <62074.85.166.31.187.1198406309.squirrel@cens.ioc.ee>

On Sun, December 23, 2007 3:29 am, Ondrej Certik wrote:
> Hi,
>
> I need to write 2D Ising model simulation into my school, so I wrote
> it in Python, then rewrote it in Fortran + f2py, and also Cython:
>
> http://hg.sharesource.org/isingmodel/
>
> And Cython solution is 2x faster than f2py. I understand, that I am
> comparing many things - wrappers, my fortran coding skills
> vs Cython C code generation and gcc vs gfortran. But still any ideas
> how to speed up fortran+f2py solution is very welcomed.

Though the problem is 2D, your implementations are essentially
1D. If you would treat the array A as 2D array (and avoid calling
subroutine p) then you would gain some 7% speed up in Fortran.

When using -DF2PY_REPORT_ATEXIT for f2py then a summary
of timings will be printed out about how much time was spent
in Fortran code and how much in the interface. In the given case
I get (nsteps=50000):

Overall time spent in ...
(a) wrapped (Fortran/C) functions           :     1962 msec
(b) f2py interface,               60 calls  :        0 msec
(c) call-back (Python) functions            :        0 msec
(d) f2py call-back interface,      0 calls  :        0 msec
(e) wrapped (Fortran/C) functions (acctual) :     1962 msec

that is, most of the time is spent in Fortran function and no time
in wrapper. The conclusion is that the cause of the
difference in timings is not in f2py or cpython generated
interfaces but in Fortran and C codes and/or compilers.

Some idiom used in Fortran code is just slower than in C..
For example, in C code you are doing calculations using
float precision but in Fortran you are forcing double precision.

HTH,
Pearu

PS: Here follows a setup.py file that I used to build the
extension modules instead of the Makefile:

#file: setup.py
def configuration(parent_package='',top_path=None):
    from numpy.distutils.misc_util import Configuration
    config = Configuration('',parent_package,top_path)
    config.add_extension('mcising', sources=['mcising.f'],
                         define_macros = [('F2PY_REPORT_ATEXIT',1)]
                         )
    #config.add_extension('pyising', sources=['pyising.pyx'])
    return config
from numpy.distutils.core import setup
setup(configuration = configuration)




From pearu at cens.ioc.ee  Sun Dec 23 05:52:16 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Sun, 23 Dec 2007 12:52:16 +0200 (EET)
Subject: [Numpy-discussion] ising model: f2py vs cython comparison
In-Reply-To: <85b5c3130712221729u542267aauad136ed184e7d9d6@mail.gmail.com>
References: <85b5c3130712221729u542267aauad136ed184e7d9d6@mail.gmail.com>
Message-ID: <62195.85.166.31.187.1198407136.squirrel@cens.ioc.ee>

On Sun, December 23, 2007 3:29 am, Ondrej Certik wrote:
> Hi,
>
> I need to write 2D Ising model simulation into my school, so I wrote
> it in Python, then rewrote it in Fortran + f2py, and also Cython:
>
> http://hg.sharesource.org/isingmodel/
>
> And Cython solution is 2x faster than f2py. I understand, that I am
> comparing many things - wrappers, my fortran coding skills
> vs Cython C code generation and gcc vs gfortran. But still any ideas
> how to speed up fortran+f2py solution is very welcomed.

When using g77 compiler instead of gfortran, I get a speed
up 4.8 times.

Btw, a line in a if statement of the fortran code
should read `A(p(i,j,N)) = - A(p(i,j,N))`.

Pearu



From ondrej at certik.cz  Sun Dec 23 07:57:32 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 23 Dec 2007 13:57:32 +0100
Subject: [Numpy-discussion] gfortran/g77+f2py vs gcc+Cython speed comparison
Message-ID: <85b5c3130712230457q26bc356ah168653168db0dab@mail.gmail.com>

Hi,

I needed to write 2D Ising model simulation into my school and I
decided to compare the two possible solutions how to do it, so I of
course wrote
it in Python, then rewrote it in Fortran + f2py, and also Cython. What
is better? Read below. :)  But for the impatient, I am going to use
Cython, reasons below.

CCing to Cython, numpy (f2py is discussed there), and sage-devel
(there are people there who could be interested in these kinds of
comparisons).

The code is available at:

http://hg.sharesource.org/isingmodel/

How to play with that - just do this (after installing Mercurial):

$ hg clone http://hg.sharesource.org/isingmodel/
[...]
$ cd isingmodel
$ hg up db7dd01cdc26                                 # just to be sure
that we are talking about the same revision / code
$ make
[...]
$ time python simulate.py
[...]
real    0m2.026s
user    0m1.988s
sys     0m0.020s

This runs Cython code. Then apply this patch to run fortran code instead:

$ hg di
diff -r db7dd01cdc26 simulate.py
--- a/simulate.py       Sun Dec 23 02:23:30 2007 +0100
+++ b/simulate.py       Sun Dec 23 02:24:33 2007 +0100
@@ -31,8 +31,8 @@ def MC(mu = 1, temp = 2, dim = 20, steps
    J=1 #coupling constant
    k=1 #Boltzman constant

-    #from mcising import mc
-    from pyising import mc
+    from mcising import mc
+    #from pyising import mc
    B = D1(A)
    mc(B, dim, steps, temp, H, mu, J, k)
    return D2(B)


And then again (and apply the patch below, otherwise it might not work for you):

$ time python simulate.py
[...]
real    0m3.600s
user    0m3.528s
sys     0m0.052s

So it's a lot slower.

We are comparing many things here - wrappers, my fortran coding skills
vs Cython C code generation and gcc vs gfortran. So I wrote to numpy
mailinglist. First Travis (the author of numpy) suggested:

"
My experience with similar kinds of comparisons is that gnu fortran
compilers are not very good, especially on 2-d problems. Try using a
different fortran compiler to see if speeds improve.
"

Then Pearu (the author of f2py) suggested:

"
Though the problem is 2D, your implementations are essentially
1D. If you would treat the array A as 2D array (and avoid calling
subroutine p) then you would gain some 7% speed up in Fortran.

When using -DF2PY_REPORT_ATEXIT for f2py then a summary
of timings will be printed out about how much time was spent
in Fortran code and how much in the interface. In the given case
I get (nsteps=50000):

Overall time spent in ...
(a) wrapped (Fortran/C) functions           :     1962 msec
(b) f2py interface,               60 calls  :        0 msec
(c) call-back (Python) functions            :        0 msec
(d) f2py call-back interface,      0 calls  :        0 msec
(e) wrapped (Fortran/C) functions (acctual) :     1962 msec

that is, most of the time is spent in Fortran function and no time
in wrapper. The conclusion is that the cause of the
difference in timings is not in f2py or cpython generated
interfaces but in Fortran and C codes and/or compilers.

Some idiom used in Fortran code is just slower than in C..
For example, in C code you are doing calculations using
float precision but in Fortran you are forcing double precision.

HTH,
Pearu

PS: Here follows a setup.py file that I used to build the
extension modules instead of the Makefile:

#file: setup.py
def configuration(parent_package='',top_path=None):
   from numpy.distutils.misc_util import Configuration
   config = Configuration('',parent_package,top_path)
   config.add_extension('mcising', sources=['mcising.f'],
                        define_macros = [('F2PY_REPORT_ATEXIT',1)]
                        )
   #config.add_extension('pyising', sources=['pyising.pyx'])
   return config
from numpy.distutils.core import setup
setup(configuration = configuration)
"

and then quickly added

"
When using g77 compiler instead of gfortran, I get a speed
up 4.8 times.

Btw, a line in a if statement of the fortran code
should read `A(p(i,j,N)) = - A(p(i,j,N))`.
"

(btw I have no idea how it could work for me without the A(p(i,j,N)) =
- A(p(i,j,N)) fix, quite embarassing)

So then we discussed it on #sympy IRC:

* Now talking on #sympy
<ondrej> hi pearu, thanks a lot for testing it!
<ondrej> 4.8 speedup, jesus christ. so the gfortran sucks a lot
<pearu> hey ondrej
* ondrej is trying g77
<pearu> gortran has advantage of being Fortran 90 compiler
<ondrej> g77 is depricated in Debian and dead upstram (imho)
<pearu> yes but I guess those guys who are maintaing g77/gfortran,
those do not crunch numbers much;)
<pearu> g77 has a long development history and is well optimized
<ondrej> I fear the fortran is a bad investment for new projects
<ondrej> I know g77 is well optimized, but it's dead
<pearu> gfortran is a write from scratch and is too young
<ondrej> do you think many people still write fortran? I use it just
because g77 was faster than gcc
<pearu> g77 is certainly not dead, scientist use it a lot because of its speed
<ondrej> btw`A(p(i,j,N)) = - A(p(i,j,N))`.
<ondrej> means my fortran code was broken
<ondrej> doesn't it?
<pearu> some think that fortran is a dead language, some use it a lot
because lots of code is written in fortran over several decades
<ondrej> yes
<pearu> yes, it was, I got segfaults because of that
<ondrej> I don't know what to think myself
<pearu> it depends on application
<pearu> if it is a research app then use fortran because of speed
<ondrej> but as you can see, you need to use g77
<ondrej> and not gfortran
<ondrej> (but all Debian is moving to use gfortran)
<pearu> I use gfortran only for f90 code
<ondrej> hm
<pearu> I think Debian is wrong in short term, may be in future
gfortran will be faster, but not at the moment
<ondrej> hm, that's bad
<ondrej> but is someone developing g77?
<pearu> no, I don't think so
<ondrej> debian moved to gfortran, because there are problems with g77
being umaintained
<pearu> but it is a complete Fortran 77 compiler and a very good one
<ondrej> hm. I think when one wants speed, one should use some
comercial compilers
<pearu> probably because new architectures are developed and g77 is
not updated to use their features
<ondrej> and when one wants robustness, it should use a free compiler
that is maintinaed (gfortran)
<pearu> commercial compilers does not mean fast compliers, in general
<ondrej> I didn't try intel, so I don't know how it compares to g77
<pearu> using intel restricts one to be on Intel platform
<ondrej> the g77 doesn't support -fdefault-real-8 for example
<pearu> AMD is daster in floating point that most scientist use
<pearu> why is this flag important, one can always write real*8
<ondrej> indeed, g77 is 1.8s and gcc 2.0s on my comp
<ondrej> as to real*8 - I thought a good strategy is to write real and
let the compiler choose the precision
<pearu> try to use float in fortran as you do in C
<ondrej> ok
<pearu> this will mess up f2py-ing as f2py assumes that real is real*4
<ondrej> yes -
<ondrej> real*4 is float, isn't it?
<pearu> yes
<ondrej> so I'll just use real*4 everywhere
<pearu> yep
<ondrej> ok
<ondrej> btw, I was writing a python script for automatically
converting real*4 to real*8 and back
<ondrej> but it's freaking hard to parse fortran
<ondrej> (I needed to also work with "real", etc)
<ondrej> I only managed it to work in simple cases
<pearu> just do re.sub
<ondrej> it's not that easy
<ondrej> there are constructs like real(something)
<ondrej> that shouldn't be converted
<ondrej> but real T should be converted
<ondrej> etc.
<ondrej> so using real*4 with g77 slowed the code done from 1.8s to 1.9s
<pearu> re.sub should be able to handle these, use cb argument, fo instance
<ondrej> ok
<pearu> you are using random numbers, could this make also a difference
<ondrej> I think it does
<ondrej> (I had to use glibc fast random to speed C up)
<pearu> a, ok, that also explains things
<ondrej> shit, the gfortran is really slow
<ondrej> 3.5s on my comp
<ondrej> no matter which precision
<pearu> try to disable random to see if loops are faster in C or fortran
<ondrej> ok
<ondrej> so gfortran is now 1.3s
<ondrej> and g77 1.04s
* ondrej is trying C
<ondrej> C is 1.05s
<pearu> so gfortran random is slow
<ondrej> yes
<ondrej> g77 random is fast
<ondrej> C random is slower than g77
<ondrej> it depends on the quality of the random generator as well
<ondrej> there are huge differences in that
<pearu> and so gfortran itsel is not too bad
<pearu> :)
<ondrej> but as you can see - there is no point for me to use g77 if I
can achieve the same speed with C
<ondrej> and gfortran is slower than both
<pearu> yep
<ondrej> ok, going to write this to the list
<pearu> can you try ifc
<ondrej> I'll also send it to the Cython guys, they'll like the result. :)
<ondrej> ifc = intel fortran compiler?
<pearu> yes
<ondrej> I like opensource and Debian
<ondrej> I think this clearly shows, that fortran is dead for me. I
think gcc is pretty good too.
<pearu> note that these results are not thanks to pyrex /cpython but
thanks to compilers:)
<pearu> i like to be fair;)
<ondrej> (yes, it's just compilers comparison, not f2py vs Cython/pyrex)
<pearu> yep
<ondrej> but there are two ways a user can choose - either cython+gcc,
or f2py+fortran
<pearu> next time we shoul probably have this discussion inscipy irc
<ondrej> and if he wants to use the most recent default free
compilers, at least I will choose cython+gcc
<pearu> there are more ways, weave, etc etc
<ondrej> I am going to paste this to my email about that
<pearu> ok
<ondrej> (if you are ok with that)
<pearu> ok with me



So, what do you think of that?

Ondrej


From efiring at hawaii.edu  Sun Dec 23 14:26:00 2007
From: efiring at hawaii.edu (Eric Firing)
Date: Sun, 23 Dec 2007 09:26:00 -1000
Subject: [Numpy-discussion] matlab to numpy help
In-Reply-To: <1198432358.6426.15.camel@gabriel-desktop>
References: <1198432358.6426.15.camel@gabriel-desktop>
Message-ID: <476EB648.7080704@hawaii.edu>

Gabriel J.L. Beckers wrote:
> I found a matlab script that I want to translate into numpy, but have
> difficulties with understanding indexing in matlab. I haven't used
> matlab very much and I was hoping that someone could help with the
> following:
> 
> It says:
> 
> 	Uns = ones(1,m);
> 	
> ... and then later
> 
> 	Xijm(:,Uns);
> 
> 
> m is a positive integer. The problem is that Xijm should be a
> 1-dimensional array (I think), so what is Uns doing in the second
> statement? Is this some way to expand Xijm into a second dimension? What
> would be a numpy equivalent?

What this is doing depends on exactly what Xijm is.  Matlab started out 
with 2-D arrays only--*everything* was a 2-D array (matrix) of 
double-precision numbers.  (Even strings were represented as 
double-precision matrices.) Later, support for more dimensions was 
tacked on, but a vector in Matlab is still a 2-D array, and by default 
it is a single row:

 >> Xijm = [2,3,4]
Xijm =
      2     3     4
 >> size(Xijm)
ans =
      1     3
 >> Xijm(:,ones(1,2))
ans =
      2     2

To make a row into a column, you can transpose it:

 >> XijmT = Xijm.'
XijmT =
      2
      3
      4
 >> XijmT(:,ones(1,2))
ans =
      2     2
      3     3
      4     4

Numpy is much more general, and supports arrays of any number of 
dimensions right from the start.

My guess is that in your Matlab application, X is a column (XijmT), as 
in the second example above, and the indexing is repeating the columns 
as shown.  Usually this is done to facilitate an array operation with 
another array that, in this example, has shape (3,2).  In numpy it is 
not necessary to make a new array with the columns repeated because 
broadcasting achieves the same result more efficiently:

In [1]:import numpy

In [2]:XijmT = numpy.array([2,3,4])

In [3]:XijmT.shape
Out[3]:(3,)

In [4]:A = numpy.arange(6)

In [5]:A.shape = 3,2

In [6]:A
Out[6]:
array([[0, 1],
        [2, 3],
        [4, 5]])

In [9]:XijmT[:,numpy.newaxis] + A
Out[9]:
array([[2, 3],
        [5, 6],
        [8, 9]])

The indexing with numpy.newaxis makes a 2-D view of the 1-D array, 
allowing the column dimension to be broadcast to match that of A.

Eric

> 
> Gabriel
> 
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From timmichelsen at gmx-topmail.de  Sun Dec 23 14:19:10 2007
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Sun, 23 Dec 2007 20:19:10 +0100
Subject: [Numpy-discussion] setting items in a matrix
In-Reply-To: <Mahogany-0.67.0-1280-20071221-095328.00@american.edu>
References: <9380affd-b743-4d65-9d94-b234cf4304a3@b1g2000pra.googlegroups.com><e76aa17f0712210002t15ad4038xcf05cd79fc3fdd05@mail.gmail.com>
	<Mahogany-0.67.0-1280-20071221-095328.00@american.edu>
Message-ID: <fkmcbe$mqf$1@ger.gmane.org>

> To learn array basics:
> <URL:http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/arrays.html>
> <URL:http://www.scipy.org/Numpy_Example_List#head-a8a8874581c2ebfc69a37ab513974a229ff3bfaa>
> <URL:http://www.scipy.org/Numpy_Example_List#head-a261b8dd10bda6a5fc268fe4f4171acee2f83968>
> <URL:http://homes.esat.kuleuven.be/~python/doc/numpy/array.html>
> 
> To learn matrix basics:
> <URL:http://www.scipy.org/NumPy_for_Matlab_Users>

Thanks you so much!

These links are really great and that's what I was really looking for.

Nice!

Kind regards and nice X-mas,
Timmie



From alex.liberzon at gmail.com  Sun Dec 23 18:00:14 2007
From: alex.liberzon at gmail.com (Alex)
Date: Sun, 23 Dec 2007 15:00:14 -0800 (PST)
Subject: [Numpy-discussion] matlab to numpy help
In-Reply-To: <1198432358.6426.15.camel@gabriel-desktop>
References: <1198432358.6426.15.camel@gabriel-desktop>
Message-ID: <776433ad-6946-4e30-b418-4c71fe04c640@j20g2000hsi.googlegroups.com>

if m = 5, then
Uns = ones(1,m) means 1x5 matrix(vector) of 1: [1 1 1 1 1]
Xijm(:,Uns) is called Tony's trick where you actually replicate the
Xijm (vector i guess) 5 times (according to the Uns.) e.g.
http://xtargets.com/snippets/posts/show/55
Matlab introduced repmat() function that does the job "explicitly",
using Tony's trick mostly.

Hope it helps,
Alex



On Dec 23, 7:52 pm, "Gabriel J.L. Beckers" <beck... at orn.mpg.de> wrote:
> I found a matlab script that I want to translate into numpy, but have
> difficulties with understanding indexing in matlab. I haven't used
> matlab very much and I was hoping that someone could help with the
> following:
>
> It says:
>
>         Uns = ones(1,m);
>
> ... and then later
>
>         Xijm(:,Uns);
>
> m is a positive integer. The problem is that Xijm should be a
> 1-dimensional array (I think), so what is Uns doing in the second
> statement? Is this some way to expand Xijm into a second dimension? What
> would be a numpy equivalent?
>
> Gabriel
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion


From jtravs at gmail.com  Mon Dec 24 06:50:29 2007
From: jtravs at gmail.com (John Travers)
Date: Mon, 24 Dec 2007 11:50:29 +0000
Subject: [Numpy-discussion] gfortran/g77+f2py vs gcc+Cython speed
	comparison
In-Reply-To: <85b5c3130712230457q26bc356ah168653168db0dab@mail.gmail.com>
References: <85b5c3130712230457q26bc356ah168653168db0dab@mail.gmail.com>
Message-ID: <3a1077e70712240350p21eab1aet49e02346f9bfc9c3@mail.gmail.com>

On Dec 23, 2007 12:57 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I needed to write 2D Ising model simulation into my school and I
> decided to compare the two possible solutions how to do it, so I of
> course wrote
> it in Python, then rewrote it in Fortran + f2py, and also Cython. What
> is better? Read below. :)  But for the impatient, I am going to use
> Cython, reasons below.

I recently tried a similar experiment with the opposite result :)

> So it's a lot slower.

This is very surprising indeed. I have just tried to run your code but
can't get the fortran generated module to work. But I wonder how you
got such a difference, because in my recent comparisons I generally
got about 0-2 % faster (i.e. negligible) results with gfortran
compared to gcc and a much bigger difference (in favour of fortran)
with the Intel compilers.

In fact for simple tests gcc and gfortran produce almost identical
assembler code (use -s switch to check). For the more mature compilers
(i.e. Intel) fortran is faster as it is inherently easier to optimize
(easier data dependency analysis and vectorisation).

The main reason I can think of is the gfortran version you are using
and the optimisation switches.

Any version 4.0.x is seriously suboptimal (first released version).
Big gains were made from 4.0-4.1 and 4.1-4.2. I would suggest 4.2 as
the minimum version to use. The development version 4.3 is faster
still (also for gcc) because of recent vectorisation work.

> When using g77 compiler instead of gfortran, I get a speed
> up 4.8 times.

The difference between g77 and gfortran with version 4.3 is now very
small, plus gfortran is going to benefit from all the new
vectorisation and optimisation work in gcc. It will not be long before
it over takes for good. In addition you can use modern fortran 95/2003
code which makes life much easier! In in addition to all that it is
very actively maintained and destined to be the default compiler on
all linux systems as it is bundled with gcc.

> So, what do you think of that?
>

If you are using a modern gfortran version then I suggest you have
found a bug, in which case I would take it to the gfortran mailing
list. If not, then try version 4.3 which can be downloaded from the
gfortran wiki: gcc.gnu.org/wiki/GFortran
In the mean time, after Christmas, I'll try and get your code working
and see for myself the difference.

Seasons greetings,
John

-- 
Telephone: (+44) (0) 7739 105209


From ondrej at certik.cz  Mon Dec 24 08:29:00 2007
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 24 Dec 2007 14:29:00 +0100
Subject: [Numpy-discussion] gfortran/g77+f2py vs gcc+Cython speed
	comparison
In-Reply-To: <3a1077e70712240350p21eab1aet49e02346f9bfc9c3@mail.gmail.com>
References: <85b5c3130712230457q26bc356ah168653168db0dab@mail.gmail.com>
	<3a1077e70712240350p21eab1aet49e02346f9bfc9c3@mail.gmail.com>
Message-ID: <85b5c3130712240529k63b65bf0r2da11d0903388ef0@mail.gmail.com>

Hi John,

thanks for the response.

> This is very surprising indeed. I have just tried to run your code but
> can't get the fortran generated module to work. But I wonder how you

Yes, I really didn't try to make it robust, I just made it work for me
on Debian unstable.

> got such a difference, because in my recent comparisons I generally
> got about 0-2 % faster (i.e. negligible) results with gfortran
> compared to gcc and a much bigger difference (in favour of fortran)
> with the Intel compilers.
>
> In fact for simple tests gcc and gfortran produce almost identical
> assembler code (use -s switch to check). For the more mature compilers
> (i.e. Intel) fortran is faster as it is inherently easier to optimize
> (easier data dependency analysis and vectorisation).
>
> The main reason I can think of is the gfortran version you are using
> and the optimisation switches.
>
> Any version 4.0.x is seriously suboptimal (first released version).
> Big gains were made from 4.0-4.1 and 4.1-4.2. I would suggest 4.2 as
> the minimum version to use. The development version 4.3 is faster
> still (also for gcc) because of recent vectorisation work.

I use:

$ gfortran --version
GNU Fortran (GCC) 4.2.3 20071123 (prerelease) (Debian 4.2.2-4)
Copyright (C) 2007 Free Software Foundation, Inc.

GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of GNU Fortran
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING

$


>
> > When using g77 compiler instead of gfortran, I get a speed
> > up 4.8 times.
>
> The difference between g77 and gfortran with version 4.3 is now very
> small, plus gfortran is going to benefit from all the new
> vectorisation and optimisation work in gcc. It will not be long before
> it over takes for good. In addition you can use modern fortran 95/2003
> code which makes life much easier! In in addition to all that it is
> very actively maintained and destined to be the default compiler on
> all linux systems as it is bundled with gcc.

Yes, I also think gfortran is the way to go, since g77 is dead upstream.
But currently, the g77 still gives quite a lot faster code than gfortran,
but I hope it will change.

>
> > So, what do you think of that?
> >
>
> If you are using a modern gfortran version then I suggest you have
> found a bug, in which case I would take it to the gfortran mailing
> list. If not, then try version 4.3 which can be downloaded from the
> gfortran wiki: gcc.gnu.org/wiki/GFortran
> In the mean time, after Christmas, I'll try and get your code working
> and see for myself the difference.

So your suggestion is to prepare a clean fortran example, that,
when compiled with gfortran, runs a lot slower, than if compiled with g77?
Yep, I can do that.

So what is your personal opinion about fortran vs C for scientific computing?
>From the social point of view, I much prefer C, since a lot more people
know it, but as you say, the good fortran compilers are a lot better, than
the C compilers. So I don't know. Probably depends on the project and
a code I want to reuse.

Ondrej


From jtravs at gmail.com  Mon Dec 24 09:37:29 2007
From: jtravs at gmail.com (John Travers)
Date: Mon, 24 Dec 2007 14:37:29 +0000
Subject: [Numpy-discussion] gfortran/g77+f2py vs gcc+Cython speed
	comparison
In-Reply-To: <85b5c3130712240529k63b65bf0r2da11d0903388ef0@mail.gmail.com>
References: <85b5c3130712230457q26bc356ah168653168db0dab@mail.gmail.com>
	<3a1077e70712240350p21eab1aet49e02346f9bfc9c3@mail.gmail.com>
	<85b5c3130712240529k63b65bf0r2da11d0903388ef0@mail.gmail.com>
Message-ID: <3a1077e70712240637v7602e1b8kcc051baca708bf9c@mail.gmail.com>

On Dec 24, 2007 1:29 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi John,
> > > When using g77 compiler instead of gfortran, I get a speed
> > > up 4.8 times.
> >
> > The difference between g77 and gfortran with version 4.3 is now very
> > small, plus gfortran is going to benefit from all the new
> > vectorisation and optimisation work in gcc. It will not be long before
> > it over takes for good. In addition you can use modern fortran 95/2003
> > code which makes life much easier! In in addition to all that it is
> > very actively maintained and destined to be the default compiler on
> > all linux systems as it is bundled with gcc.
>
> Yes, I also think gfortran is the way to go, since g77 is dead upstream.
> But currently, the g77 still gives quite a lot faster code than gfortran,
> but I hope it will change.

Yes, I just proved myself wrong (not a rare occurrence...), on my slow laptop:

With gcc (4.3.0 20071223 (experimental)):
real    0m3.272s
user    0m3.120s
sys     0m0.044s

With gfortran (4.3.0 20071223 (experimental)):
real    0m4.294s
user    0m3.976s
sys     0m0.020s

With g77 (3.4.6):
real    0m2.928s
user    0m2.812s
sys     0m0.044s

If g77=1.0 then gcc=1.27 and gfortran=1.41 (far from 4.8 times though!!)

This is in contrast to my recent experiments between gfortran and gcc,
though that code used f95 stuff extensively so was never tested
against g77.

> So your suggestion is to prepare a clean fortran example, that,
> when compiled with gfortran, runs a lot slower, than if compiled with g77?
> Yep, I can do that.

The most interesting will be to compile with g77 and gfortran with the
options to spit out the generated assembler code, then we can see
exactly where the difference is.  I'll dig back in the gfortran
mailing lists and see previous conclusions of the g77 vs gfortran
debate (which occur often). You may just be exercising a particularly
unoptimized part of gfortran.

> So what is your personal opinion about fortran vs C for scientific computing?
> >From the social point of view, I much prefer C, since a lot more people
> know it, but as you say, the good fortran compilers are a lot better, than
> the C compilers. So I don't know. Probably depends on the project and
> a code I want to reuse.

If you haven't guessed I'm pro fortran! My path was C++ -> Matlab ->
Scipy (which I still use for prototyping and short calculations) ->
fortran for big simulations. Of course, the wonderful f2py helps
combining the last two steps!

The turning point came when I discovered modern object orientated
fortran features and particularly the array intrinsics/operations in
fortran 95/2003. I realised that I could code my stuff in almost as
few lines as Matlab and Scipy and also have the fastest execution (if
using commercial compilers). My hope (slightly set back today) is that
gfortran will mature into being as fast/faster than gcc/g77 for most
code. In that case, with gcc on every linux system, then the social
opposition should disappear. Though this still will not help against
people who think f77 is fortran, ignoring the last 30 years of
development.

Anyway, I really must go home now...
John


From devnew at gmail.com  Tue Dec 25 09:47:29 2007
From: devnew at gmail.com (devnew at gmail.com)
Date: Tue, 25 Dec 2007 06:47:29 -0800 (PST)
Subject: [Numpy-discussion] matrix multipln takes too much time
Message-ID: <84f7ff0d-0f62-49c9-a060-4e70381ef676@d21g2000prf.googlegroups.com>

hi
i am doing some maths calculations involving matrices of double values
using numpy.matrix ,

java code for this is something like

int items=25;
int sample=5;
int totalcols=8100;
double[][]dblarrayone=new double[items][totalcols];
double[][]dblarraytwo=new double[items][totalcols];
//their elements are set elsewhere before calculation

double[][] resultarray = new double[items][sample];

for(int i=0;i<items;i++){
    for(int j=0;j<sample;j++){
        double tval=0.0;
        for(int p=0;p<totalcols;p++)
            tval+=dblarrayone[j][p] * dblarraytwo[i][p];
        resultarray[i][j]=Math.abs(tval);

    }

}

so I wanted to do the same  in python..(may be this code is not in the
recommended way..)
i am storing the filled matrices and other values as instance variable
of a class and access them by self.whatever...

self.items=25
self.sample=5
self.totalcols=8100
#self.matrixone,self.matrixtwo are numply matix objects with already
filled elements
#but for testing i filled it with zeros
self.matrixone=matrix(zeros((items,totalcols)))
self.matrixtwo=matrix(zeros((items,totalcols)))
resultmatrix=matrix(zeros((self.items,self.sample)))

for i in range(self.items):
    for j in range(self.sample):
         tval=0.0
         for p in range(self.totalcols):
            tval +=self.matrixone[ j , p ] * self.matrixtwo[ i , p ]
         resultmatrix[ i, j ]=abs(tval)


here I found that while the java code takes barely 110 milliseconds to
execute the code ,the
python code takes something like 53 secs to execute !!..I am baffled
by this ..can anyone advise me how i can improve this? (i want to code
in python so  I can't use c,c++ , java)

dn


From Louis.Wicker at noaa.gov  Tue Dec 25 11:16:49 2007
From: Louis.Wicker at noaa.gov (Louis Wicker)
Date: Tue, 25 Dec 2007 10:16:49 -0600
Subject: [Numpy-discussion] matrix multipln takes too much time
In-Reply-To: <84f7ff0d-0f62-49c9-a060-4e70381ef676@d21g2000prf.googlegroups.com>
References: <84f7ff0d-0f62-49c9-a060-4e70381ef676@d21g2000prf.googlegroups.com>
Message-ID: <24E24565-2BE9-4468-9F40-EF8C3C32F6E6@noaa.gov>

Hi there - quick suggestion on Xmas morning - others are much more  
familar.

You do not want to use a loop to do the matrix multiply, you want to  
use the intrinsic functions assoicated with matrix.

So you want something like

res = Math.abs( matmul(arrayone, arraytwo) )

note - that is not real code, just symbolic code.  I am sure this is  
easily found in the documentation.

cheers!

Lou


On Dec 25, 2007, at 8:47 AM, devnew at gmail.com wrote:

> hi
> i am doing some maths calculations involving matrices of double values
> using numpy.matrix ,
>
> java code for this is something like
>
> int items=25;
> int sample=5;
> int totalcols=8100;
> double[][]dblarrayone=new double[items][totalcols];
> double[][]dblarraytwo=new double[items][totalcols];
> //their elements are set elsewhere before calculation
>
> double[][] resultarray = new double[items][sample];
>
> for(int i=0;i<items;i++){
>    for(int j=0;j<sample;j++){
>        double tval=0.0;
>        for(int p=0;p<totalcols;p++)
>            tval+=dblarrayone[j][p] * dblarraytwo[i][p];
>        resultarray[i][j]=Math.abs(tval);
>
>    }
>
> }
>
> so I wanted to do the same  in python..(may be this code is not in the
> recommended way..)
> i am storing the filled matrices and other values as instance variable
> of a class and access them by self.whatever...
>
> self.items=25
> self.sample=5
> self.totalcols=8100
> #self.matrixone,self.matrixtwo are numply matix objects with already
> filled elements
> #but for testing i filled it with zeros
> self.matrixone=matrix(zeros((items,totalcols)))
> self.matrixtwo=matrix(zeros((items,totalcols)))
> resultmatrix=matrix(zeros((self.items,self.sample)))
>
> for i in range(self.items):
>    for j in range(self.sample):
>         tval=0.0
>         for p in range(self.totalcols):
>            tval +=self.matrixone[ j , p ] * self.matrixtwo[ i , p ]
>         resultmatrix[ i, j ]=abs(tval)
>
>
> here I found that while the java code takes barely 110 milliseconds to
> execute the code ,the
> python code takes something like 53 secs to execute !!..I am baffled
> by this ..can anyone advise me how i can improve this? (i want to code
> in python so  I can't use c,c++ , java)
>
> dn
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

----------------------------------------------------------------------------
| Dr. Louis J. Wicker
| NSSL/WRDD  Rm 4366
| National Weather Center
| 120 David L. Boren Boulevard, Norman, OK 73072
|
| E-mail:   Louis.Wicker at noaa.gov
| HTTP:  www.nssl.noaa.gov/~lwicker
| Phone:    (405) 325-6340
| Fax:        (405) 325-6780
|
| "Programming is not just creating strings of instructions
| for a computer to execute.  It's also 'literary' in that you
| are trying to communicate a program structure to
| other humans reading the code." - Paul Rubin
|
|"Real efficiency comes from elegant solutions, not optimized programs.
| Optimization is always just a few correctness-preserving  
transformations
| away." - Jonathan Sobel
----------------------------------------------------------------------------
|
| "The contents  of this message are mine personally and
| do not reflect any position of  the Government or NOAA."
|
----------------------------------------------------------------------------




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071225/489f3639/attachment.html>

From lbolla at gmail.com  Tue Dec 25 11:28:19 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Tue, 25 Dec 2007 17:28:19 +0100
Subject: [Numpy-discussion] matrix multipln takes too much time
In-Reply-To: <24E24565-2BE9-4468-9F40-EF8C3C32F6E6@noaa.gov>
References: <84f7ff0d-0f62-49c9-a060-4e70381ef676@d21g2000prf.googlegroups.com>
	<24E24565-2BE9-4468-9F40-EF8C3C32F6E6@noaa.gov>
Message-ID: <80c99e790712250828o417aa0d1uc3028240cf12b60e@mail.gmail.com>

you can either use matrix multiplication (see resultmatrix2) or tensordot
(see resultmatrix3).
on my computer I have:
1.    15.6 sec with your code
2.    0.072 sec with resultmatrix2
3.    0.040 sec with tensordot (resultmatrix3) (-- which is a 400x speed)

--------------------------------------------

from numpy import *

items=25
sample=5
totalcols=8100
#matrixone=matrix(zeros((items,totalcols)))
#matrixtwo=matrix(zeros((items,totalcols)))

matrixone=matrix(random.rand(items, totalcols))
matrixtwo=matrix(random.rand(items, totalcols))

resultmatrix=matrix(zeros((items,sample)))
resultmatrix2=matrix(zeros((items,sample)))
resultmatrix3=matrix(zeros((items,sample)))

# your code
for i in range(items):
    for j in range(sample):
        tval=0.0
        for p in range(totalcols):
            tval +=matrixone[ j , p ] * matrixtwo[ i , p ]
        resultmatrix[ i, j ]=abs(tval)

# matrix multiplication
for i in range(items):
    for j in range(sample):
        resultmatrix2[ i, j ] = abs(matrixone[j,:] * matrixtwo[i,:].T)

# tensordot
resulmatrix3 = tensordot(matrixone[:sample,:], matrixtwo.T, axes=1).T

---------------------------------------

hth,
L.






On Dec 25, 2007 5:16 PM, Louis Wicker <Louis.Wicker at noaa.gov> wrote:

> Hi there - quick suggestion on Xmas morning - others are much more
> familar.
> You do not want to use a loop to do the matrix multiply, you want to use
> the intrinsic functions assoicated with matrix.
>
> So you want something like
>
> res = Math.abs( matmul(arrayone, arraytwo) )
>
> note - that is not real code, just symbolic code.  I am sure this is
> easily found in the documentation.
>
> cheers!
>
> Lou
>
>
> On Dec 25, 2007, at 8:47 AM, devnew at gmail.c om <devnew at gmail.com> w
> le-interchange-newline">
>
> hi
> i am doing some maths calculations involving matrices of double values
> using numpy.matrix ,
>
> java code for this is something like
>
> int items=25;
> int sample=5;
> int totalcols=8100;
> double[][]dblarrayone=new double[items][totalcols];
> double[][]dblarraytwo=new double[items][totalcols];
> //their elements are set elsewhere before calculation
>
> double[][] resultarray = new double[items][sample];
>
> for(int i=0;i<items;i++){
>    for(int j=0;j<sample;j++){
>        double tval=0.0;
>        for(int p=0;p<totalcols;p++)
>            tval+=dblarrayone[j][p] * dblarraytwo[i][p];
>        resultarray[i][j]=Math.abs(tval);
>
>    }
>
> }
>
> so I wanted to do the same  in python ..(may b
> recommended way..)
> i am storing the filled matrices and other values as instance variable
> of a class and access them by self.whatever...
>
> self.items=25
> self.sample=5
> self.totalcols=8100
> #self.matrixone,self.matrixtwo are numply matix objects with already
> filled elements
> #but for testing i filled it with zeros
> self.matrixone=matrix(zeros((items,totalcols)))
> self.matrixtwo=matrix(zeros((items,totalcols)))
> resultmatrix=matrix(zeros((self.items,self.sample)))
>
> for i in range(self.items):
>    for j in range(self.sample):
>         tval=0.0
>         for p in range(self.totalcols):
>            tval +=self.matrixone[ j , p ] * self.matrixtwo[ i , p ]
>         resultmatrix[ i, j ]=abs(tval)
>
>
> here I found that while the java code takes ba br>execute the code ,the
> python code takes something like 53 secs to execute !!..I am baffled
> by this ..can anyone advise me how i can improve this? (i want to code
> in python so  I can't use c,c++ , java)
>
> dn
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> | Dr. Louis J. Wicker
> | NSSL/WRDD  Rm 4366
> | National Weather Center < "Apple-style-span" style="line-height: 16px;;
> font-size: 14px; ">
> | 120 David L. Boren Boulevard, Norman, OK 73072
> |
> | E-mail:   Louis.Wicker at noaa.gov
> | HTTP:  www.nssl.noaa.gov/~lwicker <http://www.nssl.noaa.gov/%7Elwicker>
> | Phone:    (405) 325-6340
> | Fax:        (405) 325-6780
> |
> | "Programming is not just creating strings of instructions
> | for a computer to execute.  It's also 'literary' in that you
> | are trying to communicate a program structure to
> | other humans reading the code." - Paul Rubin
> |
> |"Real efficiency comes from elegant solutions, not optimiz ed progr
> le="font-size: 14px; ">| Optimization is always just a few
> correctness-preserving transformations
> | away." - Jonathan Sobel
>
> ----------------------------------------------------------------------------
> |
> | "The contents  of this message are mine personally and
> | do not reflect any position of  the Government or NOAA."
> |
>
> ----------------------------------------------------------------------------
>
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071225/4267f296/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Tue Dec 25 23:30:18 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 26 Dec 2007 13:30:18 +0900
Subject: [Numpy-discussion] distutils_scons_command branch: 2nd step towards
 scons support in numpy
Message-ID: <4771D8DA.50803@ar.media.kyoto-u.ac.jp>

Hi,
      
    I have started publishing the distutils_scons_command branch in 
numpy svn repository. This implements all the distutils infrastructure 
necessary to call scons, by using the scons distutils command. In more 
details:
    - distutils setup files can ask for a different name for setup file: 
if the top setup file calls the Configuration __init__ function with 
setup_name argument = foo.py, it will recursively call foo.py instead of 
setup.py. This is useful to have several build configuration in 
parrallel (so that we can include setup.py files using the current build 
system, and setupscons.py files which use scons instead).
    - the scons distutils command is implemented. For every scons script 
file registered through the add_sconscript method, it will call scons. 
If at least one scons script is registered, it will need the numscons 
package. If no scons scripts files are registered, the command does nothing.

Once those code changes are integrated in numpy trunk, the last step for 
a working scons solution will be the release of numscons, which 
implements all the functionalities to use scons script files easily 
within numpy code. The plan is to be able to optionally build numpy 
1.0.5 using scons.

cheers,

David


From Bruce_Sherwood at ncsu.edu  Wed Dec 26 01:11:52 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Wed, 26 Dec 2007 01:11:52 -0500
Subject: [Numpy-discussion] Overloading sqrt(5.5)*myvector
In-Reply-To: <476AA418.4080108@ncsu.edu>
References: <476AA418.4080108@ncsu.edu>
Message-ID: <4771F0A8.8070304@ncsu.edu>

Sorry to repeat myself and be insistent, but could someone please at 
least comment on whether I'm doing anything obviously wrong, even if you 
don't immediately have a solution to my serious problem? There was no 
response to my question (see copy below) which I sent to both the numpy 
and Boost mailing lists.

To the numpy experts: Is there something wrong, or something I 
could/should change in how I'm trying to overload multiplication of a 
numpy square root (or other numpy function) times my own "vector" 
object? I'm seeing a huge performance hit in going from Numeric to numpy 
because Numeric sqrt returned float whereas numpy sqrt returns 
numpy.float64, so that the result is not one of my vector objects. I 
don't have a problem with myvector*sqrt(5.5).

Desperately,

Bruce Sherwood

-------------------
I'm not sure whether this is a Numpy problem or a Boost problem, so I'm 
posting to both communities. (I'm subscribed to both lists, but an 
attempt to post yesterday to this Boost list seems never have gotten to 
the archives, so I'm trying again. My apologies if this shows up twice 
here.)

In old Numeric, type(sqrt(5.5)) was float, but in numpy, type(sqrt(5.5)) 
is numpy.float64. This leads to a big performance hit in calculations in 
a beta version of VPython, using the VPython 3D "vector" class, compared 
with the old version that used Numeric (VPython is a 3D graphics module 
for Python; see vpython.org).

Operator overloading of the VPython vector class works fine for 
vector*sqrt(5.5) but not for sqrt(5.5)*vector. The following free 
function catches 5.5*vector but fails to catch sqrt(5.5)*vector, whose 
type ends up as numpy.ndarray instead of the desired vector, with 
concomitant slow conversions in later vector calculations:

inline vector
operator*( const double& s, const vector& v)
{ return vector( s*v.x, s*v.y, s*v.z); }

I've thrashed around on this, including trying to add this:

inline vector
operator*( const npy_float64& s, const vector& v)
{ return vector( s*v.x, s*v.y, s*v.z); }

But the compiler correctly complains that this is in conflict with the 
version of double*vector, since in fact npy_float64 is actually double.

It's interesting and presumably meaningful to the knowledgeable (not me) 
that vector*sqrt(5.5) yields a vector, even though the overloading 
speaks of double, not a specifically numpy name:

inline vector
operator*( const double s) const throw()
{ return vector( s*x, s*y, s*z); }

VPython uses Boost, and the glue concerning vectors includes the following:

py::class_<vector>("vector", py::init< py::optional<double, double, 
double> >())
     .def( self * double())
     .def( double() * self)

As far as I can understand from the Boost Python documentation, this is 
the proper way to specify the left-hand and right-hand overloadings. But 
do I have to add something like .def( npy_float64() * self)? Help would 
be much appreciated.

Bruce Sherwood





From devnew at gmail.com  Wed Dec 26 02:36:06 2007
From: devnew at gmail.com (devnew at gmail.com)
Date: Tue, 25 Dec 2007 23:36:06 -0800 (PST)
Subject: [Numpy-discussion] matrix multipln takes too much time
In-Reply-To: <80c99e790712250828o417aa0d1uc3028240cf12b60e@mail.gmail.com>
References: <84f7ff0d-0f62-49c9-a060-4e70381ef676@d21g2000prf.googlegroups.com>
	<24E24565-2BE9-4468-9F40-EF8C3C32F6E6@noaa.gov>
	<80c99e790712250828o417aa0d1uc3028240cf12b60e@mail.gmail.com>
Message-ID: <c8438818-21c0-4f3b-bf0e-78d324dd59cf@i29g2000prf.googlegroups.com>


> on my computer I have:
> 1.    15.6 sec with your code
> 2.    0.072 sec with resultmatrix2
> 3.    0.040 sec with tensordot (resultmatrix3) (-- which is a 400x speed)


wow ,thanks!
the tensordot fn is blinding fast..
i added /modified
 resultndarray = tensordot(matrixone[:sample,:], matrixtwo.T,
axes=1).T
resultmatrix =matrix(abs(resultndarray))

so i can get rid of the negative values in a real case

..your replies helped a lot guys,thanx a lot and happy X'Mas

dn


From gruben at bigpond.net.au  Wed Dec 26 05:49:38 2007
From: gruben at bigpond.net.au (Gary Ruben)
Date: Wed, 26 Dec 2007 21:49:38 +1100
Subject: [Numpy-discussion] Overloading sqrt(5.5)*myvector
In-Reply-To: <4771F0A8.8070304@ncsu.edu>
References: <476AA418.4080108@ncsu.edu> <4771F0A8.8070304@ncsu.edu>
Message-ID: <477231C2.2090305@bigpond.net.au>

Sorry this isn't an answer, just noise, but for those here who don't 
know, Bruce is the chief maintainer of the vpython project. I have found 
vpython aka the visual module to be a highly attractive and useful 
module for teaching physics. It would be great if someone with Boost 
experience would try to help him out. I wouldn't want him to get falsely 
disillusioned with this list as I for one have been looking forward to a 
fully numpy-compatible version of vpython.

Gary R.

Bruce Sherwood wrote:
> Sorry to repeat myself and be insistent, but could someone please at 
> least comment on whether I'm doing anything obviously wrong, even if you 
> don't immediately have a solution to my serious problem? There was no 
> response to my question (see copy below) which I sent to both the numpy 
> and Boost mailing lists.
> 
> To the numpy experts: Is there something wrong, or something I 
> could/should change in how I'm trying to overload multiplication of a 
> numpy square root (or other numpy function) times my own "vector" 
> object? I'm seeing a huge performance hit in going from Numeric to numpy 
> because Numeric sqrt returned float whereas numpy sqrt returns 
> numpy.float64, so that the result is not one of my vector objects. I 
> don't have a problem with myvector*sqrt(5.5).
> 
> Desperately,
> 
> Bruce Sherwood



From charlesr.harris at gmail.com  Wed Dec 26 14:16:58 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 26 Dec 2007 12:16:58 -0700
Subject: [Numpy-discussion] Overloading sqrt(5.5)*myvector
In-Reply-To: <477231C2.2090305@bigpond.net.au>
References: <476AA418.4080108@ncsu.edu> <4771F0A8.8070304@ncsu.edu>
	<477231C2.2090305@bigpond.net.au>
Message-ID: <e06186140712261116h40cc5c1dq1e2646c3d904e6c7@mail.gmail.com>

On Dec 26, 2007 3:49 AM, Gary Ruben <gruben at bigpond.net.au> wrote:

> Sorry this isn't an answer, just noise, but for those here who don't
> know, Bruce is the chief maintainer of the vpython project. I have found
> vpython aka the visual module to be a highly attractive and useful
> module for teaching physics. It would be great if someone with Boost
> experience would try to help him out. I wouldn't want him to get falsely
> disillusioned with this list as I for one have been looking forward to a
> fully numpy-compatible version of vpython.
>

I think the problem is that few of us are that familiar with boost/python. I
have used it myself, but only for interfacing  C++ classes or accessing
Numpy arrays through their buffer interface. I always avoided the Numeric
machinery because it looked clumsy and inefficient to me, I didn't want an
embedded version of Numeric (Numarray, Numpy), I wanted speed. Anyway, I
think numpy.float64 hides a normal C double, so the slowdown probably comes
from the boost/python machinery. I don't think boost/python was ever updated
to use Numpy in particular and the Numpy data type may be throwing it
through an interpretive loop. The speed of double vs float arithmetic on
Intel hardware is pretty much a wash, so should not show much difference
otherwise. You can get a float32 result from sqrt

In [2]: type(sqrt(float32(5.5)))
Out[2]: <type 'numpy.float32'>

But I suspect it will make little difference, probably what is needed is the
corresponding C/Python data type. I think that information is available
somewhere, but am not sure of the details. Someone else can probably help
you there (Travis?)

Chuck

<snip>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071226/8475f2ac/attachment.html>

From myeates at jpl.nasa.gov  Wed Dec 26 15:22:23 2007
From: myeates at jpl.nasa.gov (Mathew Yeates)
Date: Wed, 26 Dec 2007 12:22:23 -0800
Subject: [Numpy-discussion] how do I list all combinations
Message-ID: <4772B7FF.3060602@jpl.nasa.gov>

Hi
I've been looking at "fromfunction" and itertools but I'm flummoxed.

I have an arbitrary number of lists. I want to form all possible 
combinations from all lists. So if
r1=["dog","cat"]
r2=[1,2]

I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]

It's obvious when the number of lists is not arbitrary. But what if 
thats not known until runtime?

Mathew




From kwgoodman at gmail.com  Wed Dec 26 15:45:34 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 26 Dec 2007 12:45:34 -0800
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <4772B7FF.3060602@jpl.nasa.gov>
References: <4772B7FF.3060602@jpl.nasa.gov>
Message-ID: <f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>

On Dec 26, 2007 12:22 PM, Mathew Yeates <myeates at jpl.nasa.gov> wrote:
> I have an arbitrary number of lists. I want to form all possible
> combinations from all lists. So if
> r1=["dog","cat"]
> r2=[1,2]
>
> I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]
>
> It's obvious when the number of lists is not arbitrary. But what if
> thats not known until runtime?

Would this work?

Make a function that takes two inputs (a list of lists and a list) and
returns a list of lists that contains all possible combinations.
Iterate through all lists by calling the function with the output of
the previous call (a list of lists) and the next list.


From charlesr.harris at gmail.com  Wed Dec 26 15:52:27 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 26 Dec 2007 13:52:27 -0700
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <4772B7FF.3060602@jpl.nasa.gov>
References: <4772B7FF.3060602@jpl.nasa.gov>
Message-ID: <e06186140712261252u6e580b1ej80a74e05f34d3ba0@mail.gmail.com>

On Dec 26, 2007 1:22 PM, Mathew Yeates <myeates at jpl.nasa.gov> wrote:

> Hi
> I've been looking at "fromfunction" and itertools but I'm flummoxed.
>
> I have an arbitrary number of lists. I want to form all possible
> combinations from all lists. So if
> r1=["dog","cat"]
> r2=[1,2]
>
> I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]
>
> It's obvious when the number of lists is not arbitrary. But what if
> thats not known until runtime?


It's a mixed radix counter. Emulate the usual add and carry with a list of
digits using the length of the corresponding list as the radix at that
position. Sorta like an abacus, but with different numbers of beads in each
position.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071226/986514cb/attachment.html>

From rbastian at free.fr  Wed Dec 26 16:08:21 2007
From: rbastian at free.fr (=?iso-8859-1?q?Ren=E9=20Bastian?=)
Date: Wed, 26 Dec 2007 22:08:21 +0100
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <4772B7FF.3060602@jpl.nasa.gov>
References: <4772B7FF.3060602@jpl.nasa.gov>
Message-ID: <07122622082100.00754@rbastian>

Le Mercredi 26 D?cembre 2007 21:22, Mathew Yeates a ?crit :
> Hi
> I've been looking at "fromfunction" and itertools but I'm flummoxed.
>
> I have an arbitrary number of lists. I want to form all possible
> combinations from all lists. So if
> r1=["dog","cat"]
> r2=[1,2]
>
> I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]
>
> It's obvious when the number of lists is not arbitrary. But what if
> thats not known until runtime?
>
> Mathew

In the 'Reference Manual' of Python there is an example.

>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

-- 
Ren? Bastian
http://www.musiques-rb.org
http://pythoneon.musiques-rb.org
http://divergences.be  ?dition du 15.11.07 : musique




From sdb at cloud9.net  Wed Dec 26 16:03:59 2007
From: sdb at cloud9.net (Stuart Brorson)
Date: Wed, 26 Dec 2007 16:03:59 -0500 (EST)
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>
References: <4772B7FF.3060602@jpl.nasa.gov>
	<f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>
Message-ID: <Pine.BSF.4.64.0712261600350.36523@earl-grey.cloud9.net>

Hi --

>> I have an arbitrary number of lists. I want to form all possible
>> combinations from all lists. So if
>> r1=["dog","cat"]
>> r2=[1,2]
>>
>> I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]

Try this:

In [25]: [(x, y) for x in r1 for y in r2]
Out[25]: [('dog', 1), ('dog', 2), ('cat', 1), ('cat', 2)]


Cheers,

Stuart Brorson
Interactive Supercomputing, inc.
135 Beaver Street | Waltham | MA | 02452 | USA
http://www.interactivesupercomputing.com/


From myeates at jpl.nasa.gov  Wed Dec 26 16:04:36 2007
From: myeates at jpl.nasa.gov (Mathew Yeates)
Date: Wed, 26 Dec 2007 13:04:36 -0800
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>
References: <4772B7FF.3060602@jpl.nasa.gov>
	<f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>
Message-ID: <4772C1E4.5080006@jpl.nasa.gov>

yes, I came up with this and may use it. Seems like it would be insanely 
slow but my problem is small enough that it might be okay.

Thanks


Keith Goodman wrote:
> On Dec 26, 2007 12:22 PM, Mathew Yeates <myeates at jpl.nasa.gov> wrote:
>   
>> I have an arbitrary number of lists. I want to form all possible
>> combinations from all lists. So if
>> r1=["dog","cat"]
>> r2=[1,2]
>>
>> I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]
>>
>> It's obvious when the number of lists is not arbitrary. But what if
>> thats not known until runtime?
>>     
>
> Would this work?
>
> Make a function that takes two inputs (a list of lists and a list) and
> returns a list of lists that contains all possible combinations.
> Iterate through all lists by calling the function with the output of
> the previous call (a list of lists) and the next list.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   




From myeates at jpl.nasa.gov  Wed Dec 26 16:05:22 2007
From: myeates at jpl.nasa.gov (Mathew Yeates)
Date: Wed, 26 Dec 2007 13:05:22 -0800
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <07122622082100.00754@rbastian>
References: <4772B7FF.3060602@jpl.nasa.gov> <07122622082100.00754@rbastian>
Message-ID: <4772C212.7000102@jpl.nasa.gov>

Which reference manual?

Ren? Bastian wrote:
> Le Mercredi 26 D?cembre 2007 21:22, Mathew Yeates a ?crit :
>   
>> Hi
>> I've been looking at "fromfunction" and itertools but I'm flummoxed.
>>
>> I have an arbitrary number of lists. I want to form all possible
>> combinations from all lists. So if
>> r1=["dog","cat"]
>> r2=[1,2]
>>
>> I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]
>>
>> It's obvious when the number of lists is not arbitrary. But what if
>> thats not known until runtime?
>>
>> Mathew
>>     
>
> In the 'Reference Manual' of Python there is an example.
>
>   
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>     
>
>   




From charlesr.harris at gmail.com  Wed Dec 26 16:30:43 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 26 Dec 2007 14:30:43 -0700
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>
References: <4772B7FF.3060602@jpl.nasa.gov>
	<f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>
Message-ID: <e06186140712261330o10710091ye88bad148251299b@mail.gmail.com>

On Dec 26, 2007 1:45 PM, Keith Goodman <kwgoodman at gmail.com> wrote:

> On Dec 26, 2007 12:22 PM, Mathew Yeates <myeates at jpl.nasa.gov> wrote:
> > I have an arbitrary number of lists. I want to form all possible
> > combinations from all lists. So if
> > r1=["dog","cat"]
> > r2=[1,2]
> >
> > I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]
> >
> > It's obvious when the number of lists is not arbitrary. But what if
> > thats not known until runtime?
>
> Would this work?
>
> Make a function that takes two inputs (a list of lists and a list) and
> returns a list of lists that contains all possible combinations.
> Iterate through all lists by calling the function with the output of
> the previous call (a list of lists) and the next list.
> ____
>

Yeah, you can do it with recursion, but I don't think it would be quite as
efficient. An example of the explicit approach, define the following
generator:

def count(listoflists) :
    counter = [i[0] for i in listoflists]
    maxdigit = [len(i) - 1 for i in listoflists]
    yield counter
    while True :
        i = 0;
        while i < len(counter) and counter[i] == maxdigit[i] :
            counter[i] = 0
            i += 1
        if i < len(counter) :
            counter[i] += 1
            yield counter
        else :
            return



In [30]: a = ['dog', 'cat', 'horse']

In [31]: b = ['red', 'black']

In [32]: c = [a,b]

In [33]: for i in count.count(c) : print [c[j][i[j]] for j in range(len(c))]
   ....:
['dog', 'red']
['cat', 'red']
['horse', 'red']
['dog', 'black']
['cat', 'black']
['horse', 'black']

___________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071226/5806773c/attachment.html>

From charlesr.harris at gmail.com  Wed Dec 26 16:37:34 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 26 Dec 2007 14:37:34 -0700
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <e06186140712261330o10710091ye88bad148251299b@mail.gmail.com>
References: <4772B7FF.3060602@jpl.nasa.gov>
	<f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>
	<e06186140712261330o10710091ye88bad148251299b@mail.gmail.com>
Message-ID: <e06186140712261337u5e789769s882f77c9919849e4@mail.gmail.com>

On Dec 26, 2007 2:30 PM, Charles R Harris <charlesr.harris at gmail.com> wrote:

>
>
> On Dec 26, 2007 1:45 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>
> > On Dec 26, 2007 12:22 PM, Mathew Yeates <myeates at jpl.nasa.gov> wrote:
> > > I have an arbitrary number of lists. I want to form all possible
> > > combinations from all lists. So if
> > > r1=["dog","cat"]
> > > r2=[1,2]
> > >
> > > I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]
> > >
> > > It's obvious when the number of lists is not arbitrary. But what if
> > > thats not known until runtime?
> >
> > Would this work?
> >
> > Make a function that takes two inputs (a list of lists and a list) and
> > returns a list of lists that contains all possible combinations.
> > Iterate through all lists by calling the function with the output of
> > the previous call (a list of lists) and the next list.
> > ____
> >
>
> Yeah, you can do it with recursion, but I don't think it would be quite as
> efficient. An example of the explicit approach, define the following
> generator:
>
> def count(listoflists) :
>     counter = [i[0] for i in listoflists]
>

Make that counter = [0 for i in listoflists]. That bug slipped in going from
[0]*len(listoflists).

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071226/331007e8/attachment.html>

From myeates at jpl.nasa.gov  Wed Dec 26 17:19:56 2007
From: myeates at jpl.nasa.gov (Mathew Yeates)
Date: Wed, 26 Dec 2007 14:19:56 -0800
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <e06186140712261337u5e789769s882f77c9919849e4@mail.gmail.com>
References: <4772B7FF.3060602@jpl.nasa.gov>	<f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>	<e06186140712261330o10710091ye88bad148251299b@mail.gmail.com>
	<e06186140712261337u5e789769s882f77c9919849e4@mail.gmail.com>
Message-ID: <4772D38C.7040502@jpl.nasa.gov>

Thanks Chuck.

Charles R Harris wrote:
>
>
> On Dec 26, 2007 2:30 PM, Charles R Harris <charlesr.harris at gmail.com 
> <mailto:charlesr.harris at gmail.com>> wrote:
>
>
>
>     On Dec 26, 2007 1:45 PM, Keith Goodman <kwgoodman at gmail.com
>     <mailto:kwgoodman at gmail.com>> wrote:
>
>         On Dec 26, 2007 12:22 PM, Mathew Yeates <myeates at jpl.nasa.gov
>         <mailto:myeates at jpl.nasa.gov>> wrote:
>         > I have an arbitrary number of lists. I want to form all possible
>         > combinations from all lists. So if
>         > r1=["dog","cat"]
>         > r2=[1,2]
>         >
>         > I want to return [["dog",1],["dog",2],["cat",1],["cat",2]]
>         >
>         > It's obvious when the number of lists is not arbitrary. But
>         what if
>         > thats not known until runtime?
>
>         Would this work?
>
>         Make a function that takes two inputs (a list of lists and a
>         list) and
>         returns a list of lists that contains all possible combinations.
>         Iterate through all lists by calling the function with the
>         output of
>         the previous call (a list of lists) and the next list.
>         ____
>
>
>     Yeah, you can do it with recursion, but I don't think it would be
>     quite as efficient. An example of the explicit approach, define
>     the following generator:
>
>     def count(listoflists) :
>         counter = [i[0] for i in listoflists]
>
>
> Make that counter = [0 for i in listoflists]. That bug slipped in 
> going from [0]*len(listoflists).
>
> Chuck
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   




From tim.hochberg at ieee.org  Wed Dec 26 17:48:18 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 26 Dec 2007 15:48:18 -0700
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <4772D38C.7040502@jpl.nasa.gov>
References: <4772B7FF.3060602@jpl.nasa.gov>
	<f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>
	<e06186140712261330o10710091ye88bad148251299b@mail.gmail.com>
	<e06186140712261337u5e789769s882f77c9919849e4@mail.gmail.com>
	<4772D38C.7040502@jpl.nasa.gov>
Message-ID: <e4412d6b0712261448r7653790bhdb04307629dc415a@mail.gmail.com>

Here's a baroque way to do it using generated code:

    def cg_combinations(seqs):
        n = len(seqs)
        chunks = ["def f(%s):" % ', '.join('s%s' % i for i in range(n))]
        for i in reversed(range(n)):
            chunks.append(" " * (n -i) + "for x%s in s%s:" % (i, i))
        chunks.append(" " * n + " yield (" + ', '.join('x%s' % i for i in
range(n)) + ')')
        code = '\n'.join(chunks)
        exec code
        return f(*seqs)

It should be reasonably fast, if non-obvious. I've included a version with
some timing and testing against Chucks version below. Enjoy.

(Also, it can be simplified slightly, but I wanted to generate in the same
order as Chuck for comparison purposes).


======================================================================


    def count(seqs) :
        counter = [0 for i in seqs]
        maxdigit = [len(i) - 1 for i in seqs]
        yield counter
        while True :
            i = 0;
            while i < len(counter) and counter[i] >= maxdigit[i] :
                counter[i] = 0
                i += 1
            if i < len(counter) :
                counter[i] += 1
                yield counter
            else :
                return

    def count_combinations(seqs):
        for c in count(seqs):
            yield tuple(s[i] for (s, i) in zip(seqs, c))

    def cg_combinations(seqs):
        n = len(seqs)
        chunks = ["def f(%s):" % ', '.join('s%s' % i for i in range(n))]
        for i in reversed(range(n)):
            chunks.append(" " * (n -i) + "for x%s in s%s:" % (i, i))
        chunks.append(" " * n + " yield (" + ', '.join('x%s' % i for i in
range(n)) + ')')
        code = '\n'.join(chunks)
        exec code
        return f(*seqs)

    a = "abcde"*10
    b = range(99)
    c = [x**2 for x in range(33)]
    seqs = [a, b, c]

    if __name__ == "__main__":
        assert list(count_combinations(seqs)) == list(cg_combinations(seqs))

        import timeit
        test = timeit.Timer('list(count_combinations(seqs))',
               'from __main__ import count_combinations, seqs')
        t1 = test.timeit(number=10)
        test = timeit.Timer('list(cg_combinations(seqs))',
               'from __main__ import cg_combinations, seqs')
        t2 = test.timeit(number=10)
        print t1, t2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071226/b7ef6f25/attachment.html>

From zpincus at stanford.edu  Wed Dec 26 18:23:40 2007
From: zpincus at stanford.edu (Zachary Pincus)
Date: Wed, 26 Dec 2007 18:23:40 -0500
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <e4412d6b0712261448r7653790bhdb04307629dc415a@mail.gmail.com>
References: <4772B7FF.3060602@jpl.nasa.gov>
	<f4f93d420712261245r420cff8g4e6692dfce434bed@mail.gmail.com>
	<e06186140712261330o10710091ye88bad148251299b@mail.gmail.com>
	<e06186140712261337u5e789769s882f77c9919849e4@mail.gmail.com>
	<4772D38C.7040502@jpl.nasa.gov>
	<e4412d6b0712261448r7653790bhdb04307629dc415a@mail.gmail.com>
Message-ID: <E3BF1369-8A88-46FF-A694-15AE0B62980F@stanford.edu>

Hi all,

I use numpy's own ndindex() for tasks like these:

> In: numpy.ndindex?
> Type:           type
> Base Class:     <type 'type'>
> String Form:    <class 'numpy.lib.index_tricks.ndindex'>
> Namespace:      Interactive
> File:           /Library/Frameworks/Python.framework/Versions/2.4/ 
> lib/python2.4/site-packages/numpy/lib/index_tricks.py
> Docstring:
>     Pass in a sequence of integers corresponding
>     to the number of dimensions in the counter.  This iterator
>     will then return an N-dimensional counter.
>
>     Example:
>     >>> for index in ndindex(3,2,1):
>     ...     print index
>     (0, 0, 0)
>     (0, 1, 0)
>     (1, 0, 0)
>     (1, 1, 0)
>     (2, 0, 0)
>     (2, 1, 0)

Here's a combination function using numpy.ndindex:

def combinations(*lists):
   lens = [len(l) for l in lists]
   for indices in numpy.ndindex(*lens):
     yield [l[i] for l, i in zip(lists, indices)]

r1=["dog","cat"]
r2=[1,2]
list(combinations(r1, r2))

Out: [['dog', 1], ['dog', 2], ['cat', 1], ['cat', 2]]

Or you could use 'map' and 'operator.getitem', which might be faster(?):
def combinations(*lists):
   lens = [len(l) for l in lists]
   for indices in numpy.ndindex(*lens):
     yield map(operator.getitem, lists, indices)


In the python cookbook, there are numerous similar recipes for  
generating permutations, combinations, and the like.


Zach Pincus

Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine




On Dec 26, 2007, at 5:48 PM, Timothy Hochberg wrote:

>
> Here's a baroque way to do it using generated code:
>
>     def cg_combinations(seqs):
>         n = len(seqs)
>         chunks = ["def f(%s):" % ', '.join('s%s' % i for i in range 
> (n))]
>         for i in reversed(range(n)):
>             chunks.append(" " * (n -i) + "for x%s in s%s:" % (i, i))
>         chunks.append(" " * n + " yield (" + ', '.join('x%s' % i  
> for i in range(n)) + ')')
>         code = '\n'.join(chunks)
>         exec code
>         return f(*seqs)
>
> It should be reasonably fast, if non-obvious. I've included a  
> version with some timing and testing against Chucks version below.  
> Enjoy.
>
> (Also, it can be simplified slightly, but I wanted to generate in  
> the same order as Chuck for comparison purposes).
>
>
> ======================================================================
>
>
>     def count(seqs) :
>         counter = [0 for i in seqs]
>         maxdigit = [len(i) - 1 for i in seqs]
>         yield counter
>         while True :
>             i = 0;
>             while i < len(counter) and counter[i] >= maxdigit[i] :
>                 counter[i] = 0
>                 i += 1
>             if i < len(counter) :
>                 counter[i] += 1
>                 yield counter
>             else :
>                 return
>
>     def count_combinations(seqs):
>         for c in count(seqs):
>             yield tuple(s[i] for (s, i) in zip(seqs, c))
>
>     def cg_combinations(seqs):
>         n = len(seqs)
>         chunks = ["def f(%s):" % ', '.join('s%s' % i for i in range 
> (n))]
>         for i in reversed(range(n)):
>             chunks.append(" " * (n -i) + "for x%s in s%s:" % (i, i))
>         chunks.append(" " * n + " yield (" + ', '.join('x%s' % i  
> for i in range(n)) + ')')
>         code = '\n'.join(chunks)
>         exec code
>         return f(*seqs)
>
>     a = "abcde"*10
>     b = range(99)
>     c = [x**2 for x in range(33)]
>     seqs = [a, b, c]
>
>     if __name__ == "__main__":
>         assert list(count_combinations(seqs)) == list 
> (cg_combinations(seqs))
>
>         import timeit
>         test = timeit.Timer('list(count_combinations(seqs))',
>                'from __main__ import count_combinations, seqs')
>         t1 = test.timeit(number=10)
>         test = timeit.Timer('list(cg_combinations(seqs))',
>                'from __main__ import cg_combinations, seqs')
>         t2 = test.timeit(number=10)
>         print t1, t2
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From Bruce_Sherwood at ncsu.edu  Wed Dec 26 23:35:16 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Wed, 26 Dec 2007 23:35:16 -0500
Subject: [Numpy-discussion] [C++-sig] Overloading sqrt(5.5)*myvector
In-Reply-To: <7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>
References: <4771F0AE.40406@ncsu.edu>
	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>
Message-ID: <47732B84.70400@ncsu.edu>

Thanks for the suggestion. It hadn't occurred to me to try to override 
numpy as you suggest. However, when I try the code shown below as the 
start of a test of this scheme, I get the following error:

Traceback (most recent call last):
  File "C:\Documents and Settings\Bruce\My 
Documents\0VPythonWork\vectors.py", line 24, in <module>
    numpy.float64.__mul__ = new_mul
TypeError: can't set attributes of built-in/extension type 'numpy.float64'

I'm copying this to the numpy discussion list, as maybe someone there 
will see where to go starting from your suggestion.

Bruce Sherwood

---------------
import numpy

class vector(object):
    def __init__(self,x,y,z):
        self.data = [x,y,z]
    def __mul__(self,other):
        return vector(other*self.data[0],
                     other*self.data[1],
                     other*self.data[2])
    __rmul__ = __mul__

    def show(self):
        print self.data
       
old_mul = numpy.float64.__mul__

def new_mul( self, other ):
    if isinstance( other, vector ):
       return other*self
    else:
       return old_mul( self, other )

numpy.float64.__mul__ = new_mul

a = vector(1,2,3)
a.show()
b = 5*a
b.show()
c = a*7
c.show()


Roman Yakovenko wrote:
> On Dec 26, 2007 8:11 AM, Bruce Sherwood <Bruce_Sherwood at ncsu.edu> wrote:
>   
>> Sorry to repeat myself and be insistent, but could someone please at
>> least comment on whether I'm doing anything obviously wrong, even if you
>> don't immediately have a solution to my serious problem? There was no
>> response to my question (see copy below) which I sent to both the numpy
>> and Boost mailing lists.
>>
>> To the Boost experts: Is there something wrong, or something I
>> could/should change in how I'm trying to define to Boost the overloaded
>> multiplication of a numpy square root (or other numpy function) times my
>> own "vector" object? I'm seeing a huge performance hit in going from
>> Numeric to numpy because Numeric sqrt returned float whereas numpy sqrt
>> returns numpy.float64, so that the result is not one of my vector
>> objects. I don't have a problem with myvector*sqrt(5.5). Here is what I
>> currently am doing:
>>
>> py::class_<vector>("vector", py::init< py::optional<double, double,
>> double> >())
>>      .def( self * double())
>>      .def( double() * self)
>>
>> Desperately,
>>     
>
> If I understand you right, than you can do something like this:
>
> replace __mul__ method of numpy.float64 class:
>
> old_mul = numpy.float64.__mul__
>
> def new_mul( self, other ):
>     if other isinstance( vector ):
>        return other*self
>     else:
>        return old_mul( self, other )
>
> numpy.float64.__mul__ = new_mul
>
> HTH
>
>   


From robert.kern at gmail.com  Wed Dec 26 23:45:57 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 26 Dec 2007 23:45:57 -0500
Subject: [Numpy-discussion] Overloading sqrt(5.5)*myvector
In-Reply-To: <477231C2.2090305@bigpond.net.au>
References: <476AA418.4080108@ncsu.edu> <4771F0A8.8070304@ncsu.edu>
	<477231C2.2090305@bigpond.net.au>
Message-ID: <47732E05.9000308@gmail.com>

Gary Ruben wrote:
> Sorry this isn't an answer, just noise, but for those here who don't 
> know, Bruce is the chief maintainer of the vpython project. I have found 
> vpython aka the visual module to be a highly attractive and useful 
> module for teaching physics. It would be great if someone with Boost 
> experience would try to help him out. I wouldn't want him to get falsely 
> disillusioned with this list as I for one have been looking forward to a 
> fully numpy-compatible version of vpython.

Please keep in mind that for many of us, this is the holiday season and we are
on vacation. While I'm happy to check the list and give answers that are at the
front of my head, deeper answers that require exploration or experimentation are
beyond my available time. I'm sure others are in a similar situation.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Dec 27 00:04:29 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 27 Dec 2007 00:04:29 -0500
Subject: [Numpy-discussion] [C++-sig] Overloading sqrt(5.5)*myvector
In-Reply-To: <47732B84.70400@ncsu.edu>
References: <4771F0AE.40406@ncsu.edu>	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>
	<47732B84.70400@ncsu.edu>
Message-ID: <4773325D.3040907@gmail.com>

Bruce Sherwood wrote:
> Thanks for the suggestion. It hadn't occurred to me to try to override 
> numpy as you suggest. However, when I try the code shown below as the 
> start of a test of this scheme, I get the following error:
> 
> Traceback (most recent call last):
>   File "C:\Documents and Settings\Bruce\My 
> Documents\0VPythonWork\vectors.py", line 24, in <module>
>     numpy.float64.__mul__ = new_mul
> TypeError: can't set attributes of built-in/extension type 'numpy.float64'
> 
> I'm copying this to the numpy discussion list, as maybe someone there 
> will see where to go starting from your suggestion.

Like most (or all) builtin-types, the numpy float scalars do not permit
replacing their methods from Python.

I'm not familiar with vpython's vector. If you can make it "not look like an
ndarray", then you should be able to just implement __rmul__ on vector.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Bruce_Sherwood at ncsu.edu  Thu Dec 27 09:58:31 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Thu, 27 Dec 2007 09:58:31 -0500
Subject: [Numpy-discussion] [C++-sig] Overloading sqrt(5.5)*myvector
In-Reply-To: <4773325D.3040907@gmail.com>
References: <4771F0AE.40406@ncsu.edu>	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>	<47732B84.70400@ncsu.edu>
	<4773325D.3040907@gmail.com>
Message-ID: <4773BD97.1000703@ncsu.edu>

Thanks for the comment, which limits the range of possible solutions. 
The VPython vector class is implemented in C++, not in Python. I made up 
the simple test in my previous note to try out the solution that had 
been offered and which you have usefully ruled out. Here is the relevant 
part of the vector class, which indeed doesn't look like an ndarray:

inline vector
operator*( const double s) const throw()
{ return vector( s*x, s*y, s*z); }

and here is the free function for right multiplication:

inline vector
operator*( const double& s, const vector& v)
{ return vector( s*v.x, s*v.y, s*v.z); }

Maybe the unsolvable problem is in the Boost definitions:

py::class_<vector>("vector", py::init< py::optional<double, double, 
double> >())
     .def( self * double())
     .def( double() * self)

Left multiplication is fine, but right multiplication isn't.

Bruce Sherwood

Robert Kern wrote:
> Bruce Sherwood wrote:
>   
>> Thanks for the suggestion. It hadn't occurred to me to try to override 
>> numpy as you suggest. However, when I try the code shown below as the 
>> start of a test of this scheme, I get the following error:
>>
>> Traceback (most recent call last):
>>   File "C:\Documents and Settings\Bruce\My 
>> Documents\0VPythonWork\vectors.py", line 24, in <module>
>>     numpy.float64.__mul__ = new_mul
>> TypeError: can't set attributes of built-in/extension type 'numpy.float64'
>>
>> I'm copying this to the numpy discussion list, as maybe someone there 
>> will see where to go starting from your suggestion.
>>     
>
> Like most (or all) builtin-types, the numpy float scalars do not permit
> replacing their methods from Python.
>
> I'm not familiar with vpython's vector. If you can make it "not look like an
> ndarray", then you should be able to just implement __rmul__ on vector.
>
>   


From Bruce_Sherwood at ncsu.edu  Thu Dec 27 10:15:07 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Thu, 27 Dec 2007 10:15:07 -0500
Subject: [Numpy-discussion] [C++-sig] Overloading sqrt(5.5)*myvector
In-Reply-To: <4773BD97.1000703@ncsu.edu>
References: <4771F0AE.40406@ncsu.edu>	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>	<47732B84.70400@ncsu.edu>	<4773325D.3040907@gmail.com>
	<4773BD97.1000703@ncsu.edu>
Message-ID: <4773C17B.4050802@ncsu.edu>

I should have added: This structure worked with the older version of 
VPython which used Numeric, but it doesn't work in the beta version 
which uses numpy. Since I don't know enough about either numpy or Boost, 
I'm left guessing which subsystem is the source of my difficulties, and 
clueless about how to remedy them.

Bruce Sherwood

Bruce Sherwood wrote:
> Thanks for the comment, which limits the range of possible solutions. 
> The VPython vector class is implemented in C++, not in Python. I made up 
> the simple test in my previous note to try out the solution that had 
> been offered and which you have usefully ruled out. Here is the relevant 
> part of the vector class, which indeed doesn't look like an ndarray:
>
> inline vector
> operator*( const double s) const throw()
> { return vector( s*x, s*y, s*z); }
>
> and here is the free function for right multiplication:
>
> inline vector
> operator*( const double& s, const vector& v)
> { return vector( s*v.x, s*v.y, s*v.z); }
>
> Maybe the unsolvable problem is in the Boost definitions:
>
> py::class_<vector>("vector", py::init< py::optional<double, double, 
> double> >())
>      .def( self * double())
>      .def( double() * self)
>
> Left multiplication is fine, but right multiplication isn't.
>
> Bruce Sherwood
>
> Robert Kern wrote:
>   
>> Bruce Sherwood wrote:
>>   
>>     
>>> Thanks for the suggestion. It hadn't occurred to me to try to override 
>>> numpy as you suggest. However, when I try the code shown below as the 
>>> start of a test of this scheme, I get the following error:
>>>
>>> Traceback (most recent call last):
>>>   File "C:\Documents and Settings\Bruce\My 
>>> Documents\0VPythonWork\vectors.py", line 24, in <module>
>>>     numpy.float64.__mul__ = new_mul
>>> TypeError: can't set attributes of built-in/extension type 'numpy.float64'
>>>
>>> I'm copying this to the numpy discussion list, as maybe someone there 
>>> will see where to go starting from your suggestion.
>>>     
>>>       
>> Like most (or all) builtin-types, the numpy float scalars do not permit
>> replacing their methods from Python.
>>
>> I'm not familiar with vpython's vector. If you can make it "not look like an
>> ndarray", then you should be able to just implement __rmul__ on vector.
>>
>>   
>>     
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   


From bioinformed at gmail.com  Thu Dec 27 10:30:02 2007
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Thu, 27 Dec 2007 10:30:02 -0500
Subject: [Numpy-discussion] [C++-sig] Overloading sqrt(5.5)*myvector
In-Reply-To: <4773C17B.4050802@ncsu.edu>
References: <4771F0AE.40406@ncsu.edu>
	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>
	<47732B84.70400@ncsu.edu> <4773325D.3040907@gmail.com>
	<4773BD97.1000703@ncsu.edu> <4773C17B.4050802@ncsu.edu>
Message-ID: <2e1434c10712270730y61d694d8u7551e92b023e8cb8@mail.gmail.com>

Hi Bruce,

I have to add that I don't know the answer to your question either, but I do
know that it is solvable and that once the list experts return,
enlightenment will soon follow.  My confidence comes from knowing the Python
internals for how left and right multiplication are performed.  As long as
the "left" __mul__ operator returns NotImplemented, then the __rmul__ method
will be attempted (see http://docs.python.org/ref/numeric-types.html).  Of
course, I don't know how to declare such a beast in Boost, having never used
it, but I'm sure it is possible.  My intuition is that the first problem you
need to solve is getting Boot to generate the appropriate __rmul__ method.
The second problem, if it even exists, is ensuring that __mul__ returns
NotImplemented.

Best of luck,
-Kevin



On Dec 27, 2007 10:15 AM, Bruce Sherwood <Bruce_Sherwood at ncsu.edu> wrote:

> I should have added: This structure worked with the older version of
> VPython which used Numeric, but it doesn't work in the beta version
> which uses numpy. Since I don't know enough about either numpy or Boost,
> I'm left guessing which subsystem is the source of my difficulties, and
> clueless about how to remedy them.
>
> Bruce Sherwood
>
> Bruce Sherwood wrote:
> > Thanks for the comment, which limits the range of possible solutions.
> > The VPython vector class is implemented in C++, not in Python. I made up
> > the simple test in my previous note to try out the solution that had
> > been offered and which you have usefully ruled out. Here is the relevant
> > part of the vector class, which indeed doesn't look like an ndarray:
> >
> > inline vector
> > operator*( const double s) const throw()
> > { return vector( s*x, s*y, s*z); }
> >
> > and here is the free function for right multiplication:
> >
> > inline vector
> > operator*( const double& s, const vector& v)
> > { return vector( s*v.x, s*v.y, s*v.z); }
> >
> > Maybe the unsolvable problem is in the Boost definitions:
> >
> > py::class_<vector>("vector", py::init< py::optional<double, double,
> > double> >())
> >      .def( self * double())
> >      .def( double() * self)
> >
> > Left multiplication is fine, but right multiplication isn't.
> >
> > Bruce Sherwood
> >
> > Robert Kern wrote:
> >
> >> Bruce Sherwood wrote:
> >>
> >>
> >>> Thanks for the suggestion. It hadn't occurred to me to try to override
> >>> numpy as you suggest. However, when I try the code shown below as the
> >>> start of a test of this scheme, I get the following error:
> >>>
> >>> Traceback (most recent call last):
> >>>   File "C:\Documents and Settings\Bruce\My
> >>> Documents\0VPythonWork\vectors.py", line 24, in <module>
> >>>     numpy.float64.__mul__ = new_mul
> >>> TypeError: can't set attributes of built-in/extension type '
> numpy.float64'
> >>>
> >>> I'm copying this to the numpy discussion list, as maybe someone there
> >>> will see where to go starting from your suggestion.
> >>>
> >>>
> >> Like most (or all) builtin-types, the numpy float scalars do not permit
> >> replacing their methods from Python.
> >>
> >> I'm not familiar with vpython's vector. If you can make it "not look
> like an
> >> ndarray", then you should be able to just implement __rmul__ on vector.
> >>
> >>
> >>
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071227/b620fb7c/attachment.html>

From dikshie at gmail.com  Thu Dec 27 10:45:08 2007
From: dikshie at gmail.com (dikshie)
Date: Fri, 28 Dec 2007 00:45:08 +0900
Subject: [Numpy-discussion] non uniform random number generator
Message-ID: <910e60e80712270745n109773d4x1c7ce0a8fe870690@mail.gmail.com>

hi,
does numpy support non uniform random number generator ?

regards,

-dikshie-


From matthieu.brucher at gmail.com  Thu Dec 27 10:47:23 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 27 Dec 2007 16:47:23 +0100
Subject: [Numpy-discussion] non uniform random number generator
In-Reply-To: <910e60e80712270745n109773d4x1c7ce0a8fe870690@mail.gmail.com>
References: <910e60e80712270745n109773d4x1c7ce0a8fe870690@mail.gmail.com>
Message-ID: <e76aa17f0712270747g5cb0f153m4d4632adbeae2882@mail.gmail.com>

Hi,

Of course, look in the numpy.random module.

Matthieu

2007/12/27, dikshie <dikshie at gmail.com>:
>
> hi,
> does numpy support non uniform random number generator ?
>
> regards,
>
> -dikshie-
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071227/5e2659ff/attachment.html>

From nadavh at visionsense.com  Thu Dec 27 11:38:41 2007
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu, 27 Dec 2007 18:38:41 +0200
Subject: [Numpy-discussion] non uniform random number generator
Message-ID: <710F2847B0018641891D9A21602763600B6E78@ex3.envision.co.il>

Lot of nonuniform distributions, see numpy.random module.

   Nadav.


-----Original Message-----
From: numpy-discussion-bounces at scipy.org on behalf of dikshie
Sent: Thu 27-Dec-07 17:45
To: numpy-discussion at scipy.org
Subject: [Numpy-discussion] non uniform random number generator
 
hi,
does numpy support non uniform random number generator ?

regards,

-dikshie-
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From fperez.net at gmail.com  Thu Dec 27 16:04:34 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 27 Dec 2007 14:04:34 -0700
Subject: [Numpy-discussion] Scipy server down...
Message-ID: <db6b5ecc0712271304t2b873371mce1131fc3ea21c58@mail.gmail.com>

Any chance it might be brought back up? (I'm trying to commit the
weave cleanup work Min did at the sprint...)

Thanks,

f


From millman at berkeley.edu  Thu Dec 27 16:11:37 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 27 Dec 2007 13:11:37 -0800
Subject: [Numpy-discussion] [SciPy-user] Scipy server down...
In-Reply-To: <db6b5ecc0712271304t2b873371mce1131fc3ea21c58@mail.gmail.com>
References: <db6b5ecc0712271304t2b873371mce1131fc3ea21c58@mail.gmail.com>
Message-ID: <c7009a550712271311wcf1cbd6tc37319d213e50c29@mail.gmail.com>

I restarted httpd.  It is still slow, but seems to be responsive now.
Let me know if it still isn't working.


On Dec 27, 2007 1:04 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> Any chance it might be brought back up? (I'm trying to commit the
> weave cleanup work Min did at the sprint...)
>
> Thanks,
>
> f
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>



-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From fperez.net at gmail.com  Thu Dec 27 16:46:54 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 27 Dec 2007 14:46:54 -0700
Subject: [Numpy-discussion] [SciPy-user] Scipy server down...
In-Reply-To: <c7009a550712271311wcf1cbd6tc37319d213e50c29@mail.gmail.com>
References: <db6b5ecc0712271304t2b873371mce1131fc3ea21c58@mail.gmail.com>
	<c7009a550712271311wcf1cbd6tc37319d213e50c29@mail.gmail.com>
Message-ID: <db6b5ecc0712271346h2eaa3f57ocecffa88f6277f3a@mail.gmail.com>

On Dec 27, 2007 2:11 PM, Jarrod Millman <millman at berkeley.edu> wrote:
> I restarted httpd.  It is still slow, but seems to be responsive now.
> Let me know if it still isn't working.

SVN is working again, thanks.

Cheers,

f


From fperez.net at gmail.com  Thu Dec 27 18:04:33 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 27 Dec 2007 16:04:33 -0700
Subject: [Numpy-discussion] SciPy Sprint results
In-Reply-To: <47696873.1060008@enthought.com>
References: <47696873.1060008@enthought.com>
Message-ID: <db6b5ecc0712271504p5f8d3bb7pca04ac5f022e7d9c@mail.gmail.com>

On Dec 19, 2007 11:52 AM, Travis E. Oliphant <oliphant at enthought.com> wrote:

> Testing
> ---------
>   * scipy unit-testing will be "nose-compliant" and therefore nose will
> be required to run the SciPy tests.
>   * NumPy will still use the current testing framework but will support
> SciPy's desire to be nose-compliant.  NumPy 1.1 tests may move to just
> being "nose-compliant"
>   * The goal is to make tests easier for contributors to write.

This hadn't been committed yet, but last night I put it in, and
Matthew Brett is planning today on updating the rest of the codebase
to the nose tests, which do make a LOT of things vastly nicer and
simpler to write, at the cost of a very small new dependency (only
needed to run the actual test suite, not to use scipy).

At some point in the next few days I imagine this will get merged back
into trunk, though that merge probably needs to be done with a bit of
coordination, since the changes will touch pretty much every scipy
module (at least their test/ directory).

> Weave
> ---------
>   * weave will not move into NumPy yet, but possibly at NumPy 1.1, there
> could be a separate package containing all the "wrapping" support code
> for NumPy in a more unified fashion (if somebody is interested in this,
> it is a great time to jump in).

It's worth crediting Min (Benjamin Ragan-Kelley, ipython developer)
for spending the whole day deep in the bowels of weave clenaing up
tens of test failures that had been in for a long time.  The weave
tests aren't getting correctly picked up by scipy.test(), hence they
hadn't been reported, but you can see them with weave.test(10).

To make sure that this work doesn't fall too far out of sync with
trunk, I just committed it all back into trunk as two separate
changesets (one to update the branch itself in case anyone is going to
work further on it, one to apply the changes to the trunk itself).
Many thanks to Min for the spelunking work!

Cheers,

f


From bhendrix at enthought.com  Thu Dec 27 20:23:49 2007
From: bhendrix at enthought.com (Bryce Hendrix)
Date: Thu, 27 Dec 2007 19:23:49 -0600
Subject: [Numpy-discussion] Scipy server down...
In-Reply-To: <db6b5ecc0712271304t2b873371mce1131fc3ea21c58@mail.gmail.com>
References: <db6b5ecc0712271304t2b873371mce1131fc3ea21c58@mail.gmail.com>
Message-ID: <47745025.8080909@enthought.com>

Looks like the web site & svn are up for me.

Bryce

Fernando Perez wrote:
> Any chance it might be brought back up? (I'm trying to commit the
> weave cleanup work Min did at the sprint...)
>
> Thanks,
>
> f
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   



From oliphant at enthought.com  Thu Dec 27 22:42:56 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 27 Dec 2007 21:42:56 -0600
Subject: [Numpy-discussion] Doc-day
Message-ID: <477470C0.3030602@enthought.com>


Doc-day will start tomorrow (in about 12 hours).  It will be Friday for 
much of America and be moving into Saturday for Europe and Asia.  Join 
in on the irc.freenode.net  (channel scipy) to coordinate effort.  I 
imaging people will be in an out.  I plan on being available in IRC from 
about 9:30 am CST to 6:00 pm CST and then possibly later. 

If you are available at different times in different parts of the 
world,  jump in and pick something to work on.   Jarrod and/or I will 
put out a list of priorities in the next few hours.

-Travis O.



From devnew at gmail.com  Thu Dec 27 23:34:55 2007
From: devnew at gmail.com (devnew at gmail.com)
Date: Thu, 27 Dec 2007 20:34:55 -0800 (PST)
Subject: [Numpy-discussion] any better way to normalise a matrix
Message-ID: <4b3eadfc-074d-479b-ac85-459b15737b2d@s19g2000prg.googlegroups.com>

in my code i am trying to normalise a matrix as below

mymatrix=matrix(..# items are of double type..can be negative
values....)
numrows,numcols=mymatrix.shape

for i in range(numrows):
	temp=mymatrix[i].max()
	for j in range(numcols):
		mymatrix[i,j]=abs(mymatrix[i,j]/temp)

# i am using abs() to make neg vals positive

this way the code takes too much time to run for a case of
numrows=25, and numcols=8100 etc..
being a beginner in numpy and python ,i would like to know if this can
be done more efficiently..can anyone advise?

thanx
dn


From peridot.faceted at gmail.com  Fri Dec 28 00:14:10 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 28 Dec 2007 00:14:10 -0500
Subject: [Numpy-discussion] any better way to normalise a matrix
In-Reply-To: <4b3eadfc-074d-479b-ac85-459b15737b2d@s19g2000prg.googlegroups.com>
References: <4b3eadfc-074d-479b-ac85-459b15737b2d@s19g2000prg.googlegroups.com>
Message-ID: <ce557a360712272114q6d0153b0lfa954c96f8304760@mail.gmail.com>

On 27/12/2007, devnew at gmail.com <devnew at gmail.com> wrote:
> in my code i am trying to normalise a matrix as below
>
> mymatrix=matrix(..# items are of double type..can be negative
> values....)
> numrows,numcols=mymatrix.shape
>
> for i in range(numrows):
>         temp=mymatrix[i].max()
>         for j in range(numcols):
>                 mymatrix[i,j]=abs(mymatrix[i,j]/temp)
>
> # i am using abs() to make neg vals positive
>
> this way the code takes too much time to run for a case of
> numrows=25, and numcols=8100 etc..
> being a beginner in numpy and python ,i would like to know if this can
> be done more efficiently..can anyone advise?

Yes. A major advantage - really the reason for existence - of numpy is
that you can do operations to whole matrices at once in a single
instruction, without loops. This is convenient from a coding point of
view, and it also allows numpy to greatly accelerate calculations.
Writing code the way you have above takes no advantage of numpy's
machinery, and it would probably run faster using python lists than
numpy arrays. I strongly recommend you read some numpy documentation;
try starting with the tutorial:
http://www.scipy.org/Tentative_NumPy_Tutorial

For example, to extract an array containing the maxima of each row of
mymatrix, you can use the amax() function:

temp = numpy.amax(mymatrix, axis=1)

Multiplying a whole matrix by a constant can be done simply as well:

7*mymatrix

A final comment is that numpy is designed for computations with
multidimensional data. Its basic data type is the array. It has a
specialized data type called "matrix" which provides slightly more
convenient matrix multiplication (in the sense of linear algebra). I
recommend you do not use numpy's matrix class until you are more
accustomed to the software.

Read the documentation. Start with the tutorial. Good luck.
Anne


From millman at berkeley.edu  Fri Dec 28 00:27:09 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 27 Dec 2007 21:27:09 -0800
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <477470C0.3030602@enthought.com>
References: <477470C0.3030602@enthought.com>
Message-ID: <c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>

On Dec 27, 2007 7:42 PM, Travis E. Oliphant <oliphant at enthought.com> wrote:
> Doc-day will start tomorrow (in about 12 hours).  It will be Friday for
> much of America and be moving into Saturday for Europe and Asia.  Join
> in on the irc.freenode.net  (channel scipy) to coordinate effort.  I
> imaging people will be in an out.  I plan on being available in IRC from
> about 9:30 am CST to 6:00 pm CST and then possibly later.
>
> If you are available at different times in different parts of the
> world,  jump in and pick something to work on.

Since this is our first doc-day, it will be fairly informal.  Travis
is going to be trying to get some estimate of which packages need the
most work.  But if there is some area of NumPy or SciPy you are
familiar with, please go ahead and pitch in.  Here is the current
NumPy/ SciPy coding standard including docstring standards:
http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines

I will be working on making the roadmaps more detailed and better
documenting the discussions from the coding sprint.

Travis O. will be mostly working on NumPy docstrings and possibly
deprecation warnings for scipy.io functions.

Matthew B. will be working on converting SciPy tests to use nose per
Fernando's email.  If you are familiar with nose and want to help,
please make sure to check with Matthew or Fernando first.

I hope to see several of you on IRC tomorrow.

Happy holidays,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From fperez.net at gmail.com  Fri Dec 28 00:41:17 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 27 Dec 2007 22:41:17 -0700
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
Message-ID: <db6b5ecc0712272141r45678bbeof7e95b6cb854ae8e@mail.gmail.com>

On Dec 27, 2007 10:27 PM, Jarrod Millman <millman at berkeley.edu> wrote:

> Since this is our first doc-day, it will be fairly informal.  Travis
> is going to be trying to get some estimate of which packages need the
> most work.  But if there is some area of NumPy or SciPy you are
> familiar with, please go ahead and pitch in.  Here is the current
> NumPy/ SciPy coding standard including docstring standards:
> http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines

Care to make the Example section mandatory, instead of optional?  I
really think it should be mandatory.  We may not do a good job of it
initially, but at least we should express that it's of critical
importance that every function contains at least one small example,
whenever feasible.

I also think that the above wiki page should have a minimal,
self-contained example of a proper docstring with all 8 sections
implemented.  I'm honestly not sure at this point what the actual
changes to epydoc are (in ipython we use regular epydoc with reST),
and I think for many it would be much easier to get started by reading
a small example rather than trying to abstract out what the exact
markup should be from reading the description and the various
documents linked to (doctest, reST, epydoc...).

With such a guiding example, tomorrow people will be able to get up
and going quickly...

> I will be working on making the roadmaps more detailed and better
> documenting the discussions from the coding sprint.
>
> Travis O. will be mostly working on NumPy docstrings and possibly
> deprecation warnings for scipy.io functions.
>
> Matthew B. will be working on converting SciPy tests to use nose per
> Fernando's email.  If you are familiar with nose and want to help,
> please make sure to check with Matthew or Fernando first.

I'm afraid I won't be able to participate tomorrow, but one thing to
remember is that with nose, any and all doctest examples should be
automatically picked up (with the appropriate flags).  So a *very
easy* way for anyone to contribute is to simply add doctest examples
to the codebase.  Those serve automatically two purposes: they are
small tests for each function, and they make the library vastly easier
to use, since any function is just one

foo?

away from an example.  As a reminder, those of you using ipython >=
0.8.2 can use this feature:

In [1]: %doctest_mode
*** Pasting of code with ">>>" or "..." has been enabled.
Exception reporting mode: Plain
Doctest mode is: ON

>>> for i in range(10):
...     print i,
...
0 1 2 3 4 5 6 7 8 9

>>> >>> for i in range(10):
...     ...     print i,
...
0 1 2 3 4 5 6 7 8 9

>>> %doctest_mode
Exception reporting mode: Context
Doctest mode is: OFF


########

The %doctest_mode magic switches the ipython prompt to >>> so you can
continue using ipython but get the proper prompts for making pasteable
docstests, and it also allows you to paste input that begins with
'>>>' for execution, so you can try your doctests again.

HTH,

f


From millman at berkeley.edu  Fri Dec 28 00:50:29 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 27 Dec 2007 21:50:29 -0800
Subject: [Numpy-discussion] [SciPy-dev]  Doc-day
In-Reply-To: <db6b5ecc0712272141r45678bbeof7e95b6cb854ae8e@mail.gmail.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<db6b5ecc0712272141r45678bbeof7e95b6cb854ae8e@mail.gmail.com>
Message-ID: <c7009a550712272150j3b61bc4bxcb5ea69dd1c8186c@mail.gmail.com>

On Dec 27, 2007 9:41 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> Care to make the Example section mandatory, instead of optional?  I
> really think it should be mandatory.  We may not do a good job of it
> initially, but at least we should express that it's of critical
> importance that every function contains at least one small example,
> whenever feasible.

+1

> I also think that the above wiki page should have a minimal,
> self-contained example of a proper docstring with all 8 sections
> implemented.  I'm honestly not sure at this point what the actual
> changes to epydoc are (in ipython we use regular epydoc with reST),
> and I think for many it would be much easier to get started by reading
> a small example rather than trying to abstract out what the exact
> markup should be from reading the description and the various
> documents linked to (doctest, reST, epydoc...).

I think we already have that.  Take a look at the example at the bottom:
http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines#example

It points to a plain text example:
http://svn.scipy.org/svn/numpy/trunk/numpy/doc/example.py
and what it looks like rendered:
http://www.scipy.org/doc/example

It also explains how to check the example code out of svn and generate
the html using epydoc.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From fperez.net at gmail.com  Fri Dec 28 00:57:14 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 27 Dec 2007 22:57:14 -0700
Subject: [Numpy-discussion] [SciPy-dev]  Doc-day
In-Reply-To: <c7009a550712272150j3b61bc4bxcb5ea69dd1c8186c@mail.gmail.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<db6b5ecc0712272141r45678bbeof7e95b6cb854ae8e@mail.gmail.com>
	<c7009a550712272150j3b61bc4bxcb5ea69dd1c8186c@mail.gmail.com>
Message-ID: <db6b5ecc0712272157r1345cfbbm2f71b5c069a83226@mail.gmail.com>

On Dec 27, 2007 10:50 PM, Jarrod Millman <millman at berkeley.edu> wrote:

> > I also think that the above wiki page should have a minimal,
> > self-contained example of a proper docstring with all 8 sections
> > implemented.  I'm honestly not sure at this point what the actual
> > changes to epydoc are (in ipython we use regular epydoc with reST),
> > and I think for many it would be much easier to get started by reading
> > a small example rather than trying to abstract out what the exact
> > markup should be from reading the description and the various
> > documents linked to (doctest, reST, epydoc...).
>
> I think we already have that.  Take a look at the example at the bottom:
> http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines#example

Oops, sorry.  I read the page quickly and my brain was scanning for a
big block of text in {{{ }}}, so I didn't look at that section
carefully enough.   My mistake.

Cheers,

f


From devnew at gmail.com  Fri Dec 28 01:59:34 2007
From: devnew at gmail.com (devnew at gmail.com)
Date: Thu, 27 Dec 2007 22:59:34 -0800 (PST)
Subject: [Numpy-discussion] any better way to normalise a matrix
In-Reply-To: <ce557a360712272114q6d0153b0lfa954c96f8304760@mail.gmail.com>
References: <4b3eadfc-074d-479b-ac85-459b15737b2d@s19g2000prg.googlegroups.com>
	<ce557a360712272114q6d0153b0lfa954c96f8304760@mail.gmail.com>
Message-ID: <7c9c420f-c764-4da9-9dad-ecdedd14cab8@e25g2000prg.googlegroups.com>


> try starting with the tutorial:
> http://www.scipy.org/Tentative_NumPy_Tutorial
>
> For example, to extract an array containing the maxima of each row of
> mymatrix, you can use the amax() function:
>
> temp = numpy.amax(mymatrix, axis=1)
>
>


thanx..had a tuff time finding the functions..will go thru the
tutorial

dn


From fperez.net at gmail.com  Fri Dec 28 03:44:16 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 28 Dec 2007 01:44:16 -0700
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <db6b5ecc0712272141r45678bbeof7e95b6cb854ae8e@mail.gmail.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<db6b5ecc0712272141r45678bbeof7e95b6cb854ae8e@mail.gmail.com>
Message-ID: <db6b5ecc0712280044h29d9b07y6be7df820c3d0f11@mail.gmail.com>

Since I won't be able to participate tomorrow, here's a small
contribution.  I just added this to ipython:

http://projects.scipy.org/ipython/ipython/browser/ipython/trunk/IPython/dtutils.py

You can load it in your startup file or interactively via

from IPython.dtutils import idoctest

Type idoctest? for calling details.

This will enable you to do things like:

In [96]: idoctest
-------> idoctest()   # at this point, you start typing/pasting your
doctest.  Hit Ctrl-D or two blanks to end:

    >>> for i in range(10):
    ...     print i,
    ...
    0 1 2 3 4 5 6 7 8 9


1 items passed all tests:
   1 tests in interactive doctest
1 tests in 1 items.
1 passed and 0 failed.
Test passed.


If you have a failing doctest that you'd like to debug, you can use
'eraise=True' and errors will be immediately raised, so you can then
call %debug to debug them.

This should come in handy for all those doctests that you're all going
to write tomorrow.  If you aren't running off SVN ipython, you can
simply copy the above file somewhere and load it interactively once
ipython is running, it doesn't depend on any other changes to ipython.

It probably still needs a bit of work to make it more convenient, so
I'm open to feedback.  But it should make life easier when writing
frequent doctests.

Cheers,

f


From stefan at sun.ac.za  Fri Dec 28 06:26:17 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 28 Dec 2007 13:26:17 +0200
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
Message-ID: <20071228112617.GC9051@mentat.za.net>

On Thu, Dec 27, 2007 at 09:27:09PM -0800, Jarrod Millman wrote:
> On Dec 27, 2007 7:42 PM, Travis E. Oliphant <oliphant at enthought.com> wrote:
> > Doc-day will start tomorrow (in about 12 hours).  It will be Friday for
> > much of America and be moving into Saturday for Europe and Asia.  Join
> > in on the irc.freenode.net  (channel scipy) to coordinate effort.  I
> > imaging people will be in an out.  I plan on being available in IRC from
> > about 9:30 am CST to 6:00 pm CST and then possibly later.
> >
> > If you are available at different times in different parts of the
> > world,  jump in and pick something to work on.
> 
> Since this is our first doc-day, it will be fairly informal.  Travis
> is going to be trying to get some estimate of which packages need the
> most work.  But if there is some area of NumPy or SciPy you are
> familiar with, please go ahead and pitch in.  Here is the current
> NumPy/ SciPy coding standard including docstring standards:
> http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines

I have some questions regarding Travis' latest modifications to the
documentation guidelines:

The following section was removed, why?

"""
A reST-documented module should define:: 

  __docformat__ = 'restructuredtext en' 
	 	 
at the top level in accordance with `PEP 258
<http://www.python.org/dev/peps/pep-0258>`__.  Note that the
``__docformat__`` variable in a package's ``__init__.py`` file does
not apply to objects defined in subpackages and submodules.
"""

We had a long discussion on the mailing list on the pros and cons of
"*Parameters*:" vs. "Parameters:".  I see now that it has been changed
to

  Parameters
  ----------

Is this still recognised as a list?

I noted that the examples are now no longer indented: does ReST allow this?


Note that building the example documentation, `epydoc example.py`, now
warns:

File /tmp/example.py, line 19, in example.foo
   Warning: Line 24: Wrong underline character for heading.
   Warning: Lines 27, 30, 32, 37, 39, 41, 43, 48, 50: Improper paragraph indentation.
   

While I understand the rationale behind

  "The guiding principle is that human readers of the text itself are
  given precedence over contorting the docstring so that epydoc_
  produces nice output."

I think that it would be impractical to break compatibility with the
only documentation builder we currently have (unless there are others
I am unaware of?).

Regards
St?fan


From matthieu.brucher at gmail.com  Fri Dec 28 08:23:19 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 28 Dec 2007 14:23:19 +0100
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
Message-ID: <e76aa17f0712280523m6e541a5aua6c27207fda633e4@mail.gmail.com>

>
> Matthew B. will be working on converting SciPy tests to use nose per
> Fernando's email.  If you are familiar with nose and want to help,
> please make sure to check with Matthew or Fernando first.
>

I must have missed Fernando's email because I can't find the references for
nose :(
What are its advantages against the current numpy.testing framework ?

Matthieu
-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071228/becee1d9/attachment.html>

From dikshie at gmail.com  Fri Dec 28 11:28:29 2007
From: dikshie at gmail.com (dikshie)
Date: Sat, 29 Dec 2007 01:28:29 +0900
Subject: [Numpy-discussion] numpy installed but can' use
Message-ID: <910e60e80712280828i749b543et6a7c490b1f75f612@mail.gmail.com>

Hi,
successfully installed numpy but i cant use it (numpy is not defined).
for example:


Python 2.5.1 (r251:54863, Nov 25 2007, 02:18:29)
[GCC 4.2.1 20070719  [FreeBSD]] on freebsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> numpy.__version__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'numpy' is not defined
>>>

any hints ?

regards,
-- 
-dikshie-


From Chris.Barker at noaa.gov  Fri Dec 28 11:57:40 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 28 Dec 2007 08:57:40 -0800
Subject: [Numpy-discussion] numpy installed but can' use
In-Reply-To: <910e60e80712280828i749b543et6a7c490b1f75f612@mail.gmail.com>
References: <910e60e80712280828i749b543et6a7c490b1f75f612@mail.gmail.com>
Message-ID: <47752B04.1040205@noaa.gov>

dikshie wrote:
>>>> from numpy import *
>>>> numpy.__version__
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NameError: name 'numpy' is not defined
> 
> any hints ?

yes, you did an "import *", which means "bring all the names in the 
numpy module into this namespace. There is no name: "numpy" in the numpy 
module -- it's the name of the module itself. Try:

import numpy
numpy.__version__

then you can use the stuff in numpy this way:

MyArray = numpy.array(some_stuff)

Or, if you don't want to type quite so much, use something like:

import numpy as N
N.__version__
MyArray = N.array(some_stuff)


"Namespaces are one honking great idea -- let's do more of those!"
-- http://www.python.org/dev/peps/pep-0020/


-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From efiring at hawaii.edu  Fri Dec 28 11:59:54 2007
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 28 Dec 2007 06:59:54 -1000
Subject: [Numpy-discussion] numpy installed but can' use
In-Reply-To: <910e60e80712280828i749b543et6a7c490b1f75f612@mail.gmail.com>
References: <910e60e80712280828i749b543et6a7c490b1f75f612@mail.gmail.com>
Message-ID: <47752B8A.20806@hawaii.edu>

dikshie wrote:
> Hi,
> successfully installed numpy but i cant use it (numpy is not defined).
> for example:
> 
> 
> Python 2.5.1 (r251:54863, Nov 25 2007, 02:18:29)
> [GCC 4.2.1 20070719  [FreeBSD]] on freebsd7
> Type "help", "copyright", "credits" or "license" for more information.
>>>> from numpy import *
>>>> numpy.__version__
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NameError: name 'numpy' is not defined
> 
> any hints ?
> 
> regards,

You did not import numpy, you imported all of its contents.  Try:

import numpy
numpy.__version__

Eric



From Chris.Barker at noaa.gov  Fri Dec 28 12:00:56 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 28 Dec 2007 09:00:56 -0800
Subject: [Numpy-discussion] any better way to normalize a matrix
In-Reply-To: <4b3eadfc-074d-479b-ac85-459b15737b2d@s19g2000prg.googlegroups.com>
References: <4b3eadfc-074d-479b-ac85-459b15737b2d@s19g2000prg.googlegroups.com>
Message-ID: <47752BC8.8050608@noaa.gov>

Anne had it right -- much of the point of numpy is to use nd-arrays as 
the powerful objects they are - not just containers. Below is  a version 
of your code for comparison.

Note to numpy devs:

I like the array methods a lot -- is there any particular reason there 
is no ndarray.abs(), or has it just not been added?

-Chris



#!/usr/bin/env python

"""
Simple exmaple of normalizing an array
"""

import numpy as N
from numpy import random


mymatrix=random.uniform(-100, 100,(3,4))

print "before:", mymatrix
mymatrix2 = mymatrix.copy()

numrows,numcols=mymatrix.shape

for i in range(numrows):
     temp=mymatrix[i].max()
     for j in range(numcols):
         mymatrix[i,j]=abs(mymatrix[i,j]/temp)

print "old way:", mymatrix

## "vectorized" way:

# the "reshape" is a bit awkward, but it makes the 1-d result the right 
shape to "broadcast" to the original array
row_max = mymatrix2.max(axis=1).reshape((-1, 1))

print row_max
mymatrix2 = N.absolute((mymatrix2 / row_max))

print "vectorized:", mymatrix2
if (mymatrix == mymatrix2).all():
     print "They are the same"


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From oliphant at enthought.com  Fri Dec 28 12:32:03 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 28 Dec 2007 11:32:03 -0600
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <20071228112617.GC9051@mentat.za.net>
References: <477470C0.3030602@enthought.com>	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<20071228112617.GC9051@mentat.za.net>
Message-ID: <47753313.2010006@enthought.com>

Stefan van der Walt wrote:
> On Thu, Dec 27, 2007 at 09:27:09PM -0800, Jarrod Millman wrote:
>   
>> On Dec 27, 2007 7:42 PM, Travis E. Oliphant <oliphant at enthought.com> wrote:
>>     
>>> Doc-day will start tomorrow (in about 12 hours).  It will be Friday for
>>> much of America and be moving into Saturday for Europe and Asia.  Join
>>> in on the irc.freenode.net  (channel scipy) to coordinate effort.  I
>>> imaging people will be in an out.  I plan on being available in IRC from
>>> about 9:30 am CST to 6:00 pm CST and then possibly later.
>>>
>>> If you are available at different times in different parts of the
>>> world,  jump in and pick something to work on.
>>>       
>> Since this is our first doc-day, it will be fairly informal.  Travis
>> is going to be trying to get some estimate of which packages need the
>> most work.  But if there is some area of NumPy or SciPy you are
>> familiar with, please go ahead and pitch in.  Here is the current
>> NumPy/ SciPy coding standard including docstring standards:
>> http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines
>>     
>
> I have some questions regarding Travis' latest modifications to the
> documentation guidelines:
>
> The following section was removed, why?
>
> """
> A reST-documented module should define:: 
>
>   __docformat__ = 'restructuredtext en' 
> 	 	 
> at the top level in accordance with `PEP 258
> <http://www.python.org/dev/peps/pep-0258>`__.  Note that the
> ``__docformat__`` variable in a package's ``__init__.py`` file does
> not apply to objects defined in subpackages and submodules.
> """
>   

I don't see the point in every file having a __docformat__ line, when 
our documentaion formatting tool should already know the standard we are 
using is.   It's just more cruft.   Besides the PEP was rejected, so I 
don't know why we should be following it. 

We obviously need a pre-processor to map our files to epydoc, so let's 
do that instead of contorting docstrings into a format demanded by the 
tool.  The tool is a "tool" for us to use not be chained by.
> We had a long discussion on the mailing list on the pros and cons of
> "*Parameters*:" vs. "Parameters:".  I see now that it has been changed
> to
>
>   Parameters
>   ----------
>
> Is this still recognised as a list?
>
> I noted that the examples are now no longer indented: does ReST allow this?
>
>
> Note that building the example documentation, `epydoc example.py`, now
> warns:
>
> File /tmp/example.py, line 19, in example.foo
>    Warning: Line 24: Wrong underline character for heading.
>    Warning: Lines 27, 30, 32, 37, 39, 41, 43, 48, 50: Improper paragraph indentation.
>    
>
> While I understand the rationale behind
>
>   "The guiding principle is that human readers of the text itself are
>   given precedence over contorting the docstring so that epydoc_
>   produces nice output."
>
> I think that it would be impractical to break compatibility with the
> only documentation builder we currently have (unless there are others
> I am unaware of?).
>
>   

In my mind, the rationale trumps the "impracticality" especially since 
epydoc still produces readable output.  It is only the formatting that 
is not pretty-printed (although it doesn't look bad there either).  So, 
AFAIK, nothing is actually broken (except a little HTML formatting on 
output of epydoc).  But,  now the docstrings don't look crufty to my 
eyes.  

Basically, the problem is that I *really* don't like the *Parameters*:  
thing and all the indentation that takes place inside of docstrings to 
satisfy epydoc.  It is also inconsistent with the docstring standard 
that Enthought uses for the ETS tools suite.   It was really hard for me 
to start writing docstrings that followed that pattern. 

It seems better to have docstrings than to have them fit into an 
epydoc_-defined pattern.  

I'm sorry if I made changes unilaterally.   I didn't think there would 
be much concern as long as docstrings were moving forward.

-Travis




From robert.kern at gmail.com  Fri Dec 28 14:55:33 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 28 Dec 2007 14:55:33 -0500
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <e76aa17f0712280523m6e541a5aua6c27207fda633e4@mail.gmail.com>
References: <477470C0.3030602@enthought.com>	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<e76aa17f0712280523m6e541a5aua6c27207fda633e4@mail.gmail.com>
Message-ID: <477554B5.2080206@gmail.com>

Matthieu Brucher wrote:
>     Matthew B. will be working on converting SciPy tests to use nose per
>     Fernando's email.  If you are familiar with nose and want to help,
>     please make sure to check with Matthew or Fernando first.
> 
> 
> I must have missed Fernando's email because I can't find the references
> for nose :(

Look in "SciPy Sprint Results". It's only a brief mention, though.

> What are its advantages against the current numpy.testing framework ?

Primarily:

* It is supported by someone else and gaining wide adoption by the rest of the
Python community.

Secondarily:

* More flexible organization of tests. For nose, if it looks like a test, it's a
test. numpy.testing collects test modules which are named like the module it is
testing. E.g. for module.py <=> tests/test_module.py.

* Test generators:

  def test_evens():
    for i in range(0, 5):
      yield check_even, i, i*3

  def check_even(n, nn):
    assert n % 2 == 0 or nn % 2 == 0

* Package- and module-level setup() and teardown() functions.

* Test functions can be simple functions. They do not need to be organized into
classes if you don't need classes.

* Integrated doctest collection.

* Detailed error/failure reporting. nose can print out the values of variables
at the location of the error.

* Integrated code coverage and profiling.

* Dropping into pdb on errors and failures.

* More flexible running of specific tests. E.g. when I'm working on getting a
particular test function running, I can specify that exact test and not run the
rest of the test suite.

* Output capture. Tests can print out anything they like to be more informative,
but they won't appear unless if the test fails.

More thoroughly:

  http://somethingaboutorange.com/mrl/projects/nose/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From fperez.net at gmail.com  Fri Dec 28 15:19:58 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 28 Dec 2007 13:19:58 -0700
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <477554B5.2080206@gmail.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<e76aa17f0712280523m6e541a5aua6c27207fda633e4@mail.gmail.com>
	<477554B5.2080206@gmail.com>
Message-ID: <db6b5ecc0712281219h1c6f4d00n191c70b8adf6860d@mail.gmail.com>

On Dec 28, 2007 12:55 PM, Robert Kern <robert.kern at gmail.com> wrote:
> Matthieu Brucher wrote:
> >     Matthew B. will be working on converting SciPy tests to use nose per
> >     Fernando's email.  If you are familiar with nose and want to help,
> >     please make sure to check with Matthew or Fernando first.
> >
> >
> > I must have missed Fernando's email because I can't find the references
> > for nose :(
>
> Look in "SciPy Sprint Results". It's only a brief mention, though.
>
> > What are its advantages against the current numpy.testing framework ?
>
> Primarily:

[...]

It's worth noting that there's already a preliminary commit of this stuff here:

http://projects.scipy.org/scipy/scipy/browser/branches/testing_cleanup/scipy/sandbox/exmplpackage/tests

For now we put it in exmplpackage just so Matthew and I could quickly
coordinate, but I think he's working on propagating that throughout
more properly across the whole codebase.

But that directory already has a small readme (to which your email
could be added) as well as a little example, the test_foo file, that
showcases the various features.

Cheers,

f


From peridot.faceted at gmail.com  Fri Dec 28 18:53:44 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 28 Dec 2007 18:53:44 -0500
Subject: [Numpy-discussion] any better way to normalize a matrix
In-Reply-To: <47752BC8.8050608@noaa.gov>
References: <4b3eadfc-074d-479b-ac85-459b15737b2d@s19g2000prg.googlegroups.com>
	<47752BC8.8050608@noaa.gov>
Message-ID: <ce557a360712281553r36efc5cdxf24d4d4d328b2bbe@mail.gmail.com>

On 28/12/2007, Christopher Barker <Chris.Barker at noaa.gov> wrote:

> I like the array methods a lot -- is there any particular reason there
> is no ndarray.abs(), or has it just not been added?

Here I have to disagree with you.

Numpy provides ufuncs as general powerful tools for operating on
matrices. More can be added relatively easily, they provide not just
the basic "apply" operation but also "outer" and others. Adding
another way to accomplish the same operation just adds bulk to numpy.
I don't even like myarray.max(), preferring numpy.amax() or
numpy.maximum.reduce.

I realize we can't remove any array methods, but I don't think we
should add an array method for every unary function - surely you don't
want to see myarray.arctan()?

Anne


From stefan at sun.ac.za  Fri Dec 28 19:25:35 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Sat, 29 Dec 2007 02:25:35 +0200
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <47753313.2010006@enthought.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<20071228112617.GC9051@mentat.za.net>
	<47753313.2010006@enthought.com>
Message-ID: <20071229002535.GF9051@mentat.za.net>

On Fri, Dec 28, 2007 at 11:32:03AM -0600, Travis E. Oliphant wrote:
> I don't see the point in every file having a __docformat__ line, when 
> our documentaion formatting tool should already know the standard we are 
> using is.   It's just more cruft.   Besides the PEP was rejected, so I 
> don't know why we should be following it. 

Sorry, I didn't see the PEP was rejected.  Moot point, then.

> We obviously need a pre-processor to map our files to epydoc, so let's 
> do that instead of contorting docstrings into a format demanded by the 
> tool.

That's a good idea, thanks.  I'll take a look.

> It seems better to have docstrings than to have them fit into an 
> epydoc_-defined pattern.  

Certainly.

Thanks for the feedback.

Regards
St?fan


From Chris.Barker at noaa.gov  Fri Dec 28 19:57:42 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 28 Dec 2007 16:57:42 -0800
Subject: [Numpy-discussion] any better way to normalize a matrix
In-Reply-To: <ce557a360712281553r36efc5cdxf24d4d4d328b2bbe@mail.gmail.com>
References: <4b3eadfc-074d-479b-ac85-459b15737b2d@s19g2000prg.googlegroups.com>
	<47752BC8.8050608@noaa.gov>
	<ce557a360712281553r36efc5cdxf24d4d4d328b2bbe@mail.gmail.com>
Message-ID: <47759B86.3050203@noaa.gov>

Anne Archibald wrote:
> Numpy provides ufuncs as general powerful tools for operating on
> matrices. More can be added relatively easily, they provide not just
> the basic "apply" operation but also "outer" and others. Adding
> another way to accomplish the same operation just adds bulk to numpy.

Maybe so, but if it were up to me, I'd have just the methods, and not 
the functions -- I like namespaces and OO design. I've always thought it 
was a spurious argument that these kinds of functions can operate on 
things that aren't numpy arrays -- it's true, but they all return 
arrays, to it's not like they really are universal.

> Surely you don't want to see myarray.arctan()?

This was debated a fair bit on this list when the other methods were 
added. Personally, I like functions, rather than methods for things that 
"feel" like standard math -- trig functions, etc. I guess abs() kind of 
falls on the line.

All I know is that I expected abs()  to be a method this time.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From peridot.faceted at gmail.com  Fri Dec 28 21:01:43 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 28 Dec 2007 21:01:43 -0500
Subject: [Numpy-discussion] any better way to normalize a matrix
In-Reply-To: <47759B86.3050203@noaa.gov>
References: <4b3eadfc-074d-479b-ac85-459b15737b2d@s19g2000prg.googlegroups.com>
	<47752BC8.8050608@noaa.gov>
	<ce557a360712281553r36efc5cdxf24d4d4d328b2bbe@mail.gmail.com>
	<47759B86.3050203@noaa.gov>
Message-ID: <ce557a360712281801i36e9abax4e577eb9ce6fddb0@mail.gmail.com>

On 28/12/2007, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> Anne Archibald wrote:
> > Numpy provides ufuncs as general powerful tools for operating on
> > matrices. More can be added relatively easily, they provide not just
> > the basic "apply" operation but also "outer" and others. Adding
> > another way to accomplish the same operation just adds bulk to numpy.
>
> Maybe so, but if it were up to me, I'd have just the methods, and not
> the functions -- I like namespaces and OO design. I've always thought it
> was a spurious argument that these kinds of functions can operate on
> things that aren't numpy arrays -- it's true, but they all return
> arrays, to it's not like they really are universal.

It doesn't make a whole lot of sense to me to have n-ary methods
(though I don't think we currently have any ternary ufuncs). How would
you accomodate all the other operations ufuncs support?

The big problem with methods, it seems to me, is that if I want to add
a new ufunc, it's relatively easy. In fact, with vectorize() I do that
all the time (with all sorts of arities). But adding compiled ufuncs
in a new module is relatively straightforward; it's not at all clear
how a separate package could ever add new methods to the ndarray
object. This is not hypothetical - scipy.special introduces many
mathematical functions that either are or should be ufuncs.

As for whether this is "object-oriented", well, ufuncs are objects
with a variety of methods; one could productively inherit from them
(at least in principle, though I've never had occasion to).

Anyway, I don't know that we need to rehash the old discussion. I just
wanted to point out the value of ufuncs as objects.

Anne


From dikshie at gmail.com  Fri Dec 28 22:23:45 2007
From: dikshie at gmail.com (dikshie)
Date: Sat, 29 Dec 2007 12:23:45 +0900
Subject: [Numpy-discussion] numpy installed but can' use
In-Reply-To: <47752B04.1040205@noaa.gov>
References: <910e60e80712280828i749b543et6a7c490b1f75f612@mail.gmail.com>
	<47752B04.1040205@noaa.gov>
Message-ID: <910e60e80712281923pd13aa8radd713c061306b40@mail.gmail.com>

On Dec 29, 2007 1:57 AM, Christopher Barker <Chris.Barker at noaa.gov> wrote:
the names in the
> numpy module into this namespace. There is no name: "numpy" in the numpy
> module -- it's the name of the module itself. Try:
>
> import numpy
> numpy.__version__


so import numpy and from numpy import *
are different ?




-dikshie-


From oliphant at enthought.com  Fri Dec 28 23:38:45 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 28 Dec 2007 22:38:45 -0600
Subject: [Numpy-discussion] numpy installed but can' use
In-Reply-To: <910e60e80712281923pd13aa8radd713c061306b40@mail.gmail.com>
References: <910e60e80712280828i749b543et6a7c490b1f75f612@mail.gmail.com>	<47752B04.1040205@noaa.gov>
	<910e60e80712281923pd13aa8radd713c061306b40@mail.gmail.com>
Message-ID: <4775CF55.6000601@enthought.com>

dikshie wrote:
> On Dec 29, 2007 1:57 AM, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> the names in the
>   
>> numpy module into this namespace. There is no name: "numpy" in the numpy
>> module -- it's the name of the module itself. Try:
>>
>> import numpy
>> numpy.__version__
>>     
>
>
> so import numpy and from numpy import *
> are different ?
>   

Yes, the key is what "names" are "imported" into the current namespace 
(what names are visible to your code).

import numpy

loads the module and places the name "numpy" in the current namespace 
which points to the loaded module. 

from numpy import *

loads the module and places all the "exported" names from numpy into the 
current namespace.  The list of exported names, however, does not 
include the module numpy itself and so the current namespace does not 
get a "numpy" entry.

-Travis O.



From dikshie at gmail.com  Sat Dec 29 00:42:36 2007
From: dikshie at gmail.com (dikshie)
Date: Sat, 29 Dec 2007 14:42:36 +0900
Subject: [Numpy-discussion] numpy installed but can' use
In-Reply-To: <4775CF55.6000601@enthought.com>
References: <910e60e80712280828i749b543et6a7c490b1f75f612@mail.gmail.com>
	<47752B04.1040205@noaa.gov>
	<910e60e80712281923pd13aa8radd713c061306b40@mail.gmail.com>
	<4775CF55.6000601@enthought.com>
Message-ID: <910e60e80712282142l6f89d2ffgd4fc317e73e00284@mail.gmail.com>

On Dec 29, 2007 1:38 PM, Travis E. Oliphant <oliphant at enthought.com> wrote:

> Yes, the key is what "names" are "imported" into the current namespace
> (what names are visible to your code).
>
> import numpy
>
> loads the module and places the name "numpy" in the current namespace
> which points to the loaded module.
>
> from numpy import *
>
> loads the module and places all the "exported" names from numpy into the
> current namespace.  The list of exported names, however, does not
> include the module numpy itself and so the current namespace does not
> get a "numpy" entry.

i see.
thank you very much for the explanation.

-dikshie-


From Bruce_Sherwood at ncsu.edu  Sat Dec 29 00:47:09 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Sat, 29 Dec 2007 00:47:09 -0500
Subject: [Numpy-discussion] [C++-sig] Overloading sqrt(5.5)*myvector
In-Reply-To: <4773C17B.4050802@ncsu.edu>
References: <4771F0AE.40406@ncsu.edu>	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>	<47732B84.70400@ncsu.edu>	<4773325D.3040907@gmail.com>	<4773BD97.1000703@ncsu.edu>
	<4773C17B.4050802@ncsu.edu>
Message-ID: <4775DF5D.3030400@ncsu.edu>

I realized belatedly that I should upgrade from Boost 1.33 to 1.34. 
Alas, that didn't cure my problem.

Bruce Sherwood

Bruce Sherwood wrote:
> I should have added: This structure worked with the older version of 
> VPython which used Numeric, but it doesn't work in the beta version 
> which uses numpy. Since I don't know enough about either numpy or Boost, 
> I'm left guessing which subsystem is the source of my difficulties, and 
> clueless about how to remedy them.
>
> Bruce Sherwood
>
> Bruce Sherwood wrote:
>   
>> Thanks for the comment, which limits the range of possible solutions. 
>> The VPython vector class is implemented in C++, not in Python. I made up 
>> the simple test in my previous note to try out the solution that had 
>> been offered and which you have usefully ruled out. Here is the relevant 
>> part of the vector class, which indeed doesn't look like an ndarray:
>>
>> inline vector
>> operator*( const double s) const throw()
>> { return vector( s*x, s*y, s*z); }
>>
>> and here is the free function for right multiplication:
>>
>> inline vector
>> operator*( const double& s, const vector& v)
>> { return vector( s*v.x, s*v.y, s*v.z); }
>>
>> Maybe the unsolvable problem is in the Boost definitions:
>>
>> py::class_<vector>("vector", py::init< py::optional<double, double, 
>> double> >())
>>      .def( self * double())
>>      .def( double() * self)
>>
>> Left multiplication is fine, but right multiplication isn't.
>>
>> Bruce Sherwood
>>
>> Robert Kern wrote:
>>   
>>     
>>> Bruce Sherwood wrote:
>>>   
>>>     
>>>       
>>>> Thanks for the suggestion. It hadn't occurred to me to try to override 
>>>> numpy as you suggest. However, when I try the code shown below as the 
>>>> start of a test of this scheme, I get the following error:
>>>>
>>>> Traceback (most recent call last):
>>>>   File "C:\Documents and Settings\Bruce\My 
>>>> Documents\0VPythonWork\vectors.py", line 24, in <module>
>>>>     numpy.float64.__mul__ = new_mul
>>>> TypeError: can't set attributes of built-in/extension type 'numpy.float64'
>>>>
>>>> I'm copying this to the numpy discussion list, as maybe someone there 
>>>> will see where to go starting from your suggestion.
>>>>     
>>>>       
>>>>         
>>> Like most (or all) builtin-types, the numpy float scalars do not permit
>>> replacing their methods from Python.
>>>
>>> I'm not familiar with vpython's vector. If you can make it "not look like an
>>> ndarray", then you should be able to just implement __rmul__ on vector.
>>>
>>>   
>>>     
>>>       
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>   
>>     
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   


From Bruce_Sherwood at ncsu.edu  Sat Dec 29 02:29:05 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Sat, 29 Dec 2007 02:29:05 -0500
Subject: [Numpy-discussion] numpy and math sqrt timings
In-Reply-To: <fl0m0q$mor$1@ger.gmane.org>
References: <47709E5F.8020009@ncsu.edu>	<fktvg3$tij$1@ger.gmane.org>	<4772E796.4040207@ncsu.edu>	<fl0k58$h6b$1@ger.gmane.org>
	<fl0m0q$mor$1@ger.gmane.org>
Message-ID: <4775F741.1060103@ncsu.edu>

On the VPython list Scott Daniels suggested using try/except to deal 
with the problem of sqrt(5.5) being numpy.float64 and thereby making 
sqrt(5.5)*(VPython vector) not a (VPython vector), which ends up as a 
big performance hit on existing programs. I tried his suggestion and did 
some timing using the program shown below.

Using "from numpy import *", the numpy sqrt(5.5) gives 5.7 microsec per 
sqrt, whereas using "from math import *" a sqrt is only 0.8 microsec. 
Why is numpy so much slower than math on this simple case? For 
completeness I also timed the old Numeric sqrt, which was 14 microsec, 
so numpy is a big improvement, but still very slow compared to math.

Using Daniels's suggestion of first trying the math sqrt, falling 
through to the numpy sqrt only if the argument isn't a simple scalar, 
gives 1.3 microsec per sqrt on the simple case of a scalar argument. 
Shouldn't/couldn't numpy do something like this internally?

Bruce Sherwood

----------------------------
from math import *
mathsqrt = sqrt
from numpy import *
numpysqrt = sqrt
from time import clock

# 0.8 microsec for "raw" math sqrt
# 5.7 microsec for "raw" numpy sqrt
# 1.3 microsec if we try math sqrt first

def sqrt(x):
    try: return mathsqrt(x)
    except TypeError: return numpysqrt(x)

# Check that numpy sqrt is invoked on an array:
nums = array([1,2,3])
print sqrt(nums)
  
x = 5.5
N = 500000
t1 = clock()
for n in range(N):
    y = sqrt(x)
    y = sqrt(x)
    y = sqrt(x)
    y = sqrt(x)
    y = sqrt(x)
    y = sqrt(x)
    y = sqrt(x)
    y = sqrt(x)
    y = sqrt(x)
    y = sqrt(x)
t2 = clock()
for n in range(N):
    pass
t3 = clock()
# t3-t2 is the loop overhead (turns out negligible)
print "%i loops over 10 sqrt's takes %.1f seconds" % (N,t2-t1)
print "Total loop overhead = %.2f seconds (negligible)" % (t3-t2)
print "One sqrt takes %.1f microseconds" % (1e6*((t2-t1)-(t3-t2))/(10*N))




From matthieu.brucher at gmail.com  Sat Dec 29 02:38:11 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Sat, 29 Dec 2007 08:38:11 +0100
Subject: [Numpy-discussion] numpy and math sqrt timings
In-Reply-To: <4775F741.1060103@ncsu.edu>
References: <47709E5F.8020009@ncsu.edu> <fktvg3$tij$1@ger.gmane.org>
	<4772E796.4040207@ncsu.edu> <fl0k58$h6b$1@ger.gmane.org>
	<fl0m0q$mor$1@ger.gmane.org> <4775F741.1060103@ncsu.edu>
Message-ID: <e76aa17f0712282338u6fe26f86k4e06525bd6cd21cc@mail.gmail.com>

Hi,

This is a know fact, you should use Python default functions if you have
only one value.
If Numpy uses math.sqrt for floatting point number, it would have to use
cmath for complex values as well. Now, I don't know if an additionnal test
will slow down Numpy, if this is the case, then we should stay with the
current situation ; if I have a signle value to compute, I always use math
instead of Numpy.

Matthieu

2007/12/29, Bruce Sherwood <Bruce_Sherwood at ncsu.edu>:
>
> On the VPython list Scott Daniels suggested using try/except to deal
> with the problem of sqrt(5.5) being numpy.float64 and thereby making
> sqrt(5.5)*(VPython vector) not a (VPython vector), which ends up as a
> big performance hit on existing programs. I tried his suggestion and did
> some timing using the program shown below.
>
> Using "from numpy import *", the numpy sqrt(5.5) gives 5.7 microsec per
> sqrt, whereas using "from math import *" a sqrt is only 0.8 microsec.
> Why is numpy so much slower than math on this simple case? For
> completeness I also timed the old Numeric sqrt, which was 14 microsec,
> so numpy is a big improvement, but still very slow compared to math.
>
> Using Daniels's suggestion of first trying the math sqrt, falling
> through to the numpy sqrt only if the argument isn't a simple scalar,
> gives 1.3 microsec per sqrt on the simple case of a scalar argument.
> Shouldn't/couldn't numpy do something like this internally?
>
> Bruce Sherwood
>
> ----------------------------
> from math import *
> mathsqrt = sqrt
> from numpy import *
> numpysqrt = sqrt
> from time import clock
>
> # 0.8 microsec for "raw" math sqrt
> # 5.7 microsec for "raw" numpy sqrt
> # 1.3 microsec if we try math sqrt first
>
> def sqrt(x):
>     try: return mathsqrt(x)
>     except TypeError: return numpysqrt(x)
>
> # Check that numpy sqrt is invoked on an array:
> nums = array([1,2,3])
> print sqrt(nums)
>
> x = 5.5
> N = 500000
> t1 = clock()
> for n in range(N):
>     y = sqrt(x)
>     y = sqrt(x)
>     y = sqrt(x)
>     y = sqrt(x)
>     y = sqrt(x)
>     y = sqrt(x)
>     y = sqrt(x)
>     y = sqrt(x)
>     y = sqrt(x)
>     y = sqrt(x)
> t2 = clock()
> for n in range(N):
>     pass
> t3 = clock()
> # t3-t2 is the loop overhead (turns out negligible)
> print "%i loops over 10 sqrt's takes %.1f seconds" % (N,t2-t1)
> print "Total loop overhead = %.2f seconds (negligible)" % (t3-t2)
> print "One sqrt takes %.1f microseconds" % (1e6*((t2-t1)-(t3-t2))/(10*N))
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071229/524a3bf6/attachment.html>

From aisaac at american.edu  Sat Dec 29 07:55:12 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Sat, 29 Dec 2007 07:55:12 -0500
Subject: [Numpy-discussion] numpy installed but can' use
In-Reply-To: <910e60e80712281923pd13aa8radd713c061306b40@mail.gmail.com>
References: <910e60e80712280828i749b543et6a7c490b1f75f612@mail.gmail.com><47752B04.1040205@noaa.gov><910e60e80712281923pd13aa8radd713c061306b40@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1576-20071229-075512.00@american.edu>

On Sat, 29 Dec 2007, dikshie apparently wrote:
> so import numpy and from numpy import *
> are different ? 

http://docs.python.org/tut/node8.html

hth,
Alan Isaac






From Bruce_Sherwood at ncsu.edu  Sat Dec 29 15:41:53 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Sat, 29 Dec 2007 15:41:53 -0500
Subject: [Numpy-discussion] [C++-sig]  Overloading sqrt(5.5)*myvector
In-Reply-To: <7465b6170712291047k2933ceb5o20a0faad77bd85c0@mail.gmail.com>
References: <4771F0AE.40406@ncsu.edu>	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>	<47732B84.70400@ncsu.edu>
	<4773325D.3040907@gmail.com>	<4773BD97.1000703@ncsu.edu>
	<4773C17B.4050802@ncsu.edu>	<4775DF5D.3030400@ncsu.edu>
	<7465b6170712291047k2933ceb5o20a0faad77bd85c0@mail.gmail.com>
Message-ID: <4776B111.9020607@ncsu.edu>

Roman Yakovenko wrote:
> On Dec 29, 2007 7:47 AM, Bruce Sherwood <Bruce_Sherwood at ncsu.edu> wrote:
>   
>> I realized belatedly that I should upgrade from Boost 1.33 to 1.34.
>> Alas, that didn't cure my problem.
>>     
> Can you post small and complete example of what you are trying to achieve?
>   
I don't have a "small and complete" example available, but I'll 
summarize from earlier posts. VPython (vpython.org) has its own vector 
class to mimic the properties of 3D vectors used in physics, in the 
service of easy creation of 3D animations. There is a beta version which 
imports numpy and uses it internally; the older production version uses 
Numeric. Boost python and thread libraries are used to connect the C++ 
VPython code to Python.

There is operator overloading that includes scalar*vector and 
vector*scalar, both producing vector. With Numeric, sqrt produced a 
float, which was a scalar for the operator overloading. With numpy, sqrt 
produces a numpy.float64 which is caught by vector*scalar but not by 
scalar*vector, which means that scalar*vector produces an ndarray rather 
than a vector, which leads to a big performance hit in existing VPython 
programs. The overloading and Boost code is the same in the 
VPython/Numeric and VPython/numpy versions. I don't know whether the 
problem is with numpy or with Boost or with the combination of the two.

Here is the relevant part of the vector class:

inline vector
operator*( const double s) const throw()
{ return vector( s*x, s*y, s*z); }

and here is the free function for right multiplication:

inline vector
operator*( const double& s, const vector& v)
{ return vector( s*v.x, s*v.y, s*v.z); }

Maybe the problem is in the Boost definitions:

py::class_<vector>("vector", py::init< py::optional<double, double, 
double> >())
    .def( self * double())
    .def( double() * self)

Left multiplication is fine, but right multiplication isn't.

A colleague suggested the following Boost declarations but cautioned 
that he wasn't sure of the syntax for referring to operator, and indeed 
this doesn't compile:

.def( "__mul__", &vector::operator*(double), "Multiply vector times scalar")
.def( "__rmul__", &operator*(const double&, const vector&), "Multiply 
scalar times vector")

I would really appreciate a Boost or numpy expert being able to tell me 
what's wrong (if anything) with these forms. However, I may have a 
useful workaround as I described in a post to the numpy discussion list. 
A colleague suggested that I do something like this for sqrt and other 
such mathematical functions:

def sqrt(x):
   try: return mathsqrt(x)
   except TypeError: return numpysqrt(x)

That is, first try the simple case of a scalar argument, handled by the 
math module sqrt, and only use the numpy sqrt routine in the case of an 
array argument. Even with the overhead of the try/except machinery, one 
gets must faster square roots for scalar arguments this way than with 
the numpy sqrt.

Bruce Sherwood



From Bruce_Sherwood at ncsu.edu  Sat Dec 29 17:02:36 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Sat, 29 Dec 2007 17:02:36 -0500
Subject: [Numpy-discussion] [C++-sig]  Overloading sqrt(5.5)*myvector
In-Reply-To: <4776B111.9020607@ncsu.edu>
References: <4771F0AE.40406@ncsu.edu>	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>	<47732B84.70400@ncsu.edu>
	<4773325D.3040907@gmail.com>	<4773BD97.1000703@ncsu.edu>
	<4773C17B.4050802@ncsu.edu>	<4775DF5D.3030400@ncsu.edu>
	<7465b6170712291047k2933ceb5o20a0faad77bd85c0@mail.gmail.com>
	<4776B111.9020607@ncsu.edu>
Message-ID: <4776C3FC.7030101@ncsu.edu>

I found by timing measurements that a faster scheme with less penalty 
for the case of sqrt(array) looks like this:

def sqrt(x):
     if type(x) is float: return mathsqrt(x)
     return numpysqrt(x)

Bruce Sherwood wrote:
> Roman Yakovenko wrote:
>> On Dec 29, 2007 7:47 AM, Bruce Sherwood <Bruce_Sherwood at ncsu.edu> wrote:
>>  
>>> I realized belatedly that I should upgrade from Boost 1.33 to 1.34.
>>> Alas, that didn't cure my problem.
>>>     
>> Can you post small and complete example of what you are trying to 
>> achieve?
>>   
> I don't have a "small and complete" example available, but I'll 
> summarize from earlier posts. VPython (vpython.org) has its own vector 
> class to mimic the properties of 3D vectors used in physics, in the 
> service of easy creation of 3D animations. There is a beta version 
> which imports numpy and uses it internally; the older production 
> version uses Numeric. Boost python and thread libraries are used to 
> connect the C++ VPython code to Python.
>
> There is operator overloading that includes scalar*vector and 
> vector*scalar, both producing vector. With Numeric, sqrt produced a 
> float, which was a scalar for the operator overloading. With numpy, 
> sqrt produces a numpy.float64 which is caught by vector*scalar but not 
> by scalar*vector, which means that scalar*vector produces an ndarray 
> rather than a vector, which leads to a big performance hit in existing 
> VPython programs. The overloading and Boost code is the same in the 
> VPython/Numeric and VPython/numpy versions. I don't know whether the 
> problem is with numpy or with Boost or with the combination of the two.
>
> Here is the relevant part of the vector class:
>
> inline vector
> operator*( const double s) const throw()
> { return vector( s*x, s*y, s*z); }
>
> and here is the free function for right multiplication:
>
> inline vector
> operator*( const double& s, const vector& v)
> { return vector( s*v.x, s*v.y, s*v.z); }
>
> Maybe the problem is in the Boost definitions:
>
> py::class_<vector>("vector", py::init< py::optional<double, double, 
> double> >())
>    .def( self * double())
>    .def( double() * self)
>
> Left multiplication is fine, but right multiplication isn't.
>
> A colleague suggested the following Boost declarations but cautioned 
> that he wasn't sure of the syntax for referring to operator, and 
> indeed this doesn't compile:
>
> .def( "__mul__", &vector::operator*(double), "Multiply vector times 
> scalar")
> .def( "__rmul__", &operator*(const double&, const vector&), "Multiply 
> scalar times vector")
>
> I would really appreciate a Boost or numpy expert being able to tell 
> me what's wrong (if anything) with these forms. However, I may have a 
> useful workaround as I described in a post to the numpy discussion 
> list. A colleague suggested that I do something like this for sqrt and 
> other such mathematical functions:
>
> def sqrt(x):
>   try: return mathsqrt(x)
>   except TypeError: return numpysqrt(x)
>
> That is, first try the simple case of a scalar argument, handled by 
> the math module sqrt, and only use the numpy sqrt routine in the case 
> of an array argument. Even with the overhead of the try/except 
> machinery, one gets must faster square roots for scalar arguments this 
> way than with the numpy sqrt.
>
> Bruce Sherwood
>


From Bruce_Sherwood at ncsu.edu  Sat Dec 29 18:00:53 2007
From: Bruce_Sherwood at ncsu.edu (Bruce Sherwood)
Date: Sat, 29 Dec 2007 18:00:53 -0500
Subject: [Numpy-discussion] [C++-sig]  Overloading sqrt(5.5)*myvector
In-Reply-To: <4776C3FC.7030101@ncsu.edu>
References: <4771F0AE.40406@ncsu.edu>	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>	<47732B84.70400@ncsu.edu>	<4773325D.3040907@gmail.com>	<4773BD97.1000703@ncsu.edu>	<4773C17B.4050802@ncsu.edu>	<4775DF5D.3030400@ncsu.edu>	<7465b6170712291047k2933ceb5o20a0faad77bd85c0@mail.gmail.com>	<4776B111.9020607@ncsu.edu>
	<4776C3FC.7030101@ncsu.edu>
Message-ID: <4776D1A5.60106@ncsu.edu>

Okay, I've implemented the scheme below that was proposed by Scott 
Daniels on the VPython mailing list, and it solves my problem. It's also 
much faster than using numpy directly: even with the "def "and "if" 
overhead: sqrt(scalar) is over 3 times faster than the numpy sqrt, and 
sqrt(array) is very nearly as fast as the numpy sqrt.

Thanks to those who made suggestions. There remains the question of why 
operator overloading of the kind I've described worked with Numeric and 
Boost but not with numpy and Boost. There is also the question of 
whether it would pay for numpy to make what is probably an exceedingly 
fast check and do much faster calculations of sqrt(scalar) and other 
such mathematical functions.

Bruce Sherwood

Bruce Sherwood wrote:
> I found by timing measurements that a faster scheme with less penalty 
> for the case of sqrt(array) looks like this:
>
> def sqrt(x):
>      if type(x) is float: return mathsqrt(x)
>      return numpysqrt(x)
>
> Bruce Sherwood wrote:
>   
>> Roman Yakovenko wrote:
>>     
>>> On Dec 29, 2007 7:47 AM, Bruce Sherwood <Bruce_Sherwood at ncsu.edu> wrote:
>>>  
>>>       
>>>> I realized belatedly that I should upgrade from Boost 1.33 to 1.34.
>>>> Alas, that didn't cure my problem.
>>>>     
>>>>         
>>> Can you post small and complete example of what you are trying to 
>>> achieve?
>>>   
>>>       
>> I don't have a "small and complete" example available, but I'll 
>> summarize from earlier posts. VPython (vpython.org) has its own vector 
>> class to mimic the properties of 3D vectors used in physics, in the 
>> service of easy creation of 3D animations. There is a beta version 
>> which imports numpy and uses it internally; the older production 
>> version uses Numeric. Boost python and thread libraries are used to 
>> connect the C++ VPython code to Python.
>>
>> There is operator overloading that includes scalar*vector and 
>> vector*scalar, both producing vector. With Numeric, sqrt produced a 
>> float, which was a scalar for the operator overloading. With numpy, 
>> sqrt produces a numpy.float64 which is caught by vector*scalar but not 
>> by scalar*vector, which means that scalar*vector produces an ndarray 
>> rather than a vector, which leads to a big performance hit in existing 
>> VPython programs. The overloading and Boost code is the same in the 
>> VPython/Numeric and VPython/numpy versions. I don't know whether the 
>> problem is with numpy or with Boost or with the combination of the two.
>>
>> Here is the relevant part of the vector class:
>>
>> inline vector
>> operator*( const double s) const throw()
>> { return vector( s*x, s*y, s*z); }
>>
>> and here is the free function for right multiplication:
>>
>> inline vector
>> operator*( const double& s, const vector& v)
>> { return vector( s*v.x, s*v.y, s*v.z); }
>>
>> Maybe the problem is in the Boost definitions:
>>
>> py::class_<vector>("vector", py::init< py::optional<double, double, 
>> double> >())
>>    .def( self * double())
>>    .def( double() * self)
>>
>> Left multiplication is fine, but right multiplication isn't.
>>
>> A colleague suggested the following Boost declarations but cautioned 
>> that he wasn't sure of the syntax for referring to operator, and 
>> indeed this doesn't compile:
>>
>> .def( "__mul__", &vector::operator*(double), "Multiply vector times 
>> scalar")
>> .def( "__rmul__", &operator*(const double&, const vector&), "Multiply 
>> scalar times vector")
>>
>> I would really appreciate a Boost or numpy expert being able to tell 
>> me what's wrong (if anything) with these forms. However, I may have a 
>> useful workaround as I described in a post to the numpy discussion 
>> list. A colleague suggested that I do something like this for sqrt and 
>> other such mathematical functions:
>>
>> def sqrt(x):
>>   try: return mathsqrt(x)
>>   except TypeError: return numpysqrt(x)
>>
>> That is, first try the simple case of a scalar argument, handled by 
>> the math module sqrt, and only use the numpy sqrt routine in the case 
>> of an array argument. Even with the overhead of the try/except 
>> machinery, one gets must faster square roots for scalar arguments this 
>> way than with the numpy sqrt.
>>
>> Bruce Sherwood
>>
>>     
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   


From robert.kern at gmail.com  Sat Dec 29 18:24:52 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 29 Dec 2007 18:24:52 -0500
Subject: [Numpy-discussion] [C++-sig]  Overloading sqrt(5.5)*myvector
In-Reply-To: <4776D1A5.60106@ncsu.edu>
References: <4771F0AE.40406@ncsu.edu>	<7465b6170712252340o159aa52qe1dae3707e3ab625@mail.gmail.com>	<47732B84.70400@ncsu.edu>	<4773325D.3040907@gmail.com>	<4773BD97.1000703@ncsu.edu>	<4773C17B.4050802@ncsu.edu>	<4775DF5D.3030400@ncsu.edu>	<7465b6170712291047k2933ceb5o20a0faad77bd85c0@mail.gmail.com>	<4776B111.9020607@ncsu.edu>	<4776C3FC.7030101@ncsu.edu>
	<4776D1A5.60106@ncsu.edu>
Message-ID: <4776D744.5010601@gmail.com>

Bruce Sherwood wrote:
> There is also the question of 
> whether it would pay for numpy to make what is probably an exceedingly 
> fast check and do much faster calculations of sqrt(scalar) and other 
> such mathematical functions.

There is no question that it would pay. It takes time and effort to implement,
though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Sat Dec 29 20:51:57 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 29 Dec 2007 19:51:57 -0600
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <20071228112617.GC9051@mentat.za.net>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<20071228112617.GC9051@mentat.za.net>
Message-ID: <e06186140712291751s757b2882ufffa28892e5e784a@mail.gmail.com>

On Dec 28, 2007 5:26 AM, Stefan van der Walt <stefan at sun.ac.za> wrote:

> On Thu, Dec 27, 2007 at 09:27:09PM -0800, Jarrod Millman wrote:
> > On Dec 27, 2007 7:42 PM, Travis E. Oliphant <oliphant at enthought.com>
> wrote:
> > > Doc-day will start tomorrow (in about 12 hours).  It will be Friday
> for
> > > much of America and be moving into Saturday for Europe and Asia.  Join
> > > in on the irc.freenode.net  (channel scipy) to coordinate effort.  I
> > > imaging people will be in an out.  I plan on being available in IRC
> from
> > > about 9:30 am CST to 6:00 pm CST and then possibly later.
> > >
> > > If you are available at different times in different parts of the
> > > world,  jump in and pick something to work on.
> >
> > Since this is our first doc-day, it will be fairly informal.  Travis
> > is going to be trying to get some estimate of which packages need the
> > most work.  But if there is some area of NumPy or SciPy you are
> > familiar with, please go ahead and pitch in.  Here is the current
> > NumPy/ SciPy coding standard including docstring standards:
> > http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines
>
> I have some questions regarding Travis' latest modifications to the
> documentation guidelines:
>
> The following section was removed, why?
>
> """
> A reST-documented module should define::
>
>  __docformat__ = 'restructuredtext en'
>
> at the top level in accordance with `PEP 258
> <http://www.python.org/dev/peps/pep-0258>`__.  Note that the
> ``__docformat__`` variable in a package's ``__init__.py`` file does
> not apply to objects defined in subpackages and submodules.
> """
>
> We had a long discussion on the mailing list on the pros and cons of
> "*Parameters*:" vs. "Parameters:".  I see now that it has been changed
> to
>
>  Parameters
>  ----------
>

If we are going to avoid ReST and html definition lists, i.e.

Parameters:

then we should probably avoid ReST and headings

Parameters
----------

because that will likely be moved about. In fact, we should probably throw
ReST overboard if we are going to avoid ReST'isms. I personally think the
underline is ugly (hi Travis), but I suppose that's beside the point. If we
stay with ReST it would be nice to retain the links in the see also section.

I'm not at home at the moment, so I don't know how the new style is rendered
by epydoc, or even if we are still going to use epydoc. If not, we should
definitely decide on the structure of the docstrings and stick to it. With a
bit more pain in the appearence, we could also format for doxygen, which is
pretty standard and would also work for the C code. I suspect that's
not going to fly, though ;)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071229/e5edb5fa/attachment.html>

From fperez.net at gmail.com  Sat Dec 29 20:59:27 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Sat, 29 Dec 2007 18:59:27 -0700
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <e06186140712291751s757b2882ufffa28892e5e784a@mail.gmail.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<20071228112617.GC9051@mentat.za.net>
	<e06186140712291751s757b2882ufffa28892e5e784a@mail.gmail.com>
Message-ID: <db6b5ecc0712291759m7900ff94we2ca1983d90edea5@mail.gmail.com>

On Dec 29, 2007 6:51 PM, Charles R Harris <charlesr.harris at gmail.com> wrote:

> If not, we should
> definitely decide on the structure of the docstrings and stick to it.

+100

I'm about to commit docstrings for scimath (what I started yesterday).
 After some fixes to the numpy build, I can now work in-place and get
things like

tlon[~]> nosetests --with-doctest numpy.lib.scimath
.......
----------------------------------------------------------------------
Ran 7 tests in 0.660s

OK


Which is nice.  For now I'm adhering to what was in the guide:

>>> log?
Type:           function
Base Class:     <type 'function'>
Namespace:      Interactive
File:           /home/installers/src/scipy/numpy/numpy/lib/scimath.py
Definition:     log(x)
Docstring:
    Return the natural logarithm of x.

    If x contains negative inputs, the answer is computed and returned in the
    complex domain.

    Parameters
    ----------
    x : array_like

    Returns
    -------
    array_like

    Examples
    --------
    >>> import math

    >>> log(math.exp(1))
    1.0

    Negative arguments are correctly handled (recall that for negative
    arguments, the identity exp(log(z))==z does not hold anymore):

    >>> log(-math.exp(1)) == (1+1j*math.pi)
    True


But it's easy enough to fix them if a change is made.  But let's
settle this *soon* please, so that if changes need to be made it's
just a few, and we can just move on.

At this point we just need *a* decision.  It doesn't need to be
perfect, but any docstring is better than our canonical:

>>> np.cumsum?
Type:           function
Base Class:     <type 'function'>
Namespace:      Interactive
File:
/home/fperez/usr/opt/lib/python2.5/site-packages/numpy/core/fromnumeric.py
Definition:     np.cumsum(a, axis=None, dtype=None, out=None)
Docstring:
    Sum the array over the given axis.

    Blah, Blah.


I got that 'Blah blah' while lecturing to 20 developers and scientists
at a workshop at NCAR (the National Center for Atmospheric Research in
Boulder), projected on a 20 foot-wide screen.  A bit embarrassing, I
must admit.

So please, no more code without full docstrings in numpy/scipy....

Cheers,


f


From charlesr.harris at gmail.com  Sat Dec 29 23:52:08 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 29 Dec 2007 22:52:08 -0600
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <db6b5ecc0712291759m7900ff94we2ca1983d90edea5@mail.gmail.com>
References: <477470C0.3030602@enthought.com>
	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>
	<20071228112617.GC9051@mentat.za.net>
	<e06186140712291751s757b2882ufffa28892e5e784a@mail.gmail.com>
	<db6b5ecc0712291759m7900ff94we2ca1983d90edea5@mail.gmail.com>
Message-ID: <e06186140712292052h61679f2fu6bd12e8eca147b2@mail.gmail.com>

On Dec 29, 2007 7:59 PM, Fernando Perez <fperez.net at gmail.com> wrote:

> On Dec 29, 2007 6:51 PM, Charles R Harris <charlesr.harris at gmail.com>
> wrote:
>
> > If not, we should
> > definitely decide on the structure of the docstrings and stick to it.
>
> +100
>

I have raised the topic of documentation formats on the list several times,
and the discussions have always petered out. So I made a decision, posted
what *worked* with epydoc, and also generated a significant amount of
documentation. Now that is tossed out with hardly a mention. I would like to
propose that anyone who submits a new documentation standard also
submit code to parse the result, compare it to the existing
practice, discuss why it is an improvement, and generate documentation to
show how it works. I did all of those things, I expect nothing less from
others if we are going to reinvent the wheel. And the end result should be
available as soon the the formatting changes are made, not at some
indefinite point in the future.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071229/9a986309/attachment.html>

From oliphant at enthought.com  Sun Dec 30 00:52:43 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Sat, 29 Dec 2007 23:52:43 -0600
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <e06186140712292052h61679f2fu6bd12e8eca147b2@mail.gmail.com>
References: <477470C0.3030602@enthought.com>	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>	<20071228112617.GC9051@mentat.za.net>	<e06186140712291751s757b2882ufffa28892e5e784a@mail.gmail.com>	<db6b5ecc0712291759m7900ff94we2ca1983d90edea5@mail.gmail.com>
	<e06186140712292052h61679f2fu6bd12e8eca147b2@mail.gmail.com>
Message-ID: <4777322B.6090702@enthought.com>

Charles R Harris wrote:
>
>
> On Dec 29, 2007 7:59 PM, Fernando Perez <fperez.net at gmail.com 
> <mailto:fperez.net at gmail.com>> wrote:
>
>     On Dec 29, 2007 6:51 PM, Charles R Harris
>     <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
>
>     > If not, we should
>     > definitely decide on the structure of the docstrings and stick
>     to it.
>
>     +100
>
>  
> I have raised the topic of documentation formats on the list several 
> times, and the discussions have always petered out. So I made a 
> decision, posted what *worked* with epydoc, and also generated a 
> significant amount of documentation. Now that is tossed out with 
> hardly a mention. I would like to propose that anyone who submits a 
> new documentation standard also submit code to parse the result, 
> compare it to the existing practice, discuss why it is an 
> improvement, and generate documentation to show how it works. I did 
> all of those things, I expect nothing less from others if we are going 
> to reinvent the wheel. And the end result should be available as soon 
> the the formatting changes are made, not at some indefinite point in 
> the future.

Hey Chuck,

I'm the guilty party in the docstring revisions, and I'm very sorry that 
my actions seemed to undermine your contributions (which are very much 
appreciated).

The changes to the docstring format that I pushed are not much different 
from what was put in place.  However, they make it so that the 
documentation standard for SciPy and NumPy is not different from the ETS 
standard.  In addition, I think it conserves horizontal real-estate and 
looks better when just printed as plain text (which is mostly how 
docstrings get read).  

I ran epydoc (on the example.py file) and it still produces output that 
is very readable even if it is not currently as good looking as the 
previously rendered result.   A relatively simple pre-processer should 
be able to produce fancier output with epydoc.

As we were going to have a Doc-Day, and several people were going to be 
adding documentation, I wanted to cement the docstring standard and I 
did not want it to be based on a "technological" reason (i.e. we have 
all this stuff in the docstring so that epydoc can produce pretty 
output).    My experience is that >90% of docstring-reading is done in 
plain-text, so this should drive the decision, and not the availability 
of a tool to render it.

We had to cement a decision, so I did it.  I am very sorry for stepping 
on your toes.  I should have looked at who committed the most recent 
changes to the documentation information pages and taken more time to 
work things out with you privately (but it was very early in the morning 
on Friday (2-4 am ish).   I also did not want to have another 
mailing-list discussion on docstring formats because as you've 
witnessed, it is not usually helpful on something which is primarily in 
the eye of the beholder.

I'll take responsibility to get something in place to render the 
docstrings more beautifully.  If anybody is interested in helping with 
that, please let me know.

Sheepishly yours,

-Travis O.





From nicholasinparis at gmail.com  Sun Dec 30 03:42:18 2007
From: nicholasinparis at gmail.com (Nicholas)
Date: Sun, 30 Dec 2007 09:42:18 +0100
Subject: [Numpy-discussion] Passing a Null pointer after ndarray prototype
	definition
Message-ID: <cc213fa80712300042u46d0c272mcae473633ecbb9a2@mail.gmail.com>

I have a problem in that I have a DLL function which accepts several numpy
double arrays, but you need to pass a NULL pointer for the ones from which
you do not require results. The prototype is as follows:

T_ndDp = numpy.ctypeslib.ndpointer(dtype=float, ndim=1,
flags='C_CONTIGUOUS')
Prototype_example = CFUNCTYPE(None, c_long, T_ndDp, T_ndDp, T_ndDp)

In ctypes None is equivelent to NULL, though in order to get past the
prototype argchecks you need to cast the object to None. If for example the
pointer was to a simple double one could write something like: cast(None,
POINTER(c_double))

the problem is in casting to None for the above (T_ndDp) type

Any help would be greatly appreciated

Nicholas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071230/ab8c2152/attachment.html>

From aisaac at american.edu  Sun Dec 30 09:49:48 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Sun, 30 Dec 2007 09:49:48 -0500
Subject: [Numpy-discussion] [SciPy-dev] Doc-day
In-Reply-To: <4777322B.6090702@enthought.com>
References: <477470C0.3030602@enthought.com>	<c7009a550712272127w24e0f1e1mab31bb0b8b073e15@mail.gmail.com>	<20071228112617.GC9051@mentat.za.net>	<e06186140712291751s757b2882ufffa28892e5e784a@mail.gmail.com>	<db6b5ecc0712291759m7900ff94we2ca1983d90edea5@mail.gmail.com><e06186140712292052h61679f2fu6bd12e8eca147b2@mail.gmail.com><4777322B.6090702@enthought.com>
Message-ID: <Mahogany-0.67.0-1524-20071230-094948.00@american.edu>

On Sat, 29 Dec 2007, "Travis E. Oliphant" apparently wrote:
> they make it so that the documentation standard for SciPy 
> and NumPy is not different from the ETS standard 


1. Is the Enthought Tool Suite standard published?

2. I think it miscasts things to say that the previous
standard was built around a rendering tool.  The previous
standard was tied to reStructuredText, which happens to
be renderable by epydoc.  In the future, other tools will
render reStructuredText.  Indeed, we are all free to offer
one right now (or to contribute to epydoc).

3. The point of the reStructuredText, which has been 
developed over many years, is to make (with very light
markup) a number of semantic distinctions which have
proved useful and desirable.  If these distinctions are
not drawn in the markup, they cannot be used by any
renderer.

4. reST offers way to render math (as MathML or LaTeX->PDF).
At one point, iirc, this was considered a plus.

I am NOT trying to push for any particular documentation
standard.  I'm just a NumPy user, not a NumPy developer.  
But I would like to suggest that it might be useful to ask
David Goodger for a quick overview of what will be lost
by using the ETS markup standard instead of reST.

fwiw,
Alan Isaac





From ndbecker2 at gmail.com  Sun Dec 30 10:02:13 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sun, 30 Dec 2007 10:02:13 -0500
Subject: [Numpy-discussion] iterator speed
Message-ID: <fl8btl$4or$1@ger.gmane.org>

In numpybook, in discussing iterators, it says that if you know your array
is contiguous, it may be faster to just use C-style pointers instead of
iterators.

Looking at the code a little, it appears that PyArray_ITER_NEXT(it) attempts
to optimize for certain cases.  I wonder if anyone has done any benchmarks
on modern compilers?

In particular, I'm hoping that PyArray_ITER_NEXT on a modern compiler will
optimize the contiguous case to give equivalent performance to a c-style
pointer.  In that case, I could happily use the generic iterator without a
speed penalty.



From ndbecker2 at gmail.com  Sun Dec 30 11:23:13 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sun, 30 Dec 2007 11:23:13 -0500
Subject: [Numpy-discussion] random access iterators
Message-ID: <fl8glh$hf8$1@ger.gmane.org>

I'm looking at writing some c++ code to interoperate with numpy.  A c++
random access iterator must include a 'distance' function.  distance (i1,
i2) must return the number of times i1 has to be incremented to reach i2.

Is there a way to get this from PyArrayIterObject?



From aisaac at american.edu  Sun Dec 30 13:25:07 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Sun, 30 Dec 2007 13:25:07 -0500
Subject: [Numpy-discussion] how do I list all combinations
In-Reply-To: <4772B7FF.3060602@jpl.nasa.gov>
References: <4772B7FF.3060602@jpl.nasa.gov>
Message-ID: <Mahogany-0.67.0-1524-20071230-132507.00@american.edu>

On Wed, 26 Dec 2007, Mathew Yeates apparently wrote: 
> r1=["dog","cat"] 
> r2=[1,2] 
> I want to return [["dog",1],["dog",2],["cat",1],["cat",2]] 


This is a Cartesian product.

Sage has ``cartesian_product_iterator`` for this.
Also 
<URL:http://www.sagemath.org/doc/html/ref/module-sage.combinat.cartesian-product.html>

Here is a Cookbook implementation.
<URL:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/302478>
The generator may be adequate to your needs.

Here is a recursive implementation that does not use 
multi-dimensional indexing:

def set_product(*sets):
    last_set = sets[-1]
    drop_last = sets[:-1]
    if drop_last:
        result = set( x+(y,)
                      for x in set_product(*drop_last)
                      for y in last_set )
    else:
        result = set( (y,) for y in last_set )
    return result

Sorry for a late reply.  I'm catching up on email...

Cheers,
Alan Isaac





From oliphant at enthought.com  Sun Dec 30 18:35:42 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Sun, 30 Dec 2007 17:35:42 -0600
Subject: [Numpy-discussion] random access iterators
In-Reply-To: <fl8glh$hf8$1@ger.gmane.org>
References: <fl8glh$hf8$1@ger.gmane.org>
Message-ID: <47782B4E.4000207@enthought.com>

Neal Becker wrote:
> I'm looking at writing some c++ code to interoperate with numpy.  A c++
> random access iterator must include a 'distance' function.  distance (i1,
> i2) must return the number of times i1 has to be incremented to reach i2.
>
> Is there a way to get this from PyArrayIterObject?
>   
In the general case,  not directly.  However, you may be able to figure 
something out using the "coordinate" array that is stored with each 
iterator. 

-Travis O.



From millman at berkeley.edu  Mon Dec 31 17:43:37 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 31 Dec 2007 14:43:37 -0800
Subject: [Numpy-discussion] planet.scipy.org
Message-ID: <c7009a550712311443r6c1e70a6q5afaaba2a08bcf95@mail.gmail.com>

Hey,

I just wanted to announce that we now have a NumPy/SciPy blog
aggregator thanks to Ga?l Varoquaux:  http://planet.scipy.org/

Feel free to contact me if you have a blog that you would like included.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/