From asmi.capri at gmail.com Tue Feb 1 03:39:12 2011 From: asmi.capri at gmail.com (Asmi Shah) Date: Tue, 1 Feb 2011 09:39:12 +0100 Subject: [Numpy-discussion] create a numpy array of images Message-ID: Thanks a lot Friedrich and Chris.. It came in handy to use PIL and numpy.. :) @Zach, m aware of the poor handling of 16bit images in PIL, for that I am using imagemagick to convert it into 8 bit first and then PIL for rest of the processing.. I have one more question: how to avoid the limitation of memoryerror in numpy. as I have like 200 images to stack in the numpy array of say 1024x1344 resolution.. have any idea apart from downsampling? -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Tue Feb 1 03:42:14 2011 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 1 Feb 2011 09:42:14 +0100 (CET) Subject: [Numpy-discussion] using loadtxt() for given number of rows? In-Reply-To: <4D47421B.8070501@noaa.gov> References: <4D47421B.8070501@noaa.gov> Message-ID: On Mon, 31 Jan 2011, Christopher Barker wrote: > On 1/31/11 4:39 AM, Robert Cimrman wrote: >> I work with text files which contain several arrays separated by a few >> lines of other information, for example: >> >> POINTS 4 float >> -5.000000e-01 -5.000000e-01 0.000000e+00 >> 5.000000e-01 -5.000000e-01 0.000000e+00 >> 5.000000e-01 5.000000e-01 0.000000e+00 >> -5.000000e-01 5.000000e-01 0.000000e+00 >> >> CELLS 2 8 >> 3 0 1 2 >> 3 2 3 0 > >> I have used custom Python code with loops to read similar files, so the >> speed was not too good. Now I wonder if it would be possible to use the >> numpy.loadtxt() function for the "array-like" parts. It supports passing >> an open file object in, which is good, but it wants to read the entire >> file, which does not work in this case. >> >> It seems to me, that an additional parameter to loadtxt(), say "nrows" or >> "numrows", would do the job, > > I agree that that would be a useful feature. However, I'm not sure it > would help performance much -- I think loadtxt is written in python as well. I see. Anyway, it would allow me to reduce my code size, which counts as well to be a good thing. So there is now a new enhancement ticket [1]. > One option in the meantime. If you know how many rows, you presumable > know how many items on each row. IN that case, you can use: > > np.fromfile(open_file, sep=' ', count=num_items_to_read) > > It'll only work for multi-line text if the separator is whitespace, > which it was in your example. But if it does, it should be pretty fast. Good idea, the prerequisites are not met always, but often enough. Thanks! r. [1] http://projects.scipy.org/numpy/ticket/1731 From totonixsame at gmail.com Tue Feb 1 05:20:52 2011 From: totonixsame at gmail.com (totonixsame at gmail.com) Date: Tue, 1 Feb 2011 08:20:52 -0200 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: References: Message-ID: Hi, On Tue, Feb 1, 2011 at 6:39 AM, Asmi Shah wrote: > Thanks a lot Friedrich and Chris.. It came in handy to use PIL and numpy.. > :) > @Zach, m aware of the poor handling of 16bit images in PIL, for that I am > using imagemagick to convert it into 8 bit first and then PIL for rest of > the processing.. You could try VTK to open those files and use vtk functions to transform to numpy arrays. > I have one more question: how to avoid the limitation of memoryerror in > numpy. as I have like 200 images to stack in the numpy array of say > 1024x1344 resolution.. have any idea apart from downsampling? Take a look at numpy.memmap or h5py [1]. _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion [1] - http://code.google.com/p/h5py/ From sturla at molden.no Tue Feb 1 08:49:39 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 1 Feb 2011 14:49:39 +0100 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: References: Message-ID: <3851DA3B-0ECE-4C9A-9B1E-8EF58701D23B@molden.no> Den 1. feb. 2011 kl. 11.20 skrev "totonixsame at gmail.com" : > I have one more question: how to avoid the limitation of memoryerror > in >> >> numpy. as I have like 200 images to stack in the numpy array of say >> 1024x1344 resolution.. have any idea apart from downsampling? > > Take a look at numpy.memmap or h5py [1]. > memmap will not help unless he uses 64 bit Python, in which case he can just buy more RAM if he has too little. I suspect he is running out of virtual memory, not physical, for which 64 bit is the easiest solution. It is not possible to compensate for lack of virtual memory (typically 2GB limit with 32 bit) by memory mapping file into the already exhausted memory space. Using a database like h5py will help too, unless he tries to extract them all at once. Sturla From asmi.capri at gmail.com Tue Feb 1 09:07:35 2011 From: asmi.capri at gmail.com (Asmi Shah) Date: Tue, 1 Feb 2011 15:07:35 +0100 Subject: [Numpy-discussion] create a numpy array of images Message-ID: Hi Zach and Sturla, Well I am a "she" :)) Thanks for your inputs.. I am using 32 bit python as have so many libraries integrated with it.. and moreover, i plan to put this volume rendered on a web page or distribute the exe in the end, so want to be minimal for the memory requirements on the clients' systems.. the physical memory should not be a problem as I have 8GB RAM.. specially when the images are RGB then it gets into trouble as it adds the 4th dimension already in my case.. - asmi ---------- Forwarded message ---------- From: Sturla Molden To: Discussion of Numerical Python Date: Tue, 1 Feb 2011 14:49:39 +0100 Subject: Re: [Numpy-discussion] create a numpy array of images Den 1. feb. 2011 kl. 11.20 skrev "totonixsame at gmail.com" < totonixsame at gmail.com>: I have one more question: how to avoid the limitation of memoryerror in > >> >> numpy. as I have like 200 images to stack in the numpy array of say >> 1024x1344 resolution.. have any idea apart from downsampling? >> > > Take a look at numpy.memmap or h5py [1]. > > memmap will not help unless he uses 64 bit Python, in which case he can just buy more RAM if he has too little. I suspect he is running out of virtual memory, not physical, for which 64 bit is the easiest solution. It is not possible to compensate for lack of virtual memory (typically 2GB limit with 32 bit) by memory mapping file into the already exhausted memory space. Using a database like h5py will help too, unless he tries to extract them all at once. Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From sienkiew at stsci.edu Tue Feb 1 10:02:12 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Tue, 01 Feb 2011 10:02:12 -0500 Subject: [Numpy-discussion] current numpy in python 2.5 ? Message-ID: <4D482074.6060203@stsci.edu> Hi, I looked on the web site, but I couldn't find a list of python versions that numpy expects to work on. Is the trunk still expected to work in Python 2.5 ? I'm asking because it doesn't, but I don't want to file a bug report if it is not expected to. The problem shows up when I run numpy.test(): File "/usr/stsci/pyssgdev/2.5.4/numpy/testing/utils.py", line 619 except ValueError as e: ^ SyntaxError: invalid syntax Mark S. From asmi.capri at gmail.com Tue Feb 1 10:06:00 2011 From: asmi.capri at gmail.com (Asmi Shah) Date: Tue, 1 Feb 2011 16:06:00 +0100 Subject: [Numpy-discussion] current numpy in python 2.5 ? In-Reply-To: <4D482074.6060203@stsci.edu> References: <4D482074.6060203@stsci.edu> Message-ID: http://sourceforge.net/projects/numpy/files/NumPy/1.5.1/ here you will fine the latest version of numpy with different python versions.. On Tue, Feb 1, 2011 at 4:02 PM, Mark Sienkiewicz wrote: > Hi, > > I looked on the web site, but I couldn't find a list of python versions > that numpy expects to work on. Is the trunk still expected to work in > Python 2.5 ? > > I'm asking because it doesn't, but I don't want to file a bug report if it > is not expected to. The problem shows up when I run numpy.test(): > > File "/usr/stsci/pyssgdev/2.5.4/numpy/testing/utils.py", line 619 > except ValueError as e: > ^ > SyntaxError: invalid syntax > > Mark S. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Regards, Asmi Shah -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Feb 1 10:16:45 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 1 Feb 2011 08:16:45 -0700 Subject: [Numpy-discussion] current numpy in python 2.5 ? In-Reply-To: <4D482074.6060203@stsci.edu> References: <4D482074.6060203@stsci.edu> Message-ID: On Tue, Feb 1, 2011 at 8:02 AM, Mark Sienkiewicz wrote: > Hi, > > I looked on the web site, but I couldn't find a list of python versions > that numpy expects to work on. Is the trunk still expected to work in > Python 2.5 ? > > I'm asking because it doesn't, but I don't want to file a bug report if it > is not expected to. The problem shows up when I run numpy.test(): > > File "/usr/stsci/pyssgdev/2.5.4/numpy/testing/utils.py", line 619 > except ValueError as e: > ^ > SyntaxError: invalid syntax > > Python >= 2.4 is supported, so using the new exception syntax is a bug. Please open a ticket. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Tue Feb 1 10:22:27 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 1 Feb 2011 09:22:27 -0600 Subject: [Numpy-discussion] current numpy in python 2.5 ? In-Reply-To: References: <4D482074.6060203@stsci.edu> Message-ID: On Tue, Feb 1, 2011 at 9:16 AM, Charles R Harris wrote: > > > On Tue, Feb 1, 2011 at 8:02 AM, Mark Sienkiewicz wrote: > >> Hi, >> >> I looked on the web site, but I couldn't find a list of python versions >> that numpy expects to work on. Is the trunk still expected to work in >> Python 2.5 ? >> >> I'm asking because it doesn't, but I don't want to file a bug report if it >> is not expected to. The problem shows up when I run numpy.test(): >> >> File "/usr/stsci/pyssgdev/2.5.4/numpy/testing/utils.py", line 619 >> except ValueError as e: >> ^ >> SyntaxError: invalid syntax >> >> > Python >= 2.4 is supported, so using the new exception syntax is a bug. > Please open a ticket. > > That's from a very recent commit: https://github.com/numpy/numpy/commit/c4a556e1fcdede5fe366e4373a7f2dc802a0e898#diff-7 and as Chuck says, it is a bug, because numpy must support python 2.4. Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Feb 1 11:01:34 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 1 Feb 2011 09:01:34 -0700 Subject: [Numpy-discussion] current numpy in python 2.5 ? In-Reply-To: References: <4D482074.6060203@stsci.edu> Message-ID: On Tue, Feb 1, 2011 at 8:22 AM, Warren Weckesser < warren.weckesser at enthought.com> wrote: > > > On Tue, Feb 1, 2011 at 9:16 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Tue, Feb 1, 2011 at 8:02 AM, Mark Sienkiewicz wrote: >> >>> Hi, >>> >>> I looked on the web site, but I couldn't find a list of python versions >>> that numpy expects to work on. Is the trunk still expected to work in >>> Python 2.5 ? >>> >>> I'm asking because it doesn't, but I don't want to file a bug report if >>> it is not expected to. The problem shows up when I run numpy.test(): >>> >>> File "/usr/stsci/pyssgdev/2.5.4/numpy/testing/utils.py", line 619 >>> except ValueError as e: >>> ^ >>> SyntaxError: invalid syntax >>> >>> >> Python >= 2.4 is supported, so using the new exception syntax is a bug. >> Please open a ticket. >> >> > > That's from a very recent commit: > > > https://github.com/numpy/numpy/commit/c4a556e1fcdede5fe366e4373a7f2dc802a0e898#diff-7 > > and as Chuck says, it is a bug, because numpy must support python 2.4. > > I fixed it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Feb 1 11:16:55 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 01 Feb 2011 17:16:55 +0100 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: References: Message-ID: <4D4831F7.2020103@molden.no> Den 01.02.2011 15:07, skrev Asmi Shah: > Hi Zach and Sturla, > > Well I am a "she" :)) I apologize, I did not deduce correct gender from your name :) > Thanks for your inputs.. I am using 32 bit python as have so many > libraries integrated with it.. and moreover, i plan to put this volume > rendered on a web page or distribute the exe in the end, so want to be > minimal for the memory requirements on the clients' systems.. > the physical memory should not be a problem as I have 8GB RAM.. > specially when the images are RGB then it gets into trouble as it adds > the 4th dimension already in my case.. > With 32 bit each process only has 2 or 3 GB (dependent on OS setting) of those 8 GB available in user space. If 32 bit is a requirement, you have to store some of the data on disk to prevent exhausting the 2 GB virtual memory limit. Actually, if you have 200 16-bit images in 1024 x 1344 resolution, that is only 525 MB if stored compactly with dtype np.uint16. So check how those images are stored in your stack. Do you use floating point instead of the smallest possible integer? Do you use separate dimensions for RGBA channels? Is there anything else wasting memory besides this stack of images? Do you have a memory leak? Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedrichromstedt at gmail.com Tue Feb 1 11:31:59 2011 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Tue, 1 Feb 2011 17:31:59 +0100 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: References: Message-ID: 2011/2/1 Asmi Shah : > Thanks a lot Friedrich and Chris.. It came in handy to use PIL and numpy.. > :) :-) > I have one more question: how to avoid the limitation of memoryerror in > numpy. as I have like 200 images to stack in the numpy array of say > 1024x1344 resolution.. have any idea apart from downsampling? In case you *have* to downsample: I also ran into this, with the example about my 5 images ... im.resize((newx newy), PIL.Image.ANTIALIAS) will be your friend. http://www.pythonware.com/library/pil/handbook/image.htm. Note, you might take advatage of floating-point images ('F' spec), I don't know what the trade-offs are here. 'F' most probably takes 4x(8bit), so ... The PIL handbook does not state what PIL.Image.ANTIALIAS actually does, we can only hope that it's real sinc interpolation or similar (if your images are frequency bounded this would be best to my knowledge). In this case you do not even lose information as long as the spacial resolution of the downsampled images is still sufficient to make the signal frequency bounded. You might do a FFT (spacial) to check if your images *are* actually bounded in frequency domain. I think it does not need to be perfect. I strongly believe sinc is in scipy, but I never looked for. Friedrich From kanterburiad at gmail.com Tue Feb 1 11:38:29 2011 From: kanterburiad at gmail.com (K. B.) Date: Tue, 1 Feb 2011 17:38:29 +0100 Subject: [Numpy-discussion] SWIG examples from the Cookbook Message-ID: Hi, I asked a question on this mailing list few days ago but nobody could help me. So I wonder if it is more appropriate to post this question on the Scipy-Dev mailing list or to fill a bug repport. Thank you ! KB ----------------- original post : Hi, I'd like to ask for your help regarding the use of SWIG with numpy. ** problem description ** While I can compile successfully the examples provided in ./numpy/doc/swig I can't compile the first example provided in the Cookbook. http://www.scipy.org/Cookbook/SWIG_NumPy_examples "A simple ARGOUT_ARRAY1 example" I have this error: ezrange_wrap.c: In function ?SWIG_AsVal_long?: ezrange_wrap.c:3227: error: initializer element is not constant ** hypothesis and ugly workaround ** Comparing the examples from ./numpy/doc/swig and from the Cookbook, I noticed the main difference lies in the fact the first are in C++ while the latter in plain C. I "converted" the example from the Coobook to look like a c++ project. Essentially I renamed the .c file with a .cxx extension and I modified setup.py accordingly. This time it compiles successfully and the module is usable in python. ** questions ** 1/ What should I do to compile the Cookbook example with a C compiler? 2/ If it appears swig is now only compatible with the C++ compiler, do you have a practical workaround to propose? (without renaming the C files with a .cxx extension) ? ** configuration ** debian testing i686 swig version 1.3.40 python-numpy version 1.5.1 (I also tried 1.4.1) python version 2.6 From Chris.Barker at noaa.gov Tue Feb 1 12:58:09 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 01 Feb 2011 09:58:09 -0800 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: References: Message-ID: <4D4849B1.30607@noaa.gov> On 2/1/11 12:39 AM, Asmi Shah wrote: > I have one more question: how to avoid the limitation of memoryerror in > numpy. as I have like 200 images to stack in the numpy array of say > 1024x1344 resolution.. have any idea apart from downsampling? If I'm doing my math right, that's 262 MB, shouldn't be a problem in modern systems. That's 8bit, but 786MB if 24 bit RGB. If you are careful about how many copies you're keeping around (including temporaries), you mau be OK still. But if you really have big collections of images, you might try memory mapped arrays -- as Sturla pointed out they wont' let you create monster arrays on a 32 bit python, but maybe they do help with not clogging up memory too much? I don't know -- I haven't used them -- presumably they have a purpose. Also, pytables is worth a look, as another way to get HDF5 on disk, but I think more "natural" access. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Tue Feb 1 12:58:17 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 01 Feb 2011 09:58:17 -0800 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: References: Message-ID: <4D4849B9.2060204@noaa.gov> On 2/1/11 8:31 AM, Friedrich Romstedt wrote: > In case you *have* to downsample: > > I also ran into this, with the example about my 5 images ... > im.resize((newx newy), PIL.Image.ANTIALIAS) will be your friend. > http://www.pythonware.com/library/pil/handbook/image.htm. If you want to downsample by a integer amount (i.e a factor of 2) in each dimension, I have some Cython code that optimizes that. I'm happy to send it along. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From sturla at molden.no Tue Feb 1 13:58:16 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 01 Feb 2011 19:58:16 +0100 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: <4D4849B1.30607@noaa.gov> References: <4D4849B1.30607@noaa.gov> Message-ID: <4D4857C8.2010404@molden.no> Den 01.02.2011 18:58, skrev Christopher Barker: > But if you really have big collections of images, you might try memory > mapped arrays -- as Sturla pointed out they wont' let you create monster > arrays on a 32 bit python, But they will on 64 bit Python :D We can just memory map a temporary file, and fake as much "memory" as we need. The OS will optimize the disk access. Also consider that hardware is cheap compared to labour, at least in Europe. Is programming for memory limited 32 bit Python worth the effort? Sturla From DParker at chromalloy.com Tue Feb 1 14:22:16 2011 From: DParker at chromalloy.com (DParker at chromalloy.com) Date: Tue, 1 Feb 2011 14:22:16 -0500 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: Message-ID: Thanks for the advice. Using Sebastian's advice I was able to write a version that worked when the input arguments are both arrays with the same length. The code provided by eat works when t is an array, but not for an array of far. The numpy.vectorize version works with any combination of scalar or array input. I still haven't figured out how to rewrite my function to be as flexible as the numpy.vectorize version at accepting either scalars or array inputs and properly broadcasting the scalar arguments to the array arguments. David Parker Chromalloy - TDAG From: eat To: Discussion of Numerical Python Date: 01/31/2011 11:37 AM Subject: Re: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? Sent by: numpy-discussion-bounces at scipy.org Hi, On Mon, Jan 31, 2011 at 5:15 PM, wrote: I have several functions like the example below that I would like to make compatible with array inputs. The problem is the conditional statements give a ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all(). I can use numpy.vectorize, but if possible I'd prefer to rewrite the function. Does anyone have any advice the best way to modify the code to accept array inputs? Thanks in advance for any assistance. If I understod your question correctly, then air_gamma could be coded as: def air_gamma_0(t, far=0.0): """ Specific heat ratio (gamma) of Air/JP8 t - static temperature, Rankine [far] - fuel air ratio [- defaults to 0.0 (dry air)] air_gamma - specific heat ratio """ if far< 0.: return NAN elif far < 0.005: ag= air_gamma_1(t) ag[np.logical_or(t< 379., t> 4731.)]= NAN return ag elif far< 0.069: ag= air_gamma_2(t, far) ag[np.logical_or(t< 699., t> 4731.)]= NAN return ag else: return NAN Rest of the code is in the attachment. My two cents, eat NAN = float('nan') def air_gamma(t, far=0.0): """ Specific heat ratio (gamma) of Air/JP8 t - static temperature, Rankine [far] - fuel air ratio [- defaults to 0.0 (dry air)] air_gamma - specific heat ratio """ if far < 0.: return NAN elif far < 0.005: if t < 379. or t > 4731.: return NAN else: air_gamma = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - 4.428098e-14 * t ** 4. + 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + 0.0001684666 * t + 1.368652 elif far < 0.069: if t < 699. or t > 4731.: return NAN else: a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + 3.103507e-21 * far - 3.391308e-22 a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - 5.469399e-17 * far + 6.058125e-18 a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + 3.865306e-13 * far - 4.302534e-14 a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** 2. - 0.000000001392629 * far + 1.520583e-10 a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + 0.000002688007 * far - 0.0000002651616 a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - 0.002676877 * far + 0.0001580424 a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * far + 1.372714 air_gamma = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t ** 3. + a2 * t ** 2. + a1 * t + a0 elif far >= 0.069: return NAN else: return NAN return air_gamma David Parker Chromalloy - TDAG _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion [attachment "air_gamma.py" deleted by Dave Parker/Chromalloy] _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Tue Feb 1 14:30:29 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Tue, 1 Feb 2011 11:30:29 -0800 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: Message-ID: Have you thought about using cython to work with the numpy C-API ( http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI)? This will be fast, simple (you can mix and match Python and Cython). As for your specific issue: you can simply cast to all the inputs to numpy arrays (using asarray http://docs.scipy.org/doc/numpy/reference/generated/numpy.asarray.html) to deal with scalars. This will make sure they get broadcast correctly. On Tue, Feb 1, 2011 at 11:22 AM, wrote: > Thanks for the advice. > > Using Sebastian's advice I was able to write a version that worked when the > input arguments are both arrays with the same length. The code provided by > eat works when t is an array, but not for an array of far. > > The numpy.vectorize version works with any combination of scalar or array > input. I still haven't figured out how to rewrite my function to be as > flexible as the numpy.vectorize version at accepting either scalars or array > inputs and properly broadcasting the scalar arguments to the array > arguments. > > David Parker > Chromalloy - TDAG > > > > From: eat > To: Discussion of Numerical Python > Date: 01/31/2011 11:37 AM > Subject: Re: [Numpy-discussion] Vectorize or rewrite function to > work with array inputs? > Sent by: numpy-discussion-bounces at scipy.org > ------------------------------ > > > > Hi, > > On Mon, Jan 31, 2011 at 5:15 PM, <*DParker at chromalloy.com*> > wrote: > I have several functions like the example below that I would like to make > compatible with array inputs. The problem is the conditional statements give > a *ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all()*. I can use numpy.vectorize, but if > possible I'd prefer to rewrite the function. Does anyone have any advice the > best way to modify the code to accept array inputs? Thanks in advance for > any assistance. > > If I understod your question correctly, then air_gamma could be coded as: > def air_gamma_0(t, far=0.0): > """ > Specific heat ratio (gamma) of Air/JP8 > t - static temperature, Rankine > [far] - fuel air ratio [- defaults to 0.0 (dry air)] > air_gamma - specific heat ratio > """ > if far< 0.: > return NAN > elif far < 0.005: > ag= air_gamma_1(t) > ag[np.logical_or(t< 379., t> 4731.)]= NAN > return ag > elif far< 0.069: > ag= air_gamma_2(t, far) > ag[np.logical_or(t< 699., t> 4731.)]= NAN > return ag > else: > return NAN > Rest of the code is in the attachment. > > > My two cents, > eat > > > > NAN = float('nan') > > def air_gamma(t, far=0.0): > """ > Specific heat ratio (gamma) of Air/JP8 > t - static temperature, Rankine > [far] - fuel air ratio [- defaults to 0.0 (dry air)] > air_gamma - specific heat ratio > """ > if far < 0.: > return NAN > elif far < 0.005: > if t < 379. or t > 4731.: > return NAN > else: > air_gamma = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. > + 0.0001684666 * t + 1.368652 > elif far < 0.069: > if t < 699. or t > 4731.: > return NAN > else: > a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + > 3.103507 e-21 * far - 3.391308e-22 > a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - > 5.469399e-17 * far + 6.058125e-18 > a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + > 3.865306e-13 * far - 4.302534e-14 > a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** > 2. - 0.000000001392629 * far + 1.520583e-10 > a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + > 0.000002688007 * far - 0.0000002651616 > a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - > 0.002676877 * far + 0.0001580424 > a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * > far + 1.372714 > air_gamma = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t > ** 3. + a2 * t ** 2. + a1 * t + a0 > elif far >= 0.069: > return NAN > else: > return NAN > return air_gamma > > David Parker > Chromalloy - TDAG > _______________________________________________ > NumPy-Discussion mailing list* > **NumPy-Discussion at scipy.org* * > **http://mail.scipy.org/mailman/listinfo/numpy-discussion* > > [attachment "air_gamma.py" deleted by Dave Parker/Chromalloy] > _______________________________________________ > > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Feb 1 14:41:26 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 01 Feb 2011 20:41:26 +0100 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: Message-ID: <4D4861E6.3040603@molden.no> Den 01.02.2011 20:30, skrev John Salvatier: > Have you thought about using cython to work with the numpy C-API > (http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI)? This will > be fast, simple (you can mix and match Python and Cython). Never do this. Cython has NumPy array support now: http://docs.cython.org/src/tutorial/numpy.html Sturla From jsalvati at u.washington.edu Tue Feb 1 14:50:53 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Tue, 1 Feb 2011 11:50:53 -0800 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: <4D4861E6.3040603@molden.no> References: <4D4861E6.3040603@molden.no> Message-ID: I am curious: why you recommend against this? Using the C-API through cython seems more attractive than using the Cython-specific numpy features since they need a specific number of dimensions (and I don't think they broadcast) and don't seem that much easier. Are there hidden disadvantages to using the C-API? Have I misunderstood something? John On Tue, Feb 1, 2011 at 11:41 AM, Sturla Molden wrote: > Den 01.02.2011 20:30, skrev John Salvatier: > > Have you thought about using cython to work with the numpy C-API > > (http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI)? This will > > be fast, simple (you can mix and match Python and Cython). > > Never do this. Cython has NumPy array support now: > > http://docs.cython.org/src/tutorial/numpy.html > > > Sturla > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Tue Feb 1 14:53:36 2011 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 1 Feb 2011 13:53:36 -0600 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: Message-ID: On Tue, Feb 1, 2011 at 1:30 PM, John Salvatier wrote: > Have you thought about using cython to work with the numpy C-API ( > http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI)? This will be > fast, simple (you can mix and match Python and Cython). > > As for your specific issue: you can simply cast to all the inputs to numpy > arrays (using asarray > http://docs.scipy.org/doc/numpy/reference/generated/numpy.asarray.html) to > deal with scalars. This will make sure they get broadcast correctly. > > Don't use "asarray" blindly! Using asarray causes masked_arrays to become regular arrays. If all of the operations in your function are vectorized already, then you are causing your function to unnecessarially break support for subclassed arrays like masked_arrays. Use asanyarray() by default unless it has to be asarray(). Sorry for my rant. I just love masked arrays too much... Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Tue Feb 1 14:57:57 2011 From: faltet at pytables.org (Francesc Alted) Date: Tue, 1 Feb 2011 20:57:57 +0100 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: <4D4857C8.2010404@molden.no> References: <4D4849B1.30607@noaa.gov> <4D4857C8.2010404@molden.no> Message-ID: <201102012057.57263.faltet@pytables.org> A Tuesday 01 February 2011 19:58:16 Sturla Molden escrigu?: > Den 01.02.2011 18:58, skrev Christopher Barker: > > But if you really have big collections of images, you might try > > memory mapped arrays -- as Sturla pointed out they wont' let you > > create monster arrays on a 32 bit python, > > But they will on 64 bit Python :D We can just memory map a temporary > file, and fake as much "memory" as we need. The OS will optimize the > disk access. What do you mean by 'optimize the disk access'? One of the drawbacks of memory mapped files is precisely that OS cannot distinguish between data that belongs to 'disk' and data that belongs to 'memory'. This normally introduces extreme slowness in other programs when datasets in files exceeds physical memory but have to loaded by OS -- the reason being that OS swaps out most of programs/shared libraries that were in memory in order to be able to load new 'disk' data. The other important drawback of the memory mapped files is that you need to have at very least an amount of virtual memory that is enough to keep all of these data files. In general, you only have virtual memory that is between 1.5x and 2x the physical memory (having more than this is generally regarded as a waste of disk space). This is why I prefer very much reading directly from a file: the OS in this case is able to distinguish between data belonging to 'disk' and data belonging to 'memory'. Is in this case when the OS can really optimize disk access (unless you have complicated setups). > Also consider that hardware is cheap compared to labour, > at least in Europe. Is programming for memory limited 32 bit Python > worth the effort? -- Francesc Alted From sturla at molden.no Tue Feb 1 15:19:02 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 01 Feb 2011 21:19:02 +0100 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: <4D4861E6.3040603@molden.no> Message-ID: <4D486AB6.1030100@molden.no> Den 01.02.2011 20:50, skrev John Salvatier: > I am curious: why you recommend against this? For the same reason I prefer Fortran 95 over C for numerical programming: clarity. > Using the C-API through cython seems more attractive than using the > Cython-specific numpy features since they need a specific number of > dimensions They do not, but the generated code will be faster if you do. If you use the NumPy C API, you can get faster indexing if the number of dimensions is known. Sturla From DParker at chromalloy.com Tue Feb 1 15:20:07 2011 From: DParker at chromalloy.com (DParker at chromalloy.com) Date: Tue, 1 Feb 2011 15:20:07 -0500 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: Message-ID: I'm not sure I need to dive into cython or C for this - performance is not an issue for my problem - I just want a flexible function that will accept scalars or arrays. Both Sebastian's and eat's suggestions show using indexing to handle the conditional statements in the original function. The problem I'm having implementing this is in getting the input arguments and outputs to a common array size. Here's how I can do this but it seems ugly: # t and far are function arguments which may be scalars or arrays # ag is the output array # need to make everything array with common length t = np.array(t, ndmin=1) # Convert t to an array far = np.array(far, ndmin=1) # Convert far to an array ag = t*far*np.nan # Make an output array of the proper length using broadcasting rules t = np.zeros_like(ag)+t # Expand t to the length of the output array far = np.zeros_like(ag)+far # Expand far to the length of the output array Now with all arrays the same length I can use indexing with logical statements: ag[far<0.005] = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - 4.428098e-14 * t ** 4. + \ 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + 0.0001684666 * t + 1.368652 The resulting code looks like this: import numpy as np def air_gamma_dp(t, far=0.0): """ Specific heat ratio (gamma) of Air/JP8 t - static temperature, Rankine [far] - fuel air ratio [- defaults to 0.0 (dry air)] air_gamma - specific heat ratio """ t = np.array(t, ndmin=1) far = np.array(far, ndmin=1) ag = t*far*np.nan t = np.zeros_like(ag)+t far = np.zeros_like(ag)+far far[(far<0.) | (far>0.069)] = np.nan t[(t < 379.) | (t > 4731.)] = np.nan ag[(far<0.005)] = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - 4.428098e-14 * t ** 4. + 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + 0.0001684666 * t + 1.368652 t[(t < 699.) | (t > 4731.)] = np.nan a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + 3.103507e-21 * far - 3.391308e-22 a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - 5.469399e-17 * far + 6.058125e-18 a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + 3.865306e-13 * far - 4.302534e-14 a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** 2. - 0.000000001392629 * far + 1.520583e-10 a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + 0.000002688007 * far - 0.0000002651616 a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - 0.002676877 * far + 0.0001580424 a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * far + 1.372714 ag[far>=0.005] = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t ** 3. + a2 * t ** 2. + a1 * t + a0 return ag I was hoping there was a more elegant way to do this. David Parker Chromalloy - TDAG From: John Salvatier To: Discussion of Numerical Python Date: 02/01/2011 02:29 PM Subject: Re: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? Sent by: numpy-discussion-bounces at scipy.org Have you thought about using cython to work with the numpy C-API ( http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI)? This will be fast, simple (you can mix and match Python and Cython). As for your specific issue: you can simply cast to all the inputs to numpy arrays (using asarray http://docs.scipy.org/doc/numpy/reference/generated/numpy.asarray.html) to deal with scalars. This will make sure they get broadcast correctly. On Tue, Feb 1, 2011 at 11:22 AM, wrote: Thanks for the advice. Using Sebastian's advice I was able to write a version that worked when the input arguments are both arrays with the same length. The code provided by eat works when t is an array, but not for an array of far. The numpy.vectorize version works with any combination of scalar or array input. I still haven't figured out how to rewrite my function to be as flexible as the numpy.vectorize version at accepting either scalars or array inputs and properly broadcasting the scalar arguments to the array arguments. David Parker Chromalloy - TDAG From: eat To: Discussion of Numerical Python Date: 01/31/2011 11:37 AM Subject: Re: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? Sent by: numpy-discussion-bounces at scipy.org Hi, On Mon, Jan 31, 2011 at 5:15 PM, wrote: I have several functions like the example below that I would like to make compatible with array inputs. The problem is the conditional statements give a ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all(). I can use numpy.vectorize, but if possible I'd prefer to rewrite the function. Does anyone have any advice the best way to modify the code to accept array inputs? Thanks in advance for any assistance. If I understod your question correctly, then air_gamma could be coded as: def air_gamma_0(t, far=0.0): """ Specific heat ratio (gamma) of Air/JP8 t - static temperature, Rankine [far] - fuel air ratio [- defaults to 0.0 (dry air)] air_gamma - specific heat ratio """ if far< 0.: return NAN elif far < 0.005: ag= air_gamma_1(t) ag[np.logical_or(t< 379., t> 4731.)]= NAN return ag elif far< 0.069: ag= air_gamma_2(t, far) ag[np.logical_or(t< 699., t> 4731.)]= NAN return ag else: return NAN Rest of the code is in the attachment. My two cents, eat NAN = float('nan') def air_gamma(t, far=0.0): """ Specific heat ratio (gamma) of Air/JP8 t - static temperature, Rankine [far] - fuel air ratio [- defaults to 0.0 (dry air)] air_gamma - specific heat ratio """ if far < 0.: return NAN elif far < 0.005: if t < 379. or t > 4731.: return NAN else: air_gamma = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - 4.428098e-14 * t ** 4. + 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + 0.0001684666 * t + 1.368652 elif far < 0.069: if t < 699. or t > 4731.: return NAN else: a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + 3.103507e-21 * far - 3.391308e-22 a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - 5.469399e-17 * far + 6.058125e-18 a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + 3.865306e-13 * far - 4.302534e-14 a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** 2. - 0.000000001392629 * far + 1.520583e-10 a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + 0.000002688007 * far - 0.0000002651616 a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - 0.002676877 * far + 0.0001580424 a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * far + 1.372714 air_gamma = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t ** 3. + a2 * t ** 2. + a1 * t + a0 elif far >= 0.069: return NAN else: return NAN return air_gamma David Parker Chromalloy - TDAG _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion [attachment "air_gamma.py" deleted by Dave Parker/Chromalloy] _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Feb 1 16:10:20 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 01 Feb 2011 22:10:20 +0100 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: <4D4861E6.3040603@molden.no> Message-ID: <4D4876BC.5060107@molden.no> Den 01.02.2011 20:50, skrev John Salvatier: > I am curious: why you recommend against this? Using the C-API through > cython seems more attractive than using the Cython-specific numpy > features since they need a specific number of dimensions (and I don't > think they broadcast) and don't seem that much easier. Are there > hidden disadvantages to using the C-API? Have I misunderstood something? There is one more thing which should be mentioned: If the algorithm needs to pass an ndarray to a function, this might cause excessive reference counting and boxing/unboxing with Cython. It gets even worse if we want to pass a subarray view of an ndarray, for which Cython will create a new array object. Cython will only play nicely with NumPy arrays if there are no function calls. If there are function calls, we must give up ndarrays and use C pointers directly. Users who don't know the internals of Cython might think that using an ndarray as a function argument is a good idea, or even use slicing to create a view of a subarray. Cython does not forbid this, but it hurts performance immensely. In the NumPy C API we can pass a pointer to a PyArrayObject, adding no overhead beyond the function call. But subarrays are as primitive as in Cython. This makes anything but trivial "computational kernels" painful in Cython or the NumPy C API. Personally I avoid both Cython and the NumPy C API for this reason. There is a very nive language called Fortran 95, for which the compiler knows how to work with arrays. For those that don't use Fortran 95, there are libraries for C++ such as Blitz++. Sturla From jsalvati at u.washington.edu Tue Feb 1 16:16:48 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Tue, 1 Feb 2011 13:16:48 -0800 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: <4D4876BC.5060107@molden.no> References: <4D4861E6.3040603@molden.no> <4D4876BC.5060107@molden.no> Message-ID: Good things to know. On Tue, Feb 1, 2011 at 1:10 PM, Sturla Molden wrote: > Den 01.02.2011 20:50, skrev John Salvatier: > > I am curious: why you recommend against this? Using the C-API through > > cython seems more attractive than using the Cython-specific numpy > > features since they need a specific number of dimensions (and I don't > > think they broadcast) and don't seem that much easier. Are there > > hidden disadvantages to using the C-API? Have I misunderstood something? > > There is one more thing which should be mentioned: > > If the algorithm needs to pass an ndarray to a function, this might > cause excessive reference counting and boxing/unboxing with Cython. It > gets even worse if we want to pass a subarray view of an ndarray, for > which Cython will create a new array object. Cython will only play > nicely with NumPy arrays if there are no function calls. If there are > function calls, we must give up ndarrays and use C pointers directly. > Users who don't know the internals of Cython might think that using an > ndarray as a function argument is a good idea, or even use slicing to > create a view of a subarray. Cython does not forbid this, but it hurts > performance immensely. In the NumPy C API we can pass a pointer to a > PyArrayObject, adding no overhead beyond the function call. But > subarrays are as primitive as in Cython. This makes anything but trivial > "computational kernels" painful in Cython or the NumPy C API. > > Personally I avoid both Cython and the NumPy C API for this reason. > There is a very nive language called Fortran 95, for which the compiler > knows how to work with arrays. For those that don't use Fortran 95, > there are libraries for C++ such as Blitz++. > > > Sturla > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Feb 1 16:40:04 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 1 Feb 2011 16:40:04 -0500 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: Message-ID: On Tue, Feb 1, 2011 at 3:20 PM, wrote: > I'm not sure I need to dive into cython or C for this - performance is not > an issue for my problem - I just want a flexible function that will accept > scalars or arrays. > > Both Sebastian's and eat's suggestions show using indexing to handle the > conditional statements in the original function. The problem I'm having > implementing this is in getting the input arguments and outputs to a common > array size. Here's how I can do this but it seems ugly: > > # t and far are function arguments which may be scalars or arrays > # ag is the output array > # need to make everything array with common length > t = np.array(t, ndmin=1) ? ? ? ?# Convert t to an array > far = np.array(far, ndmin=1) ? ? ? ?# Convert far to an array > ag = t*far*np.nan ? ? ? ? ? ? ? ? ? ? ? ?# Make an output array of the > proper length using broadcasting rules > t = np.zeros_like(ag)+t ? ? ? ? ? ? ? ?# Expand t to the length of the > output array > far = np.zeros_like(ag)+far ? ? ? ?# Expand far to the length of the output > array > > Now with all arrays the same length I can use indexing with logical > statements: > ag[far<0.005] = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + \ > ? ? ? ? ? ? ? ? 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + > 0.0001684666 * t + 1.368652 Does this work? Shouldn't this raise an exception ? >>> x.shape (5, 4) >>> y.shape (5, 4) >>> x[y<10] = y Traceback (most recent call last): File "", line 1, in x[y<10] = y ValueError: array is not broadcastable to correct shape > > The resulting code looks like this: > import numpy as np > > def air_gamma_dp(t, far=0.0): > ? ? """ > ? ? Specific heat ratio (gamma) of Air/JP8 > ? ? t - static temperature, Rankine > ? ? [far] - fuel air ratio [- defaults to 0.0 (dry air)] > ? ? air_gamma - specific heat ratio > ? ? """ > ? ? t = np.array(t, ndmin=1) > ? ? far = np.array(far, ndmin=1) > ? ? ag = t*far*np.nan > ? ? t = np.zeros_like(ag)+t > ? ? far = np.zeros_like(ag)+far > > ? ? far[(far<0.) | (far>0.069)] = np.nan > ? ? t[(t < 379.) | (t > 4731.)] = np.nan > ? ? ag[(far<0.005)] = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + > ? ? ? ? ? ? ? ? ? ? ? ?1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + > 0.0001684666 * t + 1.368652 > ? ? t[(t < 699.) | (t > 4731.)] = np.nan > ? ? a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + 3.103507e-21 > * far - 3.391308e-22 > ? ? a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - 5.469399e-17 > * far + 6.058125e-18 > ? ? a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + 3.865306e-13 > * far - 4.302534e-14 > ? ? a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** 2. - > 0.000000001392629 * far + 1.520583e-10 > ? ? a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + > 0.000002688007 * far - 0.0000002651616 > ? ? a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - 0.002676877 * > far + 0.0001580424 > ? ? a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * far + > 1.372714 > ? ? ag[far>=0.005] = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t ** > 3. + a2 * t ** 2. + a1 * t + a0 > ? ? return ag > > I was hoping there was a more elegant way to do this. I think delaying the broadcasting to before the assignment to ag might be better. Since your expressions are either in far or in t, delaying the broadcasting would avoid the calculations on the broadcasted array and only broad cast before the assignment maybe something like this with t that is not broadcasted cond = (far<0.005)*((t < 379.) | (t > 4731.)) ag[cond] = (np.zeros(ag.shape) + (-3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + > 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + > 0.0001684666 * t + 1.368652))[cond] Josef > > David Parker > Chromalloy - TDAG > > > > From: ? ? ? ?John Salvatier > To: ? ? ? ?Discussion of Numerical Python > Date: ? ? ? ?02/01/2011 02:29 PM > Subject: ? ? ? ?Re: [Numpy-discussion] Vectorize or rewrite function to work > with array inputs? > Sent by: ? ? ? ?numpy-discussion-bounces at scipy.org > ________________________________ > > > Have you thought about using cython to work with the numpy C-API > (http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI)? This will be > fast, simple (you can mix and match Python and Cython). > > As for your specific issue: you can simply cast to all the inputs to numpy > arrays (using > asarray?http://docs.scipy.org/doc/numpy/reference/generated/numpy.asarray.html) > to deal with scalars. This will make sure they get broadcast correctly. > > On Tue, Feb 1, 2011 at 11:22 AM, wrote: > Thanks for the advice. > > Using Sebastian's advice I was able to write a version that worked when the > input arguments are both arrays with the same length. The code provided by > eat works when t is an array, but not for an array of far. > > The numpy.vectorize version works with any combination of scalar or array > input. I still haven't figured out how to rewrite my function to be as > flexible as the numpy.vectorize version at accepting either scalars or array > inputs and properly broadcasting the scalar arguments to the array > arguments. > > David Parker > Chromalloy - TDAG > > > > From: ? ? ? ?eat > To: ? ? ? ?Discussion of Numerical Python > Date: ? ? ? ?01/31/2011 11:37 AM > Subject: ? ? ? ?Re: [Numpy-discussion] Vectorize or rewrite function to work > with array inputs? > Sent by: ? ? ? ?numpy-discussion-bounces at scipy.org > ________________________________ > > > > Hi, > > On Mon, Jan 31, 2011 at 5:15 PM, wrote: > I have several functions like the example below that I would like to make > compatible with array inputs. The problem is the conditional statements give > a ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all(). I can use numpy.vectorize, but if > possible I'd prefer to rewrite the function. Does anyone have any advice the > best way to modify the code to accept array inputs? Thanks in advance for > any assistance. > > If I understod your question correctly, then air_gamma could be coded as: > def air_gamma_0(t, far=0.0): > ??? """ > ??? Specific heat ratio (gamma) of Air/JP8 > ??? t - static temperature, Rankine > ??? [far] - fuel air ratio [- defaults to 0.0 (dry air)] > ??? air_gamma - specific heat ratio > ??? """ > ??? if far< 0.: > ??????? return NAN > ??? elif far < 0.005: > ??????? ag= air_gamma_1(t) > ??????? ag[np.logical_or(t< 379., t> 4731.)]= NAN > ??????? return ag > ??? elif far< 0.069: > ??????? ag= air_gamma_2(t, far) > ??????? ag[np.logical_or(t< 699., t> 4731.)]= NAN > ??????? return ag > ??? else: > ??????? return NAN > Rest of the code is in the attachment. > > > My two cents, > eat > > > > NAN = float('nan') > > def air_gamma(t, far=0.0): > ? ? """ > ? ? Specific heat ratio (gamma) of Air/JP8 > ? ? t - static temperature, Rankine > ? ? [far] - fuel air ratio [- defaults to 0.0 (dry air)] > ? ? air_gamma - specific heat ratio > ? ? """ > ? ? if far < 0.: > ? ? ? ? return NAN > ? ? elif far < 0.005: > ? ? ? ? if t < 379. or t > 4731.: > ? ? ? ? ? ? return NAN > ? ? ? ? else: > ? ? ? ? ? ? air_gamma = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. > + 0.0001684666 * t + 1.368652 > ? ? elif far < 0.069: > ? ? ? ? if t < 699. or t > 4731.: > ? ? ? ? ? ? return NAN > ? ? ? ? else: > ? ? ? ? ? ? a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + > 3.103507e-21 * far - 3.391308e-22 > ? ? ? ? ? ? a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - > 5.469399e-17 * far + 6.058125e-18 > ? ? ? ? ? ? a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + > 3.865306e-13 * far - 4.302534e-14 > ? ? ? ? ? ? a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** > 2. - 0.000000001392629 * far + 1.520583e-10 > ? ? ? ? ? ? a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + > 0.000002688007 * far - 0.0000002651616 > ? ? ? ? ? ? a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - > 0.002676877 * far + 0.0001580424 > ? ? ? ? ? ? a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * > far + 1.372714 > ? ? ? ? ? ? air_gamma = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t > ** 3. + a2 * t ** 2. + a1 * t + a0 > ? ? elif far >= 0.069: > ? ? ? ? return NAN > ? ? else: > ? ? ? ? return NAN > ? ? return air_gamma > > David Parker > Chromalloy - TDAG > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > [attachment "air_gamma.py" deleted by Dave Parker/Chromalloy] > _______________________________________________ > > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From b.telenczuk at biologie.hu-berlin.de Tue Feb 1 16:59:24 2011 From: b.telenczuk at biologie.hu-berlin.de (Bartosz Telenczuk) Date: Tue, 1 Feb 2011 22:59:24 +0100 Subject: [Numpy-discussion] numpy.var fails on memmap Message-ID: <9D9E9E45-E6FE-4EEF-92F1-820AD1CB3DB7@biologie.hu-berlin.de> Hi, numpy.var on a memory mapped array returns a 0-d memmap object instead of a scalar (like with numpy arrays): >>> import numpy as np >>> fp = np.memmap('test.mm', dtype='float32', mode='w+', shape=(10,)) >>> fp[:] = np.random.rand(10) >>> fp_var= np.var(fp) >>> print type(fp_var) >>> print fp_var.ndim 0 Moreover, the obtained object is missing 'filename' attribute, so that it can not be reused in further calculations: >>> fp_std = np.sqrt(fp_var) ... AttributeError: 'memmap' object has no attribute 'filename' I can produce this bug in numpy 1.5.1 with python2.6 (from macports). Cheers, Bartosz Institute for Theoretical Biology Humboldt University of Berlin Germany http://neuroscience.telenczuk.pl From DParker at chromalloy.com Tue Feb 1 17:15:17 2011 From: DParker at chromalloy.com (DParker at chromalloy.com) Date: Tue, 1 Feb 2011 17:15:17 -0500 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: Message-ID: I tested with mixed scalar and array inputs for t and far, and same size arrays for both. I did not test with improperly sized arrays but I would expect (desire) an exception to be raised when the arrays are not broadcastable. Still the code seems like an ugly hack... David Parker Chromalloy - TDAG From: josef.pktd at gmail.com To: Discussion of Numerical Python Date: 02/01/2011 04:39 PM Subject: Re: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? Sent by: numpy-discussion-bounces at scipy.org On Tue, Feb 1, 2011 at 3:20 PM, wrote: > I'm not sure I need to dive into cython or C for this - performance is not > an issue for my problem - I just want a flexible function that will accept > scalars or arrays. > > Both Sebastian's and eat's suggestions show using indexing to handle the > conditional statements in the original function. The problem I'm having > implementing this is in getting the input arguments and outputs to a common > array size. Here's how I can do this but it seems ugly: > > # t and far are function arguments which may be scalars or arrays > # ag is the output array > # need to make everything array with common length > t = np.array(t, ndmin=1) # Convert t to an array > far = np.array(far, ndmin=1) # Convert far to an array > ag = t*far*np.nan # Make an output array of the > proper length using broadcasting rules > t = np.zeros_like(ag)+t # Expand t to the length of the > output array > far = np.zeros_like(ag)+far # Expand far to the length of the output > array > > Now with all arrays the same length I can use indexing with logical > statements: > ag[far<0.005] = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + \ > 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + > 0.0001684666 * t + 1.368652 Does this work? Shouldn't this raise an exception ? >>> x.shape (5, 4) >>> y.shape (5, 4) >>> x[y<10] = y Traceback (most recent call last): File "", line 1, in x[y<10] = y ValueError: array is not broadcastable to correct shape > > The resulting code looks like this: > import numpy as np > > def air_gamma_dp(t, far=0.0): > """ > Specific heat ratio (gamma) of Air/JP8 > t - static temperature, Rankine > [far] - fuel air ratio [- defaults to 0.0 (dry air)] > air_gamma - specific heat ratio > """ > t = np.array(t, ndmin=1) > far = np.array(far, ndmin=1) > ag = t*far*np.nan > t = np.zeros_like(ag)+t > far = np.zeros_like(ag)+far > > far[(far<0.) | (far>0.069)] = np.nan > t[(t < 379.) | (t > 4731.)] = np.nan > ag[(far<0.005)] = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + > 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + > 0.0001684666 * t + 1.368652 > t[(t < 699.) | (t > 4731.)] = np.nan > a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + 3.103507e-21 > * far - 3.391308e-22 > a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - 5.469399e-17 > * far + 6.058125e-18 > a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + 3.865306e-13 > * far - 4.302534e-14 > a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** 2. - > 0.000000001392629 * far + 1.520583e-10 > a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + > 0.000002688007 * far - 0.0000002651616 > a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - 0.002676877 * > far + 0.0001580424 > a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * far + > 1.372714 > ag[far>=0.005] = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t ** > 3. + a2 * t ** 2. + a1 * t + a0 > return ag > > I was hoping there was a more elegant way to do this. I think delaying the broadcasting to before the assignment to ag might be better. Since your expressions are either in far or in t, delaying the broadcasting would avoid the calculations on the broadcasted array and only broad cast before the assignment maybe something like this with t that is not broadcasted cond = (far<0.005)*((t < 379.) | (t > 4731.)) ag[cond] = (np.zeros(ag.shape) + (-3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + > 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + > 0.0001684666 * t + 1.368652))[cond] Josef > > David Parker > Chromalloy - TDAG > > > > From: John Salvatier > To: Discussion of Numerical Python > Date: 02/01/2011 02:29 PM > Subject: Re: [Numpy-discussion] Vectorize or rewrite function to work > with array inputs? > Sent by: numpy-discussion-bounces at scipy.org > ________________________________ > > > Have you thought about using cython to work with the numpy C-API > (http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI)? This will be > fast, simple (you can mix and match Python and Cython). > > As for your specific issue: you can simply cast to all the inputs to numpy > arrays (using > asarray http://docs.scipy.org/doc/numpy/reference/generated/numpy.asarray.html) > to deal with scalars. This will make sure they get broadcast correctly. > > On Tue, Feb 1, 2011 at 11:22 AM, wrote: > Thanks for the advice. > > Using Sebastian's advice I was able to write a version that worked when the > input arguments are both arrays with the same length. The code provided by > eat works when t is an array, but not for an array of far. > > The numpy.vectorize version works with any combination of scalar or array > input. I still haven't figured out how to rewrite my function to be as > flexible as the numpy.vectorize version at accepting either scalars or array > inputs and properly broadcasting the scalar arguments to the array > arguments. > > David Parker > Chromalloy - TDAG > > > > From: eat > To: Discussion of Numerical Python > Date: 01/31/2011 11:37 AM > Subject: Re: [Numpy-discussion] Vectorize or rewrite function to work > with array inputs? > Sent by: numpy-discussion-bounces at scipy.org > ________________________________ > > > > Hi, > > On Mon, Jan 31, 2011 at 5:15 PM, wrote: > I have several functions like the example below that I would like to make > compatible with array inputs. The problem is the conditional statements give > a ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all(). I can use numpy.vectorize, but if > possible I'd prefer to rewrite the function. Does anyone have any advice the > best way to modify the code to accept array inputs? Thanks in advance for > any assistance. > > If I understod your question correctly, then air_gamma could be coded as: > def air_gamma_0(t, far=0.0): > """ > Specific heat ratio (gamma) of Air/JP8 > t - static temperature, Rankine > [far] - fuel air ratio [- defaults to 0.0 (dry air)] > air_gamma - specific heat ratio > """ > if far< 0.: > return NAN > elif far < 0.005: > ag= air_gamma_1(t) > ag[np.logical_or(t< 379., t> 4731.)]= NAN > return ag > elif far< 0.069: > ag= air_gamma_2(t, far) > ag[np.logical_or(t< 699., t> 4731.)]= NAN > return ag > else: > return NAN > Rest of the code is in the attachment. > > > My two cents, > eat > > > > NAN = float('nan') > > def air_gamma(t, far=0.0): > """ > Specific heat ratio (gamma) of Air/JP8 > t - static temperature, Rankine > [far] - fuel air ratio [- defaults to 0.0 (dry air)] > air_gamma - specific heat ratio > """ > if far < 0.: > return NAN > elif far < 0.005: > if t < 379. or t > 4731.: > return NAN > else: > air_gamma = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. > + 0.0001684666 * t + 1.368652 > elif far < 0.069: > if t < 699. or t > 4731.: > return NAN > else: > a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + > 3.103507e-21 * far - 3.391308e-22 > a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - > 5.469399e-17 * far + 6.058125e-18 > a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + > 3.865306e-13 * far - 4.302534e-14 > a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** > 2. - 0.000000001392629 * far + 1.520583e-10 > a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + > 0.000002688007 * far - 0.0000002651616 > a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - > 0.002676877 * far + 0.0001580424 > a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * > far + 1.372714 > air_gamma = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t > ** 3. + a2 * t ** 2. + a1 * t + a0 > elif far >= 0.069: > return NAN > else: > return NAN > return air_gamma > > David Parker > Chromalloy - TDAG > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > [attachment "air_gamma.py" deleted by Dave Parker/Chromalloy] > _______________________________________________ > > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Tue Feb 1 17:27:25 2011 From: e.antero.tammi at gmail.com (eat) Date: Wed, 2 Feb 2011 00:27:25 +0200 Subject: [Numpy-discussion] Vectorize or rewrite function to work with array inputs? In-Reply-To: References: Message-ID: Hi David, (I think the discussion has divergegd, but newer mind ;-) I didn't treat far as a vector (yet, but for sure it's doable). Anyway befor going on more details I'll suggest you*ll study my new attachment. As far I can see, you just have 'simple' polynomial evaluatons, which I have refactored such a way that you just need to provide the coefficients arrays. There is no need to write explicitly polynomial evaluation in the code. My opinion is that the computational side (if your functions really are like shown sofar) will be pretty straightforward. Then it boils down how to handle the coefficient arrays reasonable (perhaps some kind of lightweigt 'database' for them ;-). Please feel free to provide any more information. Regards, eat On Tue, Feb 1, 2011 at 10:20 PM, wrote: > I'm not sure I need to dive into cython or C for this - performance is not > an issue for my problem - I just want a flexible function that will accept > scalars or arrays. > > Both Sebastian's and eat's suggestions show using indexing to handle the > conditional statements in the original function. The problem I'm having > implementing this is in getting the input arguments and outputs to a common > array size. Here's how I can do this but it seems ugly: > > # t and far are function arguments which may be scalars or arrays > # ag is the output array > # need to make everything array with common length > t = np.array(t, ndmin=1) # Convert t to an array > far = np.array(far, ndmin=1) # Convert far to an array > ag = t*far*np.nan # Make an output array of the > proper length using broadcasting rules > t = np.zeros_like(ag)+t # Expand t to the length of the > output array > far = np.zeros_like(ag)+far # Expand far to the length of the output > array > > Now with all arrays the same length I can use indexing with logical > statements: > ag[far<0.005] = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + \ > 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + > 0.0001684666 * t + 1.368652 > > The resulting code looks like this: > import numpy as np > > def air_gamma_dp(t, far=0.0): > """ > Specific heat ratio (gamma) of Air/JP8 > t - static temperature, Rankine > [far] - fuel air ratio [- defaults to 0.0 (dry air)] > air_gamma - specific heat ratio > """ > t = np.array(t, ndmin=1) > far = np.array(far, ndmin=1) > ag = t*far*np.nan > t = np.zeros_like(ag)+t > far = np.zeros_like(ag)+far > > far[(far<0.) | (far>0.069)] = np.nan > t[(t < 379.) | (t > 4731.)] = np.nan > ag[(far<0.005)] = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + > 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. + > 0.0001684666 * t + 1.368652 > t[(t < 699.) | (t > 4731.)] = np.nan > a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + 3.103507e-21 > * far - 3.391308e-22 > a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - > 5.469399e-17 * far + 6.058125e-18 > a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + 3.865306e-13 > * far - 4.302534e-14 > a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** 2. - > 0.000000001392629 * far + 1.520583e-10 > a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + > 0.000002688007 * far - 0.0000002651616 > a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - 0.002676877 * > far + 0.0001580424 > a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * far + > 1.372714 > ag[far>=0.005] = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t ** > 3. + a2 * t ** 2. + a1 * t + a0 > return ag > > I was hoping there was a more elegant way to do this. > > David Parker > Chromalloy - TDAG > > > > From: John Salvatier > To: Discussion of Numerical Python > Date: 02/01/2011 02:29 PM > Subject: Re: [Numpy-discussion] Vectorize or rewrite function to > work with array inputs? > Sent by: numpy-discussion-bounces at scipy.org > ------------------------------ > > > > Have you thought about using cython to work with the numpy C-API (* > http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI*)? > This will be fast, simple (you can mix and match Python and Cython). > > As for your specific issue: you can simply cast to all the inputs to numpy > arrays (using asarray * > http://docs.scipy.org/doc/numpy/reference/generated/numpy.asarray.html*) > to deal with scalars. This will make sure they get broadcast correctly. > > On Tue, Feb 1, 2011 at 11:22 AM, <*DParker at chromalloy.com*> > wrote: > Thanks for the advice. > > Using Sebastian's advice I was able to write a version that worked when the > input arguments are both arrays with the same length. The code provided by > eat works when t is an array, but not for an array of far. > > The numpy.vectorize version works with any combination of scalar or array > input. I still haven't figured out how to rewrite my function to be as > flexible as the numpy.vectorize version at accepting either scalars or array > inputs and properly broadcasting the scalar arguments to the array > arguments. > > David Parker > Chromalloy - TDAG > > > > From: eat <*e.antero.tammi at gmail.com* > > To: Discussion of Numerical Python <*numpy-discussion at scipy.org* > > > Date: 01/31/2011 11:37 AM > Subject: Re: [Numpy-discussion] Vectorize or rewrite function to > work with array inputs? > Sent by: *numpy-discussion-bounces at scipy.org* > ------------------------------ > > > > > Hi, > > On Mon, Jan 31, 2011 at 5:15 PM, <*DParker at chromalloy.com*> > wrote: > I have several functions like the example below that I would like to make > compatible with array inputs. The problem is the conditional statements give > a *ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all()*. I can use numpy.vectorize, but if > possible I'd prefer to rewrite the function. Does anyone have any advice the > best way to modify the code to accept array inputs? Thanks in advance for > any assistance. > > If I understod your question correctly, then air_gamma could be coded as: > def air_gamma_0(t, far=0.0): > """ > Specific heat ratio (gamma) of Air/JP8 > t - static temperature, Rankine > [far] - fuel air ratio [- defaults to 0.0 (dry air)] > air_gamma - specific heat ratio > """ > if far< 0.: > return NAN > elif far < 0.005: > ag= air_gamma_1(t) > ag[np.logical_or(t< 379., t> 4731.)]= NAN > return ag > elif far< 0.069: > ag= air_gamma_2(t, far) > ag[np.logical_or(t< 699., t> 4731.)]= NAN > return ag > else: > return NAN > Rest of the code is in the attachment. > > > My two cents, > eat > > > > NAN = float('nan') > > def air_gamma(t, far=0.0): > """ > Specific heat ratio (gamma) of Air/JP8 > t - static temperature, Rankine > [far] - fuel air ratio [- defaults to 0.0 (dry air)] > air_gamma - specific heat ratio > """ > if far < 0.: > return NAN > elif far < 0.005: > if t < 379. or t > 4731.: > return NAN > else: > air_gamma = -3.472487e-22 * t ** 6. + 6.218811e-18 * t ** 5. - > 4.428098e-14 * t ** 4. + 1.569889e-10 * t ** 3. - 0.0000002753524 * t ** 2. > + 0.0001684666 * t + 1.368652 > elif far < 0.069: > if t < 699. or t > 4731.: > return NAN > else: > a6 = 4.114808e-20 * far ** 3. - 1.644588e-20 * far ** 2. + * > 3.103507* e-21 * far - 3.391308e-22 > a5 = -6.819015e-16 * far ** 3. + 2.773945e-16 * far ** 2. - > 5.469399e-17 * far + 6.058125e-18 > a4 = 4.684637e-12 * far ** 3. - 1.887227e-12 * far ** 2. + > 3.865306e-13 * far - 4.302534e-14 > a3 = -0.00000001700602 * far ** 3. + 0.000000006593809 * far ** > 2. - 0.000000001392629 * far + 1.520583e-10 > a2 = 0.00003431136 * far ** 3. - 0.00001248285 * far ** 2. + > 0.000002688007 * far - 0.0000002651616 > a1 = -0.03792449 * far ** 3. + 0.01261025 * far ** 2. - > 0.002676877 * far + 0.0001580424 > a0 = 13.65379 * far ** 3. - 3.311225 * far ** 2. + 0.3573201 * > far + 1.372714 > air_gamma = a6 * t ** 6. + a5 * t ** 5. + a4 * t ** 4. + a3 * t > ** 3. + a2 * t ** 2. + a1 * t + a0 > elif far >= 0.069: > return NAN > else: > return NAN > return air_gamma > > David Parker > Chromalloy - TDAG > _______________________________________________ > NumPy-Discussion mailing list* > **NumPy-Discussion at scipy.org* * > **http://mail.scipy.org/mailman/listinfo/numpy-discussion* > > [attachment "air_gamma.py" deleted by Dave Parker/Chromalloy] > _______________________________________________ > > NumPy-Discussion mailing list* > **NumPy-Discussion at scipy.org* > *http://mail.scipy.org/mailman/listinfo/numpy-discussion* > > > _______________________________________________ > NumPy-Discussion mailing list* > **NumPy-Discussion at scipy.org* * > **http://mail.scipy.org/mailman/listinfo/numpy-discussion* > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: air_gamma.py Type: application/octet-stream Size: 3305 bytes Desc: not available URL: From pav at iki.fi Tue Feb 1 18:58:06 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 1 Feb 2011 23:58:06 +0000 (UTC) Subject: [Numpy-discussion] Please keep Numpy master working on Py3 Message-ID: Hi, The master branch did not build today on Python 3. Please make sure that your code works correctly also on Python 3, before pushing it. *** I mostly fixed the stuff for now, mostly just the usual bytes vs unicode. On Python 3, the tests however give two non-obvious failures -- I'm not sure if it's just a matter of the raised exception having a different type on Py2 vs Py3, or if it reflects something going wrong. Mark, do you have ideas? Pauli ====================================================================== ERROR: test_new_iterator.test_iter_buffering_delayed_alloc ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.1/dist-packages/nose-1.0.0.dev-py3.1.egg/ nose/case.py", line 188, in runTest self.test(*self.arg) File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- packages/numpy/core/tests/test_new_iterator.py", line 1471, in test_iter_buffering_delayed_alloc assert_raises(ValueError, lambda i:i[0:2], i) File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- packages/numpy/testing/utils.py", line 983, in assert_raises return nose.tools.assert_raises(*args,**kwargs) File "/usr/lib/python3.1/unittest.py", line 589, in assertRaises callableObj(*args, **kwargs) File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- packages/numpy/core/tests/test_new_iterator.py", line 1471, in assert_raises(ValueError, lambda i:i[0:2], i) TypeError: sequence index must be integer, not 'slice' ====================================================================== ERROR: test_object_array_self_reference (test_regression.TestRegression) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- packages/numpy/core/tests/test_regression.py", line 1499, in test_object_array_self_reference assert_raises(ValueError, oct, a) File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- packages/numpy/testing/utils.py", line 983, in assert_raises return nose.tools.assert_raises(*args,**kwargs) File "/usr/lib/python3.1/unittest.py", line 589, in assertRaises callableObj(*args, **kwargs) TypeError: only integer arrays with one element can be converted to an index From mwwiebe at gmail.com Tue Feb 1 19:42:04 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Tue, 1 Feb 2011 16:42:04 -0800 Subject: [Numpy-discussion] Please keep Numpy master working on Py3 In-Reply-To: References: Message-ID: On Tue, Feb 1, 2011 at 3:58 PM, Pauli Virtanen wrote: > Hi, > > The master branch did not build today on Python 3. Please make sure that > your code works correctly also on Python 3, before pushing it. > Sorry about that, I had hoped to be able to use the build-bot to test/diagnose the issues, but it's down and I didn't pursue it further. > > *** > > I mostly fixed the stuff for now, mostly just the usual bytes vs unicode. > > Thanks. > On Python 3, the tests however give two non-obvious failures -- I'm not > sure if it's just a matter of the raised exception having a different > type on Py2 vs Py3, or if it reflects something going wrong. Mark, do you > have ideas? > Sure, taking a look. ====================================================================== > ERROR: test_new_iterator.test_iter_buffering_delayed_alloc > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/local/lib/python3.1/dist-packages/nose-1.0.0.dev-py3.1.egg/ > nose/case.py", line 188, in runTest > self.test(*self.arg) > File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- > packages/numpy/core/tests/test_new_iterator.py", line 1471, in > test_iter_buffering_delayed_alloc > assert_raises(ValueError, lambda i:i[0:2], i) > File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- > packages/numpy/testing/utils.py", line 983, in assert_raises > return nose.tools.assert_raises(*args,**kwargs) > File "/usr/lib/python3.1/unittest.py", line 589, in assertRaises > callableObj(*args, **kwargs) > File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- > packages/numpy/core/tests/test_new_iterator.py", line 1471, in > assert_raises(ValueError, lambda i:i[0:2], i) > TypeError: sequence index must be integer, not 'slice' > This looks like it's a problem in new_iterator_pywrap.c, in PySequenceMethods npyiter_as_sequence. I got this error when trying to slice before implementing the slot sq_slice. I copied the definition from array_as_sequence, which I thought would already have any adjustments for Py3. The value in that slot shouldn't be NULL, though, so I don't know why it's giving the error, maybe the struct definition changed, and somehow in the array sequence methods that gets masked? Or maybe the assignment slice slot needs to be filled as well, but that doesn't seem right to me. > > ====================================================================== > ERROR: test_object_array_self_reference (test_regression.TestRegression) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- > packages/numpy/core/tests/test_regression.py", line 1499, in > test_object_array_self_reference > assert_raises(ValueError, oct, a) > File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site- > packages/numpy/testing/utils.py", line 983, in assert_raises > return nose.tools.assert_raises(*args,**kwargs) > File "/usr/lib/python3.1/unittest.py", line 589, in assertRaises > callableObj(*args, **kwargs) > TypeError: only integer arrays with one element can be converted to an > index > I see, in Py3 oct and hex call array_index instead of array_oct and array_hex. I think changing the self-reference exceptions in int/long/float/oct/hex to TypeError instead of ValueError is probably a good way to make this consistent. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Feb 1 20:11:19 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 2 Feb 2011 01:11:19 +0000 (UTC) Subject: [Numpy-discussion] Please keep Numpy master working on Py3 References: Message-ID: On Tue, 01 Feb 2011 16:42:04 -0800, Mark Wiebe wrote: [clip: problem 1] > This looks like it's a problem in new_iterator_pywrap.c, in > PySequenceMethods npyiter_as_sequence. I got this error when trying to > slice before implementing the slot sq_slice. I copied the definition > from array_as_sequence, which I thought would already have any > adjustments for Py3. The value in that slot shouldn't be NULL, though, > so I don't know why it's giving the error, maybe the struct definition > changed, and somehow in the array sequence methods that gets masked? Or > maybe the assignment slice slot needs to be filled as well, but that > doesn't seem right to me. The slot sq_slice is in the struct on Python 3, but the entry is a dummy no-op. It appears slices are supposed to be handled in PyMappingMethod. (If you walk through the __getitem__ code for ndarray, you'll note that if it encounters a single slice, it falls back to calling the PySequence methods :O) The Python C-API docs don't seem to say much on the status of sq_slice on Python 2, but at least on the Python side I know that the corresponding __*slice__ hooks are deprecated in favor of the __*item__ routines since Python 2.6. [clip: problem 2] > I see, in Py3 oct and hex call array_index instead of array_oct and > array_hex. This is because the __oct__ and __hex__ C-slots were removed from PyNumberMethods on Py3. > I think changing the self-reference exceptions in > int/long/float/oct/hex to TypeError instead of ValueError is probably > a good way to make this consistent. Probably a good idea, since the other errors there are also TypeErrors. Pauli From pav at iki.fi Tue Feb 1 20:31:00 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 2 Feb 2011 01:31:00 +0000 (UTC) Subject: [Numpy-discussion] Please keep Numpy master working on Py3 References: Message-ID: On Tue, 01 Feb 2011 16:42:04 -0800, Mark Wiebe wrote: [clip] > Sorry about that, I had hoped to be able to use the build-bot to > test/diagnose the issues, but it's down and I didn't pursue it further. Hmm, I see Github's SVN gateway is broken again. Perhaps it could be possible to make it work directly against Git... Personally, I would like to keep the master branch at a known-good-no-experimental-code state at all times; different Python versions one can install locally without too much trouble, but for cross-architecture testing of experimental code using the buildbot would be very helpful. If we are going to fix up a "successor" to the current buildbot at some point, this use case probably should be kept in mind. Pauli From ecarlson at eng.ua.edu Tue Feb 1 22:12:43 2011 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Tue, 01 Feb 2011 21:12:43 -0600 Subject: [Numpy-discussion] Postdoctoral Positions at Univ of Alabama Message-ID: Hello All, I apologize for using this venue, and flame me if you must, but I wish to announce two Postdoctoral positions in the Department of Chemical and Biological Engineering at the University of Alabama, in Tuscaloosa, AL, USA. The funding concerns modeling of geological sequestration of CO2. The actual activities for the project involve development of some open source software for petroleum reservoir simulation, development of vast amounts of educational documentation and software (again open source), theoretical developments - really, these positions can be almost anything. For example: 1. Efficient parallel coding 2. Theoretical formulations for reactive transport in porous media 3. Algorithm development 4. Scientific Python educational documentation 5. GUI development 6. Visualization 7. Video production 8. Give me some suggestions If interested, you probably should contact me via email (ecarlson .at. eng. ua .edu )with a very clear subject line of "Sequestration Postdoctoral Position Inquiry" and if I like what you present, I will provide the additional instructions to formally apply through out HR dept. The positions are available now, and can last until at least 12/8/2012, with a good chance for extension for a few months after that. Regards, Eric Carlson, Associate Professor University of Alabama From david at silveregg.co.jp Tue Feb 1 23:53:25 2011 From: david at silveregg.co.jp (David) Date: Wed, 02 Feb 2011 13:53:25 +0900 Subject: [Numpy-discussion] Please keep Numpy master working on Py3 In-Reply-To: References: Message-ID: <4D48E345.4040301@silveregg.co.jp> On 02/02/2011 08:58 AM, Pauli Virtanen wrote: > Hi, > > The master branch did not build today on Python 3. Please make sure that > your code works correctly also on Python 3, before pushing it. > > *** > > I mostly fixed the stuff for now, mostly just the usual bytes vs unicode. > > On Python 3, the tests however give two non-obvious failures -- I'm not > sure if it's just a matter of the raised exception having a different > type on Py2 vs Py3, or if it reflects something going wrong. Mark, do you > have ideas? Following the merge on Mark's code, I am stlightly concerned about the dependency between ufunc and multiarray (i.e. ufunc include header in multiarray). In the meantime, I put the relevant header in numpy/core/src/private, to make the dependency clearer. cheers, David From david at silveregg.co.jp Wed Feb 2 00:49:37 2011 From: david at silveregg.co.jp (David) Date: Wed, 02 Feb 2011 14:49:37 +0900 Subject: [Numpy-discussion] Please keep Numpy master working on Py3 In-Reply-To: <4D48E345.4040301@silveregg.co.jp> References: <4D48E345.4040301@silveregg.co.jp> Message-ID: <4D48F071.2000907@silveregg.co.jp> On 02/02/2011 01:53 PM, David wrote: > On 02/02/2011 08:58 AM, Pauli Virtanen wrote: >> Hi, >> >> The master branch did not build today on Python 3. Please make sure that >> your code works correctly also on Python 3, before pushing it. >> >> *** >> >> I mostly fixed the stuff for now, mostly just the usual bytes vs unicode. >> >> On Python 3, the tests however give two non-obvious failures -- I'm not >> sure if it's just a matter of the raised exception having a different >> type on Py2 vs Py3, or if it reflects something going wrong. Mark, do you >> have ideas? > > Following the merge on Mark's code, I am stlightly concerned about the > dependency between ufunc and multiarray (i.e. ufunc include header in > multiarray). > > In the meantime, I put the relevant header in numpy/core/src/private, to > make the dependency clearer. Following that argument, there are other unwanted dependencies between multiarray and ufunc, causing circular dependencies. I don't think they were there before, and it makes building numpy with a dependency-based tool like scons or waf extremely difficult. cheers, David From mwwiebe at gmail.com Wed Feb 2 00:57:29 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Tue, 1 Feb 2011 21:57:29 -0800 Subject: [Numpy-discussion] Please keep Numpy master working on Py3 In-Reply-To: <4D48F071.2000907@silveregg.co.jp> References: <4D48E345.4040301@silveregg.co.jp> <4D48F071.2000907@silveregg.co.jp> Message-ID: On Tue, Feb 1, 2011 at 9:49 PM, David wrote: > > > In the meantime, I put the relevant header in numpy/core/src/private, to > > make the dependency clearer. > > Following that argument, there are other unwanted dependencies between > multiarray and ufunc, causing circular dependencies. I don't think they > were there before, and it makes building numpy with a dependency-based > tool like scons or waf extremely difficult. > > This particular separation of two components felt somewhat artificial to me. For the iterator buffering, as an example, I initially thought it would be good to use the same default as the ufuncs do. Instead I ended up just using a hard-coded default. I think the error handling policy in the ufuncs could also be useful for computations in the core. This can't be changed while maintaining ABI compatibility, though. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Wed Feb 2 01:09:35 2011 From: david at silveregg.co.jp (David) Date: Wed, 02 Feb 2011 15:09:35 +0900 Subject: [Numpy-discussion] Please keep Numpy master working on Py3 In-Reply-To: References: <4D48E345.4040301@silveregg.co.jp> <4D48F071.2000907@silveregg.co.jp> Message-ID: <4D48F51F.6090205@silveregg.co.jp> On 02/02/2011 02:57 PM, Mark Wiebe wrote: > On Tue, Feb 1, 2011 at 9:49 PM, David > wrote: > > > > > > In the meantime, I put the relevant header in > numpy/core/src/private, to > > make the dependency clearer. > > Following that argument, there are other unwanted dependencies between > multiarray and ufunc, causing circular dependencies. I don't think they > were there before, and it makes building numpy with a dependency-based > tool like scons or waf extremely difficult. > > > This particular separation of two components felt somewhat artificial to > me. For the iterator buffering, as an example, I initially thought it > would be good to use the same default as the ufuncs do. Instead I ended > up just using a hard-coded default. Not sure I understand the exact argument, but if it is just a matter of getting default values, it is easy to share them using a common header. > I think the error handling policy > in the ufuncs could also be useful for computations in the core. At the moment, ufunc and multiarray are separate extensions. If each depends on the other API-wise, it is an issue. If there are some commonalities, then they can be put in a separate extensions. They are already way too big as they are (historical reasons), we should really fight this for maintanability. I realize the code organization and the build stuff is a bit of a mess at the moment, so if you have questions on that aspect, feel free to ask clarification or help, cheers, David From asmi.capri at gmail.com Wed Feb 2 05:22:15 2011 From: asmi.capri at gmail.com (Asmi Shah) Date: Wed, 2 Feb 2011 11:22:15 +0100 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: <201102012057.57263.faltet@pytables.org> References: <4D4849B1.30607@noaa.gov> <4D4857C8.2010404@molden.no> <201102012057.57263.faltet@pytables.org> Message-ID: Hi all, It seems that using 64 bit python is the solution. But the thing is i would compile my code and wanna distribute it to the clients.. and that is the only reason why i want to work on 32 bit system. Sturla, how I can make it sure that some part of the data is kept on the disk and only the necessary one in the memory; as this seems to be a solution to my problem. As i said i want a 3d visualization out of the numpy array. it works fine for the downsampled dataset. And to visualize, i have to convert the 16bit data into 8bit as PIL doesnt support 16 bit data.. the only thing i do to create my array is this: stack = numpy.empty((120, 1024, 1024)) i = 0 os.chdir(dirr) for f in os.listdir(dirr): im = Image.open(f) im = im.convert("L") a = numpy.asarray(im) print a.dtype stack[i] = a i += 1 one more thing, it really doesnt work for tiff files at all, i have to convert them into jpgs as a prior step to this. and it at max lets me create an array for around 60 slices only, where as my requirement would be around 100 to 200 images.. any ideas? can diagnose the problem?? thanks a lot.. asmi On Tue, Feb 1, 2011 at 8:57 PM, Francesc Alted wrote: > A Tuesday 01 February 2011 19:58:16 Sturla Molden escrigu?: > > Den 01.02.2011 18:58, skrev Christopher Barker: > > > But if you really have big collections of images, you might try > > > memory mapped arrays -- as Sturla pointed out they wont' let you > > > create monster arrays on a 32 bit python, > > > > But they will on 64 bit Python :D We can just memory map a temporary > > file, and fake as much "memory" as we need. The OS will optimize the > > disk access. > > What do you mean by 'optimize the disk access'? One of the drawbacks of > memory mapped files is precisely that OS cannot distinguish between data > that belongs to 'disk' and data that belongs to 'memory'. This normally > introduces extreme slowness in other programs when datasets in files > exceeds physical memory but have to loaded by OS -- the reason being > that OS swaps out most of programs/shared libraries that were in memory > in order to be able to load new 'disk' data. > > The other important drawback of the memory mapped files is that you need > to have at very least an amount of virtual memory that is enough to keep > all of these data files. In general, you only have virtual memory that > is between 1.5x and 2x the physical memory (having more than this is > generally regarded as a waste of disk space). > > This is why I prefer very much reading directly from a file: the OS in > this case is able to distinguish between data belonging to 'disk' and > data belonging to 'memory'. Is in this case when the OS can really > optimize disk access (unless you have complicated setups). > > > Also consider that hardware is cheap compared to labour, > > at least in Europe. Is programming for memory limited 32 bit Python > > worth the effort? > > -- > Francesc Alted > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From totonixsame at gmail.com Wed Feb 2 05:41:28 2011 From: totonixsame at gmail.com (totonixsame at gmail.com) Date: Wed, 2 Feb 2011 08:41:28 -0200 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: References: <4D4849B1.30607@noaa.gov> <4D4857C8.2010404@molden.no> <201102012057.57263.faltet@pytables.org> Message-ID: On Wed, Feb 2, 2011 at 8:22 AM, Asmi Shah wrote: > Hi all, > It seems that using 64 bit python is the solution. But the thing is i would > compile my code and wanna distribute it to the clients.. and that is the > only reason why i want to work on 32 bit system. Sturla, how I can make it > sure that some part of the data is kept on the disk and only the necessary > one in the memory; as this seems to be a solution to my problem. As i said i > want a 3d visualization out of the numpy array. it works fine for the > downsampled dataset. And to visualize, i have to convert the 16bit data into > 8bit as PIL doesnt support 16 bit data.. the only thing i do to create my > array is this: > stack = numpy.empty((120, 1024, 1024)) > i = 0 > os.chdir(dirr) > for f in os.listdir(dirr): > ?? ?im = Image.open(f) > ?? ?im = im.convert("L") > ?? ?a = numpy.asarray(im) > ?? ?print a.dtype > ?? ?stack[i] = a > ?? ?i += 1 > one more thing, it really doesnt work for tiff files at all, i have to > convert them into jpgs as a prior step to this. and it at max lets me create > an array for around 60 slices only, where as my requirement would be around > 100 to 200 images.. > any ideas? can diagnose the problem?? > thanks a lot.. asmi Give a try to pylibtiff [1], the cool thing is it give support to get metainformations from tiff files. There is support to read tiff files in VTK [2]. [1] - http://code.google.com/p/pylibtiff/ [2] - http://www.vtk.org/doc/nightly/html/classvtkTIFFReader.html From josef.pktd at gmail.com Wed Feb 2 09:07:34 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 2 Feb 2011 09:07:34 -0500 Subject: [Numpy-discussion] Non-deterministic floating point behavior in numpy 1.5.1 ? In-Reply-To: References: Message-ID: On Wed, Jan 12, 2011 at 1:05 PM, Pauli Virtanen wrote: > Wed, 12 Jan 2011 17:34:32 +0000, Gr?gory Guyomarc'h wrote: > [clip] >> ? ? ? y = (numpy.dot(m, x) - 13.90901663) * 1000.0 print y[0] > [clip] >> Also, I could not reproduce it with older versions of numpy such as 1.3. >> Is this behavior expected ? Is there a way to make sure the results of a >> numpy floating point computations remains the same for multiple runs? > > There are essentially no changes in the dot() routine since 1.3.0 in > Numpy. The non-determinism is probably in the BLAS linear algebra library > you have linked Numpy with. What platform are you using? (Windows? Linux? > Where did you obtain Numpy binaries?) > > What do you get if you replace `numpy.dot` with > `numpy.core.multiarray.dot` (which does not use BLAS)? > > There's another thread on a similar issue here: > > http://permalink.gmane.org/gmane.comp.python.scientific.user/27444 The question seems to show up now pretty often, for Win32 np.core.multiarray.dot always produces the same result Lapack dot has variation, but not always, and it sometimes produces consistently a result different from np.core.multiarray.dot I copied some results to the attached script file, running inside IDLE - variation, running as script on commandline - no variation, but different, running repeatedly with execfile in commandline session - same as IDLE session. I'm using the official numpy 1.5.1 release Josef > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- '''Non-deterministic behavior of Lapack from january 12, 2010, numpy mailing list Non-deterministic floating point behavior in numpy 1.5.1 ''' import numpy as np #numpy.core.multiarray.dot x = np.array([[0.00010876945607980702], [0.22568137594619658], [5.6435218858623557]]) print '\nusing lapack' for i in range(10): m = np.array([[36.0 * 36.0, 36.0, 1.0] for j in range(6)]) y = (np.dot(m, x) - 13.90901663) * 1000.0 print y[0] print '\nusing np.core.multiarray.dot' for i in range(10): m = np.array([[36.0 * 36.0, 36.0, 1.0] for j in range(6)]) y = (np.core.multiarray.dot(m, x) - 13.90901663) * 1000.0 print y[0] ''' in IDLE session, running several times the same script using lapack [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] using np.core.multiarray.dot [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] >>> using lapack [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] using np.core.multiarray.dot [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] on the command line, new run of script each time C:\Josef\work-oth>try_numpy_determ.py using lapack [ 5.00486408e-06] [ 5.00486408e-06] [ 5.00486408e-06] [ 5.00486408e-06] [ 5.00486408e-06] [ 5.00486408e-06] [ 5.00486408e-06] [ 5.00486408e-06] [ 5.00486408e-06] [ 5.00486408e-06] using np.core.multiarray.dot [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] session on commandline, without IDLE C:\Josef\work-oth>python -i try_numpy_determ.py using lapack [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] using np.core.multiarray.dot [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] >>> execfile('try_numpy_determ.py') using lapack [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] using np.core.multiarray.dot [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] >>> execfile('try_numpy_determ.py') using lapack [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] [ 5.00486230e-06] [ 5.00486408e-06] using np.core.multiarray.dot [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] [ 5.00486230e-06] >>> >>> np.show_config() atlas_threads_info: NOT AVAILABLE blas_opt_info: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\local\\lib\\yop\\sse2'] define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')] language = c atlas_blas_threads_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\local\\lib\\yop\\sse2'] define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')] language = f77 atlas_info: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\local\\lib\\yop\\sse2'] language = f77 lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\local\\lib\\yop\\sse2'] language = c mkl_info: NOT AVAILABLE >>> np.__version__ '1.5.1' ''' From Chris.Barker at noaa.gov Wed Feb 2 12:12:47 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 02 Feb 2011 09:12:47 -0800 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: References: <4D4849B1.30607@noaa.gov> <4D4857C8.2010404@molden.no> <201102012057.57263.faltet@pytables.org> Message-ID: <4D49908F.9080904@noaa.gov> > It seems that using 64 bit python is the solution. It's certainly the easy way to access a lot of memory -- and memory is cheap these days. > But the thing is i would > compile my code and wanna distribute it to the clients.. I don't think 64 bit gets in the way of that -- except that it will only runon 64 bit systems, which may be an issue. > only reason why i want to work on 32 bit system. Sturla, how I can make it > sure that some part of the data is kept on the disk and only the necessary > one in the memory; as this seems to be a solution to my problem. You can "roll your own" and have a disk cache of some sort -- it would be pretty easy to store each image in a *.npz file and load the up as you need them. But it would probably be even easier to use one of the hdf-based libarries, such as pytables -- I htink it will do it all for you. One other option, that I've never tried, is carray, which is an array compressed in memory. Depending on your images, perhaps they would compress a lot (or not ....): https://github.com/FrancescAlted/carray http://mail.scipy.org/pipermail/numpy-discussion/2010-August/052378.html > As i said i > want a 3d visualization out of the numpy array. it works fine for the > downsampled dataset. And to visualize, i have to convert the 16bit data into > 8bit as PIL doesnt support 16 bit data. It's unclear to me what you native data really is: 16 bitgreyscale? 8bit greyscale? either one should fit OK into 32 bit memory, and if 8bit is accurate enough foryour needs, then it should be pretty easy. > stack = numpy.empty((120, 1024, 1024)) numpy defaults to double precision float, np.float64, i.e. 8 bytes per element -- you probably don't want that if you are concerned about memoery use, and have 8 or 16 bit greyscale images. Try: stack = np.empty((120, 1024, 1024), dtype=np.uint8) # (or dtype=np.uint16) > i = 0 > > os.chdir(dirr) > for f in os.listdir(dirr): > > im = Image.open(f) > im = im.convert("L") you ight want to try mode "I". That should give you a 32 bit integer grey scale, which should hold all the 16 bit data without loss -- then you can convert to 16 bit when you bring it into numpy. > a = numpy.asarray(im) > print a.dtype what does this print? it should be np.uint8 > stack[i] = a here, if a is a uint8, numpy will convert it to a float64, to fit into array a -- that's why you want to set the dtype of array a when you create it. > one more thing, it really doesnt work for tiff files at all, i have to > convert them into jpgs as a prior step to this. probably not the best choice either, jpeg is lossy -- is will smear things out a bit, which you may not want. It also only holds 24 bit RGB (I think), which is both a waste and will lose information from 16 bit greyscale. If you have to convert, try PNG, though I'm not sure if it handles 16 bit greyscale, either. I'd look at a lib that can read tiff properly -- some have been suggested here, and you can also use GDAL, which is meant for geo-referenced data, but you can ignore the geo information and just get an image if you want. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mdroe at stsci.edu Wed Feb 2 14:11:12 2011 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 02 Feb 2011 14:11:12 -0500 Subject: [Numpy-discussion] BUG: add.accumulate on an array of shape (0, ) causes divide-by-zero Message-ID: <4D49AC50.60103@stsci.edu> Just wanted to make the developers aware of this bug that causes one of our pyfits unit tests to fail: http://projects.scipy.org/numpy/ticket/1733 Mike From ilya_shl at alum.mit.edu Wed Feb 2 16:42:49 2011 From: ilya_shl at alum.mit.edu (Ilya Shlyakhter) Date: Wed, 2 Feb 2011 16:42:49 -0500 Subject: [Numpy-discussion] relational join Message-ID: Does numpy have a relational join operation for joining recordarrays? thanks, ilya From robert.kern at gmail.com Wed Feb 2 16:46:36 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 2 Feb 2011 21:46:36 +0000 Subject: [Numpy-discussion] relational join In-Reply-To: References: Message-ID: On Wed, Feb 2, 2011 at 21:42, Ilya Shlyakhter wrote: > Does numpy have a relational join operation for joining recordarrays? [~] |1> from numpy.lib import recfunctions [~] |2> recfunctions.join_by? Type: function Base Class: String Form: Namespace: Interactive File: /Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/numpy/lib/recfunctions.py Definition: recfunctions.join_by(key, r1, r2, jointype='inner', r1postfix='1', r2postfix='2', defaults=None, usemask=True, asrecarray=False) Docstring: Join arrays `r1` and `r2` on key `key`. The key should be either a string or a sequence of string corresponding to the fields used to join the array. An exception is raised if the `key` field cannot be found in the two input arrays. Neither `r1` nor `r2` should have any duplicates along `key`: the presence of duplicates will make the output quite unreliable. Note that duplicates are not looked for by the algorithm. Parameters ---------- key : {string, sequence} A string or a sequence of strings corresponding to the fields used for comparison. r1, r2 : arrays Structured arrays. jointype : {'inner', 'outer', 'leftouter'}, optional If 'inner', returns the elements common to both r1 and r2. If 'outer', returns the common elements as well as the elements of r1 not in r2 and the elements of not in r2. If 'leftouter', returns the common elements and the elements of r1 not in r2. r1postfix : string, optional String appended to the names of the fields of r1 that are present in r2 but absent of the key. r2postfix : string, optional String appended to the names of the fields of r2 that are present in r1 but absent of the key. defaults : {dictionary}, optional Dictionary mapping field names to the corresponding default values. usemask : {True, False}, optional Whether to return a MaskedArray (or MaskedRecords is `asrecarray==True`) or a ndarray. asrecarray : {False, True}, optional Whether to return a recarray (or MaskedRecords if `usemask==True`) or just a flexible-type ndarray. Notes ----- * The output is sorted along the key. * A temporary array is formed by dropping the fields not in the key for the two arrays and concatenating the result. This array is then sorted, and the common entries selected. The output is constructed by filling the fields with the selected entries. Matching is not preserved if there are some duplicates... -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From wesmckinn at gmail.com Wed Feb 2 17:24:03 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Wed, 2 Feb 2011 17:24:03 -0500 Subject: [Numpy-discussion] relational join In-Reply-To: References: Message-ID: On Wed, Feb 2, 2011 at 4:46 PM, Robert Kern wrote: > On Wed, Feb 2, 2011 at 21:42, Ilya Shlyakhter wrote: >> Does numpy have a relational join operation for joining recordarrays? > > [~] > |1> from numpy.lib import recfunctions > > [~] > |2> recfunctions.join_by? > Type: ? ? ? ? ? function > Base Class: ? ? > String Form: ? ? > Namespace: ? ? ?Interactive > File: > /Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/numpy/lib/recfunctions.py > Definition: ? ? recfunctions.join_by(key, r1, r2, jointype='inner', > r1postfix='1', r2postfix='2', defaults=None, usemask=True, > asrecarray=False) > Docstring: > ? ?Join arrays `r1` and `r2` on key `key`. > > ? ?The key should be either a string or a sequence of string corresponding > ? ?to the fields used to join the array. > ? ?An exception is raised if the `key` field cannot be found in the two input > ? ?arrays. > ? ?Neither `r1` nor `r2` should have any duplicates along `key`: the presence > ? ?of duplicates will make the output quite unreliable. Note that duplicates > ? ?are not looked for by the algorithm. > > ? ?Parameters > ? ?---------- > ? ?key : {string, sequence} > ? ? ? ?A string or a sequence of strings corresponding to the fields used > ? ? ? ?for comparison. > ? ?r1, r2 : arrays > ? ? ? ?Structured arrays. > ? ?jointype : {'inner', 'outer', 'leftouter'}, optional > ? ? ? ?If 'inner', returns the elements common to both r1 and r2. > ? ? ? ?If 'outer', returns the common elements as well as the elements of r1 > ? ? ? ?not in r2 and the elements of not in r2. > ? ? ? ?If 'leftouter', returns the common elements and the elements of r1 not > ? ? ? ?in r2. > ? ?r1postfix : string, optional > ? ? ? ?String appended to the names of the fields of r1 that are present in r2 > ? ? ? ?but absent of the key. > ? ?r2postfix : string, optional > ? ? ? ?String appended to the names of the fields of r2 that are present in r1 > ? ? ? ?but absent of the key. > ? ?defaults : {dictionary}, optional > ? ? ? ?Dictionary mapping field names to the corresponding default values. > ? ?usemask : {True, False}, optional > ? ? ? ?Whether to return a MaskedArray (or MaskedRecords is `asrecarray==True`) > ? ? ? ?or a ndarray. > ? ?asrecarray : {False, True}, optional > ? ? ? ?Whether to return a recarray (or MaskedRecords if `usemask==True`) or > ? ? ? ?just a flexible-type ndarray. > > ? ?Notes > ? ?----- > ? ?* The output is sorted along the key. > ? ?* A temporary array is formed by dropping the fields not in the key for the > ? ? ?two arrays and concatenating the result. This array is then sorted, and > ? ? ?the common entries selected. The output is constructed by > filling the fields > ? ? ?with the selected entries. Matching is not preserved if there are some > ? ? ?duplicates... > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > It might also be worth your while to check out Keith Goodman's la (larry) library or my pandas library, which are both designed with relational data in mind. - Wes From faltet at pytables.org Thu Feb 3 06:33:08 2011 From: faltet at pytables.org (Francesc Alted) Date: Thu, 3 Feb 2011 12:33:08 +0100 Subject: [Numpy-discussion] create a numpy array of images In-Reply-To: <4D49908F.9080904@noaa.gov> References: <4D49908F.9080904@noaa.gov> Message-ID: <201102031233.08479.faltet@pytables.org> A Wednesday 02 February 2011 18:12:47 Christopher Barker escrigu?: > One other option, that I've never tried, is carray, which is an array > compressed in memory. Depending on your images, perhaps they would > compress a lot (or not ....): > > https://github.com/FrancescAlted/carray > http://mail.scipy.org/pipermail/numpy-discussion/2010-August/052378.h > tml Nice idea. In 0.3.1 release I've just implemented preliminary support for multidimensional data. So I was curious on the kind of compression that can be achieved on images: # Preliminaries: load numpy, matplotlib an carray libs >>> import numpy as np >>> import matplotlib.image as mpimg >>> import matplotlib.pyplot as plt >>> import carray as ca First I tried the classic Lenna (http://en.wikipedia.org/wiki/Lenna): >>> img = mpimg.imread('Lenna.png') >>> cimg = ca.carray(img) >>> cimg.nbytes/float(cimg.cbytes) 1.2450163377998429 So, just a 25% compression, not too much. But trying another example (http://matplotlib.sourceforge.net/_images/stinkbug.png) gives a significantly better ratio: >>> img2 = mpimg.imread('stinkbug.png') >>> cimg2 = ca.carray(img2) >>> cimg2.nbytes/float(cimg2.cbytes) 2.7716869102466184 And finally, the beautiful NumPy container drawing by St?fan van der Walt (slide 31 of his presentation in our latest advanced Python course, https://portal.g-node.org/python-autumnschool/materials/advanced_numpy): >>> img3 = mpimg.imread('numpy-container.png') >>> cimg3 = ca.carray(img3) >>> cimg3.nbytes/float(cimg3.cbytes) 3.7915321810785132 So, yeah, depending on the images, carray could be a nice way to keep them in-memory. And although, as I said, multidimensional support is still preliminary, matplotlib already understands carray beasts: # plotting image >>> imshow(cimg3) Cheers, -- Francesc Alted From mwwiebe at gmail.com Thu Feb 3 13:07:39 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Thu, 3 Feb 2011 10:07:39 -0800 Subject: [Numpy-discussion] Please keep Numpy master working on Py3 In-Reply-To: <4D48F51F.6090205@silveregg.co.jp> References: <4D48E345.4040301@silveregg.co.jp> <4D48F071.2000907@silveregg.co.jp> <4D48F51F.6090205@silveregg.co.jp> Message-ID: On Tue, Feb 1, 2011 at 10:09 PM, David wrote: > On 02/02/2011 02:57 PM, Mark Wiebe wrote: > > On Tue, Feb 1, 2011 at 9:49 PM, David > > wrote: > > > > > > > > > > > In the meantime, I put the relevant header in > > numpy/core/src/private, to > > > make the dependency clearer. > > > > Following that argument, there are other unwanted dependencies > between > > multiarray and ufunc, causing circular dependencies. I don't think > they > > were there before, and it makes building numpy with a > dependency-based > > tool like scons or waf extremely difficult. > > > > > > This particular separation of two components felt somewhat artificial to > > me. For the iterator buffering, as an example, I initially thought it > > would be good to use the same default as the ufuncs do. Instead I ended > > up just using a hard-coded default. > > Not sure I understand the exact argument, but if it is just a matter of > getting default values, it is easy to share them using a common header. > I guess my argument was just that a few more ufunc features need to migrate to the core. I don't think this needs to be done at the moment, though. > > > I think the error handling policy > > in the ufuncs could also be useful for computations in the core. > > At the moment, ufunc and multiarray are separate extensions. If each > depends on the other API-wise, it is an issue. If there are some > commonalities, then they can be put in a separate extensions. They are > already way too big as they are (historical reasons), we should really > fight this for maintanability. > > I realize the code organization and the build stuff is a bit of a mess > at the moment, so if you have questions on that aspect, feel free to ask > clarification or help, Yeah, I understand it's the result of organic growth and merging from many different sources. The core library should probably become layered in a manner roughly as follows, with each layer depending only on the previous APIs. This is what Travis was getting at, I believe, with the generator array idea affecting mainly the Storage and Iteration APIs, generalizing them so that new storage and iteration methods can be plugged in. Data Type API: data type numbers, casting, byte-swapping, etc. Array Storage API: array creation/manipulation/deletion. Array Iteration API: array iterators, copying routines. Array Calculation API: typedefs for various types of calculation functions, common calculation idioms, ufunc creation API, etc. Then, the ufunc module would use the Array Calculation API to implement all the ufuncs and other routines like inner, dot, trace, diag, tensordot, einsum, etc. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Feb 3 13:50:28 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 3 Feb 2011 11:50:28 -0700 Subject: [Numpy-discussion] Please keep Numpy master working on Py3 In-Reply-To: References: <4D48E345.4040301@silveregg.co.jp> <4D48F071.2000907@silveregg.co.jp> <4D48F51F.6090205@silveregg.co.jp> Message-ID: On Thu, Feb 3, 2011 at 11:07 AM, Mark Wiebe wrote: > On Tue, Feb 1, 2011 at 10:09 PM, David wrote: > >> On 02/02/2011 02:57 PM, Mark Wiebe wrote: >> > On Tue, Feb 1, 2011 at 9:49 PM, David > > > wrote: >> > >> > >> > >> > >> > > In the meantime, I put the relevant header in >> > numpy/core/src/private, to >> > > make the dependency clearer. >> > >> > Following that argument, there are other unwanted dependencies >> between >> > multiarray and ufunc, causing circular dependencies. I don't think >> they >> > were there before, and it makes building numpy with a >> dependency-based >> > tool like scons or waf extremely difficult. >> > >> > >> > This particular separation of two components felt somewhat artificial to >> > me. For the iterator buffering, as an example, I initially thought it >> > would be good to use the same default as the ufuncs do. Instead I ended >> > up just using a hard-coded default. >> >> Not sure I understand the exact argument, but if it is just a matter of >> getting default values, it is easy to share them using a common header. >> > > I guess my argument was just that a few more ufunc features need to migrate > to the core. I don't think this needs to be done at the moment, though. > > >> >> > I think the error handling policy >> > in the ufuncs could also be useful for computations in the core. >> >> At the moment, ufunc and multiarray are separate extensions. If each >> depends on the other API-wise, it is an issue. If there are some >> commonalities, then they can be put in a separate extensions. They are >> already way too big as they are (historical reasons), we should really >> fight this for maintanability. >> >> I realize the code organization and the build stuff is a bit of a mess >> at the moment, so if you have questions on that aspect, feel free to ask >> clarification or help, > > > Yeah, I understand it's the result of organic growth and merging from many > different sources. The core library should probably become layered in a > manner roughly as follows, with each layer depending only on the previous > APIs. This is what Travis was getting at, I believe, with the generator > array idea affecting mainly the Storage and Iteration APIs, generalizing > them so that new storage and iteration methods can be plugged in. > > Data Type API: data type numbers, casting, byte-swapping, etc. > Array Storage API: array creation/manipulation/deletion. > Array Iteration API: array iterators, copying routines. > Array Calculation API: typedefs for various types of calculation > functions, common calculation idioms, ufunc creation API, etc. > > Then, the ufunc module would use the Array Calculation API to implement all > the ufuncs and other routines like inner, dot, trace, diag, tensordot, > einsum, etc. > > I like the lower two levels if, as I assume, they are basically aimed at allocating, deallocating blocks of memory (or equivalent) and doing basic manipulations such as dealing with endianess and casting. Where do you see array methods making an appearance? The original Numeric only had three (IIRC) rather basic methods and everything else was function based, an approach which is probably easier to maintain. The extensive use of methods came from numarray and might be something that could be added at a higher level so that the current ndarrays would be objects combining ow level arrays and ufuncs. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Thu Feb 3 14:07:18 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Thu, 3 Feb 2011 11:07:18 -0800 Subject: [Numpy-discussion] core library structure Message-ID: Moving this to a new thread. On Thu, Feb 3, 2011 at 10:50 AM, Charles R Harris wrote: > > On Thu, Feb 3, 2011 at 11:07 AM, Mark Wiebe wrote: > >> On Tue, Feb 1, 2011 at 10:09 PM, David wrote: >> >>> On 02/02/2011 02:57 PM, Mark Wiebe wrote: >>> [snip] >>> >> >>> > I think the error handling policy >>> > in the ufuncs could also be useful for computations in the core. >>> >>> At the moment, ufunc and multiarray are separate extensions. If each >>> depends on the other API-wise, it is an issue. If there are some >>> commonalities, then they can be put in a separate extensions. They are >>> already way too big as they are (historical reasons), we should really >>> fight this for maintanability. >>> >>> I realize the code organization and the build stuff is a bit of a mess >>> at the moment, so if you have questions on that aspect, feel free to ask >>> clarification or help, >> >> >> Yeah, I understand it's the result of organic growth and merging from many >> different sources. The core library should probably become layered in a >> manner roughly as follows, with each layer depending only on the previous >> APIs. This is what Travis was getting at, I believe, with the generator >> array idea affecting mainly the Storage and Iteration APIs, generalizing >> them so that new storage and iteration methods can be plugged in. >> >> Data Type API: data type numbers, casting, byte-swapping, etc. >> Array Storage API: array creation/manipulation/deletion. >> Array Iteration API: array iterators, copying routines. >> Array Calculation API: typedefs for various types of calculation >> functions, common calculation idioms, ufunc creation API, etc. >> >> Then, the ufunc module would use the Array Calculation API to implement >> all the ufuncs and other routines like inner, dot, trace, diag, tensordot, >> einsum, etc. >> >> > I like the lower two levels if, as I assume, they are basically aimed at > allocating, deallocating blocks of memory (or equivalent) and doing basic > manipulations such as dealing with endianess and casting. Where do you see > array methods making an appearance? > That's correct. Currently, for example, the cast functions take array objects as parameters, something that would no longer be the case. The array methods vs functions only shows up in the Python exposure, I believe. The above structure only affects the C library, and how its exposed to Python could remain as it is now. The original Numeric only had three (IIRC) rather basic methods and > everything else was function based, an approach which is probably easier to > maintain. The extensive use of methods came from numarray and might be > something that could be added at a higher level so that the current ndarrays > would be objects combining ow level arrays and ufuncs. > Sounds reasonable to me. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Thu Feb 3 16:50:50 2011 From: oliphant at enthought.com (Travis Oliphant) Date: Thu, 3 Feb 2011 15:50:50 -0600 Subject: [Numpy-discussion] core library structure In-Reply-To: References: Message-ID: <37FAEB91-3739-40AC-AB46-B69ECC09E243@enthought.com> I like the thoughts on core-architecture. These are all things that we were not able to do as part of the NumPy for .NET discussions, but with the right interested parties could be acted upon. > > I like the lower two levels if, as I assume, they are basically aimed at allocating, deallocating blocks of memory (or equivalent) and doing basic manipulations such as dealing with endianess and casting. Where do you see array methods making an appearance? > > That's correct. Currently, for example, the cast functions take array objects as parameters, something that would no longer be the case. The array methods vs functions only shows up in the Python exposure, I believe. The above structure only affects the C library, and how its exposed to Python could remain as it is now. Right now, the reason the array objects are passed to the cast functions is because variable-sized data-types need the elsize information that comes from the array structure. This could be handled differently, of course, but the low-levels will need some way of passing variable information like this. For some data-types you know the size exactly, for others it's a parameter. > > The original Numeric only had three (IIRC) rather basic methods and everything else was function based, an approach which is probably easier to maintain. The extensive use of methods came from numarray and might be something that could be added at a higher level so that the current ndarrays would be objects combining ow level arrays and ufuncs. > > Sounds reasonable to me. I agree that the low-level objects should use a functional approach with very few methods. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsdale24 at gmail.com Thu Feb 3 17:35:11 2011 From: dsdale24 at gmail.com (Darren Dale) Date: Thu, 3 Feb 2011 17:35:11 -0500 Subject: [Numpy-discussion] core library structure In-Reply-To: References: Message-ID: On Thu, Feb 3, 2011 at 2:07 PM, Mark Wiebe wrote: > Moving this to a new thread. > On Thu, Feb 3, 2011 at 10:50 AM, Charles R > Harris??wrote: >> >> On Thu, Feb 3, 2011 at 11:07 AM, Mark Wiebe??wrote: [...] >>> Yeah, I understand it's the result of organic growth and merging from >>> many different sources. The core library should probably become layered in a >>> manner roughly as follows, with each layer depending only on the previous >>> APIs. ?This is what Travis was getting at, I believe, with the generator >>> array idea affecting mainly the Storage and Iteration APIs, generalizing >>> them so that new storage and iteration methods can be plugged in. >>> Data Type API: data type numbers, casting, byte-swapping, etc. >>> Array Storage API: array creation/manipulation/deletion. >>> Array Iteration API: array iterators, copying routines. >>> Array Calculation API: typedefs for various types of calculation >>> functions,?common calculation idioms,?ufunc creation API, etc. >>> Then, the ufunc module would use the Array Calculation API to implement >>> all the ufuncs and other routines like inner, dot, trace, diag, tensordot, >>> einsum, etc. >> >> I like the lower two levels if, as I assume, they are basically aimed at >> allocating, deallocating blocks of memory (or equivalent) and doing basic >> manipulations such as dealing with endianess and casting. Where do you see >> array methods making an appearance? > > That's correct. Currently, for example, the cast functions take array > objects as parameters, something that would no longer be the case. ?The > array methods vs functions only shows up in the Python exposure, I believe. > ?The above structure only affects the C library, and how its exposed to > Python could remain as it is now. >> >> The original Numeric only had three (IIRC) rather basic methods and >> everything else was function based, an approach which is probably easier to >> maintain. The extensive use of methods came from numarray and might be >> something that could be added at a higher level so that the current ndarrays >> would be objects combining ow level arrays and ufuncs. Concerning ufuncs: I wonder if we could consider generic functions as a replacement for the current __array_prepare__/__array_wrap__ mechanism. For example, if I have an ndarray, a masked array, and quantity, and I want to multiply the three together, it would be great to be able to do so with two calls to a single mul ufunc. Also, a generic function approach might provide a better mechanism to allow changes to the arrays on their way into the ufunc: import numpy as np import quantities as pq a = [1,2,3]*pq.deg # yields a subclass of ndarray np.sin(a) This is not currently possible with __array_prepare__/__array_wrap__, because __array_prepare__ is called too late by the ufunc to return process the input array and rescale it to radians. I suggested on this list that it might be possible to so with the addition of a *third* method, call it __input_prepare__... at which point Chuck rightly complained that things were getting way out of hand. Imagine if we could do something like @np.sin.register(pq.Quantity) def my_sin(q): return np.sin.default(q.rescale('radians')) np.sin(a) With generic functions, you wouldn't have to remember to use the ufunc provided by masked array for one type, or the default numpy for another type. This is something I have been meaning to revisit on the list for a while (along with the possibility of merging quantities into numpy), but keep forgetting to do so. Darren From david at silveregg.co.jp Thu Feb 3 20:56:20 2011 From: david at silveregg.co.jp (David) Date: Fri, 04 Feb 2011 10:56:20 +0900 Subject: [Numpy-discussion] core library structure In-Reply-To: <37FAEB91-3739-40AC-AB46-B69ECC09E243@enthought.com> References: <37FAEB91-3739-40AC-AB46-B69ECC09E243@enthought.com> Message-ID: <4D4B5CC4.7050905@silveregg.co.jp> On 02/04/2011 06:50 AM, Travis Oliphant wrote: > I like the thoughts on core-architecture. These are all things that we > were not able to do as part of the NumPy for .NET discussions, but with > the right interested parties could be acted upon. I will be at Pycon this year from 11th to 13th february, the whole 3 conf days. So if anyone else from the team can make it, we can discuss this further face to face. As for the changes being discussed, I think it would be beneficial to have a strategy on how to deal with new features and the release roadmap. I am a bit worried about feature creap and a too ambitious numpy 2.0 release. Can we prepare a 2.0 with refactoring as a good basis for ongoing development, how does the .net "fork" fits into this picture, etc... ? Like everyone, I am excited by those discussions about new feature, design, but I would suggest that we get a basic agreement on the main goals of numpy 2.0, what can be delayed, what cannot. cheers, David From jhtu at princeton.edu Fri Feb 4 09:13:29 2011 From: jhtu at princeton.edu (Jonathan Tu) Date: Fri, 4 Feb 2011 09:13:29 -0500 Subject: [Numpy-discussion] Complex matrix seg fault Message-ID: <1D7E99C8-E5FA-4BBC-AFC2-F2A99B13D245@princeton.edu> Hi, When I run the following code, I get a seg fault on certain cluster nodes, but not others: >>> import numpy as N >>> A = N.random.rand(262144,10)+1j*N.random.rand(262144,10) >>> x = N.random.rand(262144,) >>> d, res, rank, s = N.linalg.lstsq( A, x ) My numpy is configured as such: lapack_opt_info: libraries = ['mkl_lapack', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] library_dirs = ['/home/jhtu/local/lib/mkl_links'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] blas_opt_info: libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] library_dirs = ['/home/jhtu/local/lib/mkl_links'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] lapack_mkl_info: libraries = ['mkl_lapack', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] library_dirs = ['/home/jhtu/local/lib/mkl_links'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] blas_mkl_info: libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] library_dirs = ['/home/jhtu/local/lib/mkl_links'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] mkl_info: libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] library_dirs = ['/home/jhtu/local/lib/mkl_links'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] All unit tests pass except the following, but I was told (by this mailing list) that this shouldn't be an issue: ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.7/site-packages/numpy/testing/decorators.py", line 215, in knownfailer return f(*args, **kwargs) File "/home/jhtu/local/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 312, in test_special_values assert_almost_equal(np.log(np.conj(xa[i])), np.conj(np.log(xa[i]))) File "/home/jhtu/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 443, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: array([-inf+3.14159265j]) DESIRED: array([-inf-3.14159265j]) ---------------------------------------------------------------------- Ran 2648 tests in 15.852s FAILED (KNOWNFAIL=4, SKIP=1, failures=1) Anyone have ideas why this would fail on some nodes but not others? (I mean the code snippet fails on some nodes. The above unit test always fails.) Thanks, Jonathan Tu From dagss at student.matnat.uio.no Fri Feb 4 09:32:43 2011 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 04 Feb 2011 15:32:43 +0100 Subject: [Numpy-discussion] Complex matrix seg fault In-Reply-To: <1D7E99C8-E5FA-4BBC-AFC2-F2A99B13D245@princeton.edu> References: <1D7E99C8-E5FA-4BBC-AFC2-F2A99B13D245@princeton.edu> Message-ID: <4D4C0E0B.5060707@student.matnat.uio.no> On 02/04/2011 03:13 PM, Jonathan Tu wrote: > Hi, > > When I run the following code, I get a seg fault on certain cluster nodes, but not others: > Is it a homogenous cluster or not? Try to look for differences in /proc/cpuinfo between crashing and non-crashing nodes, in particular in the flags. Dag > >>>> import numpy as N >>>> A = N.random.rand(262144,10)+1j*N.random.rand(262144,10) >>>> x = N.random.rand(262144,) >>>> d, res, rank, s = N.linalg.lstsq( A, x ) >>>> > My numpy is configured as such: > > lapack_opt_info: > libraries = ['mkl_lapack', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] > library_dirs = ['/home/jhtu/local/lib/mkl_links'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] > blas_opt_info: > libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] > library_dirs = ['/home/jhtu/local/lib/mkl_links'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] > lapack_mkl_info: > libraries = ['mkl_lapack', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] > library_dirs = ['/home/jhtu/local/lib/mkl_links'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] > blas_mkl_info: > libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] > library_dirs = ['/home/jhtu/local/lib/mkl_links'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] > mkl_info: > libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_def', 'mkl_mc', 'iomp5', 'pthread'] > library_dirs = ['/home/jhtu/local/lib/mkl_links'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.1/072/mkl/include'] > > All unit tests pass except the following, but I was told (by this mailing list) that this shouldn't be an issue: > > ====================================================================== > FAIL: test_special_values (test_umath_complex.TestClog) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jhtu/local/lib/python2.7/site-packages/numpy/testing/decorators.py", line 215, in knownfailer > return f(*args, **kwargs) > File "/home/jhtu/local/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 312, in test_special_values > assert_almost_equal(np.log(np.conj(xa[i])), np.conj(np.log(xa[i]))) > File "/home/jhtu/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 443, in assert_almost_equal > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal > ACTUAL: array([-inf+3.14159265j]) > DESIRED: array([-inf-3.14159265j]) > > ---------------------------------------------------------------------- > Ran 2648 tests in 15.852s > > FAILED (KNOWNFAIL=4, SKIP=1, failures=1) > > > Anyone have ideas why this would fail on some nodes but not others? (I mean the code snippet fails on some nodes. The above unit test always fails.) > > > > > > > Thanks, > > > > Jonathan Tu > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From xscript at gmx.net Fri Feb 4 14:23:42 2011 From: xscript at gmx.net (=?utf-8?Q?Llu=C3=ADs?=) Date: Fri, 04 Feb 2011 20:23:42 +0100 Subject: [Numpy-discussion] core library structure In-Reply-To: (Darren Dale's message of "Thu, 3 Feb 2011 17:35:11 -0500") References: Message-ID: <87y65vr4ch.fsf@ginnungagap.bsc.es> Darren Dale writes: > With generic functions, you wouldn't have to remember to use the ufunc > provided by masked array for one type, or the default numpy for > another type. Sorry, but I don't see how generic functions should be a better approach compared to redefining methods on masked_array [1]. In both cases you have to define them one-by-one. [1] assuming 'np.foo' and 'ma.foo' (which would now be obsolete) simply call 'instance.foo', which in the ndarray level is the 'foo' ufunc object. Well, yes. I do see an advantage, if the number of methods grows too large (which is probably what started your concerns). Where I do see a utility for generic functions is as cross-cutting points for class-oblivious extensions, like deferred evaluation (although then you lose concise and per-object control of when to enable/disable deferred evaluation). But then you need "implicitly chained" implementations for the generic functions (which might lead to poor performance, I just don't know). Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From dsdale24 at gmail.com Fri Feb 4 14:46:01 2011 From: dsdale24 at gmail.com (Darren Dale) Date: Fri, 4 Feb 2011 14:46:01 -0500 Subject: [Numpy-discussion] core library structure In-Reply-To: <87y65vr4ch.fsf@ginnungagap.bsc.es> References: <87y65vr4ch.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Feb 4, 2011 at 2:23 PM, Llu?s wrote: > Darren Dale writes: > >> With generic functions, you wouldn't have to remember to use the ufunc >> provided by masked array for one type, or the default numpy for >> another type. > > Sorry, but I don't see how generic functions should be a better approach > compared to redefining methods on masked_array [1]. In both cases you > have to define them one-by-one. > > [1] assuming 'np.foo' and 'ma.foo' (which would now be obsolete) simply > ? ?call 'instance.foo', which in the ndarray level is the 'foo' ufunc > ? ?object. That's a bad assumption. np.ndarray.__add__ actually calls the np.add ufunc, not the other way around. For example, np.arcsin is a ufunc that operates on ndarray, yet there is no ndarray.arcsin method. Darren From josef.pktd at gmail.com Sun Feb 6 09:44:50 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 6 Feb 2011 09:44:50 -0500 Subject: [Numpy-discussion] histogram function return type Message-ID: there is some inconsistency in the return type of the histogram function, 1d returns int, others return float >>> np.histogramdd([y, x], bins=3)[0] array([[ 3., 15., 3.], [ 10., 2., 9.], [ 4., 0., 5.]]) >>> np.histogram2d(y, x, bins=3)[0] array([[ 3., 15., 3.], [ 10., 2., 9.], [ 4., 0., 5.]]) >>> np.histogram(y, bins=3)[0] array([21, 21, 9]) Josef From fredmfp at gmail.com Mon Feb 7 07:21:15 2011 From: fredmfp at gmail.com (Fred) Date: Mon, 07 Feb 2011 13:21:15 +0100 Subject: [Numpy-discussion] 3D array -> 2D array issue... Message-ID: <4D4FE3BB.90405@gmail.com> Hi all, Let's say C1 is a 3D array, and q0 and k are 2D array. dim C1 = nx*ny*nz dim q0 = nx*ny = dim k I have to do the following: q0[0, 0] = C1[0, 0, k[0, 0]] q0[1, 1] = C1[1, 1, k[1, 1]] ... q0[i, j] = C1[i, j, k[i, j]] ... I tried q0 = C1[:, :, k] but this obviously does not work. How could I do this ala NumPy? TIA Cheers, -- Fred From josef.pktd at gmail.com Mon Feb 7 07:37:01 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 7 Feb 2011 07:37:01 -0500 Subject: [Numpy-discussion] 3D array -> 2D array issue... In-Reply-To: <4D4FE3BB.90405@gmail.com> References: <4D4FE3BB.90405@gmail.com> Message-ID: On Mon, Feb 7, 2011 at 7:21 AM, Fred wrote: > Hi all, > > Let's say C1 is a 3D array, > and q0 and k are 2D array. > > dim C1 = nx*ny*nz > > dim q0 = nx*ny = dim k > > I have to do the following: > > q0[0, 0] = C1[0, 0, k[0, 0]] > q0[1, 1] = C1[1, 1, k[1, 1]] > ... > q0[i, j] = C1[i, j, k[i, j]] > ... > > I tried > > q0 = C1[:, :, k] > > but this obviously does not work. you need to build or broadcast the indices, something like this (written, not tried out) n0, n1 = k.shape ind0 = np.arange(n0)[:,None] ind1 = np.arange(n1) q0 = C1[ind0,ind1, k[ind0,ind1]] or better q0 = C1[ind0,ind1, k] Josef > > > How could I do this ala NumPy? > > TIA > > Cheers, > > -- > Fred > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From fredmfp at gmail.com Mon Feb 7 08:19:25 2011 From: fredmfp at gmail.com (Fred) Date: Mon, 07 Feb 2011 14:19:25 +0100 Subject: [Numpy-discussion] 3D array -> 2D array issue... In-Reply-To: References: <4D4FE3BB.90405@gmail.com> Message-ID: <4D4FF15D.3040004@gmail.com> Le 07/02/2011 13:37, josef.pktd at gmail.com a ?crit : > n0, n1 = k.shape > ind0 = np.arange(n0)[:,None] > ind1 = np.arange(n1) > q0 = C1[ind0,ind1, k[ind0,ind1]] > > or better > q0 = C1[ind0,ind1, k] Great!!! Thanks a lot. Cheers, -- Fred From dagss at student.matnat.uio.no Mon Feb 7 13:39:12 2011 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 07 Feb 2011 19:39:12 +0100 Subject: [Numpy-discussion] [ANN] Invitation for Cython workshop Message-ID: <4D503C50.6030408@student.matnat.uio.no> Like the Cython project? Want to get your hands dirty with it? A great way to do so is to participate in the Cython workshop, which takes place in Munich, March 30 -- April 3. We are renting a house to live and work in, and will also give some talks at a local university, LMU Munich: http://wiki.cython.org/workshop1/OpenCythonDay Funding for travel and lodging can be arranged. Even if you don't feel up to writing any code, please consider whether you want to come and help out with our documentation sprint. It would mean a lot to the Cython project, and also teach you all the ins and outs of Cython. Also, if you are a student who plans to apply for a Google Summer of Code project with Cython, this is a great place to be. To apply for consideration, please reply with your CV and a short email telling us why you want to participate. -- Dag Sverre Seljebotn From mcenerney1 at llnl.gov Mon Feb 7 13:45:00 2011 From: mcenerney1 at llnl.gov (McEnerney, James F.) Date: Mon, 7 Feb 2011 10:45:00 -0800 Subject: [Numpy-discussion] f2py on mac Message-ID: On my mac (10.6.5) I'm running python2.6 with numpy 1.5.0. While using f2py with f2py -c --fcompiler=gnu95 --f77flags='-m32 -O1 -fno-second-underscore -fbounds-check' src/mars.pyf src/mars.f It compiles but has link warnings. -m32 is not being passed to gcc. ld: warning: in /var/folders/k-/k-SUt4dLGyWuiaA0fiYLOU++4GM/-Tmp-/tmp8xkKpp/var/folders/k-/k-SUt4dLGyWuiaA0fiYLOU++4GM/-Tmp-/tmp8xkKpp/src.macosx-10.6-i386-2.6/src/marsmodule.o, file was built for unsupported file format which is not the architecture being linked (i386) ld: warning: in /var/folders/k-/k-SUt4dLGyWuiaA0fiYLOU++4GM/-Tmp-/tmp8xkKpp/var/folders/k-/k-SUt4dLGyWuiaA0fiYLOU++4GM/-Tmp-/tmp8xkKpp/src.macosx-10.6-i386-2.6/fortranobject.o, file was built for unsupported file format which is not the architecture being linked (i386) Any help would be appreciated!!! Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilya_shl at alum.mit.edu Mon Feb 7 13:52:45 2011 From: ilya_shl at alum.mit.edu (Ilya Shlyakhter) Date: Mon, 7 Feb 2011 13:52:45 -0500 Subject: [Numpy-discussion] moving C++ binary data to numpy Message-ID: Is there a portable way to save vector of C++ structs to a binary file, and then load it as a numpy recarray? thanks, ilya From david at silveregg.co.jp Mon Feb 7 19:30:17 2011 From: david at silveregg.co.jp (David) Date: Tue, 08 Feb 2011 09:30:17 +0900 Subject: [Numpy-discussion] moving C++ binary data to numpy In-Reply-To: References: Message-ID: <4D508E99.7030304@silveregg.co.jp> On 02/08/2011 03:52 AM, Ilya Shlyakhter wrote: > Is there a portable way to save vector of C++ structs to a binary > file C++ structs are not portable, so that sounds difficult. In practice, you have compiler-specific ways to enforce some alignement within a structure, but that sounds rather nightmarish to support. Why can't you use a library with a portable file format (e.g. hdf5), or something like google protocol buffer/thrift/etc... ? Those are designed to solve your problem cheers, David From Thomas.EMMEL at 3ds.com Tue Feb 8 10:24:10 2011 From: Thomas.EMMEL at 3ds.com (EMMEL Thomas) Date: Tue, 8 Feb 2011 15:24:10 +0000 Subject: [Numpy-discussion] Need a good idea: calculate the mean of many vectors Message-ID: <3A0080EEBFB19C4993C24098DD0A78D1226DB76E@EU-DCC-MBX01.dsone.3ds.com> Hi, here is something I am thinking about for some time and I am wondering whether there is a better solution within numpy. The task is: I have an array (300000+ entries) with arrays each with length == 3, that is initially empty like this: n = 100 # for test otherwise ~300000 a1 = reshape(zeros(3*n).astype(float), (n,3)) (Speaking literally this is a field of displacements in a Finite-Element-Mesh) Now I have a lot of triangles where the corners are the nodes, each with an index between 0 and n-1 and I like to add a unique displacement for all three nodes to a1 like this a2 = zeros(n).astype(int) for indices, data in [...]: #data = array((1.,2.,3.)) #indices = (1,5,60) for index in indices: a1[index] += data a2[index] += 1 Now after filling a1 and a2 over and over (for a lot of triangles) I can finally calculate the averaged displacement on all points by this meand = a1/reshape(a2,(n,1)) I am doing this in the old numeric package (and can do it in numpy). I wonder whether there is a better way to do that in numpy. Numpy is in this actual case 20times slower than Numeric and I know that is due to the fact that I need to change only 3 values of the big array every loop. The basic problem is: How can I add very few values to a big array with a good performance? I thought about this: m = zeros(n) put(m, indices,1.) # only 3 ones in a long list of zeros! a1 += outer(m, data) a2 += m which is in fact very slow due to the function outer. Any help appriciated Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Feb 8 11:32:46 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 8 Feb 2011 10:32:46 -0600 Subject: [Numpy-discussion] Need a good idea: calculate the mean of many vectors In-Reply-To: <3A0080EEBFB19C4993C24098DD0A78D1226DB76E@EU-DCC-MBX01.dsone.3ds.com> References: <3A0080EEBFB19C4993C24098DD0A78D1226DB76E@EU-DCC-MBX01.dsone.3ds.com> Message-ID: On Tue, Feb 8, 2011 at 09:24, EMMEL Thomas wrote: > Hi, > > here is something I am thinking about for some time and I am wondering > whether there is a better solution > within numpy. > > The task is: > I have an array (300000+ entries) with arrays each with length == 3, that is > initially empty like this: > > n = 100 # for test otherwise ~300000 > a1 = reshape(zeros(3*n).astype(float), (n,3)) > > (Speaking literally this is a field of displacements in a > Finite-Element-Mesh) > Now I have a lot of triangles where the corners are the nodes, each with an > index between 0 and n-1 > and I like to add a unique displacement for all three nodes to a1 like this > > a2 = zeros(n).astype(int) > > for indices, data in [...]: > ??? #data = array((1.,2.,3.)) > ??? #indices = (1,5,60) > ??? for index in indices: > ??? ??? a1[index] += data > ??????? a2[index] += 1 > > Now after filling a1 and a2 over and over (for a lot of triangles) I can > finally calculate the > averaged displacement on all points by this > > meand = a1/reshape(a2,(n,1)) Use np.bincount() on each coordinate. [~] |10> xyz = np.random.random_sample([10, 3]) [~] |11> ind = np.array([1,3,4,3,4,5,0,0,1,2]) [~] |12> sum_xyz = np.column_stack([np.bincount(ind, xyz[:,i]) for i in range(xyz.shape[1])]) [~] |13> sum_xyz array([[ 1.56632447, 0.88193741, 0.20253585], [ 1.36661663, 0.98698521, 0.79892009], [ 0.26787528, 0.12850502, 0.76042557], [ 1.08489219, 0.70099349, 1.56665748], [ 0.56662843, 1.3502819 , 0.15531993], [ 0.34900915, 0.34282216, 0.48250042]]) [~] |14> counts = np.bincount(ind) [~] |15> counts array([2, 2, 1, 2, 2, 1]) [~] |17> mean_xyz = sum_xyz / counts[:,np.newaxis] [~] |18> mean_xyz array([[ 0.78316224, 0.4409687 , 0.10126793], [ 0.68330832, 0.4934926 , 0.39946005], [ 0.26787528, 0.12850502, 0.76042557], [ 0.54244609, 0.35049674, 0.78332874], [ 0.28331422, 0.67514095, 0.07765996], [ 0.34900915, 0.34282216, 0.48250042]]) Watch out for divisions by 0 when you calculate the mean. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pav at iki.fi Tue Feb 8 11:33:42 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 8 Feb 2011 16:33:42 +0000 (UTC) Subject: [Numpy-discussion] Need a good idea: calculate the mean of many vectors References: <3A0080EEBFB19C4993C24098DD0A78D1226DB76E@EU-DCC-MBX01.dsone.3ds.com> Message-ID: Tue, 08 Feb 2011 15:24:10 +0000, EMMEL Thomas wrote: [clip] > n = 100 # for test otherwise ~300000 > a1 = reshape(zeros(3*n).astype(float), (n,3)) > > a2 = zeros(n).astype(int) > > for indices, data in [...]: > #data = array((1.,2.,3.)) > #indices = (1,5,60) > for index in indices: > a1[index] += data > a2[index] += 1 You can (mis-)use `bincount` to vectorize summations. Bincount does not support broadcasting and takes only 1-D inputs, so some manual shape manipulation is necessary. Something probably could/should be done to optimize Numpy's performance for small arrays. ---- import numpy as np n = 100 # 100 nodes m = 1000 # 1000 triangles # synthetic data tri_nodes = np.random.randint(0, n, size=(m, 3)) tri_data = np.random.rand(m, 3) # vectorize sums via bincount a1 = np.zeros((n,3), float) a2 = np.bincount(tri_nodes.ravel()) for j in range(3): # repeat(..., 3) -> the same (3,) data vector is added to each # node in the triangle a1[:,j] = np.bincount(tri_nodes.ravel(), np.repeat(tri_data[:,j], 3)) From bgamari.foss at gmail.com Tue Feb 8 11:44:42 2011 From: bgamari.foss at gmail.com (Ben Gamari) Date: Tue, 08 Feb 2011 11:44:42 -0500 Subject: [Numpy-discussion] Computing the norm of an array of vectors Message-ID: <87r5bicw79.fsf@gmail.com> I have an array of (say, row) vectors, v = [ [ a1, a2, a3 ], [ b1, b2, b3 ], [ c1, c2, c3 ], ... ] What is the optimal way to compute the norm of each vector, norm(v)**2 = [ [ a1**2 + a2**2 + a3**2 ], [ b1**2 + b2**2 + b3**2 ], ... ] It seems clear that numpy.norm does not handle the multidimensional case as needed. The best thing I can think of is to do something like, sum(v**2, axis=0) * ones(3) but surely there must be a better way. Any ideas? Cheers, - Ben From robert.kern at gmail.com Tue Feb 8 11:46:34 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 8 Feb 2011 10:46:34 -0600 Subject: [Numpy-discussion] Computing the norm of an array of vectors In-Reply-To: <87r5bicw79.fsf@gmail.com> References: <87r5bicw79.fsf@gmail.com> Message-ID: On Tue, Feb 8, 2011 at 10:44, Ben Gamari wrote: > I have an array of (say, row) vectors, > > ?v = [ [ a1, a2, a3 ], > ? ? ? ?[ b1, b2, b3 ], > ? ? ? ?[ c1, c2, c3 ], > ? ? ? ?... > ? ? ?] > > What is the optimal way to compute the norm of each vector, > ?norm(v)**2 = [ > ? ? ?[ a1**2 + a2**2 + a3**2 ], > ? ? ?[ b1**2 + b2**2 + b3**2 ], > ? ? ?... > ? ?] > > It seems clear that numpy.norm does not handle the multidimensional case > as needed. The best thing I can think of is to do something like, > > ?sum(v**2, axis=0) * ones(3) > > but surely there must be a better way. Any ideas? (v*v).sum(axis=1)[:,np.newaxis] You can leave off the newaxis bit if you don't really need a column vector. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From bgamari.foss at gmail.com Tue Feb 8 12:55:48 2011 From: bgamari.foss at gmail.com (Ben Gamari) Date: Tue, 08 Feb 2011 12:55:48 -0500 Subject: [Numpy-discussion] Computing the norm of an array of vectors In-Reply-To: References: <87r5bicw79.fsf@gmail.com> Message-ID: <87aai6o1gb.fsf@gmail.com> On Tue, 8 Feb 2011 10:46:34 -0600, Robert Kern wrote: > (v*v).sum(axis=1)[:,np.newaxis] > > You can leave off the newaxis bit if you don't really need a column vector. > Fair enough, I unfortunately neglected to mention that I ultimately want to normalize these vectors, hence the *ones(3) in my original proposal (although looking back, the shapes would clearly be incompatible). Is there an elegant way to achieve this? Thanks again, - Ben From robert.kern at gmail.com Tue Feb 8 12:57:30 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 8 Feb 2011 11:57:30 -0600 Subject: [Numpy-discussion] Computing the norm of an array of vectors In-Reply-To: <87aai6o1gb.fsf@gmail.com> References: <87r5bicw79.fsf@gmail.com> <87aai6o1gb.fsf@gmail.com> Message-ID: On Tue, Feb 8, 2011 at 11:55, Ben Gamari wrote: > On Tue, 8 Feb 2011 10:46:34 -0600, Robert Kern wrote: >> (v*v).sum(axis=1)[:,np.newaxis] >> >> You can leave off the newaxis bit if you don't really need a column vector. >> > Fair enough, I unfortunately neglected to mention that I ultimately want > to normalize these vectors, hence the *ones(3) in my original proposal > (although looking back, the shapes would clearly be incompatible). Is > there an elegant way to achieve this? v / np.hypot.reduce(v, axis=1)[:,np.newaxis] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From a.h.jaffe at gmail.com Tue Feb 8 13:06:48 2011 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Tue, 08 Feb 2011 18:06:48 +0000 Subject: [Numpy-discussion] Computing the norm of an array of vectors In-Reply-To: <87r5bicw79.fsf@gmail.com> References: <87r5bicw79.fsf@gmail.com> Message-ID: On 08/02/2011 16:44, Ben Gamari wrote: > I have an array of (say, row) vectors, > > v = [ [ a1, a2, a3 ], > [ b1, b2, b3 ], > [ c1, c2, c3 ], > ... > ] > > What is the optimal way to compute the norm of each vector, > norm(v)**2 = [ > [ a1**2 + a2**2 + a3**2 ], > [ b1**2 + b2**2 + b3**2 ], > ... > ] > > It seems clear that numpy.norm does not handle the multidimensional case > as needed. The best thing I can think of is to do something like, > > sum(v**2, axis=0) * ones(3) For this shape=(N,3) vector, this is not what you mean: as Robert Kern also has it you want axis=1, which produces a shape=(N,) (or the [:,newaxis] version which produces shape=(N,1). But what is the point of the ones(3)? I think you intend to make a new (N,3) array where each row duplicates the norm, so that you can then divide out the norms. But through the magic of broadcasting, that's not necessary: v/np.sqrt(sum(v**2, axis=1)[:,newaxis]) does what you want. Andrew From bgamari.foss at gmail.com Tue Feb 8 13:19:57 2011 From: bgamari.foss at gmail.com (Ben Gamari) Date: Tue, 08 Feb 2011 13:19:57 -0500 Subject: [Numpy-discussion] Computing the norm of an array of vectors In-Reply-To: References: <87r5bicw79.fsf@gmail.com> Message-ID: <87mxm6bd82.fsf@gmail.com> On Tue, 08 Feb 2011 18:06:48 +0000, Andrew Jaffe wrote: > For this shape=(N,3) vector, this is not what you mean: as Robert Kern > also has it you want axis=1, which produces a shape=(N,) (or the > [:,newaxis] version which produces shape=(N,1). > > But what is the point of the ones(3)? I think you intend to make a new > (N,3) array where each row duplicates the norm, so that you can then > divide out the norms. But through the magic of broadcasting, that's not > necessary: > > v/np.sqrt(sum(v**2, axis=1)[:,newaxis]) > > does what you want. > Thanks! I've since realized the error in my ways. I had completely forgotten that newaxis existed. Robert Kern's proposal is perfect. - Ben From charlesr.harris at gmail.com Tue Feb 8 13:35:12 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Feb 2011 11:35:12 -0700 Subject: [Numpy-discussion] Permission to close ticket for Mark Wiebe Message-ID: I don't know who handles these permissions, I didn't see a way to do it myself. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Feb 8 13:49:21 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 8 Feb 2011 18:49:21 +0000 (UTC) Subject: [Numpy-discussion] Permission to close ticket for Mark Wiebe References: Message-ID: Tue, 08 Feb 2011 11:35:12 -0700, Charles R Harris wrote: > Permission to close ticket for Mark Wiebe > > I don't know who handles these permissions, I didn't see a way to do it > myself. Granted. (You'd need a shell account to do these changes.) -- Pauli Virtanen From alok at merfinllc.com Tue Feb 8 17:54:35 2011 From: alok at merfinllc.com (Alok Singhal) Date: Tue, 8 Feb 2011 14:54:35 -0800 Subject: [Numpy-discussion] Indexing a 2-d array with a 1-d mask Message-ID: Hi, I have an NxM array, which I am indexing with a 1-d, length N boolean array. For example, with a 3x5 array: In [1]: import numpy In [2]: data = numpy.arange(15) In [3]: data.shape = 3, 5 Now, I want to select rows 0 and 2, so I can do: In [4]: mask = numpy.array([True, False, True]) In [5]: data[mask] Out[5]: array([[ 0, 1, 2, 3, 4], [10, 11, 12, 13, 14]]) But when the shape of 'data' is a 0xM, this indexing fails: In [6]: data2 = numpy.zeros((0, 5), 'd') In [7]: mask2 = numpy.zeros(0, 'bool') In [8]: data2[mask2] ------------------------------------------------------------ Traceback (most recent call last): File "", line 1, in IndexError: invalid index I would have expected the above to give me a 0x5 array. Of course, I can check on "len(data)" and not use the above indexing when it is zero, but I am hoping that I don't need to special case the boundary condition and have numpy fancy indexing do the "right thing" always. Is this a bug in numpy? Is there any other way to do what I am doing? Here is my numpy setup (numpy installed from the git repository): In [1]: import numpy In [2]: numpy.__version__ Out[2]: '1.6.0.dev-13c83fd' In [3]: numpy.show_config() blas_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 lapack_info: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 atlas_threads_info: NOT AVAILABLE blas_opt_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_blas_threads_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE mkl_info: NOT AVAILABLE In [4]: import sys In [5]: print sys.version 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) [GCC 4.4.3] Thanks! From friedrichromstedt at gmail.com Wed Feb 9 04:24:16 2011 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Wed, 9 Feb 2011 10:24:16 +0100 Subject: [Numpy-discussion] Indexing a 2-d array with a 1-d mask In-Reply-To: References: Message-ID: Hi, 2011/2/8 Alok Singhal : > Hi, > > I have an NxM array, which I am indexing with a 1-d, length N boolean > array. ?For example, with a 3x5 array: > > In [1]: import numpy > In [2]: data = numpy.arange(15) > In [3]: data.shape = 3, 5 > > Now, I want to select rows 0 and 2, so I can do: > > In [4]: mask = numpy.array([True, False, True]) > In [5]: data[mask] > Out[5]: > array([[ 0, ?1, ?2, ?3, ?4], > ? ? ?[10, 11, 12, 13, 14]]) > > But when the shape of 'data' is a 0xM, this indexing fails: > > In [6]: data2 = numpy.zeros((0, 5), 'd') > In [7]: mask2 = numpy.zeros(0, 'bool') > In [8]: data2[mask2] > ------------------------------------------------------------ > Traceback (most recent call last): > ?File "", line 1, in > IndexError: invalid index > > I would have expected the above to give me a 0x5 array. > Is there any other way to do what I > am doing? Like so (works only for ndim==2): >>> d1 = numpy.arange(0).reshape((0, 5)) >>> d2 = numpy.arange(15).reshape((3, 5)) >>> index1 = numpy.asarray([], dtype=numpy.bool) >>> index2 = numpy.asarray([True, False, True], dtype=numpy.bool) >>> x = numpy.arange(5) >>> (x1, y1) = numpy.meshgrid(x, index1.nonzero()[0]) >>> (x2, y2) = numpy.meshgrid(x, index2.nonzero()[0]) >>> (x1, y1) (array([], shape=(0, 5), dtype=int32), array([], shape=(0, 5), dtype=int32)) >>> print x2, "\n", y2 [[0 1 2 3 4] [0 1 2 3 4]] [[0 0 0 0 0] [2 2 2 2 2]] >>> d1[y1, x1] array([], shape=(0, 5), dtype=int32) >>> d2[y1, x1] array([], shape=(0, 5), dtype=int32) >>> d2[y2, x2] array([[ 0, 1, 2, 3, 4], [10, 11, 12, 13, 14]]) I don't know if the other thing is a bug, but it looks like. I could imagine that it has something to do with the implicit slicing on the array without data? Rather an imperfection ... Consider this: >>> d1 = numpy.arange(0).reshape((0,)) >>> d2 = numpy.arange(0).reshape((0, 5)) >>> d3 = numpy.arange(0).reshape((5, 0)) >>> d1[[]] array([], dtype=int32) >>> d2[[]] Traceback (most recent call last): File "", line 1, in IndexError: invalid index >>> d2[[], 0] array([], dtype=int32) >>> d3[[]] array([], shape=(0, 0), dtype=int32) >>> d3[0, []] array([], dtype=int32) >>> d3[:, []] Traceback (most recent call last): File "", line 1, in IndexError: invalid index Ticket? Friedrich From markbak at gmail.com Wed Feb 9 05:00:31 2011 From: markbak at gmail.com (Mark Bakker) Date: Wed, 9 Feb 2011 11:00:31 +0100 Subject: [Numpy-discussion] question on use of newaxis Message-ID: Hello List, I have a simple array a = array([1,2,3]) I want to take the last two terms and make it a column vector: a[[1,2],newaxis] But I get a funny error: TypeError: long() argument must be a string or a number, not 'NoneType' It does work if I do a[[1,2]][:,newaxis] But I don't understand why the first wouldn't work. Is there a reason such syntax is not supported? It would be very useful. Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedrichromstedt at gmail.com Wed Feb 9 05:40:21 2011 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Wed, 9 Feb 2011 11:40:21 +0100 Subject: [Numpy-discussion] question on use of newaxis In-Reply-To: References: Message-ID: 2011/2/9 Mark Bakker : > a = array([1,2,3]) > > I want to take the last two terms and make it a column vector: > > a[[1,2],newaxis] > a[[1,2]][:,newaxis] The former is advanced indexing, while the latter is basic slicing (after advanced indexing). See http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html. There, in the advanced indexing case, "All sequences and scalars in the selection tuple are converted to intp indexing arrays", and: >>> numpy.intp . Thus it tries to interpret the None: >>> numpy.newaxis >>> print repr(numpy.newaxis) None as an ``numpy.intp``, hence the error. Since advanced indexing deletes axes of an ndarray and inserts new ones (see http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#integer, last point), a ``numpy.newaxis`` might be ambiguous. In your case, it would be not, though. Friedrich From ndbecker2 at gmail.com Wed Feb 9 10:00:14 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 09 Feb 2011 10:00:14 -0500 Subject: [Numpy-discussion] how to do this efficiently? Message-ID: In a 1-d array, find the first point where all subsequent points have values less than a threshold, T. From zachary.pincus at yale.edu Wed Feb 9 10:14:07 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 9 Feb 2011 10:14:07 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: References: Message-ID: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> > In a 1-d array, find the first point where all subsequent points > have values > less than a threshold, T. Maybe something like: last_greater = numpy.arange(arr.shape)[arr >= T][-1] first_lower = last_greater + 1 There's probably a better way to do it, without the arange, though... From ndbecker2 at gmail.com Wed Feb 9 10:35:57 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 09 Feb 2011 10:35:57 -0500 Subject: [Numpy-discussion] how to do this efficiently? References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> Message-ID: Zachary Pincus wrote: >> In a 1-d array, find the first point where all subsequent points >> have values >> less than a threshold, T. > > Maybe something like: > > last_greater = numpy.arange(arr.shape)[arr >= T][-1] > first_lower = last_greater + 1 > > There's probably a better way to do it, without the arange, though... I'm trying to find the first point in a power spectrum such that all subsequent points are below some level. I've started with: db is my power spectrum in dB, It is already reversed. mag = np.maximum.accumulate (db) - db[-1] Now all I need is to find the first point such that mag < -50. How to do this efficiently? From zachary.pincus at yale.edu Wed Feb 9 10:38:49 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 9 Feb 2011 10:38:49 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> Message-ID: <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> >>> In a 1-d array, find the first point where all subsequent points >>> have values >>> less than a threshold, T. >> >> Maybe something like: >> >> last_greater = numpy.arange(arr.shape)[arr >= T][-1] >> first_lower = last_greater + 1 >> >> There's probably a better way to do it, without the arange, though... > > I'm trying to find the first point in a power spectrum such that all > subsequent > points are below some level. I've started with: > > db is my power spectrum in dB, It is already reversed. > > mag = np.maximum.accumulate (db) - db[-1] > > Now all I need is to find the first point such that mag < -50. How > to do this > efficiently? Right -- that's what I showed above. Find the last point in mag that is >= -50, and by definition the next point is the first point such that the remainder of mag is < -50. From davide.lasagna at polito.it Wed Feb 9 10:53:06 2011 From: davide.lasagna at polito.it (Davide Lasagna) Date: Wed, 9 Feb 2011 16:53:06 +0100 Subject: [Numpy-discussion] 2d*3d matrix dot product Message-ID: <201102091653.06506.davide.lasagna@polito.it> Hi, I want to compute the following dot product: P = np.array( [[ p11, p12 ], [p21, p22]] ) C = np.array( [c1, c2] ) where c1 and c2 are m*m matrices, so that C.shape = (2,m,m) I want to compute: A = np.array([a1, a2]) where a1 and a2 are two matrices m*m, from the dot product of P and C. I would expect: a1 = p11*c1 + p12*c2 a2 = p21*c1 + p22*c2 The code should be general so that i can multiply any P and C with shapes: P.shape = (n, n) C.shape = (n, m, l) and with a result as: A.shape = (n, m, l) I had a look at np.dot? but i can't sort out how to transpose/reshape the C array. Any help is greatly appreciated. Ciao From ndbecker2 at gmail.com Wed Feb 9 10:58:03 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 09 Feb 2011 10:58:03 -0500 Subject: [Numpy-discussion] how to do this efficiently? References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> Message-ID: Zachary Pincus wrote: >>>> In a 1-d array, find the first point where all subsequent points >>>> have values >>>> less than a threshold, T. >>> >>> Maybe something like: >>> >>> last_greater = numpy.arange(arr.shape)[arr >= T][-1] >>> first_lower = last_greater + 1 >>> >>> There's probably a better way to do it, without the arange, though... >> >> I'm trying to find the first point in a power spectrum such that all >> subsequent >> points are below some level. I've started with: >> >> db is my power spectrum in dB, It is already reversed. >> >> mag = np.maximum.accumulate (db) - db[-1] >> >> Now all I need is to find the first point such that mag < -50. How >> to do this >> efficiently? > > Right -- that's what I showed above. Find the last point in mag that > is >= -50, and by definition the next point is the first point such > that the remainder of mag is < -50. But where is numpy's 'find_first' function? I can't seem to find it so I had to make my own in C++. From zachary.pincus at yale.edu Wed Feb 9 11:04:24 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 9 Feb 2011 11:04:24 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> Message-ID: <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> On Feb 9, 2011, at 10:58 AM, Neal Becker wrote: > Zachary Pincus wrote: > >>>>> In a 1-d array, find the first point where all subsequent points >>>>> have values >>>>> less than a threshold, T. >>>> >>>> Maybe something like: >>>> >>>> last_greater = numpy.arange(arr.shape)[arr >= T][-1] >>>> first_lower = last_greater + 1 >>>> >>>> There's probably a better way to do it, without the arange, >>>> though... >>> >>> I'm trying to find the first point in a power spectrum such that all >>> subsequent >>> points are below some level. I've started with: >>> >>> db is my power spectrum in dB, It is already reversed. >>> >>> mag = np.maximum.accumulate (db) - db[-1] >>> >>> Now all I need is to find the first point such that mag < -50. How >>> to do this >>> efficiently? >> >> Right -- that's what I showed above. Find the last point in mag that >> is >= -50, and by definition the next point is the first point such >> that the remainder of mag is < -50. > > But where is numpy's 'find_first' function? I can't seem to find it > so I had to > make my own in C++. As before, the line below does what you said you need, though not maximally efficiently. (Try it in an interpreter...) There may be another way in numpy that doesn't rely on constructing the index array, but this is the first thing that came to mind. last_greater = numpy.arange(arr.shape)[arr >= T][-1] Let's unpack that dense line a bit: mask = arr >= T indices = numpy.arange(arr.shape) above_threshold_indices = indices[mask] last_above_threshold_index = above_threshold_indices[-1] Does this make sense? From alan.isaac at gmail.com Wed Feb 9 11:08:20 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Wed, 09 Feb 2011 11:08:20 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> Message-ID: <4D52BBF4.2080905@gmail.com> On 2/9/2011 10:58 AM, Neal Becker wrote: > But where is numpy's 'find_first' function? np.argmax(array>T) (Of course that constructs a boolean array...) Alan Isaac From josef.pktd at gmail.com Wed Feb 9 11:09:05 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 9 Feb 2011 11:09:05 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> Message-ID: On Wed, Feb 9, 2011 at 11:04 AM, Zachary Pincus wrote: > > On Feb 9, 2011, at 10:58 AM, Neal Becker wrote: > >> Zachary Pincus wrote: >> >>>>>> In a 1-d array, find the first point where all subsequent points >>>>>> have values >>>>>> less than a threshold, T. >>>>> >>>>> Maybe something like: >>>>> >>>>> last_greater = numpy.arange(arr.shape)[arr >= T][-1] >>>>> first_lower = last_greater + 1 >>>>> >>>>> There's probably a better way to do it, without the arange, >>>>> though... >>>> >>>> I'm trying to find the first point in a power spectrum such that all >>>> subsequent >>>> points are below some level. ?I've started with: >>>> >>>> db is my power spectrum in dB, ? It is already reversed. >>>> >>>> mag = np.maximum.accumulate (db) - db[-1] >>>> >>>> Now all I need is to find the first point such that mag < -50. ?How >>>> to do this >>>> efficiently? >>> >>> Right -- that's what I showed above. Find the last point in mag that >>> is >= -50, and by definition the next point is the first point such >>> that the remainder of mag is < -50. >> >> But where is numpy's 'find_first' function? ?I can't seem to find it >> so I had to >> make my own in C++. > > > As before, the line below does what you said you need, though not > maximally efficiently. (Try it in an interpreter...) There may be > another way in numpy that doesn't rely on constructing the index > array, but this is the first thing that came to mind. > > last_greater = numpy.arange(arr.shape)[arr >= T][-1] > > Let's unpack that dense line a bit: > > mask = arr >= T > indices = numpy.arange(arr.shape) > above_threshold_indices = indices[mask] > last_above_threshold_index = above_threshold_indices[-1] > > Does this make sense? This assumes monotonicity. Is that allowed? Josef > > > > > > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From zachary.pincus at yale.edu Wed Feb 9 11:17:22 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 9 Feb 2011 11:17:22 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> Message-ID: <72C58BD0-9AA1-492B-ADF2-CB82465CD132@yale.edu> >> As before, the line below does what you said you need, though not >> maximally efficiently. (Try it in an interpreter...) There may be >> another way in numpy that doesn't rely on constructing the index >> array, but this is the first thing that came to mind. >> >> last_greater = numpy.arange(arr.shape)[arr >= T][-1] >> >> Let's unpack that dense line a bit: >> >> mask = arr >= T >> indices = numpy.arange(arr.shape) >> above_threshold_indices = indices[mask] >> last_above_threshold_index = above_threshold_indices[-1] >> >> Does this make sense? > > This assumes monotonicity. Is that allowed? The twice-stated problem was: > In a 1-d array, find the first point where all subsequent points > have values less than a threshold, T. So that should do the trick... Though Alan's argmax solution is definitely a better one than indexing an indices array. Same logic and result though, just more compact. From zachary.pincus at yale.edu Wed Feb 9 11:25:57 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 9 Feb 2011 11:25:57 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: <72C58BD0-9AA1-492B-ADF2-CB82465CD132@yale.edu> References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> <72C58BD0-9AA1-492B-ADF2-CB82465CD132@yale.edu> Message-ID: <806A8278-A440-4471-9F8A-367BC759B5C5@yale.edu> >> This assumes monotonicity. Is that allowed? > > The twice-stated problem was: [Note to avert email-miscommunications] BTW, I wasn't trying to snipe at you with that comment, Josef... I just meant to say that this solution solves the problem as Neal posed it, though that might not be the exact problem he has. From bsouthey at gmail.com Wed Feb 9 11:39:03 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 09 Feb 2011 10:39:03 -0600 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: <72C58BD0-9AA1-492B-ADF2-CB82465CD132@yale.edu> References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> <72C58BD0-9AA1-492B-ADF2-CB82465CD132@yale.edu> Message-ID: <4D52C327.3080409@gmail.com> On 02/09/2011 10:17 AM, Zachary Pincus wrote: >>> As before, the line below does what you said you need, though not >>> maximally efficiently. (Try it in an interpreter...) There may be >>> another way in numpy that doesn't rely on constructing the index >>> array, but this is the first thing that came to mind. >>> >>> last_greater = numpy.arange(arr.shape)[arr>= T][-1] >>> >>> Let's unpack that dense line a bit: >>> >>> mask = arr>= T >>> indices = numpy.arange(arr.shape) >>> above_threshold_indices = indices[mask] >>> last_above_threshold_index = above_threshold_indices[-1] >>> >>> Does this make sense? >> This assumes monotonicity. Is that allowed? > The twice-stated problem was: > >> In a 1-d array, find the first point where all subsequent points >> have values less than a threshold, T. > So that should do the trick... Though Alan's argmax solution is > definitely a better one than indexing an indices array. Same logic and > result though, just more compact. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion This is similar to Zachary's as it just uses 'np.where'. >>> x=np.array([5,4,3,6,7,3,2,1]) >>> x array([5, 4, 3, 6, 7, 3, 2, 1]) >>> np.argmax(x>5) # doesn't appear to be correct 3 >>> np.where(x>5)[0][-1]#since np.where gives a tuple and you need the last element 4 This should give the index where all subsequent points are less than some threshold. However, this and Zachary's version fail when if all points are lower than the threshold (eg T=10 in this array). Bruce From ndbecker2 at gmail.com Wed Feb 9 11:48:25 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 09 Feb 2011 11:48:25 -0500 Subject: [Numpy-discussion] how to do this efficiently? References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> Message-ID: Zachary Pincus wrote: > > On Feb 9, 2011, at 10:58 AM, Neal Becker wrote: > >> Zachary Pincus wrote: >> >>>>>> In a 1-d array, find the first point where all subsequent points >>>>>> have values >>>>>> less than a threshold, T. >>>>> >>>>> Maybe something like: >>>>> >>>>> last_greater = numpy.arange(arr.shape)[arr >= T][-1] >>>>> first_lower = last_greater + 1 >>>>> >>>>> There's probably a better way to do it, without the arange, >>>>> though... >>>> >>>> I'm trying to find the first point in a power spectrum such that all >>>> subsequent >>>> points are below some level. I've started with: >>>> >>>> db is my power spectrum in dB, It is already reversed. >>>> >>>> mag = np.maximum.accumulate (db) - db[-1] >>>> >>>> Now all I need is to find the first point such that mag < -50. How >>>> to do this >>>> efficiently? >>> >>> Right -- that's what I showed above. Find the last point in mag that >>> is >= -50, and by definition the next point is the first point such >>> that the remainder of mag is < -50. >> >> But where is numpy's 'find_first' function? I can't seem to find it >> so I had to >> make my own in C++. > > > As before, the line below does what you said you need, though not > maximally efficiently. (Try it in an interpreter...) There may be > another way in numpy that doesn't rely on constructing the index > array, but this is the first thing that came to mind. > > last_greater = numpy.arange(arr.shape)[arr >= T][-1] > > Let's unpack that dense line a bit: > > mask = arr >= T > indices = numpy.arange(arr.shape) > above_threshold_indices = indices[mask] > last_above_threshold_index = above_threshold_indices[-1] > > Does this make sense? Yes, thanks for the explanation. Still, I wonder if numpy would benefit from a 'find_first' function. From alan.isaac at gmail.com Wed Feb 9 12:17:45 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Wed, 09 Feb 2011 12:17:45 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: <4D52C327.3080409@gmail.com> References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> <72C58BD0-9AA1-492B-ADF2-CB82465CD132@yale.edu> <4D52C327.3080409@gmail.com> Message-ID: <4D52CC39.3010801@gmail.com> On 2/9/2011 11:39 AM, Bruce Southey wrote: > np.argmax(x>5) # doesn't appear to be correct It was an answer to the particular question of how to do find_first, which it does (at the cost of a boolean array): it finds the first element greater than 5. >>> x array([5, 4, 3, 6, 7, 3, 2, 1]) >>> np.argmax(x>5) 3 >>> x[3] 6 You can also find the first element less than 4, etc. >>> np.argmax(x<4) 2 The condition should be chosen for the problem. As Josef pointed out, the usefulness of this and other approaches can depend on the monotonicity of the series. fwiw, Alan From ben.root at ou.edu Wed Feb 9 12:40:39 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 9 Feb 2011 11:40:39 -0600 Subject: [Numpy-discussion] 2d*3d matrix dot product In-Reply-To: <201102091653.06506.davide.lasagna@polito.it> References: <201102091653.06506.davide.lasagna@polito.it> Message-ID: On Wed, Feb 9, 2011 at 9:53 AM, Davide Lasagna wrote: > Hi, > > I want to compute the following dot product: > > P = np.array( [[ p11, p12 ], [p21, p22]] ) > C = np.array( [c1, c2] ) > > where c1 and c2 are m*m matrices, so that > > C.shape = (2,m,m) > > I want to compute: > > A = np.array([a1, a2]) > > where a1 and a2 are two matrices m*m, from the dot product of P and C. > > I would expect: > > a1 = p11*c1 + p12*c2 > a2 = p21*c1 + p22*c2 > > The code should be general so that i can multiply any P and C with shapes: > > P.shape = (n, n) > C.shape = (n, m, l) > > and with a result as: > > A.shape = (n, m, l) > > I had a look at np.dot? but i can't sort out how to transpose/reshape the C > array. > > Any help is greatly appreciated. > > Ciao > > tensordot() should do the trick. A = np.tensordot(P, C, axes=1) Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Feb 9 12:45:38 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 9 Feb 2011 11:45:38 -0600 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> Message-ID: On Wed, Feb 9, 2011 at 10:48, Neal Becker wrote: > Still, I wonder if numpy would benefit from a 'find_first' function. I certainly could have used one many times. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Wed Feb 9 13:00:49 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 9 Feb 2011 13:00:49 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: <806A8278-A440-4471-9F8A-367BC759B5C5@yale.edu> References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> <72C58BD0-9AA1-492B-ADF2-CB82465CD132@yale.edu> <806A8278-A440-4471-9F8A-367BC759B5C5@yale.edu> Message-ID: On Wed, Feb 9, 2011 at 11:25 AM, Zachary Pincus wrote: >>> This assumes monotonicity. Is that allowed? >> >> The twice-stated problem was: > > [Note to avert email-miscommunications] BTW, I wasn't trying to snipe > at you with that comment, Josef... > > I just meant to say that this solution solves the problem as Neal > posed it, though that might not be the exact problem he has. >> In a 1-d array, find the first point where all subsequent points have values >> less than a threshold. This doesn't imply monotonicity. Suppose with have a sin curve, and I want to find the last trough. Or a business cycle and I want to find the last recession. Unless my english deteriorated recently. :) Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From mwwiebe at gmail.com Wed Feb 9 13:14:03 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Wed, 9 Feb 2011 10:14:03 -0800 Subject: [Numpy-discussion] Permission to close ticket for Mark Wiebe In-Reply-To: References: Message-ID: On Tue, Feb 8, 2011 at 10:49 AM, Pauli Virtanen wrote: > Tue, 08 Feb 2011 11:35:12 -0700, Charles R Harris wrote: > > Permission to close ticket for Mark Wiebe > > > > I don't know who handles these permissions, I didn't see a way to do it > > myself. > > Granted. (You'd need a shell account to do these changes.) This doesn't seem to have worked, I still see the same options as before. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Wed Feb 9 13:18:18 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 9 Feb 2011 13:18:18 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> <72C58BD0-9AA1-492B-ADF2-CB82465CD132@yale.edu> <806A8278-A440-4471-9F8A-367BC759B5C5@yale.edu> Message-ID: > >>> In a 1-d array, find the first point where all subsequent points >>> have values >>> less than a threshold. > > This doesn't imply monotonicity. > Suppose with have a sin curve, and I want to find the last trough. Or > a business cycle and I want to find the last recession. > > Unless my english deteriorated recently. > Not sure that I follow? I read the statement as: find the smallest non-negative index i such that array[j] < t for all i <= j < len(array) for which the various solutions proposed work, except for the corner case where array.min() >= t. I'm not sure where monotonicity comes into play, unless one interprets the "all subsequent points" clause as "some number of subsequent points" or something... For those sort of problems, I usually wind up smoothing the array [optional], and then using scipy.ndimage.maximum to calculate peak values within a given window, and then find the points that equal the peak values -- this works pretty robustly for peak/trough detection in arbitrary dimension. From josef.pktd at gmail.com Wed Feb 9 13:56:43 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 9 Feb 2011 13:56:43 -0500 Subject: [Numpy-discussion] how to do this efficiently? In-Reply-To: References: <0B4303C2-C7C7-45C4-8B2A-CA4CDA5F9D6D@yale.edu> <91F28DB6-986D-488D-8B96-400AB81765AB@yale.edu> <3A18A0DD-B8B6-41D6-9212-FE481E0EBDAD@yale.edu> <72C58BD0-9AA1-492B-ADF2-CB82465CD132@yale.edu> <806A8278-A440-4471-9F8A-367BC759B5C5@yale.edu> Message-ID: On Wed, Feb 9, 2011 at 1:18 PM, Zachary Pincus wrote: >> >>>> In a 1-d array, find the first point where all subsequent points >>>> have values >>>> less than a threshold. >> >> This doesn't imply monotonicity. >> Suppose with have a sin curve, and I want to find the last trough. Or >> a business cycle and I want to find the last recession. >> >> Unless my english deteriorated recently. >> > > Not sure that I follow? I read the statement as: > find the smallest non-negative index i such that array[j] < t for all > i <= j < len(array) > > for which the various solutions proposed work, except for the corner > case where array.min() >= t. > > I'm not sure where monotonicity comes into play, unless one interprets > the "all subsequent points" clause as "some number of subsequent > points" or something... > > For those sort of problems, I usually wind up smoothing the array > [optional], and then using scipy.ndimage.maximum to calculate peak > values within a given window, and then find the points that equal the > peak values -- this works pretty robustly for peak/trough detection in > arbitrary dimension. (Sorry, I should reply when I'm not able to actually pay attention) your code works fine with a hump, I didn't read the example carefully enough >>> x = np.arange(10,0,-1) >>> x2 = 25-(x-5)**2 >>> x2 array([ 0, 9, 16, 21, 24, 25, 24, 21, 16, 9]) >>> T=20 >>> np.argmax(x2>> np.arange(x2.shape[0])[x2 >= T][-1]+1 8 >>> len(x)-np.argmax(x2[::]>T) + 1 8 Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From friedrichromstedt at gmail.com Wed Feb 9 14:10:49 2011 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Wed, 9 Feb 2011 20:10:49 +0100 Subject: [Numpy-discussion] 2d*3d matrix dot product In-Reply-To: <201102091653.06506.davide.lasagna@polito.it> References: <201102091653.06506.davide.lasagna@polito.it> Message-ID: 2011/2/9 Davide Lasagna : > Hi, > > I want to compute the following dot product: > > P = np.array( [[ p11, p12 ], [p21, p22]] ) > C = np.array( [c1, c2] ) > > a1 = p11*c1 + p12*c2 > a2 = p21*c1 + p22*c2 > > P.shape = (n, n) > C.shape = (n, m, l) > > and with a result as: > > A.shape = (n, m, l) Interesting problem. So: A[i, j, k] = \sum_q P[i, q] * C[q, j, k] Your problem description is an interesting way to look at tensordot. Normally the sum character is emphasised, but in your description the indiced not summed over play the role of forming complete matrices which are multiplied by scalars. Alternatively, you can look at your tensordot as a (m, l) matrix of (n,) vectors which originate from dot-multiplying earch (n,) vector in a (m, l) matrix, which comprises C, with the P matrix. But this is just an interpretation, I like the equation above best of them all. Ben's suggestion is identical to: A = numpy.tensordot(P, C, axes=(1, 0)) Depends on what you want to emphasis, the concrete indices or how many of the last/first indices to take in. http://docs.scipy.org/doc/numpy/reference/generated/numpy.tensordot.html Friedrich From markbak at gmail.com Wed Feb 9 15:12:34 2011 From: markbak at gmail.com (Mark Bakker) Date: Wed, 9 Feb 2011 21:12:34 +0100 Subject: [Numpy-discussion] how to do this efficiently? Message-ID: I recently needed a find_first function and searched the list. It has been discussed a few times before. And I haven't seen an elegant solution yet. Maybe we should file a feature request, as it would be awfully nice to have? Mark > > > Still, I wonder if numpy would benefit from a 'find_first' function. > > I certainly could have used one many times. > > -- > Robert Kern > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Feb 9 18:07:01 2011 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 9 Feb 2011 15:07:01 -0800 Subject: [Numpy-discussion] int is a subtype of bool? Message-ID: This can't possibly be right, can it? Or am I missing something? (It certainly doesn't seem to match any documentation that I can find) >>> np.issubdtype(int, bool) True >>> np.__version__ '1.5.1' -- Nathaniel From ndbecker2 at gmail.com Wed Feb 9 18:41:55 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 09 Feb 2011 18:41:55 -0500 Subject: [Numpy-discussion] how to do this efficiently? References: Message-ID: Mark Bakker wrote: > I recently needed a find_first function and searched the list. > It has been discussed a few times before. > And I haven't seen an elegant solution yet. > Maybe we should file a feature request, as it would be awfully nice to have? > Mark > >> >> > Still, I wonder if numpy would benefit from a 'find_first' function. >> >> I certainly could have used one many times. >> >> -- >> Robert Kern >> >> http://projects.scipy.org/numpy/ticket/1738 From alok at merfinllc.com Wed Feb 9 19:02:37 2011 From: alok at merfinllc.com (Alok Singhal) Date: Wed, 9 Feb 2011 16:02:37 -0800 Subject: [Numpy-discussion] Indexing a 2-d array with a 1-d mask In-Reply-To: References: Message-ID: On Wed, Feb 9, 2011 at 1:24 AM, Friedrich Romstedt wrote: > 2011/2/8 Alok Singhal : >> In [6]: data2 = numpy.zeros((0, 5), 'd') >> In [7]: mask2 = numpy.zeros(0, 'bool') >> In [8]: data2[mask2] >> ------------------------------------------------------------ >> Traceback (most recent call last): >> ?File "", line 1, in >> IndexError: invalid index >> >> I would have expected the above to give me a 0x5 array. > >> Is there any other way to do what I >> am doing? > > Like so (works only for ndim==2): > >>>> d1 = numpy.arange(0).reshape((0, 5)) >>>> d2 = numpy.arange(15).reshape((3, 5)) >>>> index1 = numpy.asarray([], dtype=numpy.bool) >>>> index2 = numpy.asarray([True, False, True], dtype=numpy.bool) >>>> x = numpy.arange(5) >>>> (x1, y1) = numpy.meshgrid(x, index1.nonzero()[0]) >>>> (x2, y2) = numpy.meshgrid(x, index2.nonzero()[0]) >>>> (x1, y1) > (array([], shape=(0, 5), dtype=int32), array([], shape=(0, 5), dtype=int32)) >>>> print x2, "\n", y2 > [[0 1 2 3 4] > ?[0 1 2 3 4]] > [[0 0 0 0 0] > ?[2 2 2 2 2]] >>>> d1[y1, x1] > array([], shape=(0, 5), dtype=int32) >>>> d2[y1, x1] > array([], shape=(0, 5), dtype=int32) >>>> d2[y2, x2] > array([[ 0, ?1, ?2, ?3, ?4], > ? ? ? [10, 11, 12, 13, 14]]) Yeah, I can do it with creating the full index array, but I have huge data sets, so I was hoping to avoid them. For now, I can just check for the borderline case and keep using the memory-efficient indexing for the "regular" cases. > I don't know if the other thing is a bug, but it looks like. ?I could > imagine that it has something to do with the implicit slicing on the > array without data? ?Rather an imperfection ... > > Consider this: > >>>> d1 = numpy.arange(0).reshape((0,)) >>>> d2 = numpy.arange(0).reshape((0, 5)) >>>> d3 = numpy.arange(0).reshape((5, 0)) >>>> d1[[]] > array([], dtype=int32) >>>> d2[[]] > Traceback (most recent call last): > ?File "", line 1, in > IndexError: invalid index >>>> d2[[], 0] > array([], dtype=int32) >>>> d3[[]] > array([], shape=(0, 0), dtype=int32) >>>> d3[0, []] > array([], dtype=int32) >>>> d3[:, []] > Traceback (most recent call last): > ?File "", line 1, in > IndexError: invalid index > > Ticket? I think so too. Although, I don't know if this behavior is a feature of advanced indexing (and not a bug). Thanks, Alok From pav at iki.fi Wed Feb 9 19:05:44 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 10 Feb 2011 00:05:44 +0000 (UTC) Subject: [Numpy-discussion] int is a subtype of bool? References: Message-ID: On Wed, 09 Feb 2011 15:07:01 -0800, Nathaniel Smith wrote: > This can't possibly be right, can it? Or am I missing something? (It > certainly doesn't seem to match any documentation that I can find) > >>>> np.issubdtype(int, bool) > True >>>> np.__version__ > '1.5.1' Thanks. It's a bug in issubdtype, http://projects.scipy.org/numpy/ticket/1739 -- Pauli Virtanen From david at silveregg.co.jp Thu Feb 10 00:30:37 2011 From: david at silveregg.co.jp (David) Date: Thu, 10 Feb 2011 14:30:37 +0900 Subject: [Numpy-discussion] FYI: experimental waf support for numpy Message-ID: <4D5377FD.4000005@silveregg.co.jp> Hi there, Following recent release of waf 1.6 and its adoption by the samba project, as well as my own work on integrating waf and bento, I have spent some time to build numpy with it. Although this is experimental, it should be possible for people to play with it: https://github.com/cournape/numpy/tree/waf_build You need a recent waf checkout (trunk), and you build numpy as follows: $waf_checkout/waf-light configure $waf_checkout/waf-light build Waf is python build system, a bit like scons, but smaller, cleaner, much faster and somehow easier to integrate as a library The added code in numpy is ~ 1200 lines of code, of which almost half is just to get missing configuration from waf, IOW, the whole numpy build is described in ~ 600 lines of code, and works on linux, mac os x and windows (Visual Studio only for now). Besides the maintenance advantage, the waf build has a few interesting features: - < 1 sec no-op build with dependencies tracking (header changes, file content change, etc...) - ~ 10 sec debug build on a 2 years old machine - very easy to customize with alternative blas/lapack libraries, or use alternative compilers like clang. - nice output and full log written when a configuration error has occured, for easier build debugging. Note that this cannot be used to install numpy, and is of interest only for developers ATM - the goal is to integrate this with bento to get a robust build and deployment framework for numpy, scipy and beyond. cheers, David From david at silveregg.co.jp Thu Feb 10 00:50:17 2011 From: david at silveregg.co.jp (David) Date: Thu, 10 Feb 2011 14:50:17 +0900 Subject: [Numpy-discussion] Rationale for simple_capsule_dtor to be static but non-inline ? Message-ID: <4D537C99.7090502@silveregg.co.jp> Hi, in npy3_compat.h, one function simple_capsule_dtor is defined as static but non-inline. AFAIK, there is no reason not to put inline (if supported by the compiler of course) for a static function defined in a header. Unless I hear someone justify it, I will change it, cheers, David From davide.lasagna at polito.it Thu Feb 10 02:43:27 2011 From: davide.lasagna at polito.it (Davide Lasagna) Date: Thu, 10 Feb 2011 08:43:27 +0100 Subject: [Numpy-discussion] 2d*3d matrix dot product In-Reply-To: References: <201102091653.06506.davide.lasagna@polito.it> Message-ID: <201102100843.27466.davide.lasagna@polito.it> > Ben's suggestion is identical to: > > A = numpy.tensordot(P, C, axes=(1, 0)) Yes, that does the trick! Thank, very good idea. Since i've build atlas with threading support, in the computation of the dot product all four cpus go to 100%, which makes it quite fast. I'm starting to love numpy array facilities.... Ciao Davide From binet at cern.ch Thu Feb 10 02:45:38 2011 From: binet at cern.ch (Sebastien Binet) Date: Thu, 10 Feb 2011 08:45:38 +0100 Subject: [Numpy-discussion] FYI: experimental waf support for numpy In-Reply-To: <4D5377FD.4000005@silveregg.co.jp> References: <4D5377FD.4000005@silveregg.co.jp> Message-ID: <87ipwsxrh9.fsf@cern.ch> David, On Thu, 10 Feb 2011 14:30:37 +0900, David wrote: > Hi there, > > Following recent release of waf 1.6 and its adoption by the samba > project, as well as my own work on integrating waf and bento, I have > spent some time to build numpy with it. Although this is experimental, > it should be possible for people to play with it: > > https://github.com/cournape/numpy/tree/waf_build great news ! > > You need a recent waf checkout (trunk), and you build numpy as follows: > > $waf_checkout/waf-light configure > $waf_checkout/waf-light build why don't you just put the waf executable within the numpy repository ? (well, I had sometimes some problem with that modus operandi: my archlinux box is using python3 and some of my colleagues are still using python2, but once everybody agreed on a given virtualenv configuration, everything went fine) -s -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From david at silveregg.co.jp Thu Feb 10 03:03:23 2011 From: david at silveregg.co.jp (David) Date: Thu, 10 Feb 2011 17:03:23 +0900 Subject: [Numpy-discussion] FYI: experimental waf support for numpy In-Reply-To: <87ipwsxrh9.fsf@cern.ch> References: <4D5377FD.4000005@silveregg.co.jp> <87ipwsxrh9.fsf@cern.ch> Message-ID: <4D539BCB.6050103@silveregg.co.jp> On 02/10/2011 04:45 PM, Sebastien Binet wrote: > David, > > On Thu, 10 Feb 2011 14:30:37 +0900, David wrote: >> Hi there, >> >> Following recent release of waf 1.6 and its adoption by the samba >> project, as well as my own work on integrating waf and bento, I have >> spent some time to build numpy with it. Although this is experimental, >> it should be possible for people to play with it: >> >> https://github.com/cournape/numpy/tree/waf_build > > great news ! > >> >> You need a recent waf checkout (trunk), and you build numpy as follows: >> >> $waf_checkout/waf-light configure >> $waf_checkout/waf-light build > why don't you just put the waf executable within the numpy repository ? Because it is still experimental, and there are some issues in waf around tools on windows, etc... meaning I will have to follow trunk pretty closely after those bugs are fixed. This is also not intended to be merged. The goal really is to discover as many issues as possible before I decide to use waf exclusively as the build "backend" in bento instead of my own (if you are interested, there is already a proof of concept of building simple extensions with waf inside bento in bento repo: https://github.com/cournape/Bento/tree/master/examples/hooks/waf/waf16) cheers, David From charlesr.harris at gmail.com Thu Feb 10 06:26:40 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Feb 2011 04:26:40 -0700 Subject: [Numpy-discussion] Rationale for simple_capsule_dtor to be static but non-inline ? In-Reply-To: <4D537C99.7090502@silveregg.co.jp> References: <4D537C99.7090502@silveregg.co.jp> Message-ID: On Wed, Feb 9, 2011 at 10:50 PM, David wrote: > Hi, > > in npy3_compat.h, one function simple_capsule_dtor is defined as static > but non-inline. AFAIK, there is no reason not to put inline (if > supported by the compiler of course) for a static function defined in a > header. Unless I hear someone justify it, I will change it, > > It's only used as an argument so it needs to be defined, there is no place that it can be inlined. The NpyCapsule stuff could probably be moved out of the header into a compatibility library somewhere but that would require the builds of numpy/scipy to all link with that library. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Feb 10 08:06:35 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 10 Feb 2011 13:06:35 +0000 (UTC) Subject: [Numpy-discussion] FYI: experimental waf support for numpy References: <4D5377FD.4000005@silveregg.co.jp> Message-ID: Thu, 10 Feb 2011 14:30:37 +0900, David wrote: [clip] > Following recent release of waf 1.6 and its adoption by the samba > project, as well as my own work on integrating waf and bento, I have > spent some time to build numpy with it. Although this is experimental, > it should be possible for people to play with it: > > https://github.com/cournape/numpy/tree/waf_build Looks great to me, and it even seems to work :) The structure of the build appears to be more easy to understand than in the Scons-based system, not to speak of distutils (the waf guys clearly have their heads screwed on the right way). Pauli From e.antero.tammi at gmail.com Thu Feb 10 10:29:23 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 10 Feb 2011 17:29:23 +0200 Subject: [Numpy-discussion] odd performance of sum? Message-ID: Hi, Observing following performance: In []: m= 1e5 In []: n= 1e2 In []: o= ones(n) In []: M= randn(m, n) In []: timeit M.sum(1) 10 loops, best of 3: 38.3 ms per loop In []: timeit dot(M, o) 10 loops, best of 3: 21.1 ms per loop In []: m= 1e2 In []: n= 1e5 In []: o= ones(n) In []: M= randn(m, n) In []: timeit M.sum(1) 100 loops, best of 3: 18.3 ms per loop In []: timeit dot(M, o) 10 loops, best of 3: 21.2 ms per loop One would expect sum to outperform dot with a clear marginal. Does there exixts any 'tricks' to increase the performance of sum? In []: sys.version Out[]: '2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)]' # installed binaries from http://python.org/ In []: np.version.version Out[]: '1.5.1' # installed binaries from http://scipy.org/ Regards, eat -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Feb 10 12:10:23 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Feb 2011 10:10:23 -0700 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: On Thu, Feb 10, 2011 at 8:29 AM, eat wrote: > Hi, > > Observing following performance: > In []: m= 1e5 > In []: n= 1e2 > In []: o= ones(n) > In []: M= randn(m, n) > In []: timeit M.sum(1) > 10 loops, best of 3: 38.3 ms per loop > In []: timeit dot(M, o) > 10 loops, best of 3: 21.1 ms per loop > > In []: m= 1e2 > In []: n= 1e5 > In []: o= ones(n) > In []: M= randn(m, n) > In []: timeit M.sum(1) > 100 loops, best of 3: 18.3 ms per loop > In []: timeit dot(M, o) > 10 loops, best of 3: 21.2 ms per loop > > One would expect sum to outperform dot with a clear marginal. Does there > exixts any 'tricks' to increase the performance of sum? > > I'm not surprised, much depends on the version of ATLAS or MKL you are linked to. If you aren't linked to either and just using numpy's version then the results are a bit strange. With numpy development I get In [1]: m= 1e5 In [2]: n= 1e2 In [3]: o= ones(n) In [4]: M= randn(m, n) In [5]: timeit M.sum(1) 100 loops, best of 3: 19.2 ms per loop In [6]: timeit dot(M, o) 100 loops, best of 3: 15 ms per loop In [7]: m= 1e2 In [8]: n= 1e5 In [9]: o= ones(n) In [10]: M= randn(m, n) In [11]: timeit M.sum(1) 100 loops, best of 3: 17.4 ms per loop In [12]: timeit dot(M, o) 100 loops, best of 3: 14.2 ms per loop Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Feb 10 12:27:30 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Feb 2011 10:27:30 -0700 Subject: [Numpy-discussion] Rationale for simple_capsule_dtor to be static but non-inline ? In-Reply-To: References: <4D537C99.7090502@silveregg.co.jp> Message-ID: On Thu, Feb 10, 2011 at 4:26 AM, Charles R Harris wrote: > > > On Wed, Feb 9, 2011 at 10:50 PM, David wrote: > >> Hi, >> >> in npy3_compat.h, one function simple_capsule_dtor is defined as static >> but non-inline. AFAIK, there is no reason not to put inline (if >> supported by the compiler of course) for a static function defined in a >> header. Unless I hear someone justify it, I will change it, >> >> > It's only used as an argument so it needs to be defined, there is no place > that it can be inlined. The NpyCapsule stuff could probably be moved out of > the header into a compatibility library somewhere but that would require the > builds of numpy/scipy to all link with that library. > > That said, inline should work also, but one needs a bit more faith in the compiler... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Thu Feb 10 12:53:51 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 10 Feb 2011 19:53:51 +0200 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: Thanks Chuck, for replying. But don't you still feel very odd that dot outperforms sum in your machine? Just to get it simply; why sum can't outperform dot? Whatever architecture (computer, cache) you have, it don't make any sense at all that when performing significantly less instructions, you'll reach to spend more time ;-). Regards, eat On Thu, Feb 10, 2011 at 7:10 PM, Charles R Harris wrote: > > > On Thu, Feb 10, 2011 at 8:29 AM, eat wrote: > >> Hi, >> >> Observing following performance: >> In []: m= 1e5 >> In []: n= 1e2 >> In []: o= ones(n) >> In []: M= randn(m, n) >> In []: timeit M.sum(1) >> 10 loops, best of 3: 38.3 ms per loop >> In []: timeit dot(M, o) >> 10 loops, best of 3: 21.1 ms per loop >> >> In []: m= 1e2 >> In []: n= 1e5 >> In []: o= ones(n) >> In []: M= randn(m, n) >> In []: timeit M.sum(1) >> 100 loops, best of 3: 18.3 ms per loop >> In []: timeit dot(M, o) >> 10 loops, best of 3: 21.2 ms per loop >> >> One would expect sum to outperform dot with a clear marginal. Does there >> exixts any 'tricks' to increase the performance of sum? >> >> > > I'm not surprised, much depends on the version of ATLAS or MKL you are > linked to. If you aren't linked to either and just using numpy's version > then the results are a bit strange. With numpy development I get > > In [1]: m= 1e5 > > In [2]: n= 1e2 > > In [3]: o= ones(n) > > In [4]: M= randn(m, n) > > In [5]: timeit M.sum(1) > 100 loops, best of 3: 19.2 ms per loop > > In [6]: timeit dot(M, o) > 100 loops, best of 3: 15 ms per loop > > In [7]: m= 1e2 > > In [8]: n= 1e5 > > In [9]: o= ones(n) > > In [10]: M= randn(m, n) > > In [11]: timeit M.sum(1) > 100 loops, best of 3: 17.4 ms per loop > > In [12]: timeit dot(M, o) > 100 loops, best of 3: 14.2 ms per loop > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Feb 10 13:16:12 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Feb 2011 12:16:12 -0600 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: On Thu, Feb 10, 2011 at 11:53, eat wrote: > Thanks Chuck, > > for replying. But don't you still feel very?odd that dot outperforms sum in > your machine? Just to get it simply; why sum can't outperform dot? Whatever > architecture (computer, cache) you have, it don't make any sense at all that > when performing significantly less instructions, you'll reach to spend more > time ;-). These days, the determining factor is less often instruction count than memory latency, and the optimized BLAS implementations of dot() heavily optimize the memory access patterns. Additionally, the number of instructions in your dot() probably isn't that many more than the sum(). The sum() is pretty dumb and just does a linear accumulation using the ufunc reduce mechanism, so (m*n-1) ADDs plus quite a few instructions for traversing the array in a generic manner. With fused multiply-adds, being able to assume contiguous data and ignore the numpy iterator overhead, and applying divide-and-conquer kernels to arrange sums, the optimized dot() implementations could have a comparable instruction count. If you were willing to spend that amount of developer time and code complexity to make platform-specific backends to sum(), you could make it go really fast, too. Typically, it's not all that important to make it worthwhile, though. One thing that might be worthwhile is to make implementations of sum() and cumsum() that avoid the ufunc machinery and do their iterations more quickly, at least for some common combinations of dtype and contiguity. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pav at iki.fi Thu Feb 10 13:31:39 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 10 Feb 2011 18:31:39 +0000 (UTC) Subject: [Numpy-discussion] odd performance of sum? References: Message-ID: Thu, 10 Feb 2011 12:16:12 -0600, Robert Kern wrote: [clip] > One thing that might be worthwhile is to make > implementations of sum() and cumsum() that avoid the ufunc machinery and > do their iterations more quickly, at least for some common combinations > of dtype and contiguity. I wonder what is the balance between the iterator overhead and the time taken in the reduction inner loop. This should be straightforward to benchmark. Apparently, some overhead decreased with the new iterators, since current Numpy master outperforms 1.5.1 by a factor of 2 for this benchmark: In [8]: %timeit M.sum(1) # Numpy 1.5.1 10 loops, best of 3: 85 ms per loop In [8]: %timeit M.sum(1) # Numpy master 10 loops, best of 3: 49.5 ms per loop I don't think this is explainable by the new memory layout optimizations, since M is C-contiguous. Perhaps there would be room for more optimization, even within the ufunc framework? -- Pauli Virtanen From e.antero.tammi at gmail.com Thu Feb 10 15:29:10 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 10 Feb 2011 22:29:10 +0200 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: Hi Robert, On Thu, Feb 10, 2011 at 8:16 PM, Robert Kern wrote: > On Thu, Feb 10, 2011 at 11:53, eat wrote: > > Thanks Chuck, > > > > for replying. But don't you still feel very odd that dot outperforms sum > in > > your machine? Just to get it simply; why sum can't outperform dot? > Whatever > > architecture (computer, cache) you have, it don't make any sense at all > that > > when performing significantly less instructions, you'll reach to spend > more > > time ;-). > > These days, the determining factor is less often instruction count > than memory latency, and the optimized BLAS implementations of dot() > heavily optimize the memory access patterns. Can't we have this as well with simple sum? > Additionally, the number > of instructions in your dot() probably isn't that many more than the > sum(). The sum() is pretty dumb But does it need to be? > and just does a linear accumulation > using the ufunc reduce mechanism, so (m*n-1) ADDs plus quite a few > instructions for traversing the array in a generic manner. With fused > multiply-adds, being able to assume contiguous data and ignore the > numpy iterator overhead, and applying divide-and-conquer kernels to > arrange sums, the optimized dot() implementations could have a > comparable instruction count. Couldn't sum benefit with similar logic? > If you were willing to spend that amount of developer time and code > complexity to make platform-specific backends to sum() Actually I would, but I'm not competent at all in that detailed level (:, But I'm willing to spend more on my own time for example for testing, debugging, analysing various improvements and suggestions if such emerge. > , you could make > it go really fast, too. Typically, it's not all that important to make > it worthwhile, though. One thing that might be worthwhile is to make > implementations of sum() and cumsum() that avoid the ufunc machinery > and do their iterations more quickly, at least for some common > combinations of dtype and contiguity. > Well I'm allready perplexd before reaching that 'ufunc machinery', it's actually anyway trivial (for us more mortal ;-) to figure out what's happening with sum on fromnumeric.py! Regards, eat > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Thu Feb 10 15:38:52 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 10 Feb 2011 22:38:52 +0200 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: Hi Pauli, On Thu, Feb 10, 2011 at 8:31 PM, Pauli Virtanen wrote: > Thu, 10 Feb 2011 12:16:12 -0600, Robert Kern wrote: > [clip] > > One thing that might be worthwhile is to make > > implementations of sum() and cumsum() that avoid the ufunc machinery and > > do their iterations more quickly, at least for some common combinations > > of dtype and contiguity. > > I wonder what is the balance between the iterator overhead and the time > taken in the reduction inner loop. This should be straightforward to > benchmark. > > Apparently, some overhead decreased with the new iterators, since current > Numpy master outperforms 1.5.1 by a factor of 2 for this benchmark: > > In [8]: %timeit M.sum(1) # Numpy 1.5.1 > 10 loops, best of 3: 85 ms per loop > > In [8]: %timeit M.sum(1) # Numpy master > 10 loops, best of 3: 49.5 ms per loop > > I don't think this is explainable by the new memory layout optimizations, > since M is C-contiguous. > > Perhaps there would be room for more optimization, even within the ufunc > framework? I hope so. Please suggest if there's anything that I can do to further advance this. (My C skills are allready bit rusty, but at any higher level I'll try my best to contribute). Thanks, eat > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Feb 10 15:49:28 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 10 Feb 2011 20:49:28 +0000 (UTC) Subject: [Numpy-discussion] odd performance of sum? References: Message-ID: On Thu, 10 Feb 2011 22:38:52 +0200, eat wrote: [clip] > I hope so. Please suggest if there's anything that I can do to further > advance this. (My C skills are allready bit rusty, but at any higher > level I'll try my best to contribute). If someone wants to try to improve the situation, here's a possible plan of attack: 1. Check first if the bottleneck is in the inner reduction loop (function DOUBLE_add in loops.c.src:712) or in the outer iteration (function PyUFunc_ReductionOp in ufunc_object.c:2781). 2. If it's in the inner loop, some optimizations are possible, e.g. specialized cases for sizeof(item) strides. Think how to add them cleanly. 3. If it's in the outer iteration, try to think how to make it faster. This will be a more messy problem to solve. -- Pauli Virtanen From josh.holbrook at gmail.com Thu Feb 10 15:51:34 2011 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Thu, 10 Feb 2011 11:51:34 -0900 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: Maybe I'm missing something, but why not just implement sum() using dot() and ones() ? --Josh On Thu, Feb 10, 2011 at 11:49 AM, Pauli Virtanen wrote: > On Thu, 10 Feb 2011 22:38:52 +0200, eat wrote: > [clip] >> I hope so. Please suggest if there's anything that I can do to further >> advance this. (My C skills are allready bit rusty, but at any higher >> level I'll try my best to contribute). > > If someone wants to try to improve the situation, here's a possible plan > of attack: > > ?1. Check first if the bottleneck is in the inner reduction loop > (function DOUBLE_add in loops.c.src:712) or in the outer iteration > (function PyUFunc_ReductionOp in ufunc_object.c:2781). > > ?2. If it's in the inner loop, some optimizations are possible, e.g. > specialized cases for sizeof(item) strides. Think how to add them cleanly. > > ?3. If it's in the outer iteration, try to think how to make it faster. > This will be a more messy problem to solve. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Feb 10 15:58:55 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Feb 2011 14:58:55 -0600 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: On Thu, Feb 10, 2011 at 14:29, eat wrote: > Hi Robert, > > On Thu, Feb 10, 2011 at 8:16 PM, Robert Kern wrote: >> >> On Thu, Feb 10, 2011 at 11:53, eat wrote: >> > Thanks Chuck, >> > >> > for replying. But don't you still feel very?odd that dot outperforms sum >> > in >> > your machine? Just to get it simply; why sum can't outperform dot? >> > Whatever >> > architecture (computer, cache) you have, it don't make any sense at all >> > that >> > when performing significantly less instructions, you'll reach to spend >> > more >> > time ;-). >> >> These days, the determining factor is less often instruction count >> than memory latency, and the optimized BLAS implementations of dot() >> heavily optimize the memory access patterns. > > Can't we have this as well with simple sum? It's technically feasible to accomplish, but as I mention later, it entails quite a large cost. Those optimized BLASes represent many man-years of effort and cause substantial headaches for people building and installing numpy. However, they are frequently worth it because those operations are often bottlenecks in whole applications. sum(), even in its stupidest implementation, rarely is. In the places where it is a significant bottleneck, an ad hoc implementation in C or Cython or even FORTRAN for just that application is pretty easy to write. You can gain speed by specializing to just your use case, e.g. contiguous data, summing down to one number, or summing along one axis of only 2D data, etc. There's usually no reason to try to generalize that implementation to put it back into numpy. >> Additionally, the number >> of instructions in your dot() probably isn't that many more than the >> sum(). The sum() is pretty dumb > > But does it need to be? As I also allude to later in my email, no, but there are still costs involved. >> and just does a linear accumulation >> using the ufunc reduce mechanism, so (m*n-1) ADDs plus quite a few >> instructions for traversing the array in a generic manner. With fused >> multiply-adds, being able to assume contiguous data and ignore the >> numpy iterator overhead, and applying divide-and-conquer kernels to >> arrange sums, the optimized dot() implementations could have a >> comparable instruction count. > > Couldn't sum benefit with similar logic? Etc. I'm not going to keep repeating myself. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From robert.kern at gmail.com Thu Feb 10 15:59:37 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Feb 2011 14:59:37 -0600 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: On Thu, Feb 10, 2011 at 14:51, Joshua Holbrook wrote: > Maybe I'm missing something, but why not just implement sum() using > dot() and ones() ? You can't do everything that sum() does with just dot() and ones(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From mwwiebe at gmail.com Thu Feb 10 16:26:16 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Thu, 10 Feb 2011 13:26:16 -0800 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: On Thu, Feb 10, 2011 at 10:31 AM, Pauli Virtanen wrote: > Thu, 10 Feb 2011 12:16:12 -0600, Robert Kern wrote: > [clip] > > One thing that might be worthwhile is to make > > implementations of sum() and cumsum() that avoid the ufunc machinery and > > do their iterations more quickly, at least for some common combinations > > of dtype and contiguity. > > I wonder what is the balance between the iterator overhead and the time > taken in the reduction inner loop. This should be straightforward to > benchmark. > > Apparently, some overhead decreased with the new iterators, since current > Numpy master outperforms 1.5.1 by a factor of 2 for this benchmark: > > In [8]: %timeit M.sum(1) # Numpy 1.5.1 > 10 loops, best of 3: 85 ms per loop > > In [8]: %timeit M.sum(1) # Numpy master > 10 loops, best of 3: 49.5 ms per loop > > I don't think this is explainable by the new memory layout optimizations, > since M is C-contiguous. > > Perhaps there would be room for more optimization, even within the ufunc > framework? > I played around with this in einsum, where it's a bit easier to specialize this case than in the ufunc machinery. What I found made the biggest difference is to use SSE prefetching instructions to prepare the cache in advance. Here are the kind of numbers I get, all from the current Numpy master: In [7]: timeit M.sum(1) 10 loops, best of 3: 44.6 ms per loop In [8]: timeit dot(M, o) 10 loops, best of 3: 36.8 ms per loop In [9]: timeit einsum('ij->i', M) 10 loops, best of 3: 32.1 ms per loop ... In [14]: timeit M.sum(1) 10 loops, best of 3: 41.5 ms per loop In [15]: timeit dot(M, o) 10 loops, best of 3: 42.1 ms per loop In [16]: timeit einsum('ij->i', M) 10 loops, best of 3: 30 ms per loop -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Thu Feb 10 16:32:30 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 10 Feb 2011 23:32:30 +0200 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: Hi Robert, On Thu, Feb 10, 2011 at 10:58 PM, Robert Kern wrote: > On Thu, Feb 10, 2011 at 14:29, eat wrote: > > Hi Robert, > > > > On Thu, Feb 10, 2011 at 8:16 PM, Robert Kern > wrote: > >> > >> On Thu, Feb 10, 2011 at 11:53, eat wrote: > >> > Thanks Chuck, > >> > > >> > for replying. But don't you still feel very odd that dot outperforms > sum > >> > in > >> > your machine? Just to get it simply; why sum can't outperform dot? > >> > Whatever > >> > architecture (computer, cache) you have, it don't make any sense at > all > >> > that > >> > when performing significantly less instructions, you'll reach to spend > >> > more > >> > time ;-). > >> > >> These days, the determining factor is less often instruction count > >> than memory latency, and the optimized BLAS implementations of dot() > >> heavily optimize the memory access patterns. > > > > Can't we have this as well with simple sum? > > It's technically feasible to accomplish, but as I mention later, it > entails quite a large cost. Those optimized BLASes represent many > man-years of effort Yes I acknowledge this. But didn't they then ignore them something simpler, like sum (but which actually could benefit exactly similiar optimizations). > and cause substantial headaches for people > building and installing numpy. I appreciate this. No doubt at all. > However, they are frequently worth it > because those operations are often bottlenecks in whole applications. > sum(), even in its stupidest implementation, rarely is. In the places > where it is a significant bottleneck, an ad hoc implementation in C or > Cython or even FORTRAN for just that application is pretty easy to > write. But here I have to disagree; I'll think that at least I (if not even the majority of numpy users) don't like (nor I'm be capable/ or have enough time/ resources) go to dwell such details. I'm sorry but I'll have to restate that it's quite reasonable to expect that sum outperforms dot in any case. Lets now to start make such movements, which enables sum to outperform dot. > You can gain speed by specializing to just your use case, e.g. > contiguous data, summing down to one number, or summing along one axis > of only 2D data, etc. There's usually no reason to try to generalize > that implementation to put it back into numpy. Yes, I would really like to specialize into my case, but 'without going out the python realm.' Thanks, eat > > >> Additionally, the number > >> of instructions in your dot() probably isn't that many more than the > >> sum(). The sum() is pretty dumb > > > > But does it need to be? > > As I also allude to later in my email, no, but there are still costs > involved. > > >> and just does a linear accumulation > >> using the ufunc reduce mechanism, so (m*n-1) ADDs plus quite a few > >> instructions for traversing the array in a generic manner. With fused > >> multiply-adds, being able to assume contiguous data and ignore the > >> numpy iterator overhead, and applying divide-and-conquer kernels to > >> arrange sums, the optimized dot() implementations could have a > >> comparable instruction count. > > > > Couldn't sum benefit with similar logic? > > Etc. I'm not going to keep repeating myself. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Feb 10 12:22:00 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 10 Feb 2011 12:22:00 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu Message-ID: <20110210122200.1667933e@python.org> I hope this message is on-topic for this mailing list! I'm working on the packaging for python-numpy 1.5 in the next version of Ubuntu (11.04 - Natty), and I'm hitting an issue that I'm hoping you can help me with. For reference, the two relevant bugs are: https://bugs.launchpad.net/ubuntu/+source/python-numpy/+bug/664276 https://bugs.launchpad.net/ubuntu/+source/python-numpy/+bug/664397 Here's the problem: for Ubuntu, we've had to disable the building of the numpy documentation package, because its dependencies violate Ubuntu policy. Numpy is in our "main" archive but the documentation depends on python-matplotlib, which lives in our "universe" archive. Such cross archive dependencies break the build. We can't put python-matplotlib in main because of *its* dependencies. You get the idea: its entire dependency stack would have to be moved to main and that's not feasible. There are a number of workarounds that we might employ to re-enable the docs, but here's what I've done so far: In conf.py: * Wrapped the 'import matplotlib' in a try-except and just issue a warning if the import fails. * use_plots = False * Comment out 'plot_directive' in extensions In doc/sphinxext/tests/test_docscrape.py: * in test_plot_examples(): * cfg = dict(use_plots=False) This lets me get through the build of the docs without the dependency on matplotlib, and a cursory look at the documentation looks pretty good. I'm inclined to propose this as a permanent workaround, but I wanted to get your feedback first. Is there a better way to build the docs without matplotlib? What are the practical effects of my workaround? Do you think this will be an acceptable solution for Ubuntu users of numpy? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From robert.kern at gmail.com Thu Feb 10 17:08:38 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Feb 2011 16:08:38 -0600 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: On Thu, Feb 10, 2011 at 15:32, eat wrote: > Hi Robert, > > On Thu, Feb 10, 2011 at 10:58 PM, Robert Kern wrote: >> >> On Thu, Feb 10, 2011 at 14:29, eat wrote: >> > Hi Robert, >> > >> > On Thu, Feb 10, 2011 at 8:16 PM, Robert Kern >> > wrote: >> >> >> >> On Thu, Feb 10, 2011 at 11:53, eat wrote: >> >> > Thanks Chuck, >> >> > >> >> > for replying. But don't you still feel very?odd that dot outperforms >> >> > sum >> >> > in >> >> > your machine? Just to get it simply; why sum can't outperform dot? >> >> > Whatever >> >> > architecture (computer, cache) you have, it don't make any sense at >> >> > all >> >> > that >> >> > when performing significantly less instructions, you'll reach to >> >> > spend >> >> > more >> >> > time ;-). >> >> >> >> These days, the determining factor is less often instruction count >> >> than memory latency, and the optimized BLAS implementations of dot() >> >> heavily optimize the memory access patterns. >> > >> > Can't we have this as well with simple sum? >> >> It's technically feasible to accomplish, but as I mention later, it >> entails quite a large cost. Those optimized BLASes represent many >> man-years of effort > > Yes I acknowledge this. But didn't they then ?ignore them something simpler, > like sum (but which actually could benefit exactly similiar optimizations). Let's set aside the fact that the people who optimized the implementation of dot() (the authors of ATLAS or the MKL or whichever optimized BLAS library you linked to) are different from those who implemented sum() (the numpy devs). Let me repeat a reason why one would put a lot of effort into optimizing dot() but not sum(): """ >> However, they are frequently worth it >> because those operations are often bottlenecks in whole applications. >> sum(), even in its stupidest implementation, rarely is. """ I don't know if I'm just not communicating very clearly, or if you just reply to individual statements before reading the whole email. >> and cause substantial headaches for people >> building and installing numpy. > > I appreciate this. No doubt at all. >> >> However, they are frequently worth it >> because those operations are often bottlenecks in whole applications. >> sum(), even in its stupidest implementation, rarely is. In the places >> where it is a significant bottleneck, an ad hoc implementation in C or >> Cython or even FORTRAN for just that application is pretty easy to >> write. > > But here I have to disagree; I'll think that at least I (if not even the > majority of numpy users) don't like (nor I'm be capable/ or have enough > time/ resources) go to dwell such details. And you think we have the time and resources to do it for you? > I'm sorry but I'll have to > restate that it's quite reasonable to expect that sum outperforms dot in any > case. You don't optimize a function just because you are capable of it. You optimize a function because it is taking up a significant portion of total runtime in your real application. Anything else is a waste of time. > Lets now to start make such movements, which enables sum to outperform > dot. Sorry, you don't get to volunteer anyone's time or set anyone's priorities but your own. There are some sensible things one could do to optimize sums or even general ufunc reductions, as outlined my Mark, Pauli and myself, but please stop using the accelerated-BLAS dot() as your benchmark. It's a completely inappropriate comparison. >> You can gain speed by specializing to just your use case, e.g. >> contiguous data, summing down to one number, or summing along one axis >> of only 2D data, etc. There's usually no reason to try to generalize >> that implementation to put it back into numpy. > > Yes, I would really like to specialize into my case, but 'without going out > the python realm.' The Bottleneck project is a good place for such things. It's a nice middle-ground for somewhat-specialized routines that are still pretty common but not general enough to be in numpy yet. http://pypi.python.org/pypi/Bottleneck If you're not willing to learn how to implement it yourself in Cython, I'm afraid that you're stuck waiting for someone to do it for you. But please don't expect anyone to feel obligated to do so. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From charlesr.harris at gmail.com Thu Feb 10 17:10:41 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Feb 2011 15:10:41 -0700 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: On Thu, Feb 10, 2011 at 2:26 PM, Mark Wiebe wrote: > On Thu, Feb 10, 2011 at 10:31 AM, Pauli Virtanen wrote: > >> Thu, 10 Feb 2011 12:16:12 -0600, Robert Kern wrote: >> [clip] >> > One thing that might be worthwhile is to make >> > implementations of sum() and cumsum() that avoid the ufunc machinery and >> > do their iterations more quickly, at least for some common combinations >> > of dtype and contiguity. >> >> I wonder what is the balance between the iterator overhead and the time >> taken in the reduction inner loop. This should be straightforward to >> benchmark. >> >> Apparently, some overhead decreased with the new iterators, since current >> Numpy master outperforms 1.5.1 by a factor of 2 for this benchmark: >> >> In [8]: %timeit M.sum(1) # Numpy 1.5.1 >> 10 loops, best of 3: 85 ms per loop >> >> In [8]: %timeit M.sum(1) # Numpy master >> 10 loops, best of 3: 49.5 ms per loop >> >> I don't think this is explainable by the new memory layout optimizations, >> since M is C-contiguous. >> >> Perhaps there would be room for more optimization, even within the ufunc >> framework? >> > > I played around with this in einsum, where it's a bit easier to specialize > this case than in the ufunc machinery. What I found made the biggest > difference is to use SSE prefetching instructions to prepare the cache in > advance. Here are the kind of numbers I get, all from the current Numpy > master: > > In [7]: timeit M.sum(1) > 10 loops, best of 3: 44.6 ms per loop > > In [8]: timeit dot(M, o) > 10 loops, best of 3: 36.8 ms per loop > > In [9]: timeit einsum('ij->i', M) > 10 loops, best of 3: 32.1 ms per loop > ... > I get an even bigger speedup: In [5]: timeit M.sum(1) 10 loops, best of 3: 19.2 ms per loop In [6]: timeit dot(M, o) 100 loops, best of 3: 15.2 ms per loop In [7]: timeit einsum('ij->i', M) 100 loops, best of 3: 11.4 ms per loop Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Feb 10 17:26:49 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 10 Feb 2011 23:26:49 +0100 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <20110210122200.1667933e@python.org> References: <20110210122200.1667933e@python.org> Message-ID: <1297376809.3224.9.camel@Obelisk> to, 2011-02-10 kello 12:22 -0500, Barry Warsaw kirjoitti: [clip: Ubuntu cannot put matplotlib needed by Numpy docs into main] > This lets me get through the build of the docs without the dependency on > matplotlib, and a cursory look at the documentation looks pretty good. > > I'm inclined to propose this as a permanent workaround, but I wanted to get > your feedback first. Is there a better way to build the docs without > matplotlib? What are the practical effects of my workaround? Do you think > this will be an acceptable solution for Ubuntu users of numpy? Without matplotlib, you won't get any graphs in the documentation. Ubuntu users will then miss out e.g. on illustrations of various probability distributions. The figures are not absolutely necessary, but I wouldn't like to see that to happen just because of a packaging issue. How about splitting off the Numpy documentation to a separate source package? That wouldn't need to get into main, as it's of use only to developers. Or, using prebuilt HTML documentation supplied by upstream or pre-built by Ubuntu devs? -- Pauli Virtanen From charlesr.harris at gmail.com Thu Feb 10 17:32:13 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Feb 2011 15:32:13 -0700 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: On Thu, Feb 10, 2011 at 3:08 PM, Robert Kern wrote: > On Thu, Feb 10, 2011 at 15:32, eat wrote: > > Hi Robert, > > > > On Thu, Feb 10, 2011 at 10:58 PM, Robert Kern > wrote: > >> > >> On Thu, Feb 10, 2011 at 14:29, eat wrote: > >> > Hi Robert, > >> > > >> > On Thu, Feb 10, 2011 at 8:16 PM, Robert Kern > >> > wrote: > >> >> > >> >> On Thu, Feb 10, 2011 at 11:53, eat wrote: > >> >> > Thanks Chuck, > >> >> > > >> >> > for replying. But don't you still feel very odd that dot > outperforms > >> >> > sum > >> >> > in > >> >> > your machine? Just to get it simply; why sum can't outperform dot? > >> >> > Whatever > >> >> > architecture (computer, cache) you have, it don't make any sense at > >> >> > all > >> >> > that > >> >> > when performing significantly less instructions, you'll reach to > >> >> > spend > >> >> > more > >> >> > time ;-). > >> >> > >> >> These days, the determining factor is less often instruction count > >> >> than memory latency, and the optimized BLAS implementations of dot() > >> >> heavily optimize the memory access patterns. > >> > > >> > Can't we have this as well with simple sum? > >> > >> It's technically feasible to accomplish, but as I mention later, it > >> entails quite a large cost. Those optimized BLASes represent many > >> man-years of effort > > > > Yes I acknowledge this. But didn't they then ignore them something > simpler, > > like sum (but which actually could benefit exactly similiar > optimizations). > > Let's set aside the fact that the people who optimized the > implementation of dot() (the authors of ATLAS or the MKL or whichever > optimized BLAS library you linked to) are different from those who > implemented sum() (the numpy devs). Let me repeat a reason why one > would put a lot of effort into optimizing dot() but not sum(): > > """ > >> However, they are frequently worth it > >> because those operations are often bottlenecks in whole applications. > >> sum(), even in its stupidest implementation, rarely is. > """ > > I don't know if I'm just not communicating very clearly, or if you > just reply to individual statements before reading the whole email. > > >> and cause substantial headaches for people > >> building and installing numpy. > > > > I appreciate this. No doubt at all. > >> > >> However, they are frequently worth it > >> because those operations are often bottlenecks in whole applications. > >> sum(), even in its stupidest implementation, rarely is. In the places > >> where it is a significant bottleneck, an ad hoc implementation in C or > >> Cython or even FORTRAN for just that application is pretty easy to > >> write. > > > > But here I have to disagree; I'll think that at least I (if not even the > > majority of numpy users) don't like (nor I'm be capable/ or have enough > > time/ resources) go to dwell such details. > > And you think we have the time and resources to do it for you? > > > I'm sorry but I'll have to > > restate that it's quite reasonable to expect that sum outperforms dot in > any > > case. > > You don't optimize a function just because you are capable of it. You > optimize a function because it is taking up a significant portion of > total runtime in your real application. Anything else is a waste of > time. > > Heh. Reminds me of a passage in General Bradley's *A Soldier's Story *where he admonished one of his officers in North Africa for taking a hill and suffering casualties, telling him that one didn't take a hill because one could, but because doing so served a purpose in the larger campaign. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Feb 10 17:39:27 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Feb 2011 16:39:27 -0600 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <20110210122200.1667933e@python.org> References: <20110210122200.1667933e@python.org> Message-ID: On Thu, Feb 10, 2011 at 11:22, Barry Warsaw wrote: > Here's the problem: for Ubuntu, we've had to disable the building of the numpy > documentation package, because its dependencies violate Ubuntu policy. ?Numpy > is in our "main" archive but the documentation depends on python-matplotlib, > which lives in our "universe" archive. ?Such cross archive dependencies break > the build. > > We can't put python-matplotlib in main because of *its* dependencies. As a digression, I think the python-matplotlib dependencies could be significantly reduced. For a number of use cases (this is one of them, but there are others), you don't need any GUI backend. Independent of this issue, it would be great to be able to install python-matplotlib in a headless server environment without pulling in all of those GUI bits. Looking at the list of the hard dependencies, I don't understand why half of them are there. http://packages.ubuntu.com/natty/python/python-matplotlib Would "universe" packages that are merely "suggested" or "recommended" also preclude python-matplotlib from moving into "main"? > I'm inclined to propose this as a permanent workaround, but I wanted to get > your feedback first. ?Is there a better way to build the docs without > matplotlib? ?What are the practical effects of my workaround? ?Do you think > this will be an acceptable solution for Ubuntu users of numpy? We might be able to commit to building and releasing complete doc packages with each source release. Could you just download and use those? We currently make such doc packages, but I don't think we keep old micro revisions around, and I suspect you would want those. http://docs.scipy.org/doc/numpy-1.5.x/numpy-html.zip -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From e.antero.tammi at gmail.com Fri Feb 11 02:49:16 2011 From: e.antero.tammi at gmail.com (eat) Date: Fri, 11 Feb 2011 09:49:16 +0200 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: Hi, On Fri, Feb 11, 2011 at 12:08 AM, Robert Kern wrote: > On Thu, Feb 10, 2011 at 15:32, eat wrote: > > Hi Robert, > > > > On Thu, Feb 10, 2011 at 10:58 PM, Robert Kern > wrote: > >> > >> On Thu, Feb 10, 2011 at 14:29, eat wrote: > >> > Hi Robert, > >> > > >> > On Thu, Feb 10, 2011 at 8:16 PM, Robert Kern > >> > wrote: > >> >> > >> >> On Thu, Feb 10, 2011 at 11:53, eat wrote: > >> >> > Thanks Chuck, > >> >> > > >> >> > for replying. But don't you still feel very odd that dot > outperforms > >> >> > sum > >> >> > in > >> >> > your machine? Just to get it simply; why sum can't outperform dot? > >> >> > Whatever > >> >> > architecture (computer, cache) you have, it don't make any sense at > >> >> > all > >> >> > that > >> >> > when performing significantly less instructions, you'll reach to > >> >> > spend > >> >> > more > >> >> > time ;-). > >> >> > >> >> These days, the determining factor is less often instruction count > >> >> than memory latency, and the optimized BLAS implementations of dot() > >> >> heavily optimize the memory access patterns. > >> > > >> > Can't we have this as well with simple sum? > >> > >> It's technically feasible to accomplish, but as I mention later, it > >> entails quite a large cost. Those optimized BLASes represent many > >> man-years of effort > > > > Yes I acknowledge this. But didn't they then ignore them something > simpler, > > like sum (but which actually could benefit exactly similiar > optimizations). > > Let's set aside the fact that the people who optimized the > implementation of dot() (the authors of ATLAS or the MKL or whichever > optimized BLAS library you linked to) are different from those who > implemented sum() (the numpy devs). Let me repeat a reason why one > would put a lot of effort into optimizing dot() but not sum(): > > """ > >> However, they are frequently worth it > >> because those operations are often bottlenecks in whole applications. > >> sum(), even in its stupidest implementation, rarely is. > """ > > I don't know if I'm just not communicating very clearly, or if you > just reply to individual statements before reading the whole email. You are communicating very well. It's me who's not communicating so well. > > >> and cause substantial headaches for people > >> building and installing numpy. > > > > I appreciate this. No doubt at all. > >> > >> However, they are frequently worth it > >> because those operations are often bottlenecks in whole applications. > >> sum(), even in its stupidest implementation, rarely is. In the places > >> where it is a significant bottleneck, an ad hoc implementation in C or > >> Cython or even FORTRAN for just that application is pretty easy to > >> write. > > > > But here I have to disagree; I'll think that at least I (if not even the > > majority of numpy users) don't like (nor I'm be capable/ or have enough > > time/ resources) go to dwell such details. > > And you think we have the time and resources to do it for you? My intention was not to suggest anything like that. > > > I'm sorry but I'll have to > > restate that it's quite reasonable to expect that sum outperforms dot in > any > > case. > > You don't optimize a function just because you are capable of it. You > optimize a function because it is taking up a significant portion of > total runtime in your real application. Anything else is a waste of > time. > > > Lets now to start make such movements, which enables sum to outperform > > dot. > > Sorry, you don't get to volunteer anyone's time or set anyone's > priorities but your own. There are some sensible things one could do > to optimize sums or even general ufunc reductions, as outlined my > Mark, Pauli and myself, but please stop using the accelerated-BLAS > dot() as your benchmark. It's a completely inappropriate comparison. OK. Lets compare sum to itself: In []: M= randn(1e5, 1e2) In []: timeit M.sum(0) 10 loops, best of 3: 169 ms per loop In []: timeit M.sum(1) 10 loops, best of 3: 37.5 ms per loop In []: timeit M.sum() 10 loops, best of 3: 18.1 ms per loop In []: timeit sum(M.sum(0)) 10 loops, best of 3: 169 ms per loop In []: timeit sum(M.sum(1)) 10 loops, best of 3: 37.7 ms per loop In []: timeit M.T.sum(0) 10 loops, best of 3: 37.7 ms per loop In []: timeit M.T.sum(1) 10 loops, best of 3: 171 ms per loop In []: timeit M.T.sum() 1 loops, best of 3: 288 ms per loop In []: timeit sum(M.T.sum(0)) 10 loops, best of 3: 37.7 ms per loop In []: timeit sum(M.T.sum(1)) 10 loops, best of 3: 170 ms per loop In []: 288./ 18.1 Out[]: 15.91160220994475 > > >> You can gain speed by specializing to just your use case, e.g. > >> contiguous data, summing down to one number, or summing along one axis > >> of only 2D data, etc. There's usually no reason to try to generalize > >> that implementation to put it back into numpy. > > > > Yes, I would really like to specialize into my case, but 'without going > out > > the python realm.' > > The Bottleneck project is a good place for such things. It's a nice > middle-ground for somewhat-specialized routines that are still pretty > common but not general enough to be in numpy yet. > > http://pypi.python.org/pypi/Bottleneck > > If you're not willing to learn how to implement it yourself in Cython, > I'm afraid that you're stuck waiting for someone to do it for you. But > please don't expect anyone to feel obligated to do so. Perhaps my bad wordings, but I do not expect anyone to do it for me. I also think that there exists some real substance I addressed (as demonstrated now with comparing sum to itself). Regards, eat > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Feb 11 05:14:36 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Feb 2011 10:14:36 +0000 (UTC) Subject: [Numpy-discussion] odd performance of sum? References: Message-ID: Thu, 10 Feb 2011 20:49:28 +0000, Pauli Virtanen wrote: [clip] > 1. Check first if the bottleneck is in the inner reduction loop > (function DOUBLE_add in loops.c.src:712) or in the outer iteration > (function PyUFunc_ReductionOp in ufunc_object.c:2781). > 2. If it's in the inner loop, some optimizations are possible, e.g. > specialized cases for sizeof(item) strides. Think how to add them > cleanly. A quick check (just replace the inner loop with a no-op) shows that for 100 items, the bottleneck is in the inner loop. The cross-over between inner loop time and strided iterator overhead apparently occurs around ~20-30 items (on the machine I used for testing). Anyway, spending time for optimizing the inner loop for a 30% speed gain (max) seems questionable... -- Pauli Virtanen From Andreas.FRENK at 3ds.com Fri Feb 11 09:01:08 2011 From: Andreas.FRENK at 3ds.com (FRENK Andreas) Date: Fri, 11 Feb 2011 14:01:08 +0000 Subject: [Numpy-discussion] Get the index of a comparison of two lists Message-ID: <5F84D7606A50BF4BA04EB53E5E4AF2F518759B75@EU-DCC-MBX03.dsone.3ds.com> Hi, I need to create a construct that returns the index of entries of the first list, if values in the first and second list are equal. Take valA = [1,2,3,4,20,21,22,23,24] valB = [1,2,3,4, 5,21,22,23] The correct solution is: [0,1,2,3,5,6,7] A potential loop can be: takeList=[] for j,a in enumerate(valA): if a in valB: takeList.append(j) Please note, valA can have entries like [1,10000000,1000000001,.....], i.e. it can be very sparse. I also thought about using bincount, but due to the sparse nature the return values from bincount would allocate too much memory. Any idea how to do it fast using numpy? Thanks, Andreas -------------- next part -------------- An HTML attachment was scrubbed... URL: From amcmorl at gmail.com Fri Feb 11 09:48:06 2011 From: amcmorl at gmail.com (Angus McMorland) Date: Fri, 11 Feb 2011 09:48:06 -0500 Subject: [Numpy-discussion] Get the index of a comparison of two lists In-Reply-To: <5F84D7606A50BF4BA04EB53E5E4AF2F518759B75@EU-DCC-MBX03.dsone.3ds.com> References: <5F84D7606A50BF4BA04EB53E5E4AF2F518759B75@EU-DCC-MBX03.dsone.3ds.com> Message-ID: On 11 February 2011 09:01, FRENK Andreas wrote: > Hi, > > I need to create a construct that returns the index of entries of the first > list, if values in the first and second list are equal. > > Take > > valA = [1,2,3,4,20,21,22,23,24] > valB = [1,2,3,4,? 5,21,22,23] > The correct solution is: [0,1,2,3,5,6,7] > > A potential loop can be: > takeList=[] > for j,a in enumerate(valA): > ??? if a in valB: > ??????? takeList.append(j) > > Please note, valA can have entries like [1,10000000,1000000001,?..], i.e. it > can be very sparse. > I also thought about using bincount, but due to the sparse nature the return > values from bincount would allocate too much memory. > > Any idea how to do it fast using numpy? This probably isn't optimal yet, but seems to perform better than your for loop for large array sizes, but is less good at very small sizes. In [11]: def test(a, b): ....: takeList = [] ....: for j, A in enumerate(a): ....: if A in b: ....: takeList.append(j) ....: return takeList In [24]: a = np.random.randint(10, size=10) In [25]: b = np.random.randint(10, size=10) In [26]: %timeit test(a,b) 10000 loops, best of 3: 55.4 ?s per loop In [27]: %timeit np.arange(a.size)[np.lib.setmember1d(a,b)] 10000 loops, best of 3: 92.9 ?s per loop In [19]: a = np.random.randint(10000, size=10000) In [20]: b = np.random.randint(10000, size=10000) In [21]: %timeit np.arange(a.size)[np.lib.setmember1d(a,b)] 100 loops, best of 3: 7.99 ms per loop In [22]: %timeit test(a,b) 10 loops, best of 3: 787 ms per loop Hope that's useful, Angus -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From josef.pktd at gmail.com Fri Feb 11 09:59:13 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 11 Feb 2011 09:59:13 -0500 Subject: [Numpy-discussion] Get the index of a comparison of two lists In-Reply-To: References: <5F84D7606A50BF4BA04EB53E5E4AF2F518759B75@EU-DCC-MBX03.dsone.3ds.com> Message-ID: On Fri, Feb 11, 2011 at 9:48 AM, Angus McMorland wrote: > On 11 February 2011 09:01, FRENK Andreas wrote: >> Hi, >> >> I need to create a construct that returns the index of entries of the first >> list, if values in the first and second list are equal. >> >> Take >> >> valA = [1,2,3,4,20,21,22,23,24] >> valB = [1,2,3,4,? 5,21,22,23] >> The correct solution is: [0,1,2,3,5,6,7] >> >> A potential loop can be: >> takeList=[] >> for j,a in enumerate(valA): >> ??? if a in valB: >> ??????? takeList.append(j) >> >> Please note, valA can have entries like [1,10000000,1000000001,?..], i.e. it >> can be very sparse. >> I also thought about using bincount, but due to the sparse nature the return >> values from bincount would allocate too much memory. >> >> Any idea how to do it fast using numpy? > > This probably isn't optimal yet, but seems to perform better than your > for loop for large array sizes, but is less good at very small sizes. > > In [11]: def test(a, b): > ? ....: ? ? takeList = [] > ? ....: ? ? for j, A in enumerate(a): > ? ....: ? ? ? ? if A in b: > ? ....: ? ? ? ? ? ? takeList.append(j) > ? ....: ? ? return takeList > > In [24]: a = np.random.randint(10, size=10) > In [25]: b = np.random.randint(10, size=10) > > In [26]: %timeit test(a,b) > 10000 loops, best of 3: 55.4 ?s per loop > > In [27]: %timeit np.arange(a.size)[np.lib.setmember1d(a,b)] > 10000 loops, best of 3: 92.9 ?s per loop > > In [19]: a = np.random.randint(10000, size=10000) > In [20]: b = np.random.randint(10000, size=10000) > > In [21]: %timeit np.arange(a.size)[np.lib.setmember1d(a,b)] > 100 loops, best of 3: 7.99 ms per loop > > In [22]: %timeit test(a,b) > 10 loops, best of 3: 787 ms per loop > > Hope that's useful, > > Angus > -- > AJC McMorland > Post-doctoral research fellow > Neurobiology, University of Pittsburgh > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > maybe this does what you want >>> help(np.in1d) >>> valA = [1,2,3,4,20,21,22,23,24] >>> valB = [1,2,3,4, 5,21,22,23] >>> np.in1d(valA, valB) array([ True, True, True, True, False, True, True, True, False], dtype=bool) one of the next two should be correct (if you change the test case) >>> np.nonzero(np.in1d(valA, valB))[0] array([0, 1, 2, 3, 5, 6, 7]) >>> np.nonzero(np.in1d(valB, valA))[0] array([0, 1, 2, 3, 5, 6, 7]) Josef From jks at iki.fi Fri Feb 11 10:18:58 2011 From: jks at iki.fi (=?iso-8859-1?Q?Jouni_K=2E_Sepp=E4nen?=) Date: Fri, 11 Feb 2011 17:18:58 +0200 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu References: <20110210122200.1667933e@python.org> Message-ID: [Crossposting to matplotlib devel list] Robert Kern writes: > On Thu, Feb 10, 2011 at 11:22, Barry Warsaw wrote: > >> Here's the problem: for Ubuntu, we've had to disable the building of >> the numpy documentation package, because its dependencies violate >> Ubuntu policy. ?Numpy is in our "main" archive but the documentation >> depends on python-matplotlib, which lives in our "universe" >> archive. ?Such cross archive dependencies break the build. >> >> We can't put python-matplotlib in main because of *its* dependencies. > > As a digression, I think the python-matplotlib dependencies could be > significantly reduced. For a number of use cases (this is one of them, > but there are others), you don't need any GUI backend. Independent of > this issue, it would be great to be able to install python-matplotlib > in a headless server environment without pulling in all of those GUI > bits. Looking at the list of the hard dependencies, I don't understand > why half of them are there. > > http://packages.ubuntu.com/natty/python/python-matplotlib Would it make sense to split out each interactive backend to its own Ubuntu package, e.g. python-matplotlib-tk, etc? Each of these would depend on the relevant toolkit packages, and python-matplotlib would have a much shorter list of dependencies. -- Jouni K. Sepp?nen http://www.iki.fi/jks From barry at python.org Fri Feb 11 10:40:57 2011 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Feb 2011 10:40:57 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <1297376809.3224.9.camel@Obelisk> References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> Message-ID: <20110211104057.46393960@python.org> On Feb 10, 2011, at 11:26 PM, Pauli Virtanen wrote: >Without matplotlib, you won't get any graphs in the documentation. >Ubuntu users will then miss out e.g. on illustrations of various >probability distributions. The figures are not absolutely necessary, but >I wouldn't like to see that to happen just because of a packaging issue. Agreed that it's not ideal. We've landed my proposed workaround in Natty today, so at least folks will get some documentation. I would definitely like to find a better long-term solution. >How about splitting off the Numpy documentation to a separate source >package? That wouldn't need to get into main, as it's of use only to >developers. I like this the best, because it would allow us to resync with Debian, which AFAICT does not have the same cross-archive problem that Ubuntu has. (And which is why Ubuntu's workaround is not appropriate for Debian.) The ideal way to deal with this would be separate upstream (i.e. you guys :) tarball releases for the code and documentation. We'd then take those and split our current single source package into to separate source packages, with the code one going into main and the documentation one going into universe. For Debian, we'd simply take those two separate source packages and split the binary packages among them as appropriate. >Or, using prebuilt HTML documentation supplied by upstream or pre-built >by Ubuntu devs? Neither will be acceptable I think. Prebuilt by upstream won't fly for Debian because they'd want the source and build process, and I don't see a feasible way for them to be prebuilt by Ubuntu that's any different than the current build process (which fails because of the cross-archive issue). Do you think it would be feasible for numpy to produce two tarballs - one with just the code and the other with just the docs? Note that one of my other proposed workarounds was very similar to this (i.e. two separate source packages), but just that we'd use the same all-encompassing tarball for both. The disadvantage of that is that it diverges from Debian, which I'd like to avoid if at all possible. A split tarball approach from upstream would be easier to get into Debian, and thus keep us in sync. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Fri Feb 11 10:55:03 2011 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Feb 2011 10:55:03 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> Message-ID: <20110211105503.5b8b0bf5@python.org> Hi Robert, On Feb 10, 2011, at 04:39 PM, Robert Kern wrote: >As a digression, I think the python-matplotlib dependencies could be >significantly reduced. For a number of use cases (this is one of them, >but there are others), you don't need any GUI backend. Independent of >this issue, it would be great to be able to install python-matplotlib >in a headless server environment without pulling in all of those GUI >bits. Looking at the list of the hard dependencies, I don't understand >why half of them are there. > > http://packages.ubuntu.com/natty/python/python-matplotlib I'm not well versed enough in python-matplotlib's packaging to understand why the dependencies are what they are. Reducing dependencies is almost always good though, and I agree that the use case of being able to install it without the GUI components would be a win all around. Any insight into reducing those hard dependencies is appreciated! >Would "universe" packages that are merely "suggested" or "recommended" >also preclude python-matplotlib from moving into "main"? main is transitively closed under build-depends, depends, and recommends. suggests of universe packages isn't a problem, but recommends of universe will cause them to be listed for promotion to main. So I think that's problematic. >> I'm inclined to propose this as a permanent workaround, but I wanted to get >> your feedback first. ?Is there a better way to build the docs without >> matplotlib? ?What are the practical effects of my workaround? ?Do you think >> this will be an acceptable solution for Ubuntu users of numpy? > >We might be able to commit to building and releasing complete doc >packages with each source release. Could you just download and use >those? We currently make such doc packages, but I don't think we keep >old micro revisions around, and I suspect you would want those. > > http://docs.scipy.org/doc/numpy-1.5.x/numpy-html.zip As mentioned previously, I *think* ideally separate source tarballs for the code and docs would be ideal. I'd then work to create two different source packages, get those into Debian, and then we'd be able to sync those to Ubuntu. Our only divergence from Debian would then be to put the code package in main and the documentation package in universe. Do you think that would be feasible? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ben.root at ou.edu Fri Feb 11 10:56:41 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 11 Feb 2011 09:56:41 -0600 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> Message-ID: On Friday, February 11, 2011, Jouni K. Sepp?nen wrote: > [Crossposting to matplotlib devel list] > > Robert Kern writes: > >> On Thu, Feb 10, 2011 at 11:22, Barry Warsaw wrote: >> >>> Here's the problem: for Ubuntu, we've had to disable the building of >>> the numpy documentation package, because its dependencies violate >>> Ubuntu policy. ?Numpy is in our "main" archive but the documentation >>> depends on python-matplotlib, which lives in our "universe" >>> archive. ?Such cross archive dependencies break the build. >>> >>> We can't put python-matplotlib in main because of *its* dependencies. >> >> As a digression, I think the python-matplotlib dependencies could be >> significantly reduced. For a number of use cases (this is one of them, >> but there are others), you don't need any GUI backend. Independent of >> this issue, it would be great to be able to install python-matplotlib >> in a headless server environment without pulling in all of those GUI >> bits. Looking at the list of the hard dependencies, I don't understand >> why half of them are there. >> >> ? http://packages.ubuntu.com/natty/python/python-matplotlib > > Would it make sense to split out each interactive backend to its own > Ubuntu package, e.g. python-matplotlib-tk, etc? Each of these would > depend on the relevant toolkit packages, and python-matplotlib would > have a much shorter list of dependencies. > > -- > Jouni K. Sepp?nen > http://www.iki.fi/jks > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > There are a lot of advantages to this idea, and I wonder if it might make distributions easier and allow fuller control by the user. In particular, kubuntu could default to using the qt backend while regular ubuntu could use gym. However, how practical is this to implement? What does it require from us upstream? Ben Root From barry at python.org Fri Feb 11 10:58:10 2011 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Feb 2011 10:58:10 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> Message-ID: <20110211105810.2480fda8@python.org> On Feb 11, 2011, at 05:18 PM, Jouni K. Sepp?nen wrote: >[Crossposting to matplotlib devel list] > >Robert Kern writes: > >> On Thu, Feb 10, 2011 at 11:22, Barry Warsaw wrote: >> >>> Here's the problem: for Ubuntu, we've had to disable the building of >>> the numpy documentation package, because its dependencies violate >>> Ubuntu policy. ?Numpy is in our "main" archive but the documentation >>> depends on python-matplotlib, which lives in our "universe" >>> archive. ?Such cross archive dependencies break the build. >>> >>> We can't put python-matplotlib in main because of *its* dependencies. >> >> As a digression, I think the python-matplotlib dependencies could be >> significantly reduced. For a number of use cases (this is one of them, >> but there are others), you don't need any GUI backend. Independent of >> this issue, it would be great to be able to install python-matplotlib >> in a headless server environment without pulling in all of those GUI >> bits. Looking at the list of the hard dependencies, I don't understand >> why half of them are there. >> >> http://packages.ubuntu.com/natty/python/python-matplotlib > >Would it make sense to split out each interactive backend to its own >Ubuntu package, e.g. python-matplotlib-tk, etc? Each of these would >depend on the relevant toolkit packages, and python-matplotlib would >have a much shorter list of dependencies. Note that the main/universe distinction happens at the source package level, so we'd have to have separate source packages, ideally with different upstream tarballs. We could finesse that with two different source packages using the same upstream tarball (as suggested in a previous follow up), but I think it would be more difficult to get into Debian, thus precipitating a divergence. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ben.root at ou.edu Fri Feb 11 10:59:35 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 11 Feb 2011 09:59:35 -0600 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> Message-ID: On Friday, February 11, 2011, Benjamin Root wrote: > On Friday, February 11, 2011, Jouni K. Sepp?nen wrote: >> [Crossposting to matplotlib devel list] >> >> Robert Kern writes: >> >>> On Thu, Feb 10, 2011 at 11:22, Barry Warsaw wrote: >>> >>>> Here's the problem: for Ubuntu, we've had to disable the building of >>>> the numpy documentation package, because its dependencies violate >>>> Ubuntu policy. ?Numpy is in our "main" archive but the documentation >>>> depends on python-matplotlib, which lives in our "universe" >>>> archive. ?Such cross archive dependencies break the build. >>>> >>>> We can't put python-matplotlib in main because of *its* dependencies. >>> >>> As a digression, I think the python-matplotlib dependencies could be >>> significantly reduced. For a number of use cases (this is one of them, >>> but there are others), you don't need any GUI backend. Independent of >>> this issue, it would be great to be able to install python-matplotlib >>> in a headless server environment without pulling in all of those GUI >>> bits. Looking at the list of the hard dependencies, I don't understand >>> why half of them are there. >>> >>> ? http://packages.ubuntu.com/natty/python/python-matplotlib >> >> Would it make sense to split out each interactive backend to its own >> Ubuntu package, e.g. python-matplotlib-tk, etc? Each of these would >> depend on the relevant toolkit packages, and python-matplotlib would >> have a much shorter list of dependencies. >> >> -- >> Jouni K. Sepp?nen >> http://www.iki.fi/jks >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > There are a lot of advantages to this idea, and I wonder if it might > make distributions easier and allow fuller control by the user. ?In > particular, kubuntu could default to using the qt backend while > regular ubuntu could use gym. stupid autocorrect... Gym -> Gtk Ben Root > > However, how practical is this to implement? ?What does it require > from us upstream? > > Ben Root > From pav at iki.fi Fri Feb 11 11:07:25 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Feb 2011 17:07:25 +0100 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <20110211104057.46393960@python.org> References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> Message-ID: <1297440445.2884.117.camel@talisman> Fri, 11 Feb 2011 10:40:57 -0500, Barry Warsaw wrote: [clip] > Neither will be acceptable I think. Prebuilt by upstream won't fly > for Debian because they'd want the source and build process, and I > don't see a feasible way for them to be prebuilt by Ubuntu that's any > different than the current build process (which fails because of > the cross-archive issue). > > Do you think it would be feasible for numpy to produce two tarballs - > one with just the code and the other with just the docs? That's exactly what I meant by "pre-built by upstream". We currently *do* have separate HTML doc packages, built for each release; cf. http://docs.scipy.org/ In the future, we can ship the HTML doc packages in the same place as the source packages, so that they are easier to discover. *** @Ralf or someone else with sourceforge permissions: would you mind uploading http://docs.scipy.org/doc/numpy-1.5.x/numpy-html.zip as numpy-doc-1.5.1.zip (or .tar.gz) on sourceforge? -- Pauli Virtanen From pav at iki.fi Fri Feb 11 11:11:46 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Feb 2011 16:11:46 +0000 (UTC) Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> Message-ID: Fri, 11 Feb 2011 17:07:25 +0100, Pauli Virtanen wrote: > Fri, 11 Feb 2011 10:40:57 -0500, Barry Warsaw wrote: [clip] >> Neither will be acceptable I think. Prebuilt by upstream won't fly for >> Debian because they'd want the source and build process, and I don't >> see a feasible way for them to be prebuilt by Ubuntu that's any >> different than the current build process (which fails because of the >> cross-archive issue). >> >> Do you think it would be feasible for numpy to produce two tarballs - >> one with just the code and the other with just the docs? > > That's exactly what I meant by "pre-built by upstream". Sorry, I misunderstood what you meant. Yes, in principle it is possible to split out stuff from the numpy/doc to a separate numpy-doc source tarball containing. However, I don't see why you (Debian & Ubuntu) could not split the source package yourselves? -- Pauli Virtanen From bsouthey at gmail.com Fri Feb 11 12:34:16 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 11 Feb 2011 11:34:16 -0600 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> Message-ID: <4D557318.50305@gmail.com> On 02/11/2011 10:11 AM, Pauli Virtanen wrote: > Fri, 11 Feb 2011 17:07:25 +0100, Pauli Virtanen wrote: >> Fri, 11 Feb 2011 10:40:57 -0500, Barry Warsaw wrote: [clip] >>> Neither will be acceptable I think. Prebuilt by upstream won't fly for >>> Debian because they'd want the source and build process, and I don't >>> see a feasible way for them to be prebuilt by Ubuntu that's any >>> different than the current build process (which fails because of the >>> cross-archive issue). >>> >>> Do you think it would be feasible for numpy to produce two tarballs - >>> one with just the code and the other with just the docs? >> That's exactly what I meant by "pre-built by upstream". > Sorry, I misunderstood what you meant. > > Yes, in principle it is possible to split out stuff from the numpy/doc to > a separate numpy-doc source tarball containing. > > However, I don't see why you (Debian& Ubuntu) could not split the source > package yourselves? > The apparent problem occurs when the user has to build the documentation from source as sphinx and matlibplot are dependencies. However, this is not a bug but an user error of failing to install the necessary dependencies before building. I presume that in Ubuntu (I know I can do that in other distros) I can install some package source without installing the binary package or even a compiler so does Ubuntu consider it a bug that the source will not compile? I do not see why Ubuntu does not also provide the documentation since it already provides the numpy binary. Of course there is the issue of what type of documentation is provided but would fix/avoid the 'bug'. I agree with Pauli, if Ubuntu wants the user to be able build the documentation then Ubuntu would have to create a special package with just the numpy documentation that includes the necessary dependencies (including numpy, matlibplot and sphinx). Bruce From morph at debian.org Fri Feb 11 12:45:23 2011 From: morph at debian.org (Sandro Tosi) Date: Fri, 11 Feb 2011 18:45:23 +0100 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <20110211104057.46393960@python.org> References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> Message-ID: On Fri, Feb 11, 2011 at 16:40, Barry Warsaw wrote: > For Debian, we'd simply take those two separate source packages and split the > binary packages among them as appropriate. excuse me? I don't like this solution, or are you going to do all the work in Debian to adapt to this proposal? You have a problem in ubuntu? fix it there: take the full numpy upstream tarball, rename it in numpy-doc, build only the doc, ship it where it better fits for ubuntu policies, work done & no-one gets hurt :) >?A split > tarball approach from upstream would be easier to get into Debian, and thus > keep us in sync. ehm... you're speaking too fast here and without involving maintainers it seems. Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From sienkiew at stsci.edu Fri Feb 11 12:49:29 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Fri, 11 Feb 2011 12:49:29 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> Message-ID: <4D5576A9.7090609@stsci.edu> >> We can't put python-matplotlib in main because of *its* dependencies. >> > > As a digression, I think the python-matplotlib dependencies could be > significantly reduced. For a number of use cases (this is one of them, > but there are others), you don't need any GUI backend. Independent of > this issue, it would be great to be able to install python-matplotlib > in a headless server environment without pulling in all of those GUI > bits. Looking at the list of the hard dependencies, I don't understand > why half of them are there. > > http://packages.ubuntu.com/natty/python/python-matplotlib > This web page lists several "dependencies" that are optional. Just flipping through the list, I can see several packages that I know that I do not have, and several more that I have never heard of. "Never heard of" could mean that it is in my linux distro and I don't know it, but I am certain that I have machines that do not have cairo or gtk+ or qt. A matplotlib application selects one of the available backends to draw the graphics. If I remember correctly _all_ backends are optional in matplotlib, but there is at least one ("agg") that is available everywhere. When you install matplotlib, it builds support for any backends that it can. A backend that depends on a missing library does not get a C extension built. BUT the python code is still installed. The only way to know that a backend is not supported in this installation is to try to use it. For example, >>> import matplotlib.backends.backend_qt Traceback (most recent call last): File "", line 1, in File "/usr/stsci/pyssgdev/2.7/matplotlib/backends/backend_qt.py", line 19, in raise ImportError("Qt backend requires pyqt to be installed.") ImportError: Qt backend requires pyqt to be installed. >>> Ok - I don't have qt on this machine. So, you can try this: Get a build machine that has all the packages required by the various backends. Build a binary distribution of matplotlib. Install it on a machine that has only some of the libraries required by the backends. The result is a copy of matplotlib with _some_ working backends and some that fail because of missing libraries. As you install other supporting packages, additional backends can start working because their shared library is now present. So, if I install matplotlib and pyqt is not there I get a working matplotlib without QT support. If I use a package installer to install pyqt, presumably it will also install the QT libraries, resulting in matplotlib that does have qt support. I'm not saying you want to do this, but it is an option. If you want to experiment in this direction, there is a list that breaks out requirements for the core and requirements for each of the backends at http://matplotlib.sourceforge.net/users/installing.html . Mark S. From pav at iki.fi Fri Feb 11 12:50:49 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Feb 2011 17:50:49 +0000 (UTC) Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> <4D557318.50305@gmail.com> Message-ID: Fri, 11 Feb 2011 11:34:16 -0600, Bruce Southey wrote: [clip] > I agree with Pauli, if Ubuntu wants the user to be able build the > documentation then Ubuntu would have to create a special package with > just the numpy documentation that includes the necessary dependencies > (including numpy, matlibplot and sphinx). The point is that the Ubuntu build servers cannot build the documentation, because matplotlib is not available at build time. (First, because matplotlib depends circularly on Numpy, and second, since by policy Numpy build cannot depend on matplotlib as Numpy is in "main" and matplotlib in "universe".) I don't see other solutions apart from creating a separate source package from the contents of numpy/doc/*. However, I don't see very many reasons why we (as the Numpy upstream) should make this split, and why the Debian/Ubuntu packagers could not do it. -- Pauli Virtanen From robert.kern at gmail.com Fri Feb 11 12:48:33 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 11 Feb 2011 11:48:33 -0600 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <4D557318.50305@gmail.com> References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> <4D557318.50305@gmail.com> Message-ID: On Fri, Feb 11, 2011 at 11:34, Bruce Southey wrote: > The apparent problem occurs when the user has to build the documentation > from source as sphinx and matlibplot are dependencies. No, please reread the original post. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From bsouthey at gmail.com Fri Feb 11 13:06:42 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 11 Feb 2011 12:06:42 -0600 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> <4D557318.50305@gmail.com> Message-ID: <4D557AB2.4070608@gmail.com> On 02/11/2011 11:50 AM, Pauli Virtanen wrote: > Fri, 11 Feb 2011 11:34:16 -0600, Bruce Southey wrote: > [clip] >> I agree with Pauli, if Ubuntu wants the user to be able build the >> documentation then Ubuntu would have to create a special package with >> just the numpy documentation that includes the necessary dependencies >> (including numpy, matlibplot and sphinx). > The point is that the Ubuntu build servers cannot build the > documentation, because matplotlib is not available at build time. (First, > because matplotlib depends circularly on Numpy, and second, since by > policy Numpy build cannot depend on matplotlib as Numpy is in "main" and > matplotlib in "universe".) If Ubuntu's different repositories are a problem then that is Ubuntu's own fault for putting the numpy and matplotlib in different repositories (that applies to other distros as well). Clearly someone did not do their homework before doing so. However, I do not know that Ubuntu's numpy package also has the sphinx dependency that is also required here. But I am sure that you would take patches from Ubuntu that allowed the building of documentation without other dependencies than numpy and sphinx. > I don't see other solutions apart from creating a separate source package > from the contents of numpy/doc/*. However, I don't see very many reasons > why we (as the Numpy upstream) should make this split, and why the > Debian/Ubuntu packagers could not do it. > But that directory is still not sufficient for having the documentation alone. Bruce PS Yes, Robert and I also read the actual bug report which indicated what the real problem is. From pav at iki.fi Fri Feb 11 13:23:59 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Feb 2011 18:23:59 +0000 (UTC) Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> <4D557318.50305@gmail.com> <4D557AB2.4070608@gmail.com> Message-ID: Fri, 11 Feb 2011 12:06:42 -0600, Bruce Southey wrote: [clip: numpy/doc/*] > But that directory is still not sufficient for having the documentation > alone. It is, given a build-time dependency on the main Numpy package. Pauli From faltet at pytables.org Fri Feb 11 14:21:43 2011 From: faltet at pytables.org (Francesc Alted) Date: Fri, 11 Feb 2011 20:21:43 +0100 Subject: [Numpy-discussion] ANN: carray 0.4 released Message-ID: <201102112021.43820.faltet@pytables.org> Hi, A new release of my new toy project is out. Enjoy! Announcing carray 0.4 ===================== What's new ---------- The most prominent feature in 0.4 is the support of multidimensional carrays. Than means that, for example, you can do:: >>> a = ca.arange(6).reshape((2,3)) Now, you can access any element in any dimension:: >>> a[:] array([[0, 1, 2], [3, 4, 5]]) >>> a[1] array([3, 4, 5]) >>> a[1,::2] array([3, 5]) >>> a[1,1] 4 Also, all the iterators in carray have received a couple of new parameters that allows to `limit` or `skip` selected elements in queries. Finally, many performance improvements have been implemented (mainly related with efficient zero-detection code). This allows for improved query times when using iterators. See: https://github.com/FrancescAlted/carray/wiki/query-compress for an example on how fast the new iterators can do. For more detailed info, see the release notes in: https://github.com/FrancescAlted/carray/wiki/Release-0.4 What it is ---------- carray is a chunked container for numerical data. Chunking allows for efficient enlarging/shrinking of data container. In addition, it can also be compressed for reducing memory needs. The compression process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. carray comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Resources --------- Visit the main carray site repository at: http://github.com/FrancescAlted/carray You can download a source package from: http://carray.pytables.org/download Manual: http://carray.pytables.org/docs/manual Home of Blosc compressor: http://blosc.pytables.org User's mail list: carray at googlegroups.com http://groups.google.com/group/carray -- Francesc Alted From barry at python.org Fri Feb 11 14:39:22 2011 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Feb 2011 14:39:22 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> Message-ID: <20110211143922.540d83a5@limelight.wooz.org> On Feb 11, 2011, at 06:45 PM, Sandro Tosi wrote: >On Fri, Feb 11, 2011 at 16:40, Barry Warsaw wrote: >> For Debian, we'd simply take those two separate source packages and split the >> binary packages among them as appropriate. > >excuse me? I don't like this solution, or are you going to do all the >work in Debian to adapt to this proposal? I would of course take on the work to do the packaging and propose it to the Debian maintainers. >You have a problem in ubuntu? fix it there: take the full numpy >upstream tarball, rename it in numpy-doc, build only the doc, ship it >where it better fits for ubuntu policies, work done & no-one gets hurt >:) Colin Watson suggested exactly this. It would still mean that Ubuntu and Debian would diverge though, which I'd think everyone would like to avoid. >>?A split >> tarball approach from upstream would be easier to get into Debian, and thus >> keep us in sync. > >ehm... you're speaking too fast here and without involving maintainers it seems. Right, I didn't mean to speak for Debian, but was looking for a solution that is feasible and acceptable to upstream too. It's a complex stack to manage, and I wanted to get upstream's opinion first, since "it ain't broke" in Debian. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Fri Feb 11 14:47:30 2011 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Feb 2011 14:47:30 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> <4D557318.50305@gmail.com> Message-ID: <20110211144730.2c51a304@limelight.wooz.org> On Feb 11, 2011, at 05:50 PM, Pauli Virtanen wrote: >The point is that the Ubuntu build servers cannot build the >documentation, because matplotlib is not available at build time. (First, >because matplotlib depends circularly on Numpy, and second, since by >policy Numpy build cannot depend on matplotlib as Numpy is in "main" and >matplotlib in "universe".) Exactly so. >I don't see other solutions apart from creating a separate source package >from the contents of numpy/doc/*. However, I don't see very many reasons >why we (as the Numpy upstream) should make this split, and why the >Debian/Ubuntu packagers could not do it. I think what we would do is just build a new source package that refers to the same upstream tarball as the code source package. We'd just move the documentation building rules to this new source package, remove them from the code package and stick the doc package in universe. This would require no change for either upstream or Debian, and would solely be in Ubuntu. The downside is that it probably means Ubuntu would more or less permanently diverge from Debian for numpy, which would be unfortunate for many reasons. I can certainly appreciate that from both upstream's and Debian's perspective, nothing is broken here and that Ubuntu must adjust. If that's the best we can do, so be it, but I'm hoping there is a solution out there that would allow us to avoid either a divergence or providing Ubuntu users with a less than complete documentation package. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Fri Feb 11 14:48:53 2011 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Feb 2011 14:48:53 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <4D557AB2.4070608@gmail.com> References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> <4D557318.50305@gmail.com> <4D557AB2.4070608@gmail.com> Message-ID: <20110211144853.5ba80a75@limelight.wooz.org> On Feb 11, 2011, at 12:06 PM, Bruce Southey wrote: >If Ubuntu's different repositories are a problem then that is Ubuntu's >own fault for putting the numpy and matplotlib in different repositories >(that applies to other distros as well). Clearly someone did not do >their homework before doing so. However, I do not know that Ubuntu's >numpy package also has the sphinx dependency that is also required here. >But I am sure that you would take patches from Ubuntu that allowed the >building of documentation without other dependencies than numpy and sphinx. The sphinx dependency isn't a problem because it is in main. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From robert.kern at gmail.com Fri Feb 11 14:57:33 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 11 Feb 2011 13:57:33 -0600 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <20110211144730.2c51a304@limelight.wooz.org> References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> <4D557318.50305@gmail.com> <20110211144730.2c51a304@limelight.wooz.org> Message-ID: On Fri, Feb 11, 2011 at 13:47, Barry Warsaw wrote: > The downside is that it probably means Ubuntu would more or less permanently > diverge from Debian for numpy, which would be unfortunate for many reasons. ?I > can certainly appreciate that from both upstream's and Debian's perspective, > nothing is broken here and that Ubuntu must adjust. ?If that's the best we can > do, so be it, but I'm hoping there is a solution out there that would allow us > to avoid either a divergence or providing Ubuntu users with a less than > complete documentation package. A python-matplotlib-nogui in main? ;-) If you are going to do an Ubuntu-only fix, would that allow using our prebuilt doc packages? IIRC, the only objection to that was that it was Debian policy to build docs from sources. But then again, that might have just been the first objection. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Sat Feb 12 05:02:56 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 12 Feb 2011 18:02:56 +0800 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <1297440445.2884.117.camel@talisman> References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> Message-ID: On Sat, Feb 12, 2011 at 12:07 AM, Pauli Virtanen wrote: > Fri, 11 Feb 2011 10:40:57 -0500, Barry Warsaw wrote: > [clip] > > Neither will be acceptable I think. Prebuilt by upstream won't fly > > for Debian because they'd want the source and build process, and I > > don't see a feasible way for them to be prebuilt by Ubuntu that's any > > different than the current build process (which fails because of > > the cross-archive issue). > > > > Do you think it would be feasible for numpy to produce two tarballs - > > one with just the code and the other with just the docs? > > That's exactly what I meant by "pre-built by upstream". We currently > *do* have separate HTML doc packages, built for each release; cf. > http://docs.scipy.org/ > > In the future, we can ship the HTML doc packages in the same place as > the source packages, so that they are easier to discover. > > *** > > @Ralf or someone else with sourceforge permissions: would you mind > uploading > > http://docs.scipy.org/doc/numpy-1.5.x/numpy-html.zip > > as numpy-doc-1.5.1.zip (or .tar.gz) on sourceforge? > > Done, as numpy-doc-html-1.5.1.zip. I understand it's not what Barry wanted, but useful to do now and in the future in any case. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Feb 12 05:12:54 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 12 Feb 2011 18:12:54 +0800 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: <20110210122200.1667933e@python.org> References: <20110210122200.1667933e@python.org> Message-ID: Hi Barry, On Fri, Feb 11, 2011 at 1:22 AM, Barry Warsaw wrote: > I hope this message is on-topic for this mailing list! > > I'm working on the packaging for python-numpy 1.5 in the next version of > Ubuntu (11.04 - Natty), and I'm hitting an issue that I'm hoping you can > help > me with. > > Now that you're here to discuss Ubuntu packaging, can you tell us if you or another packager plan to include scipy 0.9.0 in Natty if it's released on time? Final release is planned for the weekend of 19/20 Feb. Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sat Feb 12 10:38:11 2011 From: sturla at molden.no (Sturla Molden) Date: Sat, 12 Feb 2011 16:38:11 +0100 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: Message-ID: <4D56A963.9020002@molden.no> Den 10.02.2011 16:29, skrev eat: > One would expect sum to outperform dot with a clear marginal. Does > there exixts any 'tricks' to increase the performance of sum? I see that others have ansvered already. The ufunc np.sum is not going going to beat np.dot. You are racing the heavy machinery of NumPy (array iterators, type chekcs, bound checks, etc.) against level-3 BLAS routine DGEMM, the most heavily optimized numerical kernel ever written. Also beware that computation is much cheaper than memory access. Although DGEMM does more arithmetics, and even is O(N3) in that respect, it is always faster except for very sparse arrays. If you need fast loops, you can always write your own Fortran or C, and even insert OpenMP pragmas. But don't expect that to beat optimized high-level BLAS kernels by any margin. The first chapters of "Numerical Methods in Fortran 90" might be worth reading. It deals with several of these issues, including dimensional expansion, which is important for writing fast numerical code -- but not intuitively obvious. "I expect this to be faster because it does less work" is a fundamental misconception in numerical computing. Whatever cause less traffic on the memory BUS (the real bottleneck) will almost always be faster, regardless of the amount of work done by the CPU. A good advice is to use high-level BLAS whenever you can. The only exception, as mentioned, is when matrices get very sparse. Sturla From ecarlson at eng.ua.edu Sat Feb 12 15:19:39 2011 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Sat, 12 Feb 2011 14:19:39 -0600 Subject: [Numpy-discussion] Python ctypes and OpenMP mystery Message-ID: Hello All, I have been toying with OpenMP through f2py and ctypes. On the whole, the results of my efforts have been very encouraging. That said, some results are a bit perplexing. I have written identical routines that I run directly as a C-derived executable, and through ctypes as a shared library. I am running the tests on a dual-Xeon Ubuntu system with 12 cores and 24 threads. The C executable is SLIGHTLY faster than the ctypes at lower thread counts, but the C eventually has a speedup ratio of 12+, while the python caps off at 7.7, as shown below: threads C-speedup Python-speedup 1 1 1 2 2.07 1.98 3 3.1 2.96 4 4.11 3.93 5 4.97 4.75 6 5.94 5.54 7 6.83 6.53 8 7.78 7.3 9 8.68 7.68 10 9.62 7.42 11 10.38 7.51 12 10.44 7.26 13 7.19 6.04 14 7.7 5.73 15 8.27 6.03 16 8.81 6.29 17 9.37 6.55 18 9.9 6.67 19 10.36 6.9 20 10.98 7.01 21 11.45 6.97 22 11.92 7.1 23 12.2 7.08 These ratios are quite consistent from 100KB double arrays to 100MB double arrays, so I do not think it reflects a Python overhead issue. There is no question the routine is memory bandwidth constrained, and I feel lucky to squeeze the eventual 12+ ratio, but I am very perplexed as to why the performance of the Python-invoked routine seems to cap off. Does anyone have an explanation for the caps? Am I seeing some effect from ctypes, or the Python engine, or what? Cheers, Eric From neilcrighton at gmail.com Sat Feb 12 17:22:47 2011 From: neilcrighton at gmail.com (Neil) Date: Sat, 12 Feb 2011 22:22:47 +0000 (UTC) Subject: [Numpy-discussion] in1d enhancement ticket Message-ID: Hi, If someone with commit access has the chance, could they take a look at ticket 1603: http://projects.scipy.org/numpy/ticket/1603 and apply it if it looks ok? It speeds up in1d(a, b) a lot for the common use case where len(b) << len(a). Thanks, Neil From e.antero.tammi at gmail.com Sat Feb 12 19:02:03 2011 From: e.antero.tammi at gmail.com (eat) Date: Sun, 13 Feb 2011 02:02:03 +0200 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: <4D56A963.9020002@molden.no> References: <4D56A963.9020002@molden.no> Message-ID: Hi Sturla, On Sat, Feb 12, 2011 at 5:38 PM, Sturla Molden wrote: > Den 10.02.2011 16:29, skrev eat: > > One would expect sum to outperform dot with a clear marginal. Does > > there exixts any 'tricks' to increase the performance of sum? > First of all, thanks for you still replying. Well, I'm still a little bit unsure how I should proceed with this discussion... I may have used bad wordings and created unneccessary mayhyem with my original question (:. Trust me, I'm only trying to discuss here with a positive criticism in my mind. Now, I'm not pretending to know what kind of a person a 'typical' numpy user is. But I'm assuming that there just exists more than me with roughly similar questions in their (our) minds and who wish to utilize numpy more 'pythonic; all batteries included' way. Ocassionally I (we) may ask really stupid questions, but please beare with us. Said that, I'm still very confident that (from a users point of view) there's some real substance on the issue I addressed. > I see that others have ansvered already. The ufunc np.sum is not going > going to beat np.dot. You are racing the heavy machinery of NumPy (array > iterators, type chekcs, bound checks, etc.) against level-3 BLAS routine > DGEMM, the most heavily optimized numerical kernel ever written. Fair enough. > Also > beware that computation is much cheaper than memory access. Sure, that's exactly where I expected the performance boost to emerge. > Although > DGEMM does more arithmetics, and even is O(N3) in that respect, it is > always faster except for very sparse arrays. If you need fast loops, you > can always write your own Fortran or C, and even insert OpenMP pragmas. That's a very important potential, but surely not all numpy users are expected to master that ;-) > But don't expect that to beat optimized high-level BLAS kernels by any > margin. The first chapters of "Numerical Methods in Fortran 90" might be > worth reading. It deals with several of these issues, including > dimensional expansion, which is important for writing fast numerical > code -- but not intuitively obvious. "I expect this to be faster because > it does less work" is a fundamental misconception in numerical > computing. Whatever cause less traffic on the memory BUS (the real > bottleneck) will almost always be faster, regardless of the amount of > work done by the CPU. And I'm totally aware of it, and actually it was exactly the original intended logic of my question: "how bout if the sum could follow the steps of dot; then, since less instructions it must be bounded below of the execution time of dot". But as R. Kern gently pointed out allready it's not fruitfull enough avenue to proceed. And I'm able to live with that. Regards, eat > A good advice is to use high-level BLAS whenever > you can. The only exception, as mentioned, is when matrices get very > sparse. > > Sturla > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Sat Feb 12 22:30:47 2011 From: oliphant at enthought.com (Travis Oliphant) Date: Sat, 12 Feb 2011 21:30:47 -0600 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: <4D56A963.9020002@molden.no> Message-ID: I think everybody would be happy if .sum() were as fast as possible, and patches for review are always welcome. There are distinct things that could be done to improve the performance of ufuncs and their methods as several people have shown. In fact, there is a lot of low-hanging fruit inside of NumPy for optimization. One of the advantages of an open source community is that different needs get addressed by the people who need them. So, if you really need a faster sum and are willing to do the hard work to make it happen (or convince someone else to do it for you), then there will be many to say thank you when you are finished. You should know though that in my experience It is harder than you might think at first to "convince someone else to do it for you." Best regards, -Travis On Feb 12, 2011, at 6:02 PM, eat wrote: > Hi Sturla, > > On Sat, Feb 12, 2011 at 5:38 PM, Sturla Molden wrote: > Den 10.02.2011 16:29, skrev eat: > > One would expect sum to outperform dot with a clear marginal. Does > > there exixts any 'tricks' to increase the performance of sum? > First of all, thanks for you still replying. Well, I'm still a little bit unsure how I should proceed with this discussion... I may have used bad wordings and created unneccessary mayhyem with my original question (:. Trust me, I'm only trying to discuss here with a positive criticism in my mind. > > Now, I'm not pretending to know what kind of a person a 'typical' numpy user is. But I'm assuming that there just exists more than me with roughly similar questions in their (our) minds and who wish to utilize numpy more 'pythonic; all batteries included' way. Ocassionally I (we) may ask really stupid questions, but please beare with us. > > Said that, I'm still very confident that (from a users point of view) there's some real substance on the issue I addressed. > I see that others have ansvered already. The ufunc np.sum is not going > going to beat np.dot. You are racing the heavy machinery of NumPy (array > iterators, type chekcs, bound checks, etc.) against level-3 BLAS routine > DGEMM, the most heavily optimized numerical kernel ever written. > Fair enough. > Also > beware that computation is much cheaper than memory access. > Sure, that's exactly where I expected the performance boost to emerge. > Although > DGEMM does more arithmetics, and even is O(N3) in that respect, it is > always faster except for very sparse arrays. If you need fast loops, you > can always write your own Fortran or C, and even insert OpenMP pragmas. > That's a very important potential, but surely not all numpy users are expected to master that ;-) > But don't expect that to beat optimized high-level BLAS kernels by any > margin. The first chapters of "Numerical Methods in Fortran 90" might be > worth reading. It deals with several of these issues, including > dimensional expansion, which is important for writing fast numerical > code -- but not intuitively obvious. "I expect this to be faster because > it does less work" is a fundamental misconception in numerical > computing. Whatever cause less traffic on the memory BUS (the real > bottleneck) will almost always be faster, regardless of the amount of > work done by the CPU. > And I'm totally aware of it, and actually it was exactly the original intended logic of my question: "how bout if the sum could follow the steps of dot; then, since less instructions it must be bounded below of the execution time of dot". But as R. Kern gently pointed out allready it's not fruitfull enough avenue to proceed. And I'm able to live with that. > > > Regards, > eat > A good advice is to use high-level BLAS whenever > you can. The only exception, as mentioned, is when matrices get very sparse. > > Sturla > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.pinto at gmail.com Sun Feb 13 04:46:44 2011 From: nicolas.pinto at gmail.com (Nicolas Pinto) Date: Sun, 13 Feb 2011 04:46:44 -0500 Subject: [Numpy-discussion] MKL libraries can't be found In-Reply-To: References: <91254948-7598-4793-9055-80EC9D427DD6@Princeton.EDU> <08A709B5-3F86-4B28-8E9E-99D711024A62@princeton.edu> Message-ID: Hey Jonathan, Not sure if it will help, but here is how I got my MKL install working: cd ~/install virtualenv numpy_mkl cd numpy_mkl source bin/activate wget http://downloads.sourceforge.net/project/numpy/NumPy/1.5.1/numpy-1.5.1.tar.gz tar xzvf numpy-1.5.1.tar.gz cd numpy-1.5.1 cat << EOF > site.cfg [DEFAULT] library_dirs = /usr/lib include_dirs = /usr/include# numpy's configuration on resonance [DEFAULT] library_dirs = /usr/lib include_dirs = /usr/include [fftw] libraries = fftw3 [mkl] library_dirs = /opt/intel/Compiler/11.1/073/mkl/lib/em64t:/opt/intel/Compiler/11.1/073/lib/intel64 include_dirs = /opt/intel/Compiler/11.1/073/mkl/include:/opt/intel/Compiler/11.1/073/include lapack_libs = mkl_lapack mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core, iomp5, guide, irc, mkl_mc, mkl_def EOF wget http://projects.scipy.org/numpy/raw-attachment/ticket/993/system_info.patch patch numpy/distutils/system_info.py < system_info.patch cat > intelccompiler.patch << EOF --- numpy/distutils/intelccompiler.py 2010-11-08 17:58:24.000000000 -0600 +++ numpy/distutils/intelccompiler.py.patched 2011-02-13 02:09:24.247386568 -0600 @@ -8,7 +8,7 @@ """ compiler_type = 'intel' - cc_exe = 'icc' + cc_exe = 'icc -fPIC -O3 -mmse4 -fast' def __init__ (self, verbose=0, dry_run=0, force=0): UnixCCompiler.__init__ (self, verbose,dry_run, force) EOF patch -p0 < intelccompiler.patch python setup.py config python setup.py build_ext --compiler=intel --fcompiler=intel install --prefix=~/local HTH Best, N On Wed, Jan 19, 2011 at 5:24 PM, Jonathan Tu wrote: > Hi, > > I am trying to install numpy with the MKL libraries available on my cluster. ?Most of the libraries are available in one directory, but the iomp5 library is in another. > > /opt/intel/Compiler/11.1/072/mkl/lib/em64t ---> mkl_intel_lp64, mkl_intel_thread, mkl_core, mkl_def, mkl_mc > /opt/intel/Compiler/11.1/072/lib/intel64 ---> iomp5 > > Using an older MKL library that was available, I found that when all libraries are in one directory, the install went through fine. ?But in this case it says the libraries cannot be found, even if I list both under the library_dirs in site.cfg > > [mkl] > library_dirs = /opt/intel/Compiler/11.1/072/mkl/lib/em64t:/opt/intel/Compiler/11.1/072/lib/intel64 > include_dirs = /opt/intel/Compiler/11.1/072/mkl/include > lapack_libs = mkl_lapack > mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, mkl_def, mkl_mc, iomp5 > > If I try to install without iomp5, then when I import numpy I get the following error > > /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_thread.so: undefined symbol: omp_in_parallel > > Any ideas? ?I tried to put symbolic links to both library directories in one place, but that didn't work either. ?I'm trying to avoid creating a directory of symbolic links to every necessary library. > > > > > Jonathan Tu > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Nicolas Pinto http://web.mit.edu/pinto From gginiu at gmail.com Sun Feb 13 09:01:34 2011 From: gginiu at gmail.com (Andrzej Giniewicz) Date: Sun, 13 Feb 2011 15:01:34 +0100 Subject: [Numpy-discussion] mkl 10.3 linking - undefined symbol: i_free? Message-ID: Hello, I'd like to ask if anyone got around the undefined symbol i_free issue? What I did was that I used link advisor from file:///home/giniu/Downloads/MKL_Linking_Adviser-1.03.htm and it told me to use -L$MKLPATH $MKLPATH/libmkl_solver_lp64.a -Wl,--start-group -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -Wl,--end-group -fopenmp -lpthread So I created site.cfg like: library_dirs = /opt/intel/composerxe-2011.2.137/mkl/lib/intel64 include_dirs = /opt/intel/composerxe-2011.2.137/mkl/include mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core lapack_libs = mkl_lapack95_lp64 and added -fopenmp to LDFLAGS. Numpy built, then I was able to install. I started python and imported numpy, and till this point it worked. When I tried to run numpy.test() and got: Running unit tests for numpy NumPy version 1.5.1 NumPy is installed in /usr/lib/python2.7/site-packages/numpy Python version 2.7.1 (r271:86832, Feb 7 2011, 19:39:54) [GCC 4.5.2 20110127 (prerelease)] nose version 1.0.0 . *** libmkl_def.so *** failed with error : /opt/intel/composerxe-2011.2.137/mkl/lib/intel64/libmkl_def.so: undefined symbol: i_free MKL FATAL ERROR: Cannot load libmkl_def.so Anyone who got this working could give me some hint about how to get around it? I would be grateful Cheers, Andrzej. From matthieu.brucher at gmail.com Sun Feb 13 09:07:51 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sun, 13 Feb 2011 15:07:51 +0100 Subject: [Numpy-discussion] mkl 10.3 linking - undefined symbol: i_free? In-Reply-To: References: Message-ID: Hi, This pops up regularly here, you can search with Google and find this page: http://matt.eifelle.com/2008/11/03/i-used-the-latest-mkl-with-numpy-and/ Matthieu 2011/2/13 Andrzej Giniewicz > Hello, > > I'd like to ask if anyone got around the undefined symbol i_free > issue? What I did was that I used link advisor from > file:///home/giniu/Downloads/MKL_Linking_Adviser-1.03.htm and it told > me to use > > -L$MKLPATH $MKLPATH/libmkl_solver_lp64.a -Wl,--start-group > -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -Wl,--end-group -fopenmp > -lpthread > > So I created site.cfg like: > > library_dirs = /opt/intel/composerxe-2011.2.137/mkl/lib/intel64 > include_dirs = /opt/intel/composerxe-2011.2.137/mkl/include > mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core > lapack_libs = mkl_lapack95_lp64 > > and added -fopenmp to LDFLAGS. Numpy built, then I was able to > install. I started python and imported numpy, and till this point it > worked. When I tried to run numpy.test() and got: > > Running unit tests for numpy > NumPy version 1.5.1 > NumPy is installed in /usr/lib/python2.7/site-packages/numpy > Python version 2.7.1 (r271:86832, Feb 7 2011, 19:39:54) [GCC 4.5.2 > 20110127 (prerelease)] > nose version 1.0.0 > . > > *** libmkl_def.so *** failed with error : > /opt/intel/composerxe-2011.2.137/mkl/lib/intel64/libmkl_def.so: > undefined symbol: i_free > MKL FATAL ERROR: Cannot load libmkl_def.so > > Anyone who got this working could give me some hint about how to get > around it? I would be grateful > > Cheers, > Andrzej. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sun Feb 13 11:11:35 2011 From: sturla at molden.no (Sturla Molden) Date: Sun, 13 Feb 2011 17:11:35 +0100 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: <4D56A963.9020002@molden.no> Message-ID: <4D5802B7.7050000@molden.no> Den 13.02.2011 01:02, skrev eat: > Now, I'm not pretending to know what kind of a person a 'typical' > numpy user is. But I'm assuming that there just exists more than me > with roughly similar questions in their (our) minds and who wish to > utilize numpy more 'pythonic; all batteries included' way. > Ocassionally I (we) may ask really stupid questions, but please beare > with us. It was not a stupid question. It is not intuitive that N multiplications and N additions can be faster than just N additions. > If you need fast loops, you > > can always write your own Fortran or C, and even insert OpenMP > pragmas. > > That's a very important potential, but surely not all numpy users are > expected to master that ;-) Anyone who is serious about scintific computing should spend some time to learn Fortran 95, C, or both. Fortran is easier to learn and use, and has arrays like NumPy. C is more portable, but no so well suited for numerical computing. One of the strengths of NumPy, compared to e.g. Matlab, is the easy integration with C and Fortran libraries using tools like ctypes, f2py, Cython, or Swig. If you have ever tried to write a MEX file for Matlab, you'll appreciate NumPy. It is also a big strength to learn to use certain numerical libraries, like BLAS and LAPACK (e.g. Intel MKL and AMD ACML), IMSL, FFTW, NAG, et al., to avoid reinventing the wheel. Most of the numerical processing in scientific computing is done by these libraries, so even learning to use them and call them from Python (e.g. with ctypes) is a very useful skill. Premature optimization might be the root of all evil in computer programming, but reinventing the wheel is the root of all evil in scienfic computing. People far to often resort to C or Fortran, when they should just have called a (well known) library function. But apart from that, knowing C and/or Fortran is a too important skill to ignore. For example, just knowing about C and Fortran data types and calling conventions makes using existing libraries with ctypes or f2py easier. Go ahead and learn it, you will not regret it. (What you probably will regret, is wasting your time on learning Fortran i/o or C++. We have Python for that.) Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sun Feb 13 12:46:54 2011 From: sturla at molden.no (Sturla Molden) Date: Sun, 13 Feb 2011 18:46:54 +0100 Subject: [Numpy-discussion] odd performance of sum? In-Reply-To: References: <4D56A963.9020002@molden.no> Message-ID: <4D58190E.8090209@molden.no> Den 13.02.2011 04:30, skrev Travis Oliphant: > > One of the advantages of an open source community is that different > needs get addressed by the people who need them. So, if you really > need a faster sum and are willing to do the hard work to make it > happen (or convince someone else to do it for you), then there will be > many to say thank you when you are finished. You should know though > that in my experience It is harder than you might think at first to > "convince someone else to do it for you." For things like sum (and may other ufuncs), one could just use Fortran instrincs and let the Fortran compiler do the rest. (Obviously NumPy should not depend on a Fortran compiler, unlike SciPy, but that is another matter.) Assuming a wrapper takes care of C and Fortran ordering, we can just f2py this: subroutine fortran_sum_1d( m, arr, out ) use, intrinsic :: sum integer :: m real :: arr(m), out out = sum(arr) end subroutine subroutine fortran_sum_2d( m, n, arr, k, out, dim ) use, intrinsic :: sum integer :: m, n, k, dim real :: arr(m,n), out(k) out = sum(arr, dim+1) ! fortran dims start at 1 end subroutine Similar things can be done for other ufuncs. We can also use Fortran's "elemental" keyword to create our own. The Fortran compilers from Absoft and Intel (and possibly Portland) can even take code like this and make it efficient for multicore CPUs. (I don't know if GNU gfortran can do this too, but I don't expect it too.) One limitation is that the Fortran compiler must know the number of dimensions, while NumPy arrays are flexible in this respect. Another limitation is strides, but we can deal with them if they are a multiple of the dtype size, but not arbitrary number of bytes like NumPy. subroutine fortran_sum_1d_with_strides( m, s, arr, out ) use, intrinsic :: sum integer :: m, s real, target :: arr(m) real, pointer, dimension(:) :: parr real :: out parr => arr(::s) out = sum(parr) end subroutine I think for most people, Fortran is much easier than using NumPy's C API. Creating fast "ufunc" like functions is very easy with Fortran 95. If we e.g. want the L2 norm of a vector, it looks like this in Fortran: subroutine l2norm( m, arr, out ) use, intrinsic :: sum, sqrt integer :: m real :: arr(m), out out = sqrt(sum(arr**2)) ! looks quite like Python... end subroutine Just notice the messyness of C in comparison (there are much worse examples): #include #include #include double l2norm(PyArrayObject *arr) { double acc = 0.0, *y; char *addr = arr->data; for (int i=0; idimensions[0]; i++) { y = (double *)addr; acc += y*y; addr += arr->strides[0]; } return sqrt(acc); } Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sun Feb 13 20:35:38 2011 From: cournape at gmail.com (David Cournapeau) Date: Mon, 14 Feb 2011 10:35:38 +0900 Subject: [Numpy-discussion] FYI: experimental waf support for numpy In-Reply-To: References: <4D5377FD.4000005@silveregg.co.jp> Message-ID: On Thu, Feb 10, 2011 at 10:06 PM, Pauli Virtanen wrote: > Thu, 10 Feb 2011 14:30:37 +0900, David wrote: > [clip] >> Following recent release of waf 1.6 and its adoption by the samba >> project, as well as my own work on integrating waf and bento, I have >> spent some time to build numpy with it. Although this is experimental, >> it should be possible for people to play with it: >> >> https://github.com/cournape/numpy/tree/waf_build > > Looks great to me, and it even seems to work :) not sure how I should take the "it even seems to work" :) > The structure of the > build appears to be more easy to understand than in the Scons-based > system, not to speak of distutils the beauty of it is that waf itself is not much bigger than numpy.distutils itself in terms of amount of code, and while "opiniated", the waf's codebase is pretty readable, cheers, David From brockp at umich.edu Mon Feb 14 11:01:00 2011 From: brockp at umich.edu (Brock Palen) Date: Mon, 14 Feb 2011 11:01:00 -0500 Subject: [Numpy-discussion] NumPy Featured on RCE-Cast Message-ID: <012BCDB6-8C99-47D8-9A1B-E02E8C776637@umich.edu> Thank you to Travis for taking time to speak about NumPy on rce-cast.com You can find the interview and links to other shows at: http://www.rce-cast.com/Podcast/rce-48-numpy.html Thank you! Brock Palen www.umich.edu/~brockp Center for Advanced Computing brockp at umich.edu (734)936-1985 From stefan at sun.ac.za Mon Feb 14 11:28:34 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 14 Feb 2011 18:28:34 +0200 Subject: [Numpy-discussion] using loadtxt() for given number of rows? In-Reply-To: References: Message-ID: Hi Robert On Mon, Jan 31, 2011 at 2:39 PM, Robert Cimrman wrote: > It seems to me, that an additional parameter to loadtxt(), say "nrows" or > "numrows", would do the job, so that the function does not try reading the > entire file. Another possibility would be to raise an exception as it > is now, but also to return the data succesfully read so far. You can always read chunks of the file into StringIO objects, and pass those into loadtxt. I think your request makes sense though, given that a person can already skip lines at the top of the file. Regards St?fan From Chris.Barker at noaa.gov Mon Feb 14 11:41:17 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 14 Feb 2011 08:41:17 -0800 Subject: [Numpy-discussion] using loadtxt() for given number of rows? In-Reply-To: References: Message-ID: <4D595B2D.2080506@noaa.gov> On 2/14/11 8:28 AM, St?fan van der Walt wrote: > You can always read chunks of the file into StringIO objects, and pass > those into loadtxt. true, but there isn't any single method for loading n lines of a file into a StringIO object, either. I"ve always thought that file.readlines() should take a number of rows as an optional parameter. not a big deal, now that we have list comprehensions, but still it would be nice, and it makes sense to put it into loadtxt() for sure. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cimrman3 at ntc.zcu.cz Mon Feb 14 11:53:49 2011 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 14 Feb 2011 17:53:49 +0100 (CET) Subject: [Numpy-discussion] using loadtxt() for given number of rows? In-Reply-To: References: Message-ID: On Mon, 14 Feb 2011, St?fan van der Walt wrote: > Hi Robert > > On Mon, Jan 31, 2011 at 2:39 PM, Robert Cimrman wrote: >> It seems to me, that an additional parameter to loadtxt(), say "nrows" or >> "numrows", would do the job, so that the function does not try reading the >> entire file. Another possibility would be to raise an exception as it >> is now, but also to return the data succesfully read so far. > > You can always read chunks of the file into StringIO objects, and pass > those into loadtxt. I think your request makes sense though, given > that a person can already skip lines at the top of the file. Thanks for the tip, St?fan! I have solved my problem by using np.fromfile(), as suggested by Chris Barker, because I know the number of items to read in advance. I also think that functionality suitable for loadtxt(), so I created the NumPy ticket 1731. r. From alok at merfinllc.com Mon Feb 14 14:21:32 2011 From: alok at merfinllc.com (Alok Singhal) Date: Mon, 14 Feb 2011 11:21:32 -0800 Subject: [Numpy-discussion] NumPy Featured on RCE-Cast In-Reply-To: <012BCDB6-8C99-47D8-9A1B-E02E8C776637@umich.edu> References: <012BCDB6-8C99-47D8-9A1B-E02E8C776637@umich.edu> Message-ID: From bwoods at aer.com Mon Feb 14 17:21:07 2011 From: bwoods at aer.com (Bryan Woods) Date: Mon, 14 Feb 2011 17:21:07 -0500 Subject: [Numpy-discussion] conditional array indexing Message-ID: <4D59AAD3.2010905@aer.com> I have spent over an hour now trying to figure out what I thought would be a really basic conditional array assignment. I am very new to python and am in need of help. I have a 2D integer array of land cover classes and a 1D array with a roughness value for each class. I am trying to create a new 2D array containing a map of the roughness values. I am trying the following code: roughness = np.array(landcover.shape,dtype='f') for i in range(np.min(landcover),np.max(landcover)): roughness[landcover == i] = z0_legend[i] Alas it tells me that I can only have a 1D boolean array index. What is the easiest way around this without having to loop through each row of the 2D array. I need to keep this vectorized as much as possible as these arrays are very large. Thanks, Bryan -------------- next part -------------- A non-text attachment was scrubbed... Name: bwoods.vcf Type: text/x-vcard Size: 341 bytes Desc: not available URL: From robert.kern at gmail.com Mon Feb 14 17:24:38 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Feb 2011 16:24:38 -0600 Subject: [Numpy-discussion] conditional array indexing In-Reply-To: <4D59AAD3.2010905@aer.com> References: <4D59AAD3.2010905@aer.com> Message-ID: On Mon, Feb 14, 2011 at 16:21, Bryan Woods wrote: > I have spent over an hour now trying to figure out what I thought would be a > really basic conditional array assignment. I am very new to python and am in > need of help. > > I have a 2D integer array of land cover classes and a 1D array with a > roughness value for each class. I am trying to create a new 2D array > containing a map of the roughness values. roughness = z0_legend[landcover] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From bwoods at aer.com Mon Feb 14 17:39:15 2011 From: bwoods at aer.com (Bryan Woods) Date: Mon, 14 Feb 2011 17:39:15 -0500 Subject: [Numpy-discussion] conditional array indexing In-Reply-To: References: <4D59AAD3.2010905@aer.com> Message-ID: <4D59AF13.4050105@aer.com> Thanks for your reply. Unfortunately it is not working that simply. This tells me that "only integer arrays with one element can be converted to an index" roughness and landcover are identically shaped 2D arrays. z0_legend in a 1D look-up table. I assume the error here comes from trying to index a 1D array with a 2D array. On 2/14/11 5:24 PM, Robert Kern wrote: > On Mon, Feb 14, 2011 at 16:21, Bryan Woods wrote: >> I have spent over an hour now trying to figure out what I thought would be a >> really basic conditional array assignment. I am very new to python and am in >> need of help. >> >> I have a 2D integer array of land cover classes and a 1D array with a >> roughness value for each class. I am trying to create a new 2D array >> containing a map of the roughness values. > roughness = z0_legend[landcover] > -------------- next part -------------- A non-text attachment was scrubbed... Name: bwoods.vcf Type: text/x-vcard Size: 341 bytes Desc: not available URL: From robert.kern at gmail.com Mon Feb 14 17:38:27 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Feb 2011 16:38:27 -0600 Subject: [Numpy-discussion] conditional array indexing In-Reply-To: <4D59AF13.4050105@aer.com> References: <4D59AAD3.2010905@aer.com> <4D59AF13.4050105@aer.com> Message-ID: On Mon, Feb 14, 2011 at 16:39, Bryan Woods wrote: > Thanks for your reply. Unfortunately it is not working that simply. > > This tells me that "only integer arrays with one element can be converted to > an index" > > roughness and landcover are identically shaped 2D arrays. z0_legend in a 1D > look-up table. I assume the error here comes from trying to index a 1D array > with a 2D array. Make sure that z0_legend is a numpy array, not a list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pav at iki.fi Mon Feb 14 18:25:45 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 14 Feb 2011 23:25:45 +0000 (UTC) Subject: [Numpy-discussion] conditional array indexing References: <4D59AAD3.2010905@aer.com> Message-ID: On Mon, 14 Feb 2011 17:21:07 -0500, Bryan Woods wrote: [clip] > roughness = np.array(landcover.shape,dtype='f') np.zeros(landcover.shape, dtype='f') From Chris.Barker at noaa.gov Mon Feb 14 18:39:00 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 14 Feb 2011 15:39:00 -0800 Subject: [Numpy-discussion] conditional array indexing In-Reply-To: <4D59AF13.4050105@aer.com> References: <4D59AAD3.2010905@aer.com> <4D59AF13.4050105@aer.com> Message-ID: <4D59BD14.4060800@noaa.gov> On 2/14/11 2:39 PM, Bryan Woods wrote: > Thanks for your reply. Unfortunately it is not working that simply. > > This tells me that "only integer arrays with one element can be > converted to an index" Examples, example, examples! I think this is what you want: In [15]: land_cover Out[15]: array([[4, 2, 0, 4], [0, 2, 1, 1], [1, 1, 4, 2]]) In [16]: z0_legend Out[16]: array([ 3.4, 5.2, 1.3, 4.2, 6.4]) In [17]: roughness = z0_legend[land_cover] In [18]: roughness Out[18]: array([[ 6.4, 1.3, 3.4, 6.4], [ 3.4, 1.3, 5.2, 5.2], [ 5.2, 5.2, 6.4, 1.3]]) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From seb.haase at gmail.com Tue Feb 15 10:50:27 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 15 Feb 2011 16:50:27 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? Message-ID: Hi, I assume that someone here could maybe help me, and I'm hoping it's not too much off topic. I have 2 arrays of 2d point coordinates and would like to calculate all pairwise distances as fast as possible. Going from Python/Numpy to a (Swigged) C extension already gave me a 55x speedup. (.9ms vs. 50ms for arrays of length 329 and 340). I'm using gcc on Linux. Now I'm wondering if I could go even faster !? My hope that the compiler might automagically do some SSE2 optimization got disappointed. Since I have a 4 core CPU I thought OpenMP might be an option; I never used that, and after some playing around I managed to get (only) 50% slowdown(!) :-( My code in short is this: (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) ------- ---------- void dists2d( double *a_ps, int nx1, int na, double *b_ps, int nx2, int nb, double *dist, int nx3, int ny3) throw (char*) { if(nx1 != 2) throw (char*) "a must be of shape (n,2)"; if(nx2 != 2) throw (char*) "b must be of shape (n,2)"; if(nx3 != nb || ny3 != na) throw (char*) "c must be of shape (na,nb)"; double *dist_; int i, j; #pragma omp parallel private(dist_, j, i) { #pragma omp for nowait for(i=0;i 4 dist_ = dist+i*nb; // dists_ is only introduced for OpenMP for(j=0;j ---------- There is probably a simple mistake in this code - as I said I never used OpenMP before. It should be not too difficult to use OpenMP correctly here or - maybe better - is there a simple SSE(2,3,4) version that might be even better than OpenMP... !? I supposed, that I did not get the #pragma omp lines right - any idea ? Or is it in general not possible to speed this kind of code up using OpenMP !? Thanks, Sebastian Haase From matthieu.brucher at gmail.com Tue Feb 15 10:54:55 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 15 Feb 2011 16:54:55 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Hi, My first move would be to add a restrict keyword to dist (i.e. dist is the only pointer to the specific memory location), and then declare dist_ inside the first loop also with a restrict. Then, I would run valgrind or a PAPI profil on your code to see what causes the issue (false sharing, ...) Matthieu 2011/2/15 Sebastian Haase > Hi, > I assume that someone here could maybe help me, and I'm hoping it's > not too much off topic. > I have 2 arrays of 2d point coordinates and would like to calculate > all pairwise distances as fast as possible. > Going from Python/Numpy to a (Swigged) C extension already gave me a > 55x speedup. > (.9ms vs. 50ms for arrays of length 329 and 340). > I'm using gcc on Linux. > Now I'm wondering if I could go even faster !? > My hope that the compiler might automagically do some SSE2 > optimization got disappointed. > Since I have a 4 core CPU I thought OpenMP might be an option; > I never used that, and after some playing around I managed to get > (only) 50% slowdown(!) :-( > > My code in short is this: > (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) > ------- ---------- > void dists2d( > double *a_ps, int nx1, int na, > double *b_ps, int nx2, int nb, > double *dist, int nx3, int ny3) throw (char*) > { > if(nx1 != 2) throw (char*) "a must be of shape (n,2)"; > if(nx2 != 2) throw (char*) "b must be of shape (n,2)"; > if(nx3 != nb || ny3 != na) throw (char*) "c must be of shape (na,nb)"; > > double *dist_; > int i, j; > > #pragma omp parallel private(dist_, j, i) > { > #pragma omp for nowait > for(i=0;i { > //num_threads=omp_get_num_threads(); --> 4 > dist_ = dist+i*nb; // dists_ is only > introduced for OpenMP > for(j=0;j { > *dist_++ = sqrt( sq(a_ps[i*nx1] - b_ps[j*nx2]) + > sq(a_ps[i*nx1+1] - > b_ps[j*nx2+1]) ); > } > } > } > } > ------- ---------- > There is probably a simple mistake in this code - as I said I never > used OpenMP before. > It should be not too difficult to use OpenMP correctly here > or - maybe better - > is there a simple SSE(2,3,4) version that might be even better than > OpenMP... !? > > I supposed, that I did not get the #pragma omp lines right - any idea ? > Or is it in general not possible to speed this kind of code up using OpenMP > !? > > Thanks, > Sebastian Haase > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Tue Feb 15 11:19:45 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 15 Feb 2011 17:19:45 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Thanks Matthieu, using __restrict__ with g++ did not change anything. How do I use valgrind with C extensions? I don't know what "PAPI profil" is ...? -Sebastian On Tue, Feb 15, 2011 at 4:54 PM, Matthieu Brucher wrote: > Hi, > My first move would be to add a restrict keyword to dist (i.e. dist is the > only pointer to the specific memory location), and then declare dist_ inside > the first loop also with a restrict. > Then, I would run valgrind or a PAPI profil on your code to see what causes > the issue (false sharing, ...) > Matthieu > > 2011/2/15 Sebastian Haase >> >> Hi, >> I assume that someone here could maybe help me, and I'm hoping it's >> not too much off topic. >> I have 2 arrays of 2d point coordinates and would like to calculate >> all pairwise distances as fast as possible. >> Going from Python/Numpy to a (Swigged) C extension already gave me a >> 55x speedup. >> (.9ms vs. 50ms for arrays of length 329 and 340). >> I'm using gcc on Linux. >> Now I'm wondering if I could go even faster !? >> My hope that the compiler might automagically do some SSE2 >> optimization got disappointed. >> Since I have a 4 core CPU I thought OpenMP might be an option; >> I never used that, and after some playing around I managed to get >> (only) 50% slowdown(!) :-( >> >> My code in short is this: >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) >> ------- ---------- >> void dists2d( >> ? ? ? ? ? ? ? ? ? double *a_ps, int nx1, int na, >> ? ? ? ? ? ? ? ? ? double *b_ps, int nx2, int nb, >> ? ? ? ? ? ? ? ? ? double *dist, int nx3, int ny3) ?throw (char*) >> { >> ?if(nx1 != 2) ?throw (char*) "a must be of shape (n,2)"; >> ?if(nx2 != 2) ?throw (char*) "b must be of shape (n,2)"; >> ?if(nx3 != nb || ny3 != na) ? ?throw (char*) "c must be of shape (na,nb)"; >> >> ?double *dist_; >> ?int i, j; >> >> #pragma omp parallel private(dist_, j, i) >> ?{ >> #pragma omp for nowait >> ? ? ? ?for(i=0;i> ? ? ? ? ?{ >> ? ? ? ? ? ? ? ?//num_threads=omp_get_num_threads(); ?--> 4 >> ? ? ? ? ? ? ? ?dist_ = dist+i*nb; ? ? ? ? ? ? ? ? // dists_ ?is ?only >> introduced for OpenMP >> ? ? ? ? ? ? ? ?for(j=0;j> ? ? ? ? ? ? ? ? ?{ >> ? ? ? ? ? ? ? ? ? ? ? ?*dist_++ ?= sqrt( sq(a_ps[i*nx1] ? - b_ps[j*nx2]) + >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?sq(a_ps[i*nx1+1] >> - b_ps[j*nx2+1]) ); >> ? ? ? ? ? ? ? ? ?} >> ? ? ? ? ?} >> ?} >> } >> ------- ---------- >> There is probably a simple mistake in this code - as I said I never >> used OpenMP before. >> It should be not too difficult to use OpenMP correctly here >> ?or - ?maybe better - >> is there a simple SSE(2,3,4) version that might be even better than >> OpenMP... !? >> >> I supposed, that I did not get the #pragma omp lines right - any idea ? >> Or is it in general not possible to speed this kind of code up using >> OpenMP !? >> >> Thanks, >> Sebastian Haase >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From matthieu.brucher at gmail.com Tue Feb 15 11:25:00 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 15 Feb 2011 17:25:00 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Use directly restrict in C99 mode (__restrict does not have exactly the same semantics). For a valgrind profil, you can check my blog ( http://matt.eifelle.com/2009/04/07/profiling-with-valgrind/) Basically, if you have a python script, you can valgrind --optionsinmyblog python myscript.py For PAPI, you have to install several packages (perf module for kernel for instance) and a GUI to analyze the results (in Eclispe, it should be possible). Matthieu 2011/2/15 Sebastian Haase > Thanks Matthieu, > using __restrict__ with g++ did not change anything. How do I use > valgrind with C extensions? > I don't know what "PAPI profil" is ...? > -Sebastian > > > On Tue, Feb 15, 2011 at 4:54 PM, Matthieu Brucher > wrote: > > Hi, > > My first move would be to add a restrict keyword to dist (i.e. dist is > the > > only pointer to the specific memory location), and then declare dist_ > inside > > the first loop also with a restrict. > > Then, I would run valgrind or a PAPI profil on your code to see what > causes > > the issue (false sharing, ...) > > Matthieu > > > > 2011/2/15 Sebastian Haase > >> > >> Hi, > >> I assume that someone here could maybe help me, and I'm hoping it's > >> not too much off topic. > >> I have 2 arrays of 2d point coordinates and would like to calculate > >> all pairwise distances as fast as possible. > >> Going from Python/Numpy to a (Swigged) C extension already gave me a > >> 55x speedup. > >> (.9ms vs. 50ms for arrays of length 329 and 340). > >> I'm using gcc on Linux. > >> Now I'm wondering if I could go even faster !? > >> My hope that the compiler might automagically do some SSE2 > >> optimization got disappointed. > >> Since I have a 4 core CPU I thought OpenMP might be an option; > >> I never used that, and after some playing around I managed to get > >> (only) 50% slowdown(!) :-( > >> > >> My code in short is this: > >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) > >> ------- ---------- > >> void dists2d( > >> double *a_ps, int nx1, int na, > >> double *b_ps, int nx2, int nb, > >> double *dist, int nx3, int ny3) throw (char*) > >> { > >> if(nx1 != 2) throw (char*) "a must be of shape (n,2)"; > >> if(nx2 != 2) throw (char*) "b must be of shape (n,2)"; > >> if(nx3 != nb || ny3 != na) throw (char*) "c must be of shape > (na,nb)"; > >> > >> double *dist_; > >> int i, j; > >> > >> #pragma omp parallel private(dist_, j, i) > >> { > >> #pragma omp for nowait > >> for(i=0;i >> { > >> //num_threads=omp_get_num_threads(); --> 4 > >> dist_ = dist+i*nb; // dists_ is only > >> introduced for OpenMP > >> for(j=0;j >> { > >> *dist_++ = sqrt( sq(a_ps[i*nx1] - b_ps[j*nx2]) > + > >> > sq(a_ps[i*nx1+1] > >> - b_ps[j*nx2+1]) ); > >> } > >> } > >> } > >> } > >> ------- ---------- > >> There is probably a simple mistake in this code - as I said I never > >> used OpenMP before. > >> It should be not too difficult to use OpenMP correctly here > >> or - maybe better - > >> is there a simple SSE(2,3,4) version that might be even better than > >> OpenMP... !? > >> > >> I supposed, that I did not get the #pragma omp lines right - any idea ? > >> Or is it in general not possible to speed this kind of code up using > >> OpenMP !? > >> > >> Thanks, > >> Sebastian Haase > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > -- > > Information System Engineer, Ph.D. > > Blog: http://matt.eifelle.com > > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From wesmckinn at gmail.com Tue Feb 15 11:26:26 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Tue, 15 Feb 2011 11:26:26 -0500 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: On Tue, Feb 15, 2011 at 11:25 AM, Matthieu Brucher wrote: > Use directly restrict in C99 mode (__restrict does not have exactly the same > semantics). > For a valgrind profil, you can check my blog > (http://matt.eifelle.com/2009/04/07/profiling-with-valgrind/) > Basically, if you have a python script, you can valgrind --optionsinmyblog > python myscript.py > For PAPI, you have to install several packages (perf module for kernel for > instance) and a GUI to analyze the results (in Eclispe, it should be > possible). > Matthieu > 2011/2/15 Sebastian Haase >> >> Thanks Matthieu, >> using __restrict__ with g++ did not change anything. How do I use >> valgrind with C extensions? >> I don't know what "PAPI profil" is ...? >> -Sebastian >> >> >> On Tue, Feb 15, 2011 at 4:54 PM, Matthieu Brucher >> wrote: >> > Hi, >> > My first move would be to add a restrict keyword to dist (i.e. dist is >> > the >> > only pointer to the specific memory location), and then declare dist_ >> > inside >> > the first loop also with a restrict. >> > Then, I would run valgrind or a PAPI profil on your code to see what >> > causes >> > the issue (false sharing, ...) >> > Matthieu >> > >> > 2011/2/15 Sebastian Haase >> >> >> >> Hi, >> >> I assume that someone here could maybe help me, and I'm hoping it's >> >> not too much off topic. >> >> I have 2 arrays of 2d point coordinates and would like to calculate >> >> all pairwise distances as fast as possible. >> >> Going from Python/Numpy to a (Swigged) C extension already gave me a >> >> 55x speedup. >> >> (.9ms vs. 50ms for arrays of length 329 and 340). >> >> I'm using gcc on Linux. >> >> Now I'm wondering if I could go even faster !? >> >> My hope that the compiler might automagically do some SSE2 >> >> optimization got disappointed. >> >> Since I have a 4 core CPU I thought OpenMP might be an option; >> >> I never used that, and after some playing around I managed to get >> >> (only) 50% slowdown(!) :-( >> >> >> >> My code in short is this: >> >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) >> >> ------- ---------- >> >> void dists2d( >> >> ? ? ? ? ? ? ? ? ? double *a_ps, int nx1, int na, >> >> ? ? ? ? ? ? ? ? ? double *b_ps, int nx2, int nb, >> >> ? ? ? ? ? ? ? ? ? double *dist, int nx3, int ny3) ?throw (char*) >> >> { >> >> ?if(nx1 != 2) ?throw (char*) "a must be of shape (n,2)"; >> >> ?if(nx2 != 2) ?throw (char*) "b must be of shape (n,2)"; >> >> ?if(nx3 != nb || ny3 != na) ? ?throw (char*) "c must be of shape >> >> (na,nb)"; >> >> >> >> ?double *dist_; >> >> ?int i, j; >> >> >> >> #pragma omp parallel private(dist_, j, i) >> >> ?{ >> >> #pragma omp for nowait >> >> ? ? ? ?for(i=0;i> >> ? ? ? ? ?{ >> >> ? ? ? ? ? ? ? ?//num_threads=omp_get_num_threads(); ?--> 4 >> >> ? ? ? ? ? ? ? ?dist_ = dist+i*nb; ? ? ? ? ? ? ? ? // dists_ ?is ?only >> >> introduced for OpenMP >> >> ? ? ? ? ? ? ? ?for(j=0;j> >> ? ? ? ? ? ? ? ? ?{ >> >> ? ? ? ? ? ? ? ? ? ? ? ?*dist_++ ?= sqrt( sq(a_ps[i*nx1] ? - >> >> b_ps[j*nx2]) + >> >> >> >> ?sq(a_ps[i*nx1+1] >> >> - b_ps[j*nx2+1]) ); >> >> ? ? ? ? ? ? ? ? ?} >> >> ? ? ? ? ?} >> >> ?} >> >> } >> >> ------- ---------- >> >> There is probably a simple mistake in this code - as I said I never >> >> used OpenMP before. >> >> It should be not too difficult to use OpenMP correctly here >> >> ?or - ?maybe better - >> >> is there a simple SSE(2,3,4) version that might be even better than >> >> OpenMP... !? >> >> >> >> I supposed, that I did not get the #pragma omp lines right - any idea ? >> >> Or is it in general not possible to speed this kind of code up using >> >> OpenMP !? >> >> >> >> Thanks, >> >> Sebastian Haase >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > >> > -- >> > Information System Engineer, Ph.D. >> > Blog: http://matt.eifelle.com >> > LinkedIn: http://www.linkedin.com/in/matthieubrucher >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > As an aside, do you have a GPU-equipped machine? This function looks like it would be an easy CUDA target. Depends on who the users of the function are (e.g. do they also have the CUDA runtime) if whether you wanted to go that route. From seb.haase at gmail.com Tue Feb 15 11:33:13 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 15 Feb 2011 17:33:13 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Wes, I think I should have a couple of GPUs. I would be ready for anything ... if you think that I could do some easy(!) CUDA programming here, maybe you could guide me into the right direction... Thanks, Sebastian. On Tue, Feb 15, 2011 at 5:26 PM, Wes McKinney wrote: > On Tue, Feb 15, 2011 at 11:25 AM, Matthieu Brucher > wrote: >> Use directly restrict in C99 mode (__restrict does not have exactly the same >> semantics). >> For a valgrind profil, you can check my blog >> (http://matt.eifelle.com/2009/04/07/profiling-with-valgrind/) >> Basically, if you have a python script, you can valgrind --optionsinmyblog >> python myscript.py >> For PAPI, you have to install several packages (perf module for kernel for >> instance) and a GUI to analyze the results (in Eclispe, it should be >> possible). >> Matthieu >> 2011/2/15 Sebastian Haase >>> >>> Thanks Matthieu, >>> using __restrict__ with g++ did not change anything. How do I use >>> valgrind with C extensions? >>> I don't know what "PAPI profil" is ...? >>> -Sebastian >>> >>> >>> On Tue, Feb 15, 2011 at 4:54 PM, Matthieu Brucher >>> wrote: >>> > Hi, >>> > My first move would be to add a restrict keyword to dist (i.e. dist is >>> > the >>> > only pointer to the specific memory location), and then declare dist_ >>> > inside >>> > the first loop also with a restrict. >>> > Then, I would run valgrind or a PAPI profil on your code to see what >>> > causes >>> > the issue (false sharing, ...) >>> > Matthieu >>> > >>> > 2011/2/15 Sebastian Haase >>> >> >>> >> Hi, >>> >> I assume that someone here could maybe help me, and I'm hoping it's >>> >> not too much off topic. >>> >> I have 2 arrays of 2d point coordinates and would like to calculate >>> >> all pairwise distances as fast as possible. >>> >> Going from Python/Numpy to a (Swigged) C extension already gave me a >>> >> 55x speedup. >>> >> (.9ms vs. 50ms for arrays of length 329 and 340). >>> >> I'm using gcc on Linux. >>> >> Now I'm wondering if I could go even faster !? >>> >> My hope that the compiler might automagically do some SSE2 >>> >> optimization got disappointed. >>> >> Since I have a 4 core CPU I thought OpenMP might be an option; >>> >> I never used that, and after some playing around I managed to get >>> >> (only) 50% slowdown(!) :-( >>> >> >>> >> My code in short is this: >>> >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) >>> >> ------- ---------- >>> >> void dists2d( >>> >> ? ? ? ? ? ? ? ? ? double *a_ps, int nx1, int na, >>> >> ? ? ? ? ? ? ? ? ? double *b_ps, int nx2, int nb, >>> >> ? ? ? ? ? ? ? ? ? double *dist, int nx3, int ny3) ?throw (char*) >>> >> { >>> >> ?if(nx1 != 2) ?throw (char*) "a must be of shape (n,2)"; >>> >> ?if(nx2 != 2) ?throw (char*) "b must be of shape (n,2)"; >>> >> ?if(nx3 != nb || ny3 != na) ? ?throw (char*) "c must be of shape >>> >> (na,nb)"; >>> >> >>> >> ?double *dist_; >>> >> ?int i, j; >>> >> >>> >> #pragma omp parallel private(dist_, j, i) >>> >> ?{ >>> >> #pragma omp for nowait >>> >> ? ? ? ?for(i=0;i>> >> ? ? ? ? ?{ >>> >> ? ? ? ? ? ? ? ?//num_threads=omp_get_num_threads(); ?--> 4 >>> >> ? ? ? ? ? ? ? ?dist_ = dist+i*nb; ? ? ? ? ? ? ? ? // dists_ ?is ?only >>> >> introduced for OpenMP >>> >> ? ? ? ? ? ? ? ?for(j=0;j>> >> ? ? ? ? ? ? ? ? ?{ >>> >> ? ? ? ? ? ? ? ? ? ? ? ?*dist_++ ?= sqrt( sq(a_ps[i*nx1] ? - >>> >> b_ps[j*nx2]) + >>> >> >>> >> ?sq(a_ps[i*nx1+1] >>> >> - b_ps[j*nx2+1]) ); >>> >> ? ? ? ? ? ? ? ? ?} >>> >> ? ? ? ? ?} >>> >> ?} >>> >> } >>> >> ------- ---------- >>> >> There is probably a simple mistake in this code - as I said I never >>> >> used OpenMP before. >>> >> It should be not too difficult to use OpenMP correctly here >>> >> ?or - ?maybe better - >>> >> is there a simple SSE(2,3,4) version that might be even better than >>> >> OpenMP... !? >>> >> >>> >> I supposed, that I did not get the #pragma omp lines right - any idea ? >>> >> Or is it in general not possible to speed this kind of code up using >>> >> OpenMP !? >>> >> >>> >> Thanks, >>> >> Sebastian Haase >>> >> _______________________________________________ >>> >> NumPy-Discussion mailing list >>> >> NumPy-Discussion at scipy.org >>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> > >>> > >>> > -- >>> > Information System Engineer, Ph.D. >>> > Blog: http://matt.eifelle.com >>> > LinkedIn: http://www.linkedin.com/in/matthieubrucher >>> > >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> > >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> -- >> Information System Engineer, Ph.D. >> Blog: http://matt.eifelle.com >> LinkedIn: http://www.linkedin.com/in/matthieubrucher >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > As an aside, do you have a GPU-equipped machine? This function looks > like it would be an easy CUDA target. Depends on who the users of the > function are (e.g. do they also have the CUDA runtime) if whether you > wanted to go that route. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From davide.cittaro at ifom-ieo-campus.it Tue Feb 15 11:42:45 2011 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Tue, 15 Feb 2011 17:42:45 +0100 Subject: [Numpy-discussion] convolving (or correlating) with sliding windows Message-ID: Hi all, I have to work with huge numpy.array (i.e. up to 250 M long) and I have to perform either np.correlate or np.convolve between those. The process can only work on big memory machines but it takes ages. I'm writing to get some hint on how to speed up things (at cost of precision, maybe...), possibly using a moving window... is it correct to perform this: - given a window W and a step size S - given data1 and data2 - pad with zeros data1 and data2 by adding W/2 0-arrays - get the np.correlation like y = np.array([np.correlate(data1[x:x+W], data2[x:x+W], mode='valid') for x in np.arange(0, len(data1) - W, S)]).ravel() instead of np.correlate(data1, data2, mode='same') - interpolate the correlation using scipy.interpolate ? Thanks d From wesmckinn at gmail.com Tue Feb 15 11:48:05 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Tue, 15 Feb 2011 11:48:05 -0500 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: On Tue, Feb 15, 2011 at 11:33 AM, Sebastian Haase wrote: > Wes, > I think I should have a couple of GPUs. I would be ready for anything > ... if you think that I could do some easy(!) CUDA programming here, > maybe you could guide me into the right direction... > Thanks, > Sebastian. > > > On Tue, Feb 15, 2011 at 5:26 PM, Wes McKinney wrote: >> On Tue, Feb 15, 2011 at 11:25 AM, Matthieu Brucher >> wrote: >>> Use directly restrict in C99 mode (__restrict does not have exactly the same >>> semantics). >>> For a valgrind profil, you can check my blog >>> (http://matt.eifelle.com/2009/04/07/profiling-with-valgrind/) >>> Basically, if you have a python script, you can valgrind --optionsinmyblog >>> python myscript.py >>> For PAPI, you have to install several packages (perf module for kernel for >>> instance) and a GUI to analyze the results (in Eclispe, it should be >>> possible). >>> Matthieu >>> 2011/2/15 Sebastian Haase >>>> >>>> Thanks Matthieu, >>>> using __restrict__ with g++ did not change anything. How do I use >>>> valgrind with C extensions? >>>> I don't know what "PAPI profil" is ...? >>>> -Sebastian >>>> >>>> >>>> On Tue, Feb 15, 2011 at 4:54 PM, Matthieu Brucher >>>> wrote: >>>> > Hi, >>>> > My first move would be to add a restrict keyword to dist (i.e. dist is >>>> > the >>>> > only pointer to the specific memory location), and then declare dist_ >>>> > inside >>>> > the first loop also with a restrict. >>>> > Then, I would run valgrind or a PAPI profil on your code to see what >>>> > causes >>>> > the issue (false sharing, ...) >>>> > Matthieu >>>> > >>>> > 2011/2/15 Sebastian Haase >>>> >> >>>> >> Hi, >>>> >> I assume that someone here could maybe help me, and I'm hoping it's >>>> >> not too much off topic. >>>> >> I have 2 arrays of 2d point coordinates and would like to calculate >>>> >> all pairwise distances as fast as possible. >>>> >> Going from Python/Numpy to a (Swigged) C extension already gave me a >>>> >> 55x speedup. >>>> >> (.9ms vs. 50ms for arrays of length 329 and 340). >>>> >> I'm using gcc on Linux. >>>> >> Now I'm wondering if I could go even faster !? >>>> >> My hope that the compiler might automagically do some SSE2 >>>> >> optimization got disappointed. >>>> >> Since I have a 4 core CPU I thought OpenMP might be an option; >>>> >> I never used that, and after some playing around I managed to get >>>> >> (only) 50% slowdown(!) :-( >>>> >> >>>> >> My code in short is this: >>>> >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) >>>> >> ------- ---------- >>>> >> void dists2d( >>>> >> ? ? ? ? ? ? ? ? ? double *a_ps, int nx1, int na, >>>> >> ? ? ? ? ? ? ? ? ? double *b_ps, int nx2, int nb, >>>> >> ? ? ? ? ? ? ? ? ? double *dist, int nx3, int ny3) ?throw (char*) >>>> >> { >>>> >> ?if(nx1 != 2) ?throw (char*) "a must be of shape (n,2)"; >>>> >> ?if(nx2 != 2) ?throw (char*) "b must be of shape (n,2)"; >>>> >> ?if(nx3 != nb || ny3 != na) ? ?throw (char*) "c must be of shape >>>> >> (na,nb)"; >>>> >> >>>> >> ?double *dist_; >>>> >> ?int i, j; >>>> >> >>>> >> #pragma omp parallel private(dist_, j, i) >>>> >> ?{ >>>> >> #pragma omp for nowait >>>> >> ? ? ? ?for(i=0;i>>> >> ? ? ? ? ?{ >>>> >> ? ? ? ? ? ? ? ?//num_threads=omp_get_num_threads(); ?--> 4 >>>> >> ? ? ? ? ? ? ? ?dist_ = dist+i*nb; ? ? ? ? ? ? ? ? // dists_ ?is ?only >>>> >> introduced for OpenMP >>>> >> ? ? ? ? ? ? ? ?for(j=0;j>>> >> ? ? ? ? ? ? ? ? ?{ >>>> >> ? ? ? ? ? ? ? ? ? ? ? ?*dist_++ ?= sqrt( sq(a_ps[i*nx1] ? - >>>> >> b_ps[j*nx2]) + >>>> >> >>>> >> ?sq(a_ps[i*nx1+1] >>>> >> - b_ps[j*nx2+1]) ); >>>> >> ? ? ? ? ? ? ? ? ?} >>>> >> ? ? ? ? ?} >>>> >> ?} >>>> >> } >>>> >> ------- ---------- >>>> >> There is probably a simple mistake in this code - as I said I never >>>> >> used OpenMP before. >>>> >> It should be not too difficult to use OpenMP correctly here >>>> >> ?or - ?maybe better - >>>> >> is there a simple SSE(2,3,4) version that might be even better than >>>> >> OpenMP... !? >>>> >> >>>> >> I supposed, that I did not get the #pragma omp lines right - any idea ? >>>> >> Or is it in general not possible to speed this kind of code up using >>>> >> OpenMP !? >>>> >> >>>> >> Thanks, >>>> >> Sebastian Haase >>>> >> _______________________________________________ >>>> >> NumPy-Discussion mailing list >>>> >> NumPy-Discussion at scipy.org >>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> > >>>> > >>>> > >>>> > -- >>>> > Information System Engineer, Ph.D. >>>> > Blog: http://matt.eifelle.com >>>> > LinkedIn: http://www.linkedin.com/in/matthieubrucher >>>> > >>>> > _______________________________________________ >>>> > NumPy-Discussion mailing list >>>> > NumPy-Discussion at scipy.org >>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> > >>>> > >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >>> >>> -- >>> Information System Engineer, Ph.D. >>> Blog: http://matt.eifelle.com >>> LinkedIn: http://www.linkedin.com/in/matthieubrucher >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> As an aside, do you have a GPU-equipped machine? This function looks >> like it would be an easy CUDA target. Depends on who the users of the >> function are (e.g. do they also have the CUDA runtime) if whether you >> wanted to go that route. >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > These days it's pretty low-hanging fruit-- but you need to learn a bit about the CUDA API and programming model through the user guide: http://developer.download.nvidia.com/compute/cuda/3_2_prod/toolkit/docs/CUDA_C_Programming_Guide.pdf I would say it's time well-invested. There is PyCUDA which makes the "boilerplate" parts of doing GPU computing a bit more pleasant (just write the kernel)-- I've been using Cython recently to interact with CUDA-based C code which is not a terrible option either (more work than PyCUDA). But if speed is your concern than the GPU will be *way* more bang for your buck than OpenMP/SSE2. I'd be surprised if you can't get at least 20x speedup over single-threaded C code even without tweaking (and maybe even a lot faster than that). But unfortunately with CUDA programming you spend a lot of your time tweaking memory transactions to get the best performance. From josef.pktd at gmail.com Tue Feb 15 12:25:47 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 15 Feb 2011 12:25:47 -0500 Subject: [Numpy-discussion] convolving (or correlating) with sliding windows In-Reply-To: References: Message-ID: On Tue, Feb 15, 2011 at 11:42 AM, Davide Cittaro wrote: > Hi all, > I have to work with huge numpy.array (i.e. up to 250 M long) and I have to perform either np.correlate or np.convolve between those. > The process can only work on big memory machines but it takes ages. I'm writing to get some hint on how to speed up things (at cost of precision, maybe...), possibly using a moving window... is it correct to perform this: > > - given a window W and a step size S > - given data1 and data2 > - pad with zeros data1 and data2 by adding W/2 0-arrays > - get the np.correlation like > > y = np.array([np.correlate(data1[x:x+W], data2[x:x+W], mode='valid') for x in ?np.arange(0, len(data1) - W, S)]).ravel() > > instead of np.correlate(data1, data2, mode='same') > > - interpolate the correlation using scipy.interpolate > ? > > Thanks > d > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Tue Feb 15 12:27:51 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 15 Feb 2011 12:27:51 -0500 Subject: [Numpy-discussion] convolving (or correlating) with sliding windows In-Reply-To: References: Message-ID: On Tue, Feb 15, 2011 at 11:42 AM, Davide Cittaro wrote: > Hi all, > I have to work with huge numpy.array (i.e. up to 250 M long) and I have to perform either np.correlate or np.convolve between those. > The process can only work on big memory machines but it takes ages. I'm writing to get some hint on how to speed up things (at cost of precision, maybe...), possibly using a moving window... is it correct to perform this: > > - given a window W and a step size S > - given data1 and data2 > - pad with zeros data1 and data2 by adding W/2 0-arrays > - get the np.correlation like > > y = np.array([np.correlate(data1[x:x+W], data2[x:x+W], mode='valid') for x in ?np.arange(0, len(data1) - W, S)]).ravel() > > instead of np.correlate(data1, data2, mode='same') If data2 is of similar length as data1, then you should use fftconvolve which is much faster for long arrays than np.correlate or convolve. I'm not sure about the rest. Josef > - interpolate the correlation using scipy.interpolate > ? > > Thanks > d > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jkhilmer at gmail.com Tue Feb 15 13:17:54 2011 From: jkhilmer at gmail.com (Jonathan Hilmer) Date: Tue, 15 Feb 2011 11:17:54 -0700 Subject: [Numpy-discussion] convolving (or correlating) with sliding windows In-Reply-To: References: Message-ID: I'm sorry that I don't have some example code for you, but you probably need to break down the problem if you can't fit it into memory: http://en.wikipedia.org/wiki/Overlap-add_method Jonathan On Tue, Feb 15, 2011 at 10:27 AM, wrote: > On Tue, Feb 15, 2011 at 11:42 AM, Davide Cittaro > wrote: >> Hi all, >> I have to work with huge numpy.array (i.e. up to 250 M long) and I have to perform either np.correlate or np.convolve between those. >> The process can only work on big memory machines but it takes ages. I'm writing to get some hint on how to speed up things (at cost of precision, maybe...), possibly using a moving window... is it correct to perform this: >> >> - given a window W and a step size S >> - given data1 and data2 >> - pad with zeros data1 and data2 by adding W/2 0-arrays >> - get the np.correlation like >> >> y = np.array([np.correlate(data1[x:x+W], data2[x:x+W], mode='valid') for x in ?np.arange(0, len(data1) - W, S)]).ravel() >> >> instead of np.correlate(data1, data2, mode='same') > > If data2 is of similar length as data1, then you should use > fftconvolve which is much faster for long arrays than np.correlate or > convolve. I'm not sure about the rest. > > Josef > > >> - interpolate the correlation using scipy.interpolate >> ? >> >> Thanks >> d >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From e.antero.tammi at gmail.com Tue Feb 15 15:11:12 2011 From: e.antero.tammi at gmail.com (eat) Date: Tue, 15 Feb 2011 22:11:12 +0200 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Hi, On Tue, Feb 15, 2011 at 5:50 PM, Sebastian Haase wrote: > Hi, > I assume that someone here could maybe help me, and I'm hoping it's > not too much off topic. > I have 2 arrays of 2d point coordinates and would like to calculate > all pairwise distances as fast as possible. > Going from Python/Numpy to a (Swigged) C extension already gave me a > 55x speedup. > (.9ms vs. 50ms for arrays of length 329 and 340). > With my very modest machine (Intel Dual CPU E2200, 2.2 Ghz) utilizing scipy.spatial.distance.pdist will take some 1.5 ms for such arrays. Even the simple linear algebra based full matrix calculations can be done less than 5 ms. My two cents, eat > I'm using gcc on Linux. > Now I'm wondering if I could go even faster !? > My hope that the compiler might automagically do some SSE2 > optimization got disappointed. > Since I have a 4 core CPU I thought OpenMP might be an option; > I never used that, and after some playing around I managed to get > (only) 50% slowdown(!) :-( > > My code in short is this: > (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) > ------- ---------- > void dists2d( > double *a_ps, int nx1, int na, > double *b_ps, int nx2, int nb, > double *dist, int nx3, int ny3) throw (char*) > { > if(nx1 != 2) throw (char*) "a must be of shape (n,2)"; > if(nx2 != 2) throw (char*) "b must be of shape (n,2)"; > if(nx3 != nb || ny3 != na) throw (char*) "c must be of shape (na,nb)"; > > double *dist_; > int i, j; > > #pragma omp parallel private(dist_, j, i) > { > #pragma omp for nowait > for(i=0;i { > //num_threads=omp_get_num_threads(); --> 4 > dist_ = dist+i*nb; // dists_ is only > introduced for OpenMP > for(j=0;j { > *dist_++ = sqrt( sq(a_ps[i*nx1] - b_ps[j*nx2]) + > sq(a_ps[i*nx1+1] - > b_ps[j*nx2+1]) ); > } > } > } > } > ------- ---------- > There is probably a simple mistake in this code - as I said I never > used OpenMP before. > It should be not too difficult to use OpenMP correctly here > or - maybe better - > is there a simple SSE(2,3,4) version that might be even better than > OpenMP... !? > > I supposed, that I did not get the #pragma omp lines right - any idea ? > Or is it in general not possible to speed this kind of code up using OpenMP > !? > > Thanks, > Sebastian Haase > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Tue Feb 15 16:42:00 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 15 Feb 2011 22:42:00 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Hi Eat, I will surely try these routines tomorrow, but I still think that neither scipy function does the complete distance calculation of all possible pairs as done by my C code. For 2 arrays, X and Y, of nX and nY 2d coordinates respectively, I need to get nX times nY distances computed. >From the online documentation I understand that pdist calculates something like nX square distances for a single array X of nX coordinates, and cdist would calculate nX distances, where nX is required to equal nY. Thanks for looking into this, Sebastian On Tue, Feb 15, 2011 at 9:11 PM, eat wrote: > Hi, > > On Tue, Feb 15, 2011 at 5:50 PM, Sebastian Haase > wrote: >> >> Hi, >> I assume that someone here could maybe help me, and I'm hoping it's >> not too much off topic. >> I have 2 arrays of 2d point coordinates and would like to calculate >> all pairwise distances as fast as possible. >> Going from Python/Numpy to a (Swigged) C extension already gave me a >> 55x speedup. >> (.9ms vs. 50ms for arrays of length 329 and 340). > > With my very modest machine (Intel Dual CPU E2200, 2.2 Ghz) utilizing > scipy.spatial.distance.pdist will take some 1.5 ms for such arrays. Even the > simple linear algebra based full matrix calculations can be done less than 5 > ms. > > My two cents, > eat >> >> I'm using gcc on Linux. >> Now I'm wondering if I could go even faster !? >> My hope that the compiler might automagically do some SSE2 >> optimization got disappointed. >> Since I have a 4 core CPU I thought OpenMP might be an option; >> I never used that, and after some playing around I managed to get >> (only) 50% slowdown(!) :-( >> >> My code in short is this: >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) >> ------- ---------- >> void dists2d( >> ? ? ? ? ? ? ? ? ? double *a_ps, int nx1, int na, >> ? ? ? ? ? ? ? ? ? double *b_ps, int nx2, int nb, >> ? ? ? ? ? ? ? ? ? double *dist, int nx3, int ny3) ?throw (char*) >> { >> ?if(nx1 != 2) ?throw (char*) "a must be of shape (n,2)"; >> ?if(nx2 != 2) ?throw (char*) "b must be of shape (n,2)"; >> ?if(nx3 != nb || ny3 != na) ? ?throw (char*) "c must be of shape (na,nb)"; >> >> ?double *dist_; >> ?int i, j; >> >> #pragma omp parallel private(dist_, j, i) >> ?{ >> #pragma omp for nowait >> ? ? ? ?for(i=0;i> ? ? ? ? ?{ >> ? ? ? ? ? ? ? ?//num_threads=omp_get_num_threads(); ?--> 4 >> ? ? ? ? ? ? ? ?dist_ = dist+i*nb; ? ? ? ? ? ? ? ? // dists_ ?is ?only >> introduced for OpenMP >> ? ? ? ? ? ? ? ?for(j=0;j> ? ? ? ? ? ? ? ? ?{ >> ? ? ? ? ? ? ? ? ? ? ? ?*dist_++ ?= sqrt( sq(a_ps[i*nx1] ? - b_ps[j*nx2]) + >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?sq(a_ps[i*nx1+1] >> - b_ps[j*nx2+1]) ); >> ? ? ? ? ? ? ? ? ?} >> ? ? ? ? ?} >> ?} >> } >> ------- ---------- >> There is probably a simple mistake in this code - as I said I never >> used OpenMP before. >> It should be not too difficult to use OpenMP correctly here >> ?or - ?maybe better - >> is there a simple SSE(2,3,4) version that might be even better than >> OpenMP... !? >> >> I supposed, that I did not get the #pragma omp lines right - any idea ? >> Or is it in general not possible to speed this kind of code up using >> OpenMP !? >> >> Thanks, >> Sebastian Haase >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From thomas at i-n.dk Tue Feb 15 18:04:23 2011 From: thomas at i-n.dk (Thomas Ingeman-Nielsen) Date: Wed, 16 Feb 2011 00:04:23 +0100 Subject: [Numpy-discussion] f2py "target file xxx not generated" Message-ID: Hi, I'm trying to get started with f2py on a Windows 7 environment using the Python(x,y) v 2.6.5.6 distribution. I'm following the introductory example of the f2py userguide and try to wrap the file FIB1.F using the command: f2py.py -c fib1.f -m fib1 from the windows command line. I get the following output: running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler opti ons running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt ions running build_src build_src building extension "fib" sources f2py options: [] f2py:> c:\users\thomas\appdata\local\temp\tmpamyxnx\src.win32-2.6\fibmodule.c creating c:\users\thomas\appdata\local\temp\tmpamyxnx creating c:\users\thomas\appdata\local\temp\tmpamyxnx\src.win32-2.6 Reading fortran codes... Reading file 'fib1.f' (format:fix,strict) Post-processing... Post-processing (stage 2)... Building modules... error: f2py target file 'c:\\users\\thomas\\appdata\\local\\temp\\tmpamyxnx\\src .win32-2.6\\fibmodule.c' not generated output of f2py.py -c --help-fcompiler: Gnu95FCompiler instance properties: archiver = ['c:\\GCC451\\bin\\gfortran.exe', '-cr'] compile_switch = '-c' compiler_f77 = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-ffixed- form', '-fno-second-underscore', '-mno-cygwin', '-O3', '- funroll-loops'] compiler_f90 = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-fno-second- underscore', '-mno-cygwin', '-O3', '-funroll-loops'] compiler_fix = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-ffixed- form', '-fno-second-underscore', '-mno-cygwin', '-Wall', ' -fno-second-underscore', '-mno-cygwin', '-O3', '-funroll- loops'] libraries = ['gfortran'] library_dirs = ['c:\\gcc451\\lib', 'c:\\gcc451\\lib\\gcc\\i686-pc- mingw32\\4.5.1'] linker_exe = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-mno- cygwin', '-Wall', '-mno-cygwin'] linker_so = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-mno- cygwin', '-Wall', '-mno-cygwin', '-shared'] object_switch = '-o ' ranlib = ['c:\\GCC451\\bin\\gfortran.exe'] version = LooseVersion ('4.5.1') version_cmd = ['c:\\GCC451\\bin\\gfortran.exe', '--version', '-mno- cygwin'] Fortran compilers found: --fcompiler=gnu95 GNU Fortran 95 compiler (4.5.1) Compilers available for this platform, but not found: --fcompiler=absoft Absoft Corp Fortran Compiler --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler --fcompiler=g95 G95 Fortran Compiler --fcompiler=gnu GNU Fortran 77 compiler --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps --fcompiler=intelvem Intel Visual Fortran Compiler for 64-bit apps Compilers not available on this platform: --fcompiler=compaq Compaq Fortran Compiler --fcompiler=hpux HP Fortran 90 Compiler --fcompiler=ibm IBM XL Fortran Compiler --fcompiler=intel Intel Fortran Compiler for 32-bit apps --fcompiler=intele Intel Fortran Compiler for Itanium apps --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler --fcompiler=mips MIPSpro Fortran Compiler --fcompiler=nag NAGWare Fortran 95 Compiler --fcompiler=none Fake Fortran compiler --fcompiler=pg Portland Group Fortran Compiler --fcompiler=sun Sun or Forte Fortran 95 Compiler --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler For compiler details, run 'config_fc --verbose' setup command. Running f2py without arguments result in the following version information: Version: 1 numpy Version: 1.5.1 Requires: Python 2.3 or higher. License: NumPy license (see LICENSE.txt in the NumPy source code) Copyright 1999 - 2005 Pearu Peterson all rights reserved. http://cens.ioc.ee/projects/f2py2e/ I'm puzzled by this, I expect I have the compiler correctly installed, since f2py recognizes it (and I am able to compile Fortran code using GCC from Eclipse). How about the f2py version number = 1? I was under the impression that development had reached version >2. Reinstalling Numpy from SourceForge numpy-1.5.1-win32-superpack-python2.6.exe result in exactly the same output as above. Trying to install f2py using easy_install and then running f2py without arguments result in the following error: Traceback (most recent call last): File "C:\Python26\Scripts\f2py.py", line 5, in pkg_resources.run_script('f2py==2.45.241-1926', 'f2py.py') File "C:\Python26\lib\site-packages\pkg_resources.py", line 489, in run_script self.require(requires)[0].run_script(script_name, ns) File "C:\Python26\lib\site-packages\pkg_resources.py", line 1207, in run_scrip t execfile(script_filename, namespace, namespace) File "c:\python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\EGG-INF O\scripts\f2py.py", line 3, in import f2py2e File "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ __init__.py", line 10, in import f2py2e File "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ f2py2e.py", line 26, in import crackfortran File "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ crackfortran.py", line 1586 as=b['args'] ^ SyntaxError: invalid syntax Any comments on how to adapt my setup to allow f2py to function correctly would be much appreciated! Best regards, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Tue Feb 15 20:28:38 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 15 Feb 2011 20:28:38 -0500 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: The `cdist` function in scipy spatial does what you want, and takes ~ 1ms on my machine. In [1]: import numpy as np In [2]: from scipy.spatial.distance import cdist In [3]: a = np.random.random((340, 2)) In [4]: b = np.random.random((329, 2)) In [5]: c = cdist(a, b) In [6]: c.shape Out[6]: (340, 329) In [7]: %timeit cdist(a, b) 1000 loops, best of 3: 1.18 ms per loop On Tue, Feb 15, 2011 at 4:42 PM, Sebastian Haase wrote: > Hi Eat, > I will surely try these routines tomorrow, > but I still think that neither scipy function does the complete > distance calculation of all possible pairs as done by my C code. > For 2 arrays, X and Y, of nX and nY 2d coordinates respectively, I > need to get nX times nY distances computed. > >From the online documentation I understand that > pdist calculates something like nX square distances for a single array > X of nX coordinates, > and cdist would calculate nX distances, where nX is required to equal nY. > > Thanks for looking into this, > Sebastian > > On Tue, Feb 15, 2011 at 9:11 PM, eat wrote: > > Hi, > > > > On Tue, Feb 15, 2011 at 5:50 PM, Sebastian Haase > > wrote: > >> > >> Hi, > >> I assume that someone here could maybe help me, and I'm hoping it's > >> not too much off topic. > >> I have 2 arrays of 2d point coordinates and would like to calculate > >> all pairwise distances as fast as possible. > >> Going from Python/Numpy to a (Swigged) C extension already gave me a > >> 55x speedup. > >> (.9ms vs. 50ms for arrays of length 329 and 340). > > > > With my very modest machine (Intel Dual CPU E2200, 2.2 Ghz) utilizing > > scipy.spatial.distance.pdist will take some 1.5 ms for such arrays. Even > the > > simple linear algebra based full matrix calculations can be done less > than 5 > > ms. > > > > My two cents, > > eat > >> > >> I'm using gcc on Linux. > >> Now I'm wondering if I could go even faster !? > >> My hope that the compiler might automagically do some SSE2 > >> optimization got disappointed. > >> Since I have a 4 core CPU I thought OpenMP might be an option; > >> I never used that, and after some playing around I managed to get > >> (only) 50% slowdown(!) :-( > >> > >> My code in short is this: > >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) > >> ------- ---------- > >> void dists2d( > >> double *a_ps, int nx1, int na, > >> double *b_ps, int nx2, int nb, > >> double *dist, int nx3, int ny3) throw (char*) > >> { > >> if(nx1 != 2) throw (char*) "a must be of shape (n,2)"; > >> if(nx2 != 2) throw (char*) "b must be of shape (n,2)"; > >> if(nx3 != nb || ny3 != na) throw (char*) "c must be of shape > (na,nb)"; > >> > >> double *dist_; > >> int i, j; > >> > >> #pragma omp parallel private(dist_, j, i) > >> { > >> #pragma omp for nowait > >> for(i=0;i >> { > >> //num_threads=omp_get_num_threads(); --> 4 > >> dist_ = dist+i*nb; // dists_ is only > >> introduced for OpenMP > >> for(j=0;j >> { > >> *dist_++ = sqrt( sq(a_ps[i*nx1] - b_ps[j*nx2]) > + > >> > sq(a_ps[i*nx1+1] > >> - b_ps[j*nx2+1]) ); > >> } > >> } > >> } > >> } > >> ------- ---------- > >> There is probably a simple mistake in this code - as I said I never > >> used OpenMP before. > >> It should be not too difficult to use OpenMP correctly here > >> or - maybe better - > >> is there a simple SSE(2,3,4) version that might be even better than > >> OpenMP... !? > >> > >> I supposed, that I did not get the #pragma omp lines right - any idea ? > >> Or is it in general not possible to speed this kind of code up using > >> OpenMP !? > >> > >> Thanks, > >> Sebastian Haase > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.taylor at utoronto.ca Tue Feb 15 21:42:50 2011 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Tue, 15 Feb 2011 21:42:50 -0500 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Take a look at a nice project coming out of my department: http://code.google.com/p/cudamat/ Best, Jon. On Tue, Feb 15, 2011 at 11:33 AM, Sebastian Haase wrote: > Wes, > I think I should have a couple of GPUs. I would be ready for anything > ... if you think that I could do some easy(!) CUDA programming here, > maybe you could guide me into the right direction... > Thanks, > Sebastian. > > > On Tue, Feb 15, 2011 at 5:26 PM, Wes McKinney wrote: >> On Tue, Feb 15, 2011 at 11:25 AM, Matthieu Brucher >> wrote: >>> Use directly restrict in C99 mode (__restrict does not have exactly the same >>> semantics). >>> For a valgrind profil, you can check my blog >>> (http://matt.eifelle.com/2009/04/07/profiling-with-valgrind/) >>> Basically, if you have a python script, you can valgrind --optionsinmyblog >>> python myscript.py >>> For PAPI, you have to install several packages (perf module for kernel for >>> instance) and a GUI to analyze the results (in Eclispe, it should be >>> possible). >>> Matthieu >>> 2011/2/15 Sebastian Haase >>>> >>>> Thanks Matthieu, >>>> using __restrict__ with g++ did not change anything. How do I use >>>> valgrind with C extensions? >>>> I don't know what "PAPI profil" is ...? >>>> -Sebastian >>>> >>>> >>>> On Tue, Feb 15, 2011 at 4:54 PM, Matthieu Brucher >>>> wrote: >>>> > Hi, >>>> > My first move would be to add a restrict keyword to dist (i.e. dist is >>>> > the >>>> > only pointer to the specific memory location), and then declare dist_ >>>> > inside >>>> > the first loop also with a restrict. >>>> > Then, I would run valgrind or a PAPI profil on your code to see what >>>> > causes >>>> > the issue (false sharing, ...) >>>> > Matthieu >>>> > >>>> > 2011/2/15 Sebastian Haase >>>> >> >>>> >> Hi, >>>> >> I assume that someone here could maybe help me, and I'm hoping it's >>>> >> not too much off topic. >>>> >> I have 2 arrays of 2d point coordinates and would like to calculate >>>> >> all pairwise distances as fast as possible. >>>> >> Going from Python/Numpy to a (Swigged) C extension already gave me a >>>> >> 55x speedup. >>>> >> (.9ms vs. 50ms for arrays of length 329 and 340). >>>> >> I'm using gcc on Linux. >>>> >> Now I'm wondering if I could go even faster !? >>>> >> My hope that the compiler might automagically do some SSE2 >>>> >> optimization got disappointed. >>>> >> Since I have a 4 core CPU I thought OpenMP might be an option; >>>> >> I never used that, and after some playing around I managed to get >>>> >> (only) 50% slowdown(!) :-( >>>> >> >>>> >> My code in short is this: >>>> >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) >>>> >> ------- ---------- >>>> >> void dists2d( >>>> >> ? ? ? ? ? ? ? ? ? double *a_ps, int nx1, int na, >>>> >> ? ? ? ? ? ? ? ? ? double *b_ps, int nx2, int nb, >>>> >> ? ? ? ? ? ? ? ? ? double *dist, int nx3, int ny3) ?throw (char*) >>>> >> { >>>> >> ?if(nx1 != 2) ?throw (char*) "a must be of shape (n,2)"; >>>> >> ?if(nx2 != 2) ?throw (char*) "b must be of shape (n,2)"; >>>> >> ?if(nx3 != nb || ny3 != na) ? ?throw (char*) "c must be of shape >>>> >> (na,nb)"; >>>> >> >>>> >> ?double *dist_; >>>> >> ?int i, j; >>>> >> >>>> >> #pragma omp parallel private(dist_, j, i) >>>> >> ?{ >>>> >> #pragma omp for nowait >>>> >> ? ? ? ?for(i=0;i>>> >> ? ? ? ? ?{ >>>> >> ? ? ? ? ? ? ? ?//num_threads=omp_get_num_threads(); ?--> 4 >>>> >> ? ? ? ? ? ? ? ?dist_ = dist+i*nb; ? ? ? ? ? ? ? ? // dists_ ?is ?only >>>> >> introduced for OpenMP >>>> >> ? ? ? ? ? ? ? ?for(j=0;j>>> >> ? ? ? ? ? ? ? ? ?{ >>>> >> ? ? ? ? ? ? ? ? ? ? ? ?*dist_++ ?= sqrt( sq(a_ps[i*nx1] ? - >>>> >> b_ps[j*nx2]) + >>>> >> >>>> >> ?sq(a_ps[i*nx1+1] >>>> >> - b_ps[j*nx2+1]) ); >>>> >> ? ? ? ? ? ? ? ? ?} >>>> >> ? ? ? ? ?} >>>> >> ?} >>>> >> } >>>> >> ------- ---------- >>>> >> There is probably a simple mistake in this code - as I said I never >>>> >> used OpenMP before. >>>> >> It should be not too difficult to use OpenMP correctly here >>>> >> ?or - ?maybe better - >>>> >> is there a simple SSE(2,3,4) version that might be even better than >>>> >> OpenMP... !? >>>> >> >>>> >> I supposed, that I did not get the #pragma omp lines right - any idea ? >>>> >> Or is it in general not possible to speed this kind of code up using >>>> >> OpenMP !? >>>> >> >>>> >> Thanks, >>>> >> Sebastian Haase >>>> >> _______________________________________________ >>>> >> NumPy-Discussion mailing list >>>> >> NumPy-Discussion at scipy.org >>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> > >>>> > >>>> > >>>> > -- >>>> > Information System Engineer, Ph.D. >>>> > Blog: http://matt.eifelle.com >>>> > LinkedIn: http://www.linkedin.com/in/matthieubrucher >>>> > >>>> > _______________________________________________ >>>> > NumPy-Discussion mailing list >>>> > NumPy-Discussion at scipy.org >>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> > >>>> > >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >>> >>> -- >>> Information System Engineer, Ph.D. >>> Blog: http://matt.eifelle.com >>> LinkedIn: http://www.linkedin.com/in/matthieubrucher >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> As an aside, do you have a GPU-equipped machine? This function looks >> like it would be an easy CUDA target. Depends on who the users of the >> function are (e.g. do they also have the CUDA runtime) if whether you >> wanted to go that route. >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ecarlson at eng.ua.edu Tue Feb 15 22:50:42 2011 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Tue, 15 Feb 2011 21:50:42 -0600 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: I don't have the slightest idea what I'm doing, but.... ____ file name - the_lib.c ___ #include #include #include #include void dists2d( double *a_ps, int na, double *b_ps, int nb, double *dist, int num_threads) { int i, j; int dynamic=0; omp_set_dynamic(dynamic); omp_set_num_threads(num_threads); double ax,ay, dif_x, dif_y; int nx1=2; int nx2=2; #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) for(i=0;i References: Message-ID: On 16/feb/2011, at 00:04, numpy-discussion-request at scipy.org wrote: > > I'm sorry that I don't have some example code for you, but you > probably need to break down the problem if you can't fit it into > memory: http://en.wikipedia.org/wiki/Overlap-add_method > > Jonathan Thanks! You saved my day, also I found I can apply this to many other problems... I only need a good "signal processing tips and tricks" book now :-) d -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Wed Feb 16 04:19:59 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 16 Feb 2011 10:19:59 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Chris, OK, sorry -- I miss read (cdist doc says A and B must have same number of "columns"(!) not "rows"). On my machine I got the exact same timing as my (non OpenMP) C code. That is really got, compared to normal ufunc based numpy code. But my question in this thread is, how to get better than that, using either SSE or OpenMP (or CUDA?) Thanks again for telling me about scipy.distance, Sebastian On Wed, Feb 16, 2011 at 2:28 AM, Chris Colbert wrote: > > The `cdist` function in scipy spatial does what you want, and takes ~ 1ms on > my machine. > > In [1]: import numpy as np > In [2]: from scipy.spatial.distance import cdist > In [3]: a = np.random.random((340, 2)) > In [4]: b = np.random.random((329, 2)) > In [5]: c = cdist(a, b) > In [6]: c.shape > Out[6]: (340, 329) > In [7]: %timeit cdist(a, b) > 1000 loops, best of 3: 1.18 ms per loop > > > On Tue, Feb 15, 2011 at 4:42 PM, Sebastian Haase > wrote: >> >> Hi Eat, >> I will surely try these routines tomorrow, >> but I still think that neither scipy function does the complete >> distance calculation of all possible pairs as done by my C code. >> For 2 arrays, X and Y, of nX and nY 2d coordinates respectively, I >> need to get nX times nY distances computed. >> >From the online documentation I understand that >> pdist calculates something like nX square distances for a single array >> X of nX coordinates, >> and cdist would calculate nX distances, where nX is required to equal nY. >> >> Thanks for looking into this, >> Sebastian >> >> On Tue, Feb 15, 2011 at 9:11 PM, eat wrote: >> > Hi, >> > >> > On Tue, Feb 15, 2011 at 5:50 PM, Sebastian Haase >> > wrote: >> >> >> >> Hi, >> >> I assume that someone here could maybe help me, and I'm hoping it's >> >> not too much off topic. >> >> I have 2 arrays of 2d point coordinates and would like to calculate >> >> all pairwise distances as fast as possible. >> >> Going from Python/Numpy to a (Swigged) C extension already gave me a >> >> 55x speedup. >> >> (.9ms vs. 50ms for arrays of length 329 and 340). >> > >> > With my very modest machine (Intel Dual CPU E2200, 2.2 Ghz) utilizing >> > scipy.spatial.distance.pdist will take some 1.5 ms for such arrays. Even >> > the >> > simple linear algebra based full matrix calculations can be done less >> > than 5 >> > ms. >> > >> > My two cents, >> > eat >> >> >> >> I'm using gcc on Linux. >> >> Now I'm wondering if I could go even faster !? >> >> My hope that the compiler might automagically do some SSE2 >> >> optimization got disappointed. >> >> Since I have a 4 core CPU I thought OpenMP might be an option; >> >> I never used that, and after some playing around I managed to get >> >> (only) 50% slowdown(!) :-( >> >> >> >> My code in short is this: >> >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) >> >> ------- ---------- >> >> void dists2d( >> >> ? ? ? ? ? ? ? ? ? double *a_ps, int nx1, int na, >> >> ? ? ? ? ? ? ? ? ? double *b_ps, int nx2, int nb, >> >> ? ? ? ? ? ? ? ? ? double *dist, int nx3, int ny3) ?throw (char*) >> >> { >> >> ?if(nx1 != 2) ?throw (char*) "a must be of shape (n,2)"; >> >> ?if(nx2 != 2) ?throw (char*) "b must be of shape (n,2)"; >> >> ?if(nx3 != nb || ny3 != na) ? ?throw (char*) "c must be of shape >> >> (na,nb)"; >> >> >> >> ?double *dist_; >> >> ?int i, j; >> >> >> >> #pragma omp parallel private(dist_, j, i) >> >> ?{ >> >> #pragma omp for nowait >> >> ? ? ? ?for(i=0;i> >> ? ? ? ? ?{ >> >> ? ? ? ? ? ? ? ?//num_threads=omp_get_num_threads(); ?--> 4 >> >> ? ? ? ? ? ? ? ?dist_ = dist+i*nb; ? ? ? ? ? ? ? ? // dists_ ?is ?only >> >> introduced for OpenMP >> >> ? ? ? ? ? ? ? ?for(j=0;j> >> ? ? ? ? ? ? ? ? ?{ >> >> ? ? ? ? ? ? ? ? ? ? ? ?*dist_++ ?= sqrt( sq(a_ps[i*nx1] ? - >> >> b_ps[j*nx2]) + >> >> >> >> ?sq(a_ps[i*nx1+1] >> >> - b_ps[j*nx2+1]) ); >> >> ? ? ? ? ? ? ? ? ?} >> >> ? ? ? ? ?} >> >> ?} >> >> } >> >> ------- ---------- >> >> There is probably a simple mistake in this code - as I said I never >> >> used OpenMP before. >> >> It should be not too difficult to use OpenMP correctly here >> >> ?or - ?maybe better - >> >> is there a simple SSE(2,3,4) version that might be even better than >> >> OpenMP... !? >> >> >> >> I supposed, that I did not get the #pragma omp lines right - any idea ? >> >> Or is it in general not possible to speed this kind of code up using >> >> OpenMP !? >> >> >> >> Thanks, >> >> Sebastian Haase >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From tmp50 at ukr.net Wed Feb 16 04:57:28 2011 From: tmp50 at ukr.net (Dmitrey) Date: Wed, 16 Feb 2011 11:57:28 +0200 Subject: [Numpy-discussion] [ANN] New package: SpaceFuncs (2D, 3D, ND geometric modeling, optimization, solving) Message-ID: Hi all, I'm glad to inform you about new, 4th OpenOpt Suite module: SpaceFuncs - a tool for 2D, 3D, N-dimensional geometric modeling with possibilities of parametrized calculations, numerical optimization and solving systems of geometrical equations with automatic differentiation. The module is written in Python + NumPy, requires FuncDesigner (and OpenOpt, DerApproximator for some operations). It has completely free license: BSD. For details see its home page http://openopt.org/SpaceFuncs and documentation http://openopt.org/SpaceFuncsDoc Also, you can try it online via our Sage-server (sometimes hangs due to high load, through) http://sage.openopt.org/welcome Regards, Dmitrey. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at i-n.dk Wed Feb 16 05:00:38 2011 From: thomas at i-n.dk (Thomas Ingeman-Nielsen) Date: Wed, 16 Feb 2011 11:00:38 +0100 Subject: [Numpy-discussion] SOLVED: f2py "target file xxx not generated" Message-ID: Found a solution myself: I tried to compile the fib1.f file directly using the gfortran compiler, and found that indentation in the (fixed format) file was wrong. Correcting this took care of the "target file xxx not generated" error, but still an error resulted: ... ... error: Unable to find vcvarsall.bat A bit of googling pointed me to the following thread: http://code.google.com/p/rdflib/issues/detail?id=104#c4 which suggests creating a "distutils.cfg" file. After creating this file as described in the above reference, everything wraps/compiles smoothly, and the fib1 subroutine is callable from Python. Best regards, Thomas 2011/2/16 Thomas Ingeman-Nielsen > Hi, > > I'm trying to get started with f2py on a Windows 7 environment using the > Python(x,y) v 2.6.5.6 distribution. > I'm following the introductory example of the f2py userguide and try to > wrap the file FIB1.F using the command: > > f2py.py -c fib1.f -m fib1 > > from the windows command line. I get the following output: > > running build > running config_cc > unifing config_cc, config, build_clib, build_ext, build commands --compiler > opti > ons > running config_fc > unifing config_fc, config, build_clib, build_ext, build commands > --fcompiler opt > ions > running build_src > build_src > building extension "fib" sources > f2py options: [] > f2py:> > c:\users\thomas\appdata\local\temp\tmpamyxnx\src.win32-2.6\fibmodule.c > creating c:\users\thomas\appdata\local\temp\tmpamyxnx > creating c:\users\thomas\appdata\local\temp\tmpamyxnx\src.win32-2.6 > Reading fortran codes... > Reading file 'fib1.f' (format:fix,strict) > Post-processing... > Post-processing (stage 2)... > Building modules... > error: f2py target file > 'c:\\users\\thomas\\appdata\\local\\temp\\tmpamyxnx\\src > .win32-2.6\\fibmodule.c' not generated > > > > output of f2py.py -c --help-fcompiler: > > > Gnu95FCompiler instance properties: > archiver = ['c:\\GCC451\\bin\\gfortran.exe', '-cr'] > compile_switch = '-c' > compiler_f77 = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-ffixed- > form', '-fno-second-underscore', '-mno-cygwin', '-O3', > '- > funroll-loops'] > compiler_f90 = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', > '-fno-second- > underscore', '-mno-cygwin', '-O3', '-funroll-loops'] > compiler_fix = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-ffixed- > form', '-fno-second-underscore', '-mno-cygwin', > '-Wall', ' > -fno-second-underscore', '-mno-cygwin', '-O3', > '-funroll- > loops'] > libraries = ['gfortran'] > library_dirs = ['c:\\gcc451\\lib', 'c:\\gcc451\\lib\\gcc\\i686-pc- > mingw32\\4.5.1'] > linker_exe = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-mno- > cygwin', '-Wall', '-mno-cygwin'] > linker_so = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-mno- > cygwin', '-Wall', '-mno-cygwin', '-shared'] > object_switch = '-o ' > ranlib = ['c:\\GCC451\\bin\\gfortran.exe'] > version = LooseVersion ('4.5.1') > version_cmd = ['c:\\GCC451\\bin\\gfortran.exe', '--version', '-mno- > cygwin'] > Fortran compilers found: > --fcompiler=gnu95 GNU Fortran 95 compiler (4.5.1) > Compilers available for this platform, but not found: > --fcompiler=absoft Absoft Corp Fortran Compiler > --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler > --fcompiler=g95 G95 Fortran Compiler > --fcompiler=gnu GNU Fortran 77 compiler > --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps > --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps > --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps > --fcompiler=intelvem Intel Visual Fortran Compiler for 64-bit apps > Compilers not available on this platform: > --fcompiler=compaq Compaq Fortran Compiler > --fcompiler=hpux HP Fortran 90 Compiler > --fcompiler=ibm IBM XL Fortran Compiler > --fcompiler=intel Intel Fortran Compiler for 32-bit apps > --fcompiler=intele Intel Fortran Compiler for Itanium apps > --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler > --fcompiler=mips MIPSpro Fortran Compiler > --fcompiler=nag NAGWare Fortran 95 Compiler > --fcompiler=none Fake Fortran compiler > --fcompiler=pg Portland Group Fortran Compiler > --fcompiler=sun Sun or Forte Fortran 95 Compiler > --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler > For compiler details, run 'config_fc --verbose' setup command. > > > > Running f2py without arguments result in the following version information: > > Version: 1 > numpy Version: 1.5.1 > Requires: Python 2.3 or higher. > License: NumPy license (see LICENSE.txt in the NumPy source code) > Copyright 1999 - 2005 Pearu Peterson all rights reserved. > http://cens.ioc.ee/projects/f2py2e/ > > > > I'm puzzled by this, I expect I have the compiler correctly installed, > since f2py recognizes it (and I am able to compile Fortran code using GCC > from Eclipse). > How about the f2py version number = 1? I was under the impression that > development had reached version >2. > > > Reinstalling Numpy from SourceForge numpy-1.5.1-win32-superpack-python2.6.exe > result in exactly the same output as above. > > Trying to install f2py using easy_install and then running f2py without > arguments result in the following error: > > Traceback (most recent call last): > File "C:\Python26\Scripts\f2py.py", line 5, in > pkg_resources.run_script('f2py==2.45.241-1926', 'f2py.py') > File "C:\Python26\lib\site-packages\pkg_resources.py", line 489, in > run_script > > self.require(requires)[0].run_script(script_name, ns) > File "C:\Python26\lib\site-packages\pkg_resources.py", line 1207, in > run_scrip > t > execfile(script_filename, namespace, namespace) > File > "c:\python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\EGG-INF > O\scripts\f2py.py", line 3, in > import f2py2e > File > "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ > __init__.py", line 10, in > import f2py2e > File > "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ > f2py2e.py", line 26, in > import crackfortran > File > "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ > crackfortran.py", line 1586 > as=b['args'] > ^ > SyntaxError: invalid syntax > > > > Any comments on how to adapt my setup to allow f2py to function correctly > would be much appreciated! > > Best regards, > > Thomas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Wed Feb 16 07:29:47 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 16 Feb 2011 13:29:47 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Matthieu, I got it to run in valgrind (using the options from your blog). For dist2d() it says under "Types": ---------------------------------------------------------------------- Event Type Incl. Self Short Formula ------------------------------------------------------------------------------------ Instruction Fetch 379090 32 lr Data Read Access 116136 7 Dr Data Write Access 29921 11 Dw L1 Instn Fetch Miss 195 4 |1mr L1 Data Read Miss 270 2 Dlmr L1 Data Write Miss 3636 0 Dlmw L2 Instn Fetch Miss 125 4 |2mr L2 Data Read Miss 144 1 D2mr L2 Data Write Miss 83 0 D2mw L1 Miss Sum 4101 6 L1m = |1mr + D1mr + D1mw L2 Miss Sum 352 5 L2m = |2mr + D2mr + D2mw Cycle Estimation 455300592 CEst = Ir + 10 L1m + 100 L2m --------------------------------------------------------------------------------------- (hope this is readable - I used some OCR software to past it here) (My source code still uses __restrict__) Do I see this right, that the problem is shown by the lines L1 Data Read Miss and L1 Miss Sum I have no idea what to look for .... Thanks for the help, -- Sebastian On Tue, Feb 15, 2011 at 5:25 PM, Matthieu Brucher wrote: > Use directly restrict in C99 mode (__restrict does not have exactly the same > semantics). > For a valgrind profil, you can check my blog > (http://matt.eifelle.com/2009/04/07/profiling-with-valgrind/) > Basically, if you have a python script, you can valgrind --optionsinmyblog > python myscript.py > For PAPI, you have to install several packages (perf module for kernel for > instance) and a GUI to analyze the results (in Eclispe, it should be > possible). > Matthieu > 2011/2/15 Sebastian Haase >> >> Thanks Matthieu, >> using __restrict__ with g++ did not change anything. How do I use >> valgrind with C extensions? >> I don't know what "PAPI profil" is ...? >> -Sebastian >> >> >> On Tue, Feb 15, 2011 at 4:54 PM, Matthieu Brucher >> wrote: >> > Hi, >> > My first move would be to add a restrict keyword to dist (i.e. dist is >> > the >> > only pointer to the specific memory location), and then declare dist_ >> > inside >> > the first loop also with a restrict. >> > Then, I would run valgrind or a PAPI profil on your code to see what >> > causes >> > the issue (false sharing, ...) >> > Matthieu >> > >> > 2011/2/15 Sebastian Haase >> >> >> >> Hi, >> >> I assume that someone here could maybe help me, and I'm hoping it's >> >> not too much off topic. >> >> I have 2 arrays of 2d point coordinates and would like to calculate >> >> all pairwise distances as fast as possible. >> >> Going from Python/Numpy to a (Swigged) C extension already gave me a >> >> 55x speedup. >> >> (.9ms vs. 50ms for arrays of length 329 and 340). >> >> I'm using gcc on Linux. >> >> Now I'm wondering if I could go even faster !? >> >> My hope that the compiler might automagically do some SSE2 >> >> optimization got disappointed. >> >> Since I have a 4 core CPU I thought OpenMP might be an option; >> >> I never used that, and after some playing around I managed to get >> >> (only) 50% slowdown(!) :-( >> >> >> >> My code in short is this: >> >> (My SWIG typemaps use obj_to_array_no_conversion() from numpy.i) >> >> ------- ---------- >> >> void dists2d( >> >> ? ? ? ? ? ? ? ? ? double *a_ps, int nx1, int na, >> >> ? ? ? ? ? ? ? ? ? double *b_ps, int nx2, int nb, >> >> ? ? ? ? ? ? ? ? ? double *dist, int nx3, int ny3) ?throw (char*) >> >> { >> >> ?if(nx1 != 2) ?throw (char*) "a must be of shape (n,2)"; >> >> ?if(nx2 != 2) ?throw (char*) "b must be of shape (n,2)"; >> >> ?if(nx3 != nb || ny3 != na) ? ?throw (char*) "c must be of shape >> >> (na,nb)"; >> >> >> >> ?double *dist_; >> >> ?int i, j; >> >> >> >> #pragma omp parallel private(dist_, j, i) >> >> ?{ >> >> #pragma omp for nowait >> >> ? ? ? ?for(i=0;i> >> ? ? ? ? ?{ >> >> ? ? ? ? ? ? ? ?//num_threads=omp_get_num_threads(); ?--> 4 >> >> ? ? ? ? ? ? ? ?dist_ = dist+i*nb; ? ? ? ? ? ? ? ? // dists_ ?is ?only >> >> introduced for OpenMP >> >> ? ? ? ? ? ? ? ?for(j=0;j> >> ? ? ? ? ? ? ? ? ?{ >> >> ? ? ? ? ? ? ? ? ? ? ? ?*dist_++ ?= sqrt( sq(a_ps[i*nx1] ? - >> >> b_ps[j*nx2]) + >> >> >> >> ?sq(a_ps[i*nx1+1] >> >> - b_ps[j*nx2+1]) ); >> >> ? ? ? ? ? ? ? ? ?} >> >> ? ? ? ? ?} >> >> ?} >> >> } >> >> ------- ---------- >> >> There is probably a simple mistake in this code - as I said I never >> >> used OpenMP before. >> >> It should be not too difficult to use OpenMP correctly here >> >> ?or - ?maybe better - >> >> is there a simple SSE(2,3,4) version that might be even better than >> >> OpenMP... !? >> >> >> >> I supposed, that I did not get the #pragma omp lines right - any idea ? >> >> Or is it in general not possible to speed this kind of code up using >> >> OpenMP !? >> >> >> >> Thanks, >> >> Sebastian Haase >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > >> > -- >> > Information System Engineer, Ph.D. >> > Blog: http://matt.eifelle.com >> > LinkedIn: http://www.linkedin.com/in/matthieubrucher >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ndbecker2 at gmail.com Wed Feb 16 07:45:46 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 16 Feb 2011 07:45:46 -0500 Subject: [Numpy-discussion] unwrap enhancement? Message-ID: I want to use unwrap to process a sequence of segments, such that the sequence appears continous. That is, unwrap maintains a state which represents a multiple of 2\pi. I want to continue that state with the next call to unwrap. Or in other words, I'd like to specify an initial phase for the start of unwrap. This could, of course, be done by concatenating the initial phase to the beginning of the segment, but that is not very efficient (memory allocation, copying, etc). It would be nice to just add an optional initial phase arg to unwrap. Thoughts? From seb.haase at gmail.com Wed Feb 16 07:50:44 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 16 Feb 2011 13:50:44 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Eric, this is amazing !! Thanks very much, I have rarely seen such a compact source example that just worked. The timings I get are: c_threads 1 time 0.00155731916428 c_threads 2 time 0.000829789638519 c_threads 3 time 0.000616888999939 c_threads 4 time 0.000704760551453 c_threads 5 time 0.000933599472046 c_threads 6 time 0.000809240341187 c_threads 7 time 0.000837240219116 c_threads 8 time 0.000817658901215 c_threads 9 time 0.000843930244446 c_threads 10 time 0.000861320495605 c_threads 11 time 0.000936930179596 c_threads 12 time 0.000847370624542 The optimum for my Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz seems to be at 3 threads with .6 ms for the given test case. I just reran my normal non-OpenMP C based code, and it takes .84 ms (1.35x slower). from scipy.spatial import distance distance.cdist(a,b) takes .9 ms -- which includes the allocation of the output array, because there is no `out` option available. So I'm happy that OpenMP works, but apparently on my CPU the speed increase is not overwhelming (yet)... Thanks, -- Sebastian On Wed, Feb 16, 2011 at 4:50 AM, Eric Carlson wrote: > I don't have the slightest idea what I'm doing, but.... > > > > ____ > file name - the_lib.c > ___ > #include > #include > #include > #include > > void dists2d( ? ? ?double *a_ps, int na, > ? ? ? ? ? ? ? ? ? double *b_ps, int nb, > ? ? ? ? ? ? ? ? ? double *dist, int num_threads) > { > > ? ?int i, j; > ? ?int dynamic=0; > ? ?omp_set_dynamic(dynamic); > ? ?omp_set_num_threads(num_threads); > ? ?double ax,ay, dif_x, dif_y; > ? ?int nx1=2; > ? ?int nx2=2; > > > #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) > ? ? ? ?for(i=0;i ? ? ? ? ?{ > ? ? ? ? ? ? ? ?ax=a_ps[i*nx1]; > ? ? ? ? ? ? ? ? ay=a_ps[i*nx1+1]; > ? ? ? ? ? ? ? ?for(j=0;j ? ? ? ? ? ? ? ? ?{ ? ? dif_x = ax - b_ps[j*nx2]; > ? ? ? ? ? ? ? ? ? ? ? ? dif_y = ay - b_ps[j*nx2+1]; > ? ? ? ? ? ? ? ? ? ? ? ? dist[2*i+j] ?= sqrt(dif_x*dif_x+dif_y*dif_y); > ? ? ? ? ? ? ? ? ?} > ? ? ? ? ?} > } > > > ________ > > > COMPILE: > __________ > gcc -c the_lib.c -fPIC -fopenmp -ffast-math > gcc -shared -o the_lib.so the_lib.o -lgomp -lm > > > ____ > > the_python_prog.py > _____________ > > from ctypes import * > my_lib=CDLL('the_lib.so') #or full path to lib > import numpy as np > import time > > na=329 > nb=340 > a=np.random.rand(na,2) > b=np.random.rand(nb,2) > c=np.zeros(na*nb) > trials=100 > max_threads = 24 > for k in range(1,max_threads): > ? ? n_threads =c_int(k) > ? ? na2=c_int(na) > ? ? nb2=c_int(nb) > > ? ? start = time.time() > ? ? for k1 in range(trials): > ? ? ? ? ret = > my_lib.dists2d(a.ctypes.data_as(c_void_p),na2,b.ctypes.data_as(c_void_p),nb2,c.ctypes.data_as(c_void_p),n_threads) > ? ? print "c_threads",k, " time ", (time.time()-start)/trials > > > > ____ > Results on my machine, dual xeon, 12 cores > na=329 > nb=340 > ____ > > 100 trials each: > c_threads 1 ?time ?0.00109949827194 > c_threads 2 ?time ?0.0005726313591 > c_threads 3 ?time ?0.000429179668427 > c_threads 4 ?time ?0.000349278450012 > c_threads 5 ?time ?0.000287139415741 > c_threads 6 ?time ?0.000252468585968 > c_threads 7 ?time ?0.000222821235657 > c_threads 8 ?time ?0.000206289291382 > c_threads 9 ?time ?0.000187981128693 > c_threads 10 ?time ?0.000172770023346 > c_threads 11 ?time ?0.000164999961853 > c_threads 12 ?time ?0.000157740116119 > > ____ > ____ > Results on my machine, dual xeon, 12 cores > na=3290 > nb=3400 > ______ > 100 trials each: > c_threads 1 ?time ?0.10744508028 > c_threads 2 ?time ?0.0542239999771 > c_threads 3 ?time ?0.037127559185 > c_threads 4 ?time ?0.0280736112595 > c_threads 5 ?time ?0.0228648614883 > c_threads 6 ?time ?0.0194904088974 > c_threads 7 ?time ?0.0165715909004 > c_threads 8 ?time ?0.0145838689804 > c_threads 9 ?time ?0.0130002498627 > c_threads 10 ?time ?0.0116940999031 > c_threads 11 ?time ?0.0107557415962 > c_threads 12 ?time ?0.00990005016327 (speedup almost 11) > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Andreas.FRENK at 3ds.com Wed Feb 16 09:09:30 2011 From: Andreas.FRENK at 3ds.com (FRENK Andreas) Date: Wed, 16 Feb 2011 14:09:30 +0000 Subject: [Numpy-discussion] Get the index of a comparison of two lists Message-ID: <5F84D7606A50BF4BA04EB53E5E4AF2F51875A422@EU-DCC-MBX03.dsone.3ds.com> Hi, Thanks for the two solutions. Indeed, there are much faster than the brute force method using in. The setmember1d is deprecate din newer python releases, therefore I already switched to in1d(). (Thanks Josef.) Andreas -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Wed Feb 16 10:02:16 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 16 Feb 2011 16:02:16 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Update: I just noticed that using Eric's OpenMP code gave me only a 1.35x speedup when comparing 3 threads vs. my non OpenMP code. However, when comparing 3 threads vs. 1 thread, I could call this a 2.55x speedup. This obviously sounds much better, but is obviously not the number that matters... (Ergo Eric's 11x speedup on 12 cores might also be more like 5x or 6x when compared to simple C. w/o any OpenMP overhead). Finally, I have to report that when plugging the C code (from scipy.spatial.distance.cdist) into my real application, which compares for one sample file point pairs from a sequence of 1000 microscopy images, the speed gain appears to be a total of 2x (only !!) -- compared to my original app that used the simple numpy version implemented like this: def pyDist2(a_ps,b_ps, ab_dist): ab_dist = N.zeros(shape=(a_ps.shape[0],b_ps.shape[0]), dtype= N.float64) for i in range(a_ps.shape[0]): d = b_ps- a_ps[i] ab_dist[i] = N.sqrt( N.sum(d**2, -1) ) return ab_dist (The test I was using before was just testing one pair of two arrays of length 329 and 340, whereas the average length for the 1000 images is 260 It could also be that I have substantial constant over from some OpenGL lists that I'm generating in my app.) Nevertheless, I'm happy having learned something about OpenMP. I'm still trying to get more info from valgrind. And, maybe someone could comment, if SSE would help for the function in question. Thanks, Sebastian. On Wed, Feb 16, 2011 at 1:50 PM, Sebastian Haase wrote: > Eric, > this is amazing !! Thanks very much, I have rarely seen such a compact > source example that just worked. > The timings I get are: > c_threads ?1 ?time ?0.00155731916428 > c_threads ?2 ?time ?0.000829789638519 > c_threads ?3 ?time ?0.000616888999939 > c_threads ?4 ?time ?0.000704760551453 > c_threads ?5 ?time ?0.000933599472046 > c_threads ?6 ?time ?0.000809240341187 > c_threads ?7 ?time ?0.000837240219116 > c_threads ?8 ?time ?0.000817658901215 > c_threads ?9 ?time ?0.000843930244446 > c_threads 10 ?time ?0.000861320495605 > c_threads 11 ?time ?0.000936930179596 > c_threads 12 ?time ?0.000847370624542 > > The optimum for my > Intel(R) Core(TM)2 Quad CPU ? ?Q9550 ?@ 2.83GHz > seems to be at 3 threads with .6 ms for the given test case. > > I just reran my normal non-OpenMP C based code, and it takes > .84 ms ? (1.35x slower). > from scipy.spatial import distance > distance.cdist(a,b) ? takes > .9 ms ?-- which includes the allocation of the output array, because > there is no `out` option available. > > > So I'm happy that OpenMP works, > but apparently on my CPU the speed increase is not overwhelming (yet)... > > Thanks, > -- Sebastian > > > On Wed, Feb 16, 2011 at 4:50 AM, Eric Carlson wrote: >> I don't have the slightest idea what I'm doing, but.... >> >> >> >> ____ >> file name - the_lib.c >> ___ >> #include >> #include >> #include >> #include >> >> void dists2d( ? ? ?double *a_ps, int na, >> ? ? ? ? ? ? ? ? ? double *b_ps, int nb, >> ? ? ? ? ? ? ? ? ? double *dist, int num_threads) >> { >> >> ? ?int i, j; >> ? ?int dynamic=0; >> ? ?omp_set_dynamic(dynamic); >> ? ?omp_set_num_threads(num_threads); >> ? ?double ax,ay, dif_x, dif_y; >> ? ?int nx1=2; >> ? ?int nx2=2; >> >> >> #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) >> ? ? ? ?for(i=0;i> ? ? ? ? ?{ >> ? ? ? ? ? ? ? ?ax=a_ps[i*nx1]; >> ? ? ? ? ? ? ? ? ay=a_ps[i*nx1+1]; >> ? ? ? ? ? ? ? ?for(j=0;j> ? ? ? ? ? ? ? ? ?{ ? ? dif_x = ax - b_ps[j*nx2]; >> ? ? ? ? ? ? ? ? ? ? ? ? dif_y = ay - b_ps[j*nx2+1]; >> ? ? ? ? ? ? ? ? ? ? ? ? dist[2*i+j] ?= sqrt(dif_x*dif_x+dif_y*dif_y); >> ? ? ? ? ? ? ? ? ?} >> ? ? ? ? ?} >> } >> >> >> ________ >> >> >> COMPILE: >> __________ >> gcc -c the_lib.c -fPIC -fopenmp -ffast-math >> gcc -shared -o the_lib.so the_lib.o -lgomp -lm >> >> >> ____ >> >> the_python_prog.py >> _____________ >> >> from ctypes import * >> my_lib=CDLL('the_lib.so') #or full path to lib >> import numpy as np >> import time >> >> na=329 >> nb=340 >> a=np.random.rand(na,2) >> b=np.random.rand(nb,2) >> c=np.zeros(na*nb) >> trials=100 >> max_threads = 24 >> for k in range(1,max_threads): >> ? ? n_threads =c_int(k) >> ? ? na2=c_int(na) >> ? ? nb2=c_int(nb) >> >> ? ? start = time.time() >> ? ? for k1 in range(trials): >> ? ? ? ? ret = >> my_lib.dists2d(a.ctypes.data_as(c_void_p),na2,b.ctypes.data_as(c_void_p),nb2,c.ctypes.data_as(c_void_p),n_threads) >> ? ? print "c_threads",k, " time ", (time.time()-start)/trials >> >> >> >> ____ >> Results on my machine, dual xeon, 12 cores >> na=329 >> nb=340 >> ____ >> >> 100 trials each: >> c_threads 1 ?time ?0.00109949827194 >> c_threads 2 ?time ?0.0005726313591 >> c_threads 3 ?time ?0.000429179668427 >> c_threads 4 ?time ?0.000349278450012 >> c_threads 5 ?time ?0.000287139415741 >> c_threads 6 ?time ?0.000252468585968 >> c_threads 7 ?time ?0.000222821235657 >> c_threads 8 ?time ?0.000206289291382 >> c_threads 9 ?time ?0.000187981128693 >> c_threads 10 ?time ?0.000172770023346 >> c_threads 11 ?time ?0.000164999961853 >> c_threads 12 ?time ?0.000157740116119 >> >> ____ >> ____ >> Results on my machine, dual xeon, 12 cores >> na=3290 >> nb=3400 >> ______ >> 100 trials each: >> c_threads 1 ?time ?0.10744508028 >> c_threads 2 ?time ?0.0542239999771 >> c_threads 3 ?time ?0.037127559185 >> c_threads 4 ?time ?0.0280736112595 >> c_threads 5 ?time ?0.0228648614883 >> c_threads 6 ?time ?0.0194904088974 >> c_threads 7 ?time ?0.0165715909004 >> c_threads 8 ?time ?0.0145838689804 >> c_threads 9 ?time ?0.0130002498627 >> c_threads 10 ?time ?0.0116940999031 >> c_threads 11 ?time ?0.0107557415962 >> c_threads 12 ?time ?0.00990005016327 (speedup almost 11) >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From faltet at pytables.org Wed Feb 16 10:57:03 2011 From: faltet at pytables.org (Francesc Alted) Date: Wed, 16 Feb 2011 16:57:03 +0100 Subject: [Numpy-discussion] Python ctypes and OpenMP mystery In-Reply-To: References: Message-ID: <201102161657.03446.faltet@pytables.org> A Saturday 12 February 2011 21:19:39 Eric Carlson escrigu?: > Hello All, > I have been toying with OpenMP through f2py and ctypes. On the whole, > the results of my efforts have been very encouraging. That said, some > results are a bit perplexing. > > I have written identical routines that I run directly as a C-derived > executable, and through ctypes as a shared library. I am running the > tests on a dual-Xeon Ubuntu system with 12 cores and 24 threads. The > C executable is SLIGHTLY faster than the ctypes at lower thread > counts, but the C eventually has a speedup ratio of 12+, while the > python caps off at 7.7, as shown below: > > threads C-speedup Python-speedup > 1 1 1 > 2 2.07 1.98 > 3 3.1 2.96 > 4 4.11 3.93 > 5 4.97 4.75 > 6 5.94 5.54 > 7 6.83 6.53 > 8 7.78 7.3 > 9 8.68 7.68 > 10 9.62 7.42 > 11 10.38 7.51 > 12 10.44 7.26 > 13 7.19 6.04 > 14 7.7 5.73 > 15 8.27 6.03 > 16 8.81 6.29 > 17 9.37 6.55 > 18 9.9 6.67 > 19 10.36 6.9 > 20 10.98 7.01 > 21 11.45 6.97 > 22 11.92 7.1 > 23 12.2 7.08 > > These ratios are quite consistent from 100KB double arrays to 100MB > double arrays, so I do not think it reflects a Python overhead issue. > There is no question the routine is memory bandwidth constrained, and > I feel lucky to squeeze the eventual 12+ ratio, but I am very > perplexed as to why the performance of the Python-invoked routine > seems to cap off. > > Does anyone have an explanation for the caps? Am I seeing some effect > from ctypes, or the Python engine, or what? It is difficult to realize what could be going on by only looking at the timings. Can you attach a small, self-contained benchmark? Not that I can offer a definitive answer, but I'm curious about this. -- Francesc Alted From ecarlson at eng.ua.edu Wed Feb 16 20:12:27 2011 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Wed, 16 Feb 2011 19:12:27 -0600 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Sebastian, Optimization appears to be important here. I used no optimization in my previous post, so you could try the -O3 compile option: gcc -O3 -c my_lib.c -fPIC -fopenmp -ffast-math for na=329 and nb=340 I get (about 7.5 speedup) c_threads 1 time 0.00103106021881 c_threads 2 time 0.000528309345245 c_threads 3 time 0.000362541675568 c_threads 4 time 0.00028993844986 c_threads 5 time 0.000287840366364 c_threads 6 time 0.000264899730682 c_threads 7 time 0.000244019031525 c_threads 8 time 0.000242137908936 c_threads 9 time 0.000232398509979 c_threads 10 time 0.000227460861206 c_threads 11 time 0.00021938085556 c_threads 12 time 0.000216970443726 c_threads 13 time 0.000215198993683 c_threads 14 time 0.00021940946579 c_threads 15 time 0.000204219818115 c_threads 16 time 0.000216958522797 c_threads 17 time 0.000219728946686 c_threads 18 time 0.000199990272522 c_threads 19 time 0.000157492160797 c_threads 20 time 0.000171000957489 c_threads 21 time 0.000147500038147 c_threads 22 time 0.000141770839691 c_threads 23 time 0.000137741565704 for na=3290 and nb=3400 (about 11.5 speedup) c_threads 1 time 0.100258581638 c_threads 2 time 0.0501346611977 c_threads 3 time 0.0335096096992 c_threads 4 time 0.0253720903397 c_threads 5 time 0.0208190107346 c_threads 6 time 0.0173784399033 c_threads 7 time 0.0148811817169 c_threads 8 time 0.0130474209785 c_threads 9 time 0.011598110199 c_threads 10 time 0.0104278612137 c_threads 11 time 0.00950778007507 c_threads 12 time 0.00870131969452 c_threads 13 time 0.015882730484 c_threads 14 time 0.0148504400253 c_threads 15 time 0.0139465212822 c_threads 16 time 0.0130301308632 c_threads 17 time 0.012240819931 c_threads 18 time 0.011567029953 c_threads 19 time 0.0109891605377 c_threads 20 time 0.0104281497002 c_threads 21 time 0.00992572069168 c_threads 22 time 0.00957406997681 c_threads 23 time 0.00936627149582 for na=329 and nb=340, cdist comes in at 0.00111914873123 which is 1.085x slower than the c version for my system. for na=3290 and nb=3400 cdist gives 0.143441538811 Cheers, Eric From ecarlson at eng.ua.edu Wed Feb 16 20:24:33 2011 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Wed, 16 Feb 2011 19:24:33 -0600 Subject: [Numpy-discussion] Python ctypes and OpenMP mystery In-Reply-To: <201102161657.03446.faltet@pytables.org> References: <201102161657.03446.faltet@pytables.org> Message-ID: Hello Francesc, The problem appears to related to my lack of optimization in the compilation. If I use gcc -O3 -c my_lib.c -fPIC -fopenmp -ffast-math the C executable and ctypes/python versions behave almost identically. Getting decent behavior takes some thought, though, far from the incredible almost-automatic behavior of numexpr. Now I've got to figure out how to scale up a bunch of vector adds/multiplies. Neither numexpr or openmp get you very far with a bunch of "z=a*x+b*y"-type calcs. Cheers, Eric From faltet at pytables.org Thu Feb 17 03:58:14 2011 From: faltet at pytables.org (Francesc Alted) Date: Thu, 17 Feb 2011 09:58:14 +0100 Subject: [Numpy-discussion] Python ctypes and OpenMP mystery In-Reply-To: References: <201102161657.03446.faltet@pytables.org> Message-ID: <201102170958.14538.faltet@pytables.org> A Thursday 17 February 2011 02:24:33 Eric Carlson escrigu?: > Hello Francesc, > The problem appears to related to my lack of optimization in the > compilation. If I use > > gcc -O3 -c my_lib.c -fPIC -fopenmp -ffast-math > > > the C executable and ctypes/python versions behave almost > identically. Ahh, good to know. > Getting decent behavior takes some thought, though, far > from the incredible almost-automatic behavior of numexpr. numexpr uses a very simple method for distributing load among the threads, so I suppose this is why it is fast. The drawback is that numexpr only can be used for operations implying the same index (i.e. like a+b**3, but not for things like a[i+1]+b[i]**3). For other operations openmp is probably the best option (I should say the *easiest* option) right now. > Now I've got to figure out how to scale up a bunch of vector > adds/multiplies. Neither numexpr or openmp get you very far with a > bunch of "z=a*x+b*y"-type calcs. For these sort of computations you are most probably hitting the memory bandwidth wall, so you are out of luck (at least until processors will be fast enough to allow compression to actually reduce the time spent in computations). Cheers, -- Francesc Alted From seb.haase at gmail.com Thu Feb 17 04:16:37 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 17 Feb 2011 10:16:37 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Eric, thanks for insisting on this. I noticed that, when I saw it first, just to forget about it again ... The new timings on my machine are: $: gcc -O3 -c the_lib.c -fPIC -fopenmp -ffast-math $: gcc -shared -o the_lib.so the_lib.o -lgomp -lm $: python2.5 the_python_prog.py c_threads 1 time 0.000897128582001 c_threads 2 time 0.000540800094604 c_threads 3 time 0.00035933971405 c_threads 4 time 0.000529370307922 c_threads 5 time 0.00049122095108 c_threads 6 time 0.000540502071381 c_threads 7 time 0.000580079555511 c_threads 8 time 0.000643739700317 c_threads 9 time 0.000622930526733 c_threads 10 time 0.000680360794067 c_threads 11 time 0.000613269805908 c_threads 12 time 0.000633401870728 That is, your OpenMP version is again fastest using 3 threads on my 4 core CPU. It is now 2.34x times faster than my non-OpenMP code (which compares to scipy...cdist). And, it is (only !?) 7% slower than then non-OpenMP code, when running on 1 thread. (The speedup 3 threads vs. 1 thread is 2.5x) So, that is pretty good !! What I don't understand, why did you start your first post with "I don't have the slightest idea what I'm doing" ;-) Do you think, one could get even better ? And, where does the 7% slow-down (for single thread) come from ? Is it possible to have the OpenMP option in a code, without _any_ penalty for 1 core machines ? Thanks, - Sebastian On Thu, Feb 17, 2011 at 2:12 AM, Eric Carlson wrote: > Sebastian, > Optimization appears to be important here. I used no optimization in my > previous post, so you could try the -O3 compile option: > > ?gcc -O3 -c my_lib.c -fPIC -fopenmp -ffast-math > > for na=329 and nb=340 I get (about 7.5 speedup) > c_threads 1 ?time ?0.00103106021881 > c_threads 2 ?time ?0.000528309345245 > c_threads 3 ?time ?0.000362541675568 > c_threads 4 ?time ?0.00028993844986 > c_threads 5 ?time ?0.000287840366364 > c_threads 6 ?time ?0.000264899730682 > c_threads 7 ?time ?0.000244019031525 > c_threads 8 ?time ?0.000242137908936 > c_threads 9 ?time ?0.000232398509979 > c_threads 10 ?time ?0.000227460861206 > c_threads 11 ?time ?0.00021938085556 > c_threads 12 ?time ?0.000216970443726 > c_threads 13 ?time ?0.000215198993683 > c_threads 14 ?time ?0.00021940946579 > c_threads 15 ?time ?0.000204219818115 > c_threads 16 ?time ?0.000216958522797 > c_threads 17 ?time ?0.000219728946686 > c_threads 18 ?time ?0.000199990272522 > c_threads 19 ?time ?0.000157492160797 > c_threads 20 ?time ?0.000171000957489 > c_threads 21 ?time ?0.000147500038147 > c_threads 22 ?time ?0.000141770839691 > c_threads 23 ?time ?0.000137741565704 > > for na=3290 and nb=3400 (about 11.5 speedup) > c_threads 1 ?time ?0.100258581638 > c_threads 2 ?time ?0.0501346611977 > c_threads 3 ?time ?0.0335096096992 > c_threads 4 ?time ?0.0253720903397 > c_threads 5 ?time ?0.0208190107346 > c_threads 6 ?time ?0.0173784399033 > c_threads 7 ?time ?0.0148811817169 > c_threads 8 ?time ?0.0130474209785 > c_threads 9 ?time ?0.011598110199 > c_threads 10 ?time ?0.0104278612137 > c_threads 11 ?time ?0.00950778007507 > c_threads 12 ?time ?0.00870131969452 > c_threads 13 ?time ?0.015882730484 > c_threads 14 ?time ?0.0148504400253 > c_threads 15 ?time ?0.0139465212822 > c_threads 16 ?time ?0.0130301308632 > c_threads 17 ?time ?0.012240819931 > c_threads 18 ?time ?0.011567029953 > c_threads 19 ?time ?0.0109891605377 > c_threads 20 ?time ?0.0104281497002 > c_threads 21 ?time ?0.00992572069168 > c_threads 22 ?time ?0.00957406997681 > c_threads 23 ?time ?0.00936627149582 > > > for na=329 and nb=340, cdist comes in at 0.00111914873123 which is > 1.085x slower than the c version for my system. > > for na=3290 and nb=3400 cdist gives ?0.143441538811 > > Cheers, > Eric > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From matthieu.brucher at gmail.com Thu Feb 17 04:29:25 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 17 Feb 2011 10:29:25 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: > Do you think, one could get even better ? > And, where does the 7% slow-down (for single thread) come from ? > Is it possible to have the OpenMP option in a code, without _any_ > penalty for 1 core machines ? > There will always be a penalty for parallel code that runs on one core. You have at least the overhead for splitting the data. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Thu Feb 17 04:39:38 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 17 Feb 2011 10:39:38 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: On Thu, Feb 17, 2011 at 10:29 AM, Matthieu Brucher wrote: > >> Do you think, one could get even better ? >> And, where does the 7% slow-down (for single thread) come from ? >> Is it possible to have the OpenMP option in a code, without _any_ >> penalty for 1 core machines ? > > There will always be a penalty for parallel code that runs on one core. You > have at least the overhead for splitting the data. > I was referring to when num_threads=1; // and omp_set_num_threads(num_threads); is explicitly called. Then, where does the overhead come from ? -- The call to omp_set_dynamic(dynamic); Or the #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) or some magic done by gcc ... -fopenmp ? (I'm referring to Eric Carlson's earlier in this thread) I'm wondering if one could have a C "if"-statement, e.g. if(num_threads == 0) to then not do any of the omp_xxx() calls. Obviously, the #pragma would have to be replaceable by some omp_xxx() call first Thanks, - Sebastian From matthieu.brucher at gmail.com Thu Feb 17 04:57:51 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 17 Feb 2011 10:57:51 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: > Then, where does the overhead come from ? -- > The call to omp_set_dynamic(dynamic); > Or the > #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) > It may be this. You initialize a thread pool, even if it has only one thread, and there is the dynamic part, so OpenMP may create several chunks instead of one big chunk. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Thu Feb 17 08:12:22 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 17 Feb 2011 14:12:22 +0100 Subject: [Numpy-discussion] why is www.numpy.org an HTML frame for numpy.scipy.org ?? Message-ID: Hi, I just noticed, that www.numpy.org is now a really nice looking page -- congratulations ! But, when I click on any link - e.g. "Older python array packages" my browser shows me a new page, but keeps pointing to www.numpy.org. Obviously, www.numpy.org shows just a frame, and the real page is at http://numpy.scipy.org. I would vote, removing that frame and redirect www.numpy.org to numpy.scipy.org Is there any other use of numpy.org which I'm overlooking ? Cheers, -- Sebastian Haase ' From seb.haase at gmail.com Thu Feb 17 10:23:45 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 17 Feb 2011 16:23:45 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: Hi, More surprises: shaase at iris:~/code/SwiggedDistOMP: gcc -O3 -c the_lib.c -fPIC -fopenmp -ffast-math shaase at iris:~/code/SwiggedDistOMP: gcc -shared -o the_lib.so the_lib.o -lgomp -lm shaase at iris:~/code/SwiggedDistOMP: priithon the_python_prog.py c_threads 0 time 0.000437839031219 # this is now, without #pragma omp parallel for ... c_threads 1 time 0.000865449905396 c_threads 2 time 0.000520548820496 c_threads 3 time 0.00033704996109 c_threads 4 time 0.000620169639587 c_threads 5 time 0.000465350151062 c_threads 6 time 0.000696349143982 This correct now the timing of, max OpenMP speed (3 threads) vs. no OpenMP to speedup of (only!) 1.3x Not 2.33x (which was the number I got when comparing OpenMP to the cdist function). The c code is now: the_lib.c ------------------------------------------------------------------------------------------ #include #include #include #include void dists2d( double *a_ps, int na, double *b_ps, int nb, double *dist, int num_threads) { int i, j; double ax,ay, dif_x, dif_y; int nx1=2; int nx2=2; if(num_threads>0) { int dynamic=0; omp_set_dynamic(dynamic); omp_set_num_threads(num_threads); #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) for(i=0;i wrote: > >> Then, where does the overhead come from ? -- >> The call to ? ?omp_set_dynamic(dynamic); >> Or the >> #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) > > It may be this. You initialize a thread pool, even if it has only one > thread, and there is the dynamic part, so OpenMP may create several chunks > instead of one big chunk. > > Matthieu > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From matthieu.brucher at gmail.com Thu Feb 17 10:31:13 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 17 Feb 2011 16:31:13 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: It may also be the sizes of the chunk OMP uses. You can/should specify them.in the OMP pragma so that it is a multiple of the cache line size or something close. Matthieu 2011/2/17 Sebastian Haase > Hi, > More surprises: > shaase at iris:~/code/SwiggedDistOMP: gcc -O3 -c the_lib.c -fPIC -fopenmp > -ffast-math > shaase at iris:~/code/SwiggedDistOMP: gcc -shared -o the_lib.so the_lib.o > -lgomp -lm > shaase at iris:~/code/SwiggedDistOMP: priithon the_python_prog.py > c_threads 0 time 0.000437839031219 # this is now, without > #pragma omp parallel for ... > c_threads 1 time 0.000865449905396 > c_threads 2 time 0.000520548820496 > c_threads 3 time 0.00033704996109 > c_threads 4 time 0.000620169639587 > c_threads 5 time 0.000465350151062 > c_threads 6 time 0.000696349143982 > > This correct now the timing of, max OpenMP speed (3 threads) vs. no > OpenMP to speedup of (only!) 1.3x > Not 2.33x (which was the number I got when comparing OpenMP to the > cdist function). > The c code is now: > > the_lib.c > > ------------------------------------------------------------------------------------------ > #include > #include > #include > #include > > void dists2d( double *a_ps, int na, > double *b_ps, int nb, > double *dist, int num_threads) > { > > int i, j; > double ax,ay, dif_x, dif_y; > int nx1=2; > int nx2=2; > > if(num_threads>0) > { > int dynamic=0; > omp_set_dynamic(dynamic); > omp_set_num_threads(num_threads); > > > #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) > for(i=0;i { > ax=a_ps[i*nx1]; > ay=a_ps[i*nx1+1]; > for(j=0;j { dif_x = ax - b_ps[j*nx2]; > dif_y = ay - b_ps[j*nx2+1]; > dist[2*i+j] = sqrt(dif_x*dif_x+dif_y*dif_y); > } > } > } else { > for(i=0;i { > ax=a_ps[i*nx1]; > ay=a_ps[i*nx1+1]; > for(j=0;j { dif_x = ax - b_ps[j*nx2]; > dif_y = ay - b_ps[j*nx2+1]; > dist[2*i+j] = sqrt(dif_x*dif_x+dif_y*dif_y); > } > } > } > } > ------------------------------------------------------------------ > $ gcc -O3 -c the_lib.c -fPIC -fopenmp -ffast-math > $ gcc -shared -o the_lib.so the_lib.o -lgomp -lm > > So, I guess I found a way of getting rid of the OpenMP overhead when > run with 1 thread, > and found that - if measured correctly, using same compiler settings > and so on - the speedup is so small that there no point in doing > OpenMP - again. > (For my case, having (only) 4 cores) > > > Cheers, > Sebastian. > > > > On Thu, Feb 17, 2011 at 10:57 AM, Matthieu Brucher > wrote: > > > >> Then, where does the overhead come from ? -- > >> The call to omp_set_dynamic(dynamic); > >> Or the > >> #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) > > > > It may be this. You initialize a thread pool, even if it has only one > > thread, and there is the dynamic part, so OpenMP may create several > chunks > > instead of one big chunk. > > > > Matthieu > > -- > > Information System Engineer, Ph.D. > > Blog: http://matt.eifelle.com > > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Feb 17 18:00:05 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 17 Feb 2011 18:00:05 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> Message-ID: <20110217180005.06931fcf@python.org> Hi Ralf, On Feb 12, 2011, at 06:12 PM, Ralf Gommers wrote: > Now that you're here to discuss Ubuntu packaging, can you tell us if you or >another packager plan to include scipy 0.9.0 in Natty if it's released on >time? Final release is planned for the weekend of 19/20 Feb. Feature freeze for Natty is February 24th, so if the timing is *just right* it may be possible. Ideally, we'd get 0.9.0 into Debian right after scipy 0.9.0 is released, and then sync it to Ubuntu. I don't have permission to upload to Debian, so we'd have to coordinate that between upstream (you guys), Debian Python Modules Team , and Ubuntu. If you remind me, I'll will try to facilitate that. One question though: is 0.9.0 API and backward compatible with 0.8.0? I see we have a couple of dozen reverse dependencies on python-scipy, so if there's any chance that 0.9.0 would break those dependencies, we may have to push that off until Natty+1. Here are a list of the reverse dependencies in Natty: spyder shogun-python-modular science-statistics science-numericalcomputation science-nanoscale-physics pytrainer python-symeig python-scitools python-scipy-dbg python-scikits-learn python-openopt python-nipype python-nipy python-nibabel python-networkx python-mvpa python-mdp python-matplotlib python-gastables psychopy model-builder mayavi2 gvb gnuradio-utils I can probably run a test build in a PPA for these packages, but the timing will be tight. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Thu Feb 17 18:11:57 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 17 Feb 2011 18:11:57 -0500 Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu In-Reply-To: References: <20110210122200.1667933e@python.org> <1297376809.3224.9.camel@Obelisk> <20110211104057.46393960@python.org> <1297440445.2884.117.camel@talisman> <4D557318.50305@gmail.com> <20110211144730.2c51a304@limelight.wooz.org> Message-ID: <20110217181157.393a7bbd@python.org> On Feb 11, 2011, at 01:57 PM, Robert Kern wrote: >If you are going to do an Ubuntu-only fix, would that allow using our >prebuilt doc packages? IIRC, the only objection to that was that it was >Debian policy to build docs from sources. But then again, that might have >just been the first objection. Probably not. We'd want to build the docs from source just like Debian would. I thought of perhaps a middle ground. We leave the python-numpy source package as it currently is, providing a python-numpy-doc binary package that disables the matplotlib extension. We'd add a new Ubuntu-only universe package that built, e.g. python-numpy-doc-complete which could include the matplotlib extension. It might be tricky to work out the exact details, since a naive approach would have file conflicts. (E.g. could add a binary package conflict or install the docs into a different location). I don't know that I'll get to this for Natty, but in any case, this bug is tracking the issue: https://bugs.launchpad.net/ubuntu/+source/python-numpy/+bug/721007 Thanks everyone for your input. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From pav at iki.fi Thu Feb 17 19:16:10 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 18 Feb 2011 00:16:10 +0000 (UTC) Subject: [Numpy-discussion] numpy docs dependency problem in Ubuntu References: <20110210122200.1667933e@python.org> <20110217180005.06931fcf@python.org> Message-ID: On Thu, 17 Feb 2011 18:00:05 -0500, Barry Warsaw wrote: [clip] > One question though: is 0.9.0 API and backward compatible with 0.8.0? I > see we have a couple of dozen reverse dependencies on python-scipy, so > if there's any chance that 0.9.0 would break those dependencies, we may > have to push that off until Natty+1. The Scipy releases in general are not fully backward API compatible (until 1.0 is reached), and this applies also to 0.8.0 vs. 0.9.0. So I guess the timing is pretty tight... -- Pauli Virtanen From ecarlson at eng.ua.edu Thu Feb 17 20:59:24 2011 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Thu, 17 Feb 2011 19:59:24 -0600 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: For 4 cores, on your system, your conclusion makes some sense. That said, I played around with this on both a core 2 duo and the 12 core system. For the 12-core system, on my tests the 0 case ran extremely close to the 2-thread case for all my sizes. The core 2 duo runs windows 7, and after downloading pthreadsGC2.dll from the pthreads project, I was able to use openmp under a year-old (32-bit) pythonxy distribution with mingw. The result, 0 threads come in slightly faster than one thread, .00102 versus .00106, and 2 threads took .00060. My current theory is that gcc under linux uses some background trick to get two thread-like streams going. As I assess scale-up under linux, I will need to consider this behavior. Creating optimal codes with OpenMP certainly requires a considerable commitment. Given the problem-specific fine tuning required, I would not expect much gain in general-purpose routines. In specific routines like cdist, it might make more sense. I talked to a Dell HPC rep today, and he said that squeezing out an extra 15% performance boost on an Intel CPU was a pleasant surprise, so the 30% improvement is maybe not so bad. Cheers, Eric From asmund.hjulstad at gmail.com Fri Feb 18 04:20:26 2011 From: asmund.hjulstad at gmail.com (=?ISO-8859-1?Q?=C5smund_Hjulstad?=) Date: Fri, 18 Feb 2011 10:20:26 +0100 Subject: [Numpy-discussion] numpy.fromiter hiding exceptions Message-ID: Hi all, I am finding it hard to debug cases where I am creating numpy arrays from generators, and the generator function throws an exception. It seems that numpy just swallows the exception, and what I get is a not too helpful ValueError: iterator too short Much more helpfull would be to see the original exception, and get the stacktrace in ipython (or wherever I am working). Is this possible in some easy way, or am I stuck with the equivalent of if debug: mygenerator = list(mygenerator) a = np.fromiter(iter(mygenerator), dtype=xxx, count=xxx) -- ?smund Hjulstad, asmund at hjulstad.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From scipy at SamuelJohn.de Fri Feb 18 05:32:07 2011 From: scipy at SamuelJohn.de (Samuel John) Date: Fri, 18 Feb 2011 11:32:07 +0100 Subject: [Numpy-discussion] How to tell if I succeeded to build numpy with amd, umfpack and lapack In-Reply-To: References: <20110127060951.GB21623@ykcyc> Message-ID: Ping. How to tell, if numpy successfully build against libamd.a and libumfpack.a? How do I know if they were successfully linked (statically)? Is it possible from within numpy, like show_config() ? I think show_config() has no information about these in it :-( Anybody? Thanks, Samuel From sccolbert at gmail.com Fri Feb 18 09:49:58 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 18 Feb 2011 09:49:58 -0500 Subject: [Numpy-discussion] How to tell if I succeeded to build numpy with amd, umfpack and lapack In-Reply-To: References: <20110127060951.GB21623@ykcyc> Message-ID: It certainly does. Here is mine, showing that numpy is linked against mkl: In [2]: np.show_config() lapack_opt_info: libraries = ['mkl_lapack95', 'mkl_intel', 'mkl_intel_thread', 'mkl_core', 'mkl_p4m', 'mkl_p4p', 'pthread'] library_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/include'] blas_opt_info: libraries = ['mkl_intel', 'mkl_intel_thread', 'mkl_core', 'mkl_p4m', 'mkl_p4p', 'pthread'] library_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/include'] lapack_mkl_info: libraries = ['mkl_lapack95', 'mkl_intel', 'mkl_intel_thread', 'mkl_core', 'mkl_p4m', 'mkl_p4p', 'pthread'] library_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/include'] blas_mkl_info: libraries = ['mkl_intel', 'mkl_intel_thread', 'mkl_core', 'mkl_p4m', 'mkl_p4p', 'pthread'] library_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/include'] mkl_info: libraries = ['mkl_intel', 'mkl_intel_thread', 'mkl_core', 'mkl_p4m', 'mkl_p4p', 'pthread'] library_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/Library/Frameworks/Python.framework/Versions/1.3.0/include'] If those lists are empty for you, then numpy is not linked properly (of course yours will be ATLAS and not mkl) On Fri, Feb 18, 2011 at 5:32 AM, Samuel John wrote: > Ping. > > How to tell, if numpy successfully build against libamd.a and libumfpack.a? > How do I know if they were successfully linked (statically)? > Is it possible from within numpy, like show_config() ? > I think show_config() has no information about these in it :-( > > Anybody? > > Thanks, > Samuel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Fri Feb 18 11:04:45 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 18 Feb 2011 11:04:45 -0500 Subject: [Numpy-discussion] help with translating some matlab Message-ID: I got the following matlab code from a colleage: initialization: h =zeros(1, N); %% initial filter coefficients lambda =1; delta =0.001; P =eye(N)/delta; loop: z =P*(x1'); g =z/(lambda+ x1*z); y = h*x1'; %% filter output e = ph_cpm_out(n) - y; %% error h = h + e*g'; %% adaptation Pnext=(P-g*z')/lambda; P =Pnext; So it looks to me: z is a vector The step g=z/(lambda+x1*z) seems to be a vector division. How do I translate this to numpy? From robince at gmail.com Fri Feb 18 11:16:59 2011 From: robince at gmail.com (Robin) Date: Fri, 18 Feb 2011 17:16:59 +0100 Subject: [Numpy-discussion] How to tell if I succeeded to build numpy with amd, umfpack and lapack In-Reply-To: References: <20110127060951.GB21623@ykcyc> Message-ID: I think numpy doesn't use umfpack. scipy.sparse used to, but now the umfpack stuff has been moved out to a scikit. So you probably won't see anything about those libraries, but if you install scikits.umfpack and it works then you must be linked correctly. Cheers Robin On Fri, Feb 18, 2011 at 11:32 AM, Samuel John wrote: > Ping. > > How to tell, if numpy successfully build against libamd.a and libumfpack.a? > How do I know if they were successfully linked (statically)? > Is it possible from within numpy, like show_config() ? > I think show_config() has no information about these in it :-( > > Anybody? > > Thanks, > ?Samuel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From warren.weckesser at enthought.com Fri Feb 18 11:16:53 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Fri, 18 Feb 2011 10:16:53 -0600 Subject: [Numpy-discussion] help with translating some matlab In-Reply-To: References: Message-ID: On Fri, Feb 18, 2011 at 10:04 AM, Neal Becker wrote: > I got the following matlab code from a colleage: > > initialization: > > h =zeros(1, N); %% initial filter coefficients > lambda =1; > delta =0.001; > > P =eye(N)/delta; > > loop: > > z =P*(x1'); > > g =z/(lambda+ x1*z); > > y = h*x1'; %% filter output > > e = ph_cpm_out(n) - y; %% error > > h = h + e*g'; %% adaptation > > Pnext=(P-g*z')/lambda; > > P =Pnext; > > So it looks to me: > z is a vector > > The step g=z/(lambda+x1*z) seems to be a vector division. > How do I translate this to numpy? > It looks like x1 is a row vector and z is a column vector, so x1*z is a scalar. With numpy, something like g = z / (lambda + np.dot(x1, z)) should work. Warren > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Fri Feb 18 13:11:39 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 18 Feb 2011 13:11:39 -0500 Subject: [Numpy-discussion] help with translating some matlab References: Message-ID: My translation is: x1 = rcv[n:n-N:-1] z = np.dot (P, x1.conj().transpose()) g = z / (_lambda + np.dot (x1, z)) y = np.dot (h, x1.conj().transpose()) e = x[n-N/2] - y h += np.dot (e, g.conj().transpose()) P = (P - np.dot (g, z.conj().transpose()))/_lambda But it doesn't work. You say z should be a column vector. I got: In [138]: x1.shape Out[138]: (64,) In [139]: z.shape Out[139]: (64,) Clearly, I did something wrong here. From ndbecker2 at gmail.com Fri Feb 18 13:50:49 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 18 Feb 2011 13:50:49 -0500 Subject: [Numpy-discussion] help with translating some matlab References: Message-ID: Neal Becker wrote: > My translation is: > > x1 = rcv[n:n-N:-1] > > z = np.dot (P, x1.conj().transpose()) > > g = z / (_lambda + np.dot (x1, z)) > > y = np.dot (h, x1.conj().transpose()) > > e = x[n-N/2] - y > > h += np.dot (e, g.conj().transpose()) > > P = (P - np.dot (g, z.conj().transpose()))/_lambda > > But it doesn't work. > > You say z should be a column vector. I got: > In [138]: x1.shape > Out[138]: (64,) > > In [139]: z.shape > Out[139]: (64,) > > Clearly, I did something wrong here. I think I've got it. In numpy, a 'vector' is a row vector. If I want to turn a row vector into a column vector, transpose doesn't work. I need to use newaxis for that. So the whole translation is: x1 = rcv[n:n-N:-1] z = np.dot (P, x1.conj()[:,np.newaxis]) g = z / (_lambda + np.dot (x1, z)) y = np.dot (h, x1.conj()[:,np.newaxis]) e = x[n] - y h += np.dot (e, g.conj().transpose()) P = (P - np.dot (g, z.conj().transpose()))/_lambda From warren.weckesser at enthought.com Fri Feb 18 14:23:20 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Fri, 18 Feb 2011 13:23:20 -0600 Subject: [Numpy-discussion] help with translating some matlab In-Reply-To: References: Message-ID: On Fri, Feb 18, 2011 at 12:50 PM, Neal Becker wrote: > Neal Becker wrote: > > > My translation is: > > > > x1 = rcv[n:n-N:-1] > > > > z = np.dot (P, x1.conj().transpose()) > > > > g = z / (_lambda + np.dot (x1, z)) > > > > y = np.dot (h, x1.conj().transpose()) > > > > e = x[n-N/2] - y > > > > h += np.dot (e, g.conj().transpose()) > > > > P = (P - np.dot (g, z.conj().transpose()))/_lambda > > > > But it doesn't work. > > > > You say z should be a column vector. I got: > > In [138]: x1.shape > > Out[138]: (64,) > > > > In [139]: z.shape > > Out[139]: (64,) > > > > Clearly, I did something wrong here. > > I think I've got it. In numpy, a 'vector' is a row vector. If I want to > turn a > row vector into a column vector, transpose doesn't work. I need to use > newaxis > for that. So the whole translation is: > > x1 = rcv[n:n-N:-1] > > z = np.dot (P, x1.conj()[:,np.newaxis]) > > g = z / (_lambda + np.dot (x1, z)) > > y = np.dot (h, x1.conj()[:,np.newaxis]) > > e = x[n] - y > > h += np.dot (e, g.conj().transpose()) > > P = (P - np.dot (g, z.conj().transpose()))/_lambda > > Sure, you can implement Matlab's row and column vectors that way. I would probably stick with true 1D arrays (rather than Nx1 2D arrays), and implement those lines something like this: x1 = rcv[n:n-N:-1] z = np.dot(P, x1.conj()) g = z / (_lambda + np.dot(x1, z)) y = np.vdot(x1, h) e = x[n] - y # Note: e is a scalar. h += e * g.conj() P = (P - np.outer(g, z.conj()))/_lambda Note the use of vdot(); vdot(a,b) conjugates a and then dots with b. Also note the use of outer() in the line that updates P. Warren > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Fri Feb 18 18:40:07 2011 From: sturla at molden.no (Sturla Molden) Date: Sat, 19 Feb 2011 00:40:07 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: Message-ID: <4D5F0357.6070801@molden.no> Den 17.02.2011 16:31, skrev Matthieu Brucher: > It may also be the sizes of the chunk OMP uses. You can/should specify > them.in > > Matthieu > > the OMP pragma so that it is a multiple of the cache line size or > something close. Also beware of "false sharing" among the threads. When one processor updates the array "dist" in Sebastian's code, the cache line is dirtied for the other processors: #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) for(i=0;i From sturla at molden.no Fri Feb 18 18:45:05 2011 From: sturla at molden.no (Sturla Molden) Date: Sat, 19 Feb 2011 00:45:05 +0100 Subject: [Numpy-discussion] help with translating some matlab In-Reply-To: References: Message-ID: <4D5F0481.9040805@molden.no> I think x.conj().transpose() is too verbose, use x.H instead :-) Sturla Den 18.02.2011 19:11, skrev Neal Becker: > My translation is: > > x1 = rcv[n:n-N:-1] > > z = np.dot (P, x1.conj().transpose()) > > g = z / (_lambda + np.dot (x1, z)) > > y = np.dot (h, x1.conj().transpose()) > > e = x[n-N/2] - y > > h += np.dot (e, g.conj().transpose()) > > P = (P - np.dot (g, z.conj().transpose()))/_lambda > > But it doesn't work. > > You say z should be a column vector. I got: > In [138]: x1.shape > Out[138]: (64,) > > In [139]: z.shape > Out[139]: (64,) > > Clearly, I did something wrong here. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From scipy at SamuelJohn.de Sat Feb 19 06:07:47 2011 From: scipy at SamuelJohn.de (Samuel John) Date: Sat, 19 Feb 2011 12:07:47 +0100 Subject: [Numpy-discussion] How to tell if I succeeded to build numpy with amd, umfpack and lapack In-Reply-To: References: <20110127060951.GB21623@ykcyc> Message-ID: Thanks Robin, that makes sense and explains why I could not find any reference. Perhaps the scipy.org wiki and install instructions should be updated. I mean how many people try to compile amd and umfpack, because they think it's good for numpy to have them, because the site.cfg contains those entries! To conclude: numpy does *NOT* use umfpack or libamd at all. Those sections in the site.cfg are outdated. From seb.haase at gmail.com Sat Feb 19 12:13:44 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Sat, 19 Feb 2011 18:13:44 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: <4D5F0357.6070801@molden.no> References: <4D5F0357.6070801@molden.no> Message-ID: Thanks a lot. Very informative. I guess what you say about "cache line is dirtied" is related to the info I got with valgrind (see my email in this thread: L1 Data Write Miss 3636). Can one assume that the cache line is always a few mega bytes ? Thanks, Sebastian On Sat, Feb 19, 2011 at 12:40 AM, Sturla Molden wrote: > Den 17.02.2011 16:31, skrev Matthieu Brucher: > > It may also be the sizes of the chunk OMP uses. You can/should specify > them.in > > Matthieu > > the OMP pragma so that it is a multiple of the cache line size or something > close. > > Also beware of "false sharing" among the threads. When one processor updates > the array "dist" in Sebastian's code, the cache line is dirtied for the > other processors: > > ? #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) > ? for(i=0;i ? ?? ax=a_ps[i*nx1]; > ? ?? ay=a_ps[i*nx1+1]; > ? ?? for(j=0;j ???????? dif_x = ax - b_ps[j*nx2]; > ? ? ? ?? dif_y = ay - b_ps[j*nx2+1]; > > ???????? /* update shared memory */ > > ? ? ? ?? dist[2*i+j] ?= sqrt(dif_x*dif_x+dif_y*dif_y); > > ???????? /* ... and poof the cache is dirty */ > > ???? } > ? } > > Whenever this happens, the processors must stop whatever they are doing to > resynchronize their cache lines. "False sharing" can therefore work as an > "invisible GIL" inside OpenMP code.The processors can appear to run in > syrup, and there is excessive traffic on the memory bus. > > This is also why MPI programs often scale better than OpenMP programs, > despite the IPC overhead. > > An advice when working with OpenMP is to let each thread write to private > data arrays, and only share read-only arrays. > > One can e.g. use OpenMP's "reduction" pragma to achieve this. E.g. intialize > the array dist with zeros, and use reduction(+:dist) in the OpenMP pragma > line. > > Sturla > From pav at iki.fi Sat Feb 19 12:50:02 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 19 Feb 2011 17:50:02 +0000 (UTC) Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? References: <4D5F0357.6070801@molden.no> Message-ID: On Sat, 19 Feb 2011 18:13:44 +0100, Sebastian Haase wrote: > Thanks a lot. Very informative. I guess what you say about "cache line > is dirtied" is related to the info I got with valgrind (see my email in > this thread: L1 Data Write Miss 3636). Can one assume that the cache > line is always a few mega bytes ? Cache lines are typically much smaller, 16-512 bytes. In this specific case, since the stride of the `i` loop is only 2*sizeof(float) = 16 bytes << cache line size, threads running with different `i` tend to write to the same cache lines. -- Pauli Virtanen From matthieu.brucher at gmail.com Sat Feb 19 15:49:52 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 19 Feb 2011 21:49:52 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: <4D5F0357.6070801@molden.no> Message-ID: Write miss are indication that data had to be imported inside L1 before it could be written. I don't know if valgrind can give indication of false sharing, unfortunately. That's why I suggested you use a multiple of the cache line so that false sharing do not occur. Matthieu 2011/2/19 Sebastian Haase > Thanks a lot. Very informative. I guess what you say about "cache line > is dirtied" is related to the info I got with valgrind (see my email > in this thread: L1 Data Write Miss 3636). > Can one assume that the cache line is always a few mega bytes ? > > Thanks, > Sebastian > > On Sat, Feb 19, 2011 at 12:40 AM, Sturla Molden wrote: > > Den 17.02.2011 16:31, skrev Matthieu Brucher: > > > > It may also be the sizes of the chunk OMP uses. You can/should specify > > them.in > > > > Matthieu > > > > the OMP pragma so that it is a multiple of the cache line size or > something > > close. > > > > Also beware of "false sharing" among the threads. When one processor > updates > > the array "dist" in Sebastian's code, the cache line is dirtied for the > > other processors: > > > > #pragma omp parallel for private(j, i,ax,ay, dif_x, dif_y) > > for(i=0;i > ax=a_ps[i*nx1]; > > ay=a_ps[i*nx1+1]; > > for(j=0;j > dif_x = ax - b_ps[j*nx2]; > > dif_y = ay - b_ps[j*nx2+1]; > > > > /* update shared memory */ > > > > dist[2*i+j] = sqrt(dif_x*dif_x+dif_y*dif_y); > > > > /* ... and poof the cache is dirty */ > > > > } > > } > > > > Whenever this happens, the processors must stop whatever they are doing > to > > resynchronize their cache lines. "False sharing" can therefore work as an > > "invisible GIL" inside OpenMP code.The processors can appear to run in > > syrup, and there is excessive traffic on the memory bus. > > > > This is also why MPI programs often scale better than OpenMP programs, > > despite the IPC overhead. > > > > An advice when working with OpenMP is to let each thread write to private > > data arrays, and only share read-only arrays. > > > > One can e.g. use OpenMP's "reduction" pragma to achieve this. E.g. > intialize > > the array dist with zeros, and use reduction(+:dist) in the OpenMP pragma > > line. > > > > Sturla > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sat Feb 19 18:01:59 2011 From: sturla at molden.no (Sturla Molden) Date: Sun, 20 Feb 2011 00:01:59 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: References: <4D5F0357.6070801@molden.no> Message-ID: <4D604BE7.1050104@molden.no> Den 19.02.2011 18:13, skrev Sebastian Haase: > Can one assume that the cache line is always a few mega bytes ? Don't confuse the size of a cache with the size of a cache line. A "cache line" (which is the unit that gets marked dirty) is typically 8-512 bytes. Make sure your OpenMP threads stay off each others cache lines, and it will scale nicely. For example, you can specify a chunk-size to the "schedule" pragma to force them apart; it applies to the loop index, so you must do calculations for the block size on the shared write buffer. If you use reduction(+:dist) the write buffer will be completely private, but you get summations after the loop. That is the limited amount of control you get with OpenMP. pthreads will give you better control than OpenMP, but are messy and painful to work with. With MPI you have separate processes, so everything is completely isolated. It's more difficult to program and debug than OpenMP code, but will usually perform better. Sturla From faltet at pytables.org Sun Feb 20 07:09:37 2011 From: faltet at pytables.org (Francesc Alted) Date: Sun, 20 Feb 2011 13:09:37 +0100 Subject: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ? In-Reply-To: <4D604BE7.1050104@molden.no> References: <4D604BE7.1050104@molden.no> Message-ID: <201102201309.37292.faltet@pytables.org> A Sunday 20 February 2011 00:01:59 Sturla Molden escrigu?: > pthreads will give you better control than OpenMP, but are messy and > painful to work with. > > With MPI you have separate processes, so everything is completely > isolated. It's more difficult to program and debug than OpenMP code, > but will usually perform better. To be more specific, MPI will perform better if you don't need to share the memory of your working set among all your processes, but in case you need to do this, threads (pthreads, OpenMP) lets you access all parts of the working set in memory much more easily. In fact, all the threads of a process can access the complete working set transparently and efficiently (although this is precisely why they are trickier to program: you have to explicitly avoid simultaneous writing in the same memory area), not to mention that threads are much cheaper to create than processes. Generally speaking, if your problem is large, CPU intensive and not very memory bounded, MPI usually leads to better results. Otherwise threads tend to do a better job. The thing is that memory is increasingly becoming more and more a bottleneck nowadays, so threads are here to stay for a long, long time (which is certainly unfortunate for programmers, specially for pthreads ones :). -- Francesc Alted From uvemas at gmail.com Mon Feb 21 03:28:01 2011 From: uvemas at gmail.com (Vicent Mas) Date: Mon, 21 Feb 2011 09:28:01 +0100 Subject: [Numpy-discussion] ANN: ViTables 2.1 released! Message-ID: <201102210928.02377.uvemas@gmail.com> ======================= Announcing ViTables 2.1 ======================= After a long period retired from development activities I'm happy to come back and present ViTables 2.1, the latest release of this viewer for PyTables/HDF5 files. As it happens with the entire PyTables family, the main strength of ViTables is its ability to manage really large datasets in a fast and comfortable manner. For example, with ViTables you can open a table with one thousand million rows in a few tenths of second, with very low memory requirements. Also important is the fact that it is designed to be a multiplatform application, i.e., it runs flawlessly in Unix (and hence, GNU/Linux), Mac OS X and Windows. The fact that it runs on top of PyTables ensures its speed and memory efficiency. In this release you will find some bug fixes and usability enhancements. New features include: - improved queries. They are faster and eat much less memory - a simple but flexible plugins framework - human friendly formatting of times and dates for PyTables native Time fields and also for time series created via scikits.timeseries package - datasets can be imported/exported from/to files with CSV format Platforms --------- At the moment, ViTables has been fully tested only on GNU/Linux and Windows platforms, but as it is made on top of Python, PyQt and PyTables, its portability should be really good and it should work just fine in other Unices. How to get it ------------- Visit: http://www.vitables.org/download and click the link 'downloads area' to find the available packages. Share your experience --------------------- I'm very interested in your feedback about ViTables. Please send your opinions, suggestions, bugs, etc. to the ViTables Users Group at http://tech.groups.yahoo.com/group/vitables-users. Thank you! Enjoy Data with ViTables, the troll of the PyTables family! :: Share what you know, learn what you don't -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From sridhar.ratna at gmail.com Mon Feb 21 12:49:13 2011 From: sridhar.ratna at gmail.com (Sridhar Ratnakumar) Date: Mon, 21 Feb 2011 09:49:13 -0800 Subject: [Numpy-discussion] Numpy 1.5.1 - Mac - with Activestate Python 3 Message-ID: Hi, Just wanted to respond to http://comments.gmane.org/gmane.comp.python.numeric.general/42084 *CORTESI: I have installed ActiveState's Python 3 packages on Mac OS X 10.6.6. [...] What can I do to persuade numpy to install? Must I build it from sourceto get it to use Python 3? RALF: [...] there is no [NumPy] binary for Python 3 at the moment. But unless you have a specific need/desire to use 3.1/3.2 I'd suggest staying with 2.6 or 2.7 from python.org for now. With Activestate you have to compile yourself or use their (paid?) pypm repo.* NumPy can be installed from PyPM repo for ActivePython 2.6, 2.7, 3.1 (and the upcoming 3.2) on Windows, MacOSX and Linux. It is built with Intel's MKL and fortran compiler. http://code.activestate.com/pypm/numpy/ Note that we recently moved NumPy, SciPy and matplotlib to the free repo; so you can install it free of charge. -- Sridhar Ratnakumar Python Developer ActiveState -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Mon Feb 21 15:09:05 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 21 Feb 2011 15:09:05 -0500 Subject: [Numpy-discussion] Different behavior for astype('str') in numpy 1.5.1 vs 1.6.0? Message-ID: I get a lot of errors and failures running our tests with most recent numpy trunk. One in particular (so far) seems to be a bug. In [1]: import numpy as np In [2]: np.__version__ Out[2]: '1.6.0.dev-c50af53' In [3]: tmp_arr = np.array([['black'],['white'],['other']]) In [4]: tmp_arr.astype('str') Out[4]: array([['b'], ['w'], ['o']], dtype='|S1') The old behavior In [1]: import numpy as np In [2]: np.__version__ Out[2]: '1.5.1' In [3]: tmp_arr = np.array([['black'],['white'],['other']]) In [4]: tmp_arr.astype('str') Out[4]: array([['black'], ['white'], ['other']], dtype='|S5') Should I file a ticket? Skipper PS. Is there an incompatibility of numpy 1.5.1 and numpy 1.6.0 trunk for packages that depend on numpy? From pav at iki.fi Mon Feb 21 15:49:59 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 21 Feb 2011 20:49:59 +0000 (UTC) Subject: [Numpy-discussion] Different behavior for astype('str') in numpy 1.5.1 vs 1.6.0? References: Message-ID: On Mon, 21 Feb 2011 15:09:05 -0500, Skipper Seabold wrote: [clip] > Should I file a ticket? Yes. [clip] > PS. Is there an incompatibility of numpy 1.5.1 and numpy 1.6.0 trunk for > packages that depend on numpy? It should be binary compatible. -- Pauli Virtanen From jsseabold at gmail.com Mon Feb 21 16:03:43 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 21 Feb 2011 16:03:43 -0500 Subject: [Numpy-discussion] Different behavior for astype('str') in numpy 1.5.1 vs 1.6.0? In-Reply-To: References: Message-ID: On Mon, Feb 21, 2011 at 3:49 PM, Pauli Virtanen wrote: > On Mon, 21 Feb 2011 15:09:05 -0500, Skipper Seabold wrote: > [clip] >> Should I file a ticket? > > Yes. http://projects.scipy.org/numpy/ticket/1748 > > [clip] >> PS. Is there an incompatibility of numpy 1.5.1 and numpy 1.6.0 trunk for >> packages that depend on numpy? > > It should be binary compatible. > I had to reinstall scipy, matplotlib, etc. when I changed from 1.6.0 back to 1.5.1. It also looked like the behavior of numpy.testing was different (Ie., I was getting failures for my tests that pass under 1.5.1 at the specified precision for assert_almost_equal), but this is a separate issue that I'll have to explore later. Skipper From pav at iki.fi Mon Feb 21 17:56:38 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 21 Feb 2011 22:56:38 +0000 (UTC) Subject: [Numpy-discussion] Different behavior for astype('str') in numpy 1.5.1 vs 1.6.0? References: Message-ID: On Mon, 21 Feb 2011 16:03:43 -0500, Skipper Seabold wrote: [clip] > I had to reinstall scipy, matplotlib, etc. when I changed from 1.6.0 > back to 1.5.1. Yes, it should be backward compatible (you should be able use binaries compiled with 1.5.1 on 1.6.0), but not forward compatible (binaries compiled with 1.6.0 won't work on 1.5.1). > It also looked like the behavior of numpy.testing was different (Ie., I > was getting failures for my tests that pass under 1.5.1 at the specified > precision for assert_almost_equal), but this is a separate issue that > I'll have to explore later. Sure, tickets are welcome. The current unit test suite is probably not very complete, esp. for numpy.testing. Pauli From sturla at molden.no Tue Feb 22 15:45:15 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 22 Feb 2011 21:45:15 +0100 Subject: [Numpy-discussion] NumPy speed tests by NASA Message-ID: <4D64205B.1030407@molden.no> I came accross some NumPy performance tests by NASA. Comparisons against pure Python, Matlab, gfortran, Intel Fortran, Intel Fortran with MKL, and Java. For those that are interested, it is here: https://modelingguru.nasa.gov/docs/DOC-1762 Sturla From wahid807 at gmail.com Tue Feb 22 16:19:29 2011 From: wahid807 at gmail.com (Akand Islam) Date: Tue, 22 Feb 2011 15:19:29 -0600 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: <4D64205B.1030407@molden.no> References: <4D64205B.1030407@molden.no> Message-ID: Thanks for posting a nice report. Akand On Tue, Feb 22, 2011 at 2:45 PM, Sturla Molden wrote: > I came accross some NumPy performance tests by NASA. Comparisons against > pure Python, Matlab, gfortran, Intel Fortran, Intel Fortran with MKL, > and Java. For those that are interested, it is here: > > https://modelingguru.nasa.gov/docs/DOC-1762 > > Sturla > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Tue Feb 22 16:44:56 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 22 Feb 2011 16:44:56 -0500 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: <4D64205B.1030407@molden.no> References: <4D64205B.1030407@molden.no> Message-ID: <4D642E58.6050704@gmail.com> On 2/22/2011 3:45 PM, Sturla Molden wrote: > I came accross some NumPy performance tests by NASA. Comparisons against > pure Python, Matlab, gfortran, Intel Fortran, Intel Fortran with MKL, > and Java. For those that are interested, it is here: > https://modelingguru.nasa.gov/docs/DOC-1762 I don't believe the matrix multiplication results. Maybe I misunderstand them ... >>> t = timeit.Timer("np.dot(A,B)","import numpy as np;N=1500;A=np.random.random((N,N));B=np.random.random((N,N))") >>> print t.timeit(number=10)/10. 1.09043075307 I'm using the precompiled Windows binaries. Alan Isaac From gael.varoquaux at normalesup.org Tue Feb 22 16:48:09 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 22 Feb 2011 22:48:09 +0100 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: <4D642E58.6050704@gmail.com> References: <4D64205B.1030407@molden.no> <4D642E58.6050704@gmail.com> Message-ID: <20110222214809.GF9360@phare.normalesup.org> On Tue, Feb 22, 2011 at 04:44:56PM -0500, Alan G Isaac wrote: > On 2/22/2011 3:45 PM, Sturla Molden wrote: > > I came accross some NumPy performance tests by NASA. Comparisons against > > pure Python, Matlab, gfortran, Intel Fortran, Intel Fortran with MKL, > > and Java. For those that are interested, it is here: > > https://modelingguru.nasa.gov/docs/DOC-1762 > I don't believe the matrix multiplication results. > Maybe I misunderstand them ... Probably because the numpy binary that the author was using was compiled without a blas implementation, and just using numpy's internal lapack_lite. This is a common problem in real life. Ga?l From pav at iki.fi Tue Feb 22 16:54:45 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 22 Feb 2011 21:54:45 +0000 (UTC) Subject: [Numpy-discussion] NumPy speed tests by NASA References: <4D64205B.1030407@molden.no> <4D642E58.6050704@gmail.com> Message-ID: On Tue, 22 Feb 2011 16:44:56 -0500, Alan G Isaac wrote: [clip] > I don't believe the matrix multiplication results. Maybe I misunderstand > them ... > > >>> t = timeit.Timer("np.dot(A,B)","import numpy as > >>> np;N=1500;A=np.random.random((N,N));B=np.random.random((N,N))") > >>> print t.timeit(number=10)/10. > 1.09043075307 > > I'm using the precompiled Windows binaries. ... which are linked with BLAS. Compare: In [2]: A=np.ones((1000,1000)) In [3]: B=np.ones((1000,1000)) In [4]: %timeit np.dot(A, B) 1 loops, best of 3: 236 ms per loop In [6]: %timeit np.core.multiarray.dot(A, B) 1 loops, best of 3: 8.21 s per loop If not BLAS is available at compile time, it falls back to using the latter `dot` --- which is 35 x slower. The reason for the difference is that the non-BLAS version can work also with object arrays etc. and so does indirect function calls for the operations. From pav at iki.fi Tue Feb 22 16:59:26 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 22 Feb 2011 21:59:26 +0000 (UTC) Subject: [Numpy-discussion] NumPy speed tests by NASA References: <4D64205B.1030407@molden.no> <4D642E58.6050704@gmail.com> <20110222214809.GF9360@phare.normalesup.org> Message-ID: On Tue, 22 Feb 2011 22:48:09 +0100, Gael Varoquaux wrote: [clip] > Probably because the numpy binary that the author was using was compiled > without a blas implementation, and just using numpy's internal > lapack_lite. This is a common problem in real life. It doesn't use blas_lite at the moment. It probably could be made to do so, though, and this would probably already give a 10x speed boost. From gael.varoquaux at normalesup.org Tue Feb 22 17:01:27 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 22 Feb 2011 23:01:27 +0100 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: References: <4D64205B.1030407@molden.no> <4D642E58.6050704@gmail.com> <20110222214809.GF9360@phare.normalesup.org> Message-ID: <20110222220127.GG9360@phare.normalesup.org> On Tue, Feb 22, 2011 at 09:59:26PM +0000, Pauli Virtanen wrote: > > Probably because the numpy binary that the author was using was compiled > > without a blas implementation, and just using numpy's internal > > lapack_lite. This is a common problem in real life. > It doesn't use blas_lite at the moment. It probably could be made to do > so, though, and this would probably already give a 10x speed boost. Ah. That explains the low performance. Thanks for correcting me. G From gokhansever at gmail.com Tue Feb 22 18:19:31 2011 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Tue, 22 Feb 2011 16:19:31 -0700 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: <4D642E58.6050704@gmail.com> References: <4D64205B.1030407@molden.no> <4D642E58.6050704@gmail.com> Message-ID: On Tue, Feb 22, 2011 at 2:44 PM, Alan G Isaac wrote: > > > I don't believe the matrix multiplication results. > Maybe I misunderstand them ... > > >>> t = timeit.Timer("np.dot(A,B)","import numpy as > np;N=1500;A=np.random.random((N,N));B=np.random.random((N,N))") > >>> print t.timeit(number=10)/10. > 1.09043075307 > > I'm using the precompiled Windows binaries. This is on Fedora 14 x86_64 --using Fedora provided builds of ATLAS, and build using gcc/gfortran. >>> t = timeit.Timer("np.dot(A,B)","import numpy as np;N=1500;A=np.random.random((N,N));B=np.random.random((N,N))") >>> print t.timeit(number=10)/10. 0.497710204124 I am guessing ATLAS is thread aware since with N=15000 each of the quad core runs at %100. Probably MKL build doesn't bring much speed advantage in this computation. Any thoughts? -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From wickedgrey at gmail.com Tue Feb 22 19:05:13 2011 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Tue, 22 Feb 2011 16:05:13 -0800 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: <20110222214809.GF9360@phare.normalesup.org> References: <4D64205B.1030407@molden.no> <4D642E58.6050704@gmail.com> <20110222214809.GF9360@phare.normalesup.org> Message-ID: On Tue, Feb 22, 2011 at 1:48 PM, Gael Varoquaux wrote: > Probably because the numpy binary that the author was using was compiled > without a blas implementation, and just using numpy's internal > lapack_lite. This is a common problem in real life. Is there an easy way to check from within numpy if it's using a blas iplementation or not? Thanks, Eli From sturla at molden.no Tue Feb 22 19:21:04 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 23 Feb 2011 01:21:04 +0100 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: References: <4D64205B.1030407@molden.no> <4D642E58.6050704@gmail.com> Message-ID: <4D6452F0.60407@molden.no> Den 23.02.2011 00:19, skrev G?khan Sever: > > I am guessing ATLAS is thread aware since with N=15000 each of the > quad core runs at %100. Probably MKL build doesn't bring much speed > advantage in this computation. Any thoughts? > There are still things like optimal cache use, SIMD extensions, etc. to consider. Some of MKL is hand-tweaked assemby and e.g. very fast on iCore. Other BLAS implementations to consider are ACML, GotoBLAS2, ACML-GPU, and CUBLAS. GotoBLAS2 is currently the fastest BLAS implementation on x64 CPUs. It can e.g. be linked with the reference implementation of LAPACK. GotoBLAS is open source and is very easy to build ("just type make"). ACML is probably better than MKL on AMD processors, but not as good as MKL on Intel processors, and currently free of charge (an MKL license costs $399). Tthe recet ACML-GPU library can move matrix multiplication (DGEMM and friends) to the GPU if there is an ATI (AMD) chip available, and the matrices are sufficiently large. The ATI GPU can also be programmed with OpenCL, but ACML-GPU just looks like an ordinary BLAS and LAPACK implementation (in addition to FFTs and PRNGs), so no special programming is needed. If one has an nVidia GPU, there is the CUBLAS library which implements BLAS, but not LAPACK. It has Fortran bindings and can probably be used with a reference LAPACK. Sturla From david at silveregg.co.jp Wed Feb 23 02:32:09 2011 From: david at silveregg.co.jp (David) Date: Wed, 23 Feb 2011 16:32:09 +0900 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: <4D64205B.1030407@molden.no> References: <4D64205B.1030407@molden.no> Message-ID: <4D64B7F9.6090109@silveregg.co.jp> On 02/23/2011 05:45 AM, Sturla Molden wrote: > I came accross some NumPy performance tests by NASA. Comparisons against > pure Python, Matlab, gfortran, Intel Fortran, Intel Fortran with MKL, > and Java. For those that are interested, it is here: This is mostly a test of the blas/lapack used, so it is not very useful IMO, except maybe to show that you can deal with non trivial problems on top of python (surprisingly, many scientists programming a fair bit are still unaware of the vectorizing concept altogether), cheers, David From dumka.dumka at yahoo.com Wed Feb 23 09:22:27 2011 From: dumka.dumka at yahoo.com (Dumka Dumka) Date: Wed, 23 Feb 2011 06:22:27 -0800 (PST) Subject: [Numpy-discussion] Problems with installation (wrong Python version) Message-ID: <506508.9231.qm@web120806.mail.ne1.yahoo.com> Dear numpy Community, I try to install numpy on my Windows 7. I take the code from here: http://sourceforge.net/projects/numpy/files/NumPy/ (I click on "Download numpy-1.5.1-win32-superpack-python2.6.exe (5.4 MB)"). After I start the installation, the installer tells me: Python version 2.6 (Found in registry). I also see the link to the "C:\Python26" directory in the installation window. However, I want to use Python2.7 (I had it on my computer together with the Python 2.6). So, I deleted the "C:\Python26" directory an set a new value to the "Path" environment variable to make a link to the newer version of the Python (version 2.7). I also restarted the computer. However, it does not help. When I try to run the numpy installer again I see the same: Python version 2.6 (Found in registry). I cannot understand where the installer find the link to the old version. If I just continue the installation, it fails (and it started to fail after I deleted the "C:\Python26" directory). So, to summarize, numpy installer "thinks" that I have Python 2.6 (and I don't) and then it fails because it is wrong. How can I force the numpy installer to use Python 2.7? Can I? Thank you in advance for any help, Dumka -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaohong86 at gmail.com Wed Feb 23 09:27:16 2011 From: shaohong86 at gmail.com (Shao Hong) Date: Wed, 23 Feb 2011 22:27:16 +0800 Subject: [Numpy-discussion] Problems with installation (wrong Python version) In-Reply-To: <506508.9231.qm@web120806.mail.ne1.yahoo.com> References: <506508.9231.qm@web120806.mail.ne1.yahoo.com> Message-ID: Hi! I think you should download the 2.7 pack ( http://sourceforge.net/projects/numpy/files/NumPy/1.5.1/numpy-1.5.1-win32-superpack-python2.7.exe/download) and it might detect your python 2.7 instead. Shao Hong On Wed, Feb 23, 2011 at 10:22 PM, Dumka Dumka wrote: > Dear numpy Community, > > I try to install numpy on my Windows 7. I take the code from here: > http://sourceforge.net/projects/numpy/files/NumPy/ (I click on "Download > numpy-1.5.1-win32-superpack-python2.6.exe (5.4 MB)"). > > After I start the installation, the installer tells me: Python version 2.6 > (Found in registry). I also see the link to the "C:\Python26" directory in > the installation window. > > However, I want to use Python2.7 (I had it on my computer together with the > Python 2.6). So, I deleted the "C:\Python26" directory an set a new value to > the "Path" environment variable to make a link to the newer version of the > Python (version 2.7). I also restarted the computer. > > However, it does not help. When I try to run the numpy installer again I > see the same: Python version 2.6 (Found in registry). I cannot understand > where the installer find the link to the old version. If I just continue the > installation, it fails (and it started to fail after I deleted the > "C:\Python26" directory). So, to summarize, numpy installer "thinks" that I > have Python 2.6 (and I don't) and then it fails because it is wrong. > > How can I force the numpy installer to use Python 2.7? Can I? > > Thank you in advance for any help, > Dumka > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Feb 23 09:29:58 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 23 Feb 2011 22:29:58 +0800 Subject: [Numpy-discussion] Problems with installation (wrong Python version) In-Reply-To: <506508.9231.qm@web120806.mail.ne1.yahoo.com> References: <506508.9231.qm@web120806.mail.ne1.yahoo.com> Message-ID: On Wed, Feb 23, 2011 at 10:22 PM, Dumka Dumka wrote: > Dear numpy Community, > I try to install numpy on my Windows 7. I take the code from here: > http://sourceforge.net/projects/numpy/files/NumPy/??(I click on "Download > numpy-1.5.1-win32-superpack-python2.6.exe (5.4 MB)"). > After I start the installation, the installer tells me: Python version 2.6 > (Found in registry). I also see the link to the "C:\Python26" directory in > the installation window. > However, I want to use Python2.7 (I had it on my computer together with the > Python 2.6). So, I deleted the "C:\Python26" directory an set a new value to > the "Path" environment variable to make a link to the newer version of the > Python (version 2.7). I also restarted the computer. > However, it does not help. When I try to run the numpy installer again I see > the same:?Python version 2.6 (Found in registry). I cannot understand where > the installer find the link to the old version. If I just continue the > installation, it fails (and it started to fail after I deleted the > "C:\Python26" directory). So, to summarize, numpy installer "thinks" that I > have Python 2.6 (and I don't) and then it fails because it is wrong. > How can I force the numpy installer to use Python 2.7? Can I? There are separate installers for each Python version. Just go back to http://sourceforge.net/projects/numpy/files/NumPy/1.5.1/ and grab the file that ends in -python2.7.exe. And no need to delete your Python26 folder, you can have multiple versions side-by-side. Ralf From dumka.dumka at yahoo.com Wed Feb 23 09:33:43 2011 From: dumka.dumka at yahoo.com (Dumka Dumka) Date: Wed, 23 Feb 2011 06:33:43 -0800 (PST) Subject: [Numpy-discussion] Problems with installation (wrong Python version) In-Reply-To: References: <506508.9231.qm@web120806.mail.ne1.yahoo.com> Message-ID: <230879.95256.qm@web120803.mail.ne1.yahoo.com> Thank you very much! It resolved the problem. I have successfully installed the numpy for Python 2.7 ________________________________ From: Shao Hong To: Discussion of Numerical Python Sent: Wed, February 23, 2011 3:27:16 PM Subject: Re: [Numpy-discussion] Problems with installation (wrong Python version) Hi! I think you should download the 2.7 pack ( http://sourceforge.net/projects/numpy/files/NumPy/1.5.1/numpy-1.5.1-win32-superpack-python2.7.exe/download ) and it might detect your python 2.7 instead. Shao Hong On Wed, Feb 23, 2011 at 10:22 PM, Dumka Dumka wrote: Dear numpy Community, > > >I try to install numpy on my Windows 7. I take the code from here: >http://sourceforge.net/projects/numpy/files/NumPy/ (I click on "Download >numpy-1.5.1-win32-superpack-python2.6.exe (5.4 MB)"). > > >After I start the installation, the installer tells me: Python version 2.6 >(Found in registry). I also see the link to the "C:\Python26" directory in the >installation window. > > >However, I want to use Python2.7 (I had it on my computer together with the >Python 2.6). So, I deleted the "C:\Python26" directory an set a new value to the >"Path" environment variable to make a link to the newer version of the Python >(version 2.7). I also restarted the computer. > > >However, it does not help. When I try to run the numpy installer again I see the >same: Python version 2.6 (Found in registry). I cannot understand where the >installer find the link to the old version. If I just continue the installation, >it fails (and it started to fail after I deleted the "C:\Python26" directory). >So, to summarize, numpy installer "thinks" that I have Python 2.6 (and I don't) >and then it fails because it is wrong. > > >How can I force the numpy installer to use Python 2.7? Can I? > > >Thank you in advance for any help, >Dumka > >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Wed Feb 23 12:00:04 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 23 Feb 2011 18:00:04 +0100 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: <4D64B7F9.6090109@silveregg.co.jp> References: <4D64205B.1030407@molden.no> <4D64B7F9.6090109@silveregg.co.jp> Message-ID: <4D653D14.2040207@molden.no> Den 23.02.2011 08:32, skrev David: > This is mostly a test of the blas/lapack used, so it is not very useful > IMO, except maybe to show that you can deal with non trivial problems on > top of python (surprisingly, many scientists programming a fair bit are > still unaware of the vectorizing concept altogether), > I know that, and I am not sure if those who posted it did. It does show that their NumPy is not worse than Matlab (albeit pure Python is), except for the part that depends on BLAS version. It also shows that vectorization in NumPy can perform comparable to na?ve do-loops in gfortran (performance within 50% or better). This implies that going from NumPy to gfortran for speed might not help a lot. Sturla From ben.root at ou.edu Wed Feb 23 12:34:11 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 23 Feb 2011 11:34:11 -0600 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: <4D653D14.2040207@molden.no> References: <4D64205B.1030407@molden.no> <4D64B7F9.6090109@silveregg.co.jp> <4D653D14.2040207@molden.no> Message-ID: On Wed, Feb 23, 2011 at 11:00 AM, Sturla Molden wrote: > Den 23.02.2011 08:32, skrev David: > > This is mostly a test of the blas/lapack used, so it is not very useful > > IMO, except maybe to show that you can deal with non trivial problems on > > top of python (surprisingly, many scientists programming a fair bit are > > still unaware of the vectorizing concept altogether), > > > > I know that, and I am not sure if those who posted it did. > > It does show that their NumPy is not worse than Matlab (albeit pure > Python is), except for the part that depends on BLAS version. > > Just to point out, this person used Matlab 2008. While I do not use Matlab on a regular basis, I have seen marked improvements in performance in subsequent versions. I have also seen differences in performance for the same version of Matlab running on the different OSs of a dual-boot machine. Or as a CS professor says at my university: "it depends... on the machine, the operating system, the day of the week and the color of your socks." Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Wed Feb 23 14:29:24 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 23 Feb 2011 20:29:24 +0100 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: References: <4D64205B.1030407@molden.no> <4D64B7F9.6090109@silveregg.co.jp> <4D653D14.2040207@molden.no> Message-ID: <4D656014.1000701@molden.no> Den 23.02.2011 18:34, skrev Benjamin Root: > > Just to point out, this person used Matlab 2008. While I do not use > Matlab on a regular basis, I have seen marked improvements in > performance in subsequent versions. They have got a JIT compiler for loops. Matlab is actually an easy language to JIT compile because all objects are immutable, assignments always make copies (implemented as copy-on-write), and there is never any aliasing. The problem with this is slowdown when working with large arrays and severe heap-fragmentation on 32-bit systems. > I have also seen differences in performance for the same version of > Matlab running on the different OSs of a dual-boot machine. The Windows version used to come with Intel MKL and the Linux version with ATLAS. Sturla From scopatz at gmail.com Wed Feb 23 18:33:54 2011 From: scopatz at gmail.com (Anthony Scopatz) Date: Wed, 23 Feb 2011 23:33:54 +0000 Subject: [Numpy-discussion] ANN: inSCIght, The Scientific Computing Podcast Message-ID: Hello All, I am very pleased to announce inSCIght, a new scientific computing podcast (press release below). I apologize for those of you in the intersection of these lists that may receive this message multiple times. As I mention in the press release, we are very open to your contributions! Be Well Anthony inSCIght: The Scientific Computing Podcast ========================================== 'inSCIght' is a podcast that focuses on scientific computing in all of its various forms. Every week we have a few panelists engage head-to-head on poignant and interesting topics. The panelists are drawn from all across the scientific computing community. From embedded systems experts to very high level language gurus, biologists and nuclear engineers, the hosts of inSCIght use computers to solve science and engineering problems everyday. This podcast throws people, ideas, and opinions into an audio-blender hoping to educate and entice each other and the world. You can find us at: * inSCIght.org (http://inscight.org/), * Twitter (http://twitter.com/inscight/), * Convore (https://convore.com/inscight/), * and GitHub (https://github.com/inscight/). Furthermore, we are are always looking to supplement our current repertoire of hosts and special guests. So if you would like to contribute to inSCIght or have something interesting to present on a show, feel free to email us at info_AT_inscight.org. We'd love to have you join the conversation! The inSCIght podcast is a co-production of Enthought, Software Carpentry, and The Hacker Within. Thanks for listening! The inSCIght podcast is licensed under the Creative Commons Attribution 3.0 Unported (CC BY 3.0) license. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonovik at gmail.com Thu Feb 24 10:49:06 2011 From: jonovik at gmail.com (Jon Olav Vik) Date: Thu, 24 Feb 2011 15:49:06 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Adding_=60offset=60_argument_to_np?= =?utf-8?q?=2Elib=2Eformat=2Eopen=5Fmemmap_and_np=2Eload?= Message-ID: https://github.com/jonovik/numpy/compare/master...offset_memmap The `offset` argument to np.memmap enables memory-mapping a portion of a file on disk to a memory-mapped Numpy array. Memory-mapping can also be done with np.load, but it uses np.lib.format.open_memmap, which has no offset argument. I have added an offset argument to np.lib.format.open_memmap and np.load as detailed in the link above, and humbly submit the changes for review. This is my first time using git, apologies for any mistakes. Note that the offset is in terms of array elements, not bytes (which is what np.memmap uses), because that was what I had use for. Also, I added a `shape` to np.load to memory-map only a portion of a file. My use case was to preallocate a big record array on disk, then start many processes writing to their separate, memory-mapped segments of the file. The end result was one big array on disk, with the correct shape and data type information. Using a record array makes the data structure more self- documenting. Using open_memmap with mode="w+" is the fastest way I've found to preallocate an array on disk; it does not create the huge array in memory. Letting multiple processes memory-map and read/write to non-overlapping portions without interfering with each other allows for fast, simple parallel I/ O. I've used this extensively on Numpy 1.4.0, but based my Git checkout on the current Numpy trunk. There have been some rearrangements in np.load since then (it used to be in np.lib.io and is now in np.lib.npyio), but as far as I can see, my modifications carry over fine. I haven't had a chance to test with Numpy trunk, though. (What is the best way to set up a test version without affecting my working 1.4.0 setup?) Hope this can be useful, Jon Olav Vik From mustapha.bouikhif at idris.fr Thu Feb 24 11:04:00 2011 From: mustapha.bouikhif at idris.fr (Mustapha BOUIKHIF) Date: Thu, 24 Feb 2011 17:04:00 +0100 Subject: [Numpy-discussion] Problems with numpy Message-ID: <4D668170.20209@idris.fr> Hello all, I am having problems with numpy (version 1.5.1) when executing this command as root, evry thing is OK: python -Wd -c 'import numpy; numpy.test()' Running unit tests for numpy NumPy version 1.5.1 NumPy is installed in /usr/local/lib64/python2.6/site-packages/numpy Python version 2.6 (r26:66714, Nov 9 2010, 01:31:57) [GCC 4.3.4 [gcc-4_3-branch revision 152973]] nose version 1.0.0 ...................... Ran 3006 tests in 14.088s OK (KNOWNFAIL=4) But as a normal user (non root), the same command show errors: python -Wd -c 'import numpy; numpy.test()' Traceback (most recent call last): File "", line 1, in File "/usr/local/lib64/python2.6/site-packages/numpy/__init__.py", line 127, in raise ImportError(msg) ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python intepreter from there. Python 2.6 Linux 2.6.32.27-0.2-default x86_64 x86_64 x86_64 GNU/Linux modules are in /usr/local/lib64/python2.6/site-packages/ Any help will be appreciated. -- Cordialement Mustapha Bouikhif CNRS - IDRIS Tel: +33 1 69 35 85 27 Fax: +33 1 69 35 37 75 E-mail: mustapha.bouikhif at idris.fr http://www.idris.fr From ben.root at ou.edu Thu Feb 24 11:25:20 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 24 Feb 2011 10:25:20 -0600 Subject: [Numpy-discussion] Problems with numpy In-Reply-To: <4D668170.20209@idris.fr> References: <4D668170.20209@idris.fr> Message-ID: On Thu, Feb 24, 2011 at 10:04 AM, Mustapha BOUIKHIF < mustapha.bouikhif at idris.fr> wrote: > Hello all, > > I am having problems with numpy (version 1.5.1) > > when executing this command as root, evry thing is OK: > python -Wd -c 'import numpy; numpy.test()' > > Running unit tests for numpy > NumPy version 1.5.1 > NumPy is installed in /usr/local/lib64/python2.6/site-packages/numpy > Python version 2.6 (r26:66714, Nov 9 2010, 01:31:57) [GCC 4.3.4 > [gcc-4_3-branch revision 152973]] > nose version 1.0.0 > > ...................... > Ran 3006 tests in 14.088s > > OK (KNOWNFAIL=4) > > But as a normal user (non root), the same command show errors: > > python -Wd -c 'import numpy; numpy.test()' > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib64/python2.6/site-packages/numpy/__init__.py", > line 127, in > raise ImportError(msg) > ImportError: Error importing numpy: you should not try to import numpy > from its source directory; please exit the numpy source tree, and > relaunch your python intepreter from there. > > > Python 2.6 > Linux 2.6.32.27-0.2-default x86_64 x86_64 x86_64 GNU/Linux > modules are in /usr/local/lib64/python2.6/site-packages/ > > Any help will be appreciated. > > > Mustapha, "ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python intepreter from there." It is generally a bad idea to import installed modules directly from their source directory. Just change your working directory to another location where "numpy" is not a directory, and you should be able to run the tests. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From mustapha.bouikhif at idris.fr Thu Feb 24 11:35:36 2011 From: mustapha.bouikhif at idris.fr (Mustapha BOUIKHIF) Date: Thu, 24 Feb 2011 17:35:36 +0100 Subject: [Numpy-discussion] Problems with numpy In-Reply-To: References: <4D668170.20209@idris.fr> Message-ID: <4D6688D8.4060602@idris.fr> Benjamin, I am not in numpy install dir and have the error however: #python Python 2.6 (r26:66714, Nov 9 2010, 01:31:57) [GCC 4.3.4 [gcc-4_3-branch revision 152973]] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.getcwd() *'/home/idris/cos/scos190'* >>> import numpy Traceback (most recent call last): File "", line 1, in File "/usr/local/lib64/python2.6/site-packages/numpy/__init__.py", line 127, in raise ImportError(msg) ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python intepreter from there. Benjamin Root wrote: > > > On Thu, Feb 24, 2011 at 10:04 AM, Mustapha BOUIKHIF > > wrote: > > Hello all, > > I am having problems with numpy (version 1.5.1) > > when executing this command as root, evry thing is OK: > python -Wd -c 'import numpy; numpy.test()' > > Running unit tests for numpy > NumPy version 1.5.1 > NumPy is installed in /usr/local/lib64/python2.6/site-packages/numpy > Python version 2.6 (r26:66714, Nov 9 2010, 01:31:57) [GCC 4.3.4 > [gcc-4_3-branch revision 152973]] > nose version 1.0.0 > > ...................... > Ran 3006 tests in 14.088s > > OK (KNOWNFAIL=4) > > But as a normal user (non root), the same command show errors: > > python -Wd -c 'import numpy; numpy.test()' > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib64/python2.6/site-packages/numpy/__init__.py", > line 127, in > raise ImportError(msg) > ImportError: Error importing numpy: you should not try to import numpy > from its source directory; please exit the numpy source tree, and > relaunch your python intepreter from there. > > > Python 2.6 > Linux 2.6.32.27-0.2-default x86_64 x86_64 x86_64 GNU/Linux > modules are in /usr/local/lib64/python2.6/site-packages/ > > Any help will be appreciated. > > > > Mustapha, > > "ImportError: Error importing numpy: you should not try to import numpy > from its source directory; please exit the numpy source tree, and > relaunch your python intepreter from there." > > It is generally a bad idea to import installed modules directly from > their source directory. Just change your working directory to another > location where "numpy" is not a directory, and you should be able to run > the tests. > > Ben Root > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Cordialement Mustapha Bouikhif CNRS - IDRIS Tel: +33 1 69 35 85 27 Fax: +33 1 69 35 37 75 E-mail: mustapha.bouikhif at idris.fr http://www.idris.fr From ben.root at ou.edu Thu Feb 24 11:40:17 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 24 Feb 2011 10:40:17 -0600 Subject: [Numpy-discussion] Problems with numpy In-Reply-To: <4D6688D8.4060602@idris.fr> References: <4D668170.20209@idris.fr> <4D6688D8.4060602@idris.fr> Message-ID: On Thu, Feb 24, 2011 at 10:35 AM, Mustapha BOUIKHIF < mustapha.bouikhif at idris.fr> wrote: > Benjamin, > > I am not in numpy install dir and have the error however: > > #python > Python 2.6 (r26:66714, Nov 9 2010, 01:31:57) > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import os > >>> os.getcwd() > *'/home/idris/cos/scos190'* > >>> import numpy > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib64/python2.6/site-packages/numpy/__init__.py", > line 127, in > raise ImportError(msg) > ImportError: Error importing numpy: you should not try to import numpy > from its source directory; please exit the numpy source tree, and > relaunch your python intepreter from there. > > > Are there any directories called "numpy" in /home/idris/cos/scos190? Could you try cd-ing over to another directory and trying again? Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Feb 24 11:47:14 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 24 Feb 2011 16:47:14 +0000 (UTC) Subject: [Numpy-discussion] Problems with numpy References: <4D668170.20209@idris.fr> <4D6688D8.4060602@idris.fr> Message-ID: Thu, 24 Feb 2011 10:40:17 -0600, Benjamin Root wrote: [clip] >> >>> import numpy >> Traceback (most recent call last): >> File "", line 1, in >> File "/usr/local/lib64/python2.6/site-packages/numpy/__init__.py", >> line 127, in >> raise ImportError(msg) >> ImportError: Error importing numpy: you should not try to import numpy >> from its source directory; please exit the numpy source tree, and >> relaunch your python intepreter from there. [clip] > Are there any directories called "numpy" in /home/idris/cos/scos190? > Could you try cd-ing over to another directory and trying again? Another possible reason is that Numpy was installed wrong (as the numpy.__config__ module is apparently missing). Numpy needs to be installed via "python setup.py install", manually copying the "numpy" directory is not enough. -- Pauli Virtanen From pav at iki.fi Thu Feb 24 11:48:35 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 24 Feb 2011 16:48:35 +0000 (UTC) Subject: [Numpy-discussion] Problems with numpy References: <4D668170.20209@idris.fr> <4D6688D8.4060602@idris.fr> Message-ID: Thu, 24 Feb 2011 16:47:14 +0000, Pauli Virtanen wrote: > Another possible reason is that Numpy was installed wrong (as the > numpy.__config__ module is apparently missing). Numpy needs to be > installed via "python setup.py install", manually copying the "numpy" > directory is not enough. Or, the permissions of "/usr/local/lib64/python2.6/site-packages/numpy/ __config__.py" (and possibly other files) are such that only root can read them. -- Pauli Virtanen From jonovik at gmail.com Thu Feb 24 11:56:36 2011 From: jonovik at gmail.com (Jon Olav Vik) Date: Thu, 24 Feb 2011 16:56:36 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Adding_=60offset=60_argument_to_np?= =?utf-8?q?=2Elib=2Eformat=2Eopen=5Fmemmap_and_np=2Eload?= References: Message-ID: Jon Olav Vik gmail.com> writes: > https://github.com/jonovik/numpy/compare/master...offset_memmap > I've used this extensively on Numpy 1.4.0, but based my Git checkout on the > current Numpy trunk. There have been some rearrangements in np.load since then > (it used to be in np.lib.io and is now in np.lib.npyio), but as far as I can > see, my modifications carry over fine. I haven't had a chance to test with > Numpy trunk, though. (What is the best way to set up a test version without > affecting my working 1.4.0 setup?) I tried to push my modifications for 1.4.0, but couldn't figure out how my Github account could hold forks of both Numpy trunk and maintenance/1.4.x. Anyhow, here is a patch for 1.4: >From c3ff71637c6c00d6cac1ee22a2cad34de2449431 Mon Sep 17 00:00:00 2001 From: Jon Olav Vik Date: Thu, 24 Feb 2011 17:38:03 +0100 Subject: [PATCH 54/54] Added `offset` parameter as in np.memmap to np.load and np.lib.format.open_memmap. Modified numpy/lib/format.py Modified numpy/lib/io.py Doctests: >>> filename = "temp.npy" >>> np.save(filename, np.arange(10)) >>> load(filename) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> mmap = load(filename, mmap_mode="r+") >>> mmap memmap([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> mmap[3:7] = 42 >>> del mmap >>> np.load(filename) array([ 0, 1, 2, 42, 42, 42, 42, 7, 8, 9]) >>> mmap = load(filename, mmap_mode="r+", offset=2, shape=6) >>> mmap[-1] = 123 >>> del mmap >>> np.load(filename) array([ 0, 1, 2, 42, 42, 42, 42, 123, 8, 9]) >>> import os >>> os.remove(filename) --- numpy/lib/format.py | 17 +++++++++++++---- numpy/lib/io.py | 7 +++++-- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/numpy/lib/format.py b/numpy/lib/format.py index 3c5fe32..7c28b09 100644 --- a/numpy/lib/format.py +++ b/numpy/lib/format.py @@ -460,7 +460,7 @@ def read_array(fp): def open_memmap(filename, mode='r+', dtype=None, shape=None, - fortran_order=False, version=(1,0)): + fortran_order=False, version=(1,0), offset=0): """ Open a .npy file as a memory-mapped array. @@ -479,13 +479,15 @@ def open_memmap(filename, mode='r+', dtype=None, shape=None, mode. shape : tuple of int, optional The shape of the array if we are creating a new file in "write" - mode. + mode. Shape of (contiguous) slice if opening an existing file. fortran_order : bool, optional Whether the array should be Fortran-contiguous (True) or C-contiguous (False) if we are creating a new file in "write" mode. version : tuple of int (major, minor) If the mode is a "write" mode, then this is the version of the file format used to create the file. + offset : int, optional + Number of elements to skip along the first dimension. Returns ------- @@ -509,6 +511,7 @@ def open_memmap(filename, mode='r+', dtype=None, shape=None, " existing file handles.") if 'w' in mode: + assert offset == 0, "Cannot specify offset when creating memmap" # We are creating the file, not reading it. # Check if we ought to create the file. if version != (1, 0): @@ -541,11 +544,17 @@ def open_memmap(filename, mode='r+', dtype=None, shape=None, if version != (1, 0): msg = "only support version (1,0) of file format, not %r" raise ValueError(msg % (version,)) - shape, fortran_order, dtype = read_array_header_1_0(fp) + fullshape, fortran_order, dtype = read_array_header_1_0(fp) + if shape is None: + shape = fullshape + if offset: + shape = list(fullshape) + shape[0] = shape[0] - offset + shape = tuple(shape) if dtype.hasobject: msg = "Array can't be memory-mapped: Python objects in dtype." raise ValueError(msg) - offset = fp.tell() + offset = fp.tell() + offset * dtype.itemsize finally: fp.close() diff --git a/numpy/lib/io.py b/numpy/lib/io.py index 262d20d..694bae2 100644 --- a/numpy/lib/io.py +++ b/numpy/lib/io.py @@ -212,7 +212,7 @@ class NpzFile(object): return self.files.__contains__(key) -def load(file, mmap_mode=None): +def load(file, mmap_mode=None, offset=0, shape=None): """ Load a pickled, ``.npy``, or ``.npz`` binary file. @@ -272,6 +272,9 @@ def load(file, mmap_mode=None): memmap([4, 5, 6]) """ + if (not mmap_mode) and (offset or shape): + raise ValueError("Offset and shape should be used only with mmap_mode") + import gzip if isinstance(file, basestring): @@ -290,7 +293,7 @@ def load(file, mmap_mode=None): return NpzFile(fid) elif magic == format.MAGIC_PREFIX: # .npy file if mmap_mode: - return format.open_memmap(file, mode=mmap_mode) + return open_memmap(file, mode=mmap_mode, shape=shape, offset=offset) else: return format.read_array(fid) else: # Try a pickle -- 1.7.4.msysgit.0 From bwoods at aer.com Thu Feb 24 13:00:09 2011 From: bwoods at aer.com (Bryan Woods) Date: Thu, 24 Feb 2011 13:00:09 -0500 Subject: [Numpy-discussion] blank values '--' Message-ID: <4D669CA9.3090909@aer.com> Hello, I am loading in data from the GRIB2 files using PyNIO. Once I load them into numpy arrays I am left with a halo of values near the array boundaries that appear to be empty. When I print them or use them in conditional statements I see values of '--'. I'd like to turn these values into zeroes at the very least. As an example if I print the array I see: [[-- -- -- ..., -- -- --] [-- -- -- ..., -- -- --] [-- -- -- ..., -- -- --] ..., [-- -- -- ..., -- -- --] [-- -- -- ..., -- -- --] [-- -- -- ..., -- -- --]] though I know that there are valid values in the interior of the domain which plot properly. I can't find anything in the numpy documentation that addresses this type of missing value. Has anyone dealt with this before? Thanks, Bryan -------------- next part -------------- A non-text attachment was scrubbed... Name: bwoods.vcf Type: text/x-vcard Size: 341 bytes Desc: not available URL: From pav at iki.fi Thu Feb 24 13:05:07 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 24 Feb 2011 18:05:07 +0000 (UTC) Subject: [Numpy-discussion] blank values '--' References: <4D669CA9.3090909@aer.com> Message-ID: Thu, 24 Feb 2011 13:00:09 -0500, Bryan Woods wrote: [clip] > As an example if I print the array I see: > > [[-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --] > ..., > [-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --]] > > though I know that there are valid values in the interior of the domain > which plot properly. > > I can't find anything in the numpy documentation that addresses this > type of missing value. Has anyone dealt with this before? Here you go: http://docs.scipy.org/doc/numpy/reference/maskedarray.html -- Pauli Virtanen From ben.root at ou.edu Thu Feb 24 13:10:44 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 24 Feb 2011 12:10:44 -0600 Subject: [Numpy-discussion] blank values '--' In-Reply-To: <4D669CA9.3090909@aer.com> References: <4D669CA9.3090909@aer.com> Message-ID: On Thu, Feb 24, 2011 at 12:00 PM, Bryan Woods wrote: > Hello, > > I am loading in data from the GRIB2 files using PyNIO. Once I load them > into numpy arrays I am left with a halo of values near the array boundaries > that appear to be empty. When I print them or use them in conditional > statements I see values of '--'. I'd like to turn these values into zeroes > at the very least. > > As an example if I print the array I see: > > [[-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --] > ..., > [-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --]] > > though I know that there are valid values in the interior of the domain > which plot properly. > > I can't find anything in the numpy documentation that addresses this type > of missing value. Has anyone dealt with this before? > > Thanks, > Bryan > > This is called MaskedArrays, and they are a wonderful thing. http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html This is better than using magic numbers to indicate missing data, and sort of similiar in concept to using NaNs in place of missing data. Most of matplotlib's plotting functions support masked arrays (although we are still improving in this regard). You can easily get a regular numpy array like so: unmasked = maskedData.filled(0.0) I hope that helps! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From dbrown at ucar.edu Thu Feb 24 13:18:23 2011 From: dbrown at ucar.edu (David Brown) Date: Thu, 24 Feb 2011 11:18:23 -0700 Subject: [Numpy-discussion] blank values '--' In-Reply-To: <4D669CA9.3090909@aer.com> References: <4D669CA9.3090909@aer.com> Message-ID: Hi Bryan, By default PyNIO returns a NumPy masked array if a _FillValue attribute is found for the variable you are trying to load. It looks like Pauli Virtenen has pointed to the relevant NumPy documentation. Using masked array methods you can discover the values that are masked and print as '--'. From PyNIO it is also possible to control the creation of masked arrays from your data files. See http://www.pyngl.ucar.edu/Nio.shtml under Class NioOptions / generic options / MaskedArrayMode. You can indeed create a regular non-masked NumPy array if you set the MaskedArrayMode option to 'MaskedNever'. However, this will give you an array with 1e20 (in the case of GRIB files) in all the locations where you now see a '--'. You can of course change this to something else using the 'where' construct. -dave On Feb 24, 2011, at 11:00 AM, Bryan Woods wrote: > Hello, > > I am loading in data from the GRIB2 files using PyNIO. Once I load them into numpy arrays I am left with a halo of values near the array boundaries that appear to be empty. When I print them or use them in conditional statements I see values of '--'. I'd like to turn these values into zeroes at the very least. > > As an example if I print the array I see: > > [[-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --] > ..., > [-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --] > [-- -- -- ..., -- -- --]] > > though I know that there are valid values in the interior of the domain which plot properly. > > I can't find anything in the numpy documentation that addresses this type of missing value. Has anyone dealt with this before? > > Thanks, > Bryan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From nouiz at nouiz.org Thu Feb 24 14:26:01 2011 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Thu, 24 Feb 2011 14:26:01 -0500 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: References: <4D64205B.1030407@molden.no> <4D642E58.6050704@gmail.com> <20110222214809.GF9360@phare.normalesup.org> Message-ID: Hi, To print the information you can do: python -c 'import numpy;numpy.__config__.show()' You can access the info directly with: numpy.distutils.__config__.blas_opt_info['library_dirs']] numpy.distutils.__config__.blas_opt_info['libraries']] numpy.distutils.__config__.blas_opt_info['include_dirs']] This is what I do in Theano. Fred On Tue, Feb 22, 2011 at 7:05 PM, Eli Stevens (Gmail) wrote: > On Tue, Feb 22, 2011 at 1:48 PM, Gael Varoquaux > wrote: >> Probably because the numpy binary that the author was using was compiled >> without a blas implementation, and just using numpy's internal >> lapack_lite. This is a common problem in real life. > > Is there an easy way to check from within numpy if it's using a blas > iplementation or not? > > Thanks, > Eli > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From nwagner at iam.uni-stuttgart.de Thu Feb 24 14:32:10 2011 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 24 Feb 2011 20:32:10 +0100 Subject: [Numpy-discussion] NumPy speed tests by NASA In-Reply-To: References: <4D64205B.1030407@molden.no> <4D642E58.6050704@gmail.com> <20110222214809.GF9360@phare.normalesup.org> Message-ID: On Thu, 24 Feb 2011 14:26:01 -0500 Fr?d?ric Bastien wrote: > Hi, > > To print the information you can do: > > python -c 'import numpy;numpy.__config__.show()' > > You can access the info directly with: > > numpy.distutils.__config__.blas_opt_info['library_dirs']] > numpy.distutils.__config__.blas_opt_info['libraries']] > numpy.distutils.__config__.blas_opt_info['include_dirs']] > > This is what I do in Theano. > >Fred You can do >>> from numpy import show_config >>> show_config() atlas_threads_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/nwagner/src/ATLAS3.8.2/mybuild/lib'] define_macros = [('ATLAS_INFO', '"\\"3.8.2\\""')] language = f77 include_dirs = ['/home/nwagner/src/ATLAS3.8.2/include'] blas_opt_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/nwagner/src/ATLAS3.8.2/mybuild/lib'] define_macros = [('ATLAS_INFO', '"\\"3.8.2\\""')] language = c include_dirs = ['/home/nwagner/src/ATLAS3.8.2/include'] atlas_blas_threads_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/nwagner/src/ATLAS3.8.2/mybuild/lib'] define_macros = [('ATLAS_INFO', '"\\"3.8.2\\""')] language = c include_dirs = ['/home/nwagner/src/ATLAS3.8.2/include'] lapack_opt_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/nwagner/src/ATLAS3.8.2/mybuild/lib'] define_macros = [('ATLAS_INFO', '"\\"3.8.2\\""')] language = f77 include_dirs = ['/home/nwagner/src/ATLAS3.8.2/include'] lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE > > On Tue, Feb 22, 2011 at 7:05 PM, Eli Stevens (Gmail) > wrote: >> On Tue, Feb 22, 2011 at 1:48 PM, Gael Varoquaux >> wrote: >>> Probably because the numpy binary that the author was >>>using was compiled >>> without a blas implementation, and just using numpy's >>>internal >>> lapack_lite. This is a common problem in real life. >> >> Is there an easy way to check from within numpy if it's >>using a blas >> iplementation or not? >> >> Thanks, >> Eli >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From akabaila at pcug.org.au Thu Feb 24 21:06:49 2011 From: akabaila at pcug.org.au (Algis Kabaila) Date: Fri, 25 Feb 2011 13:06:49 +1100 Subject: [Numpy-discussion] Porting numpy to Python3 Message-ID: <201102251306.49886.akabaila@pcug.org.au> Are there plans to port numpy to Python3? In particular, when will the packages of Linear Algebra (viz matrix inversion) be available in Python 3 compatible modules. Because of the importance of numpy in many scientific endeavours is so great, information of the availability in Python 3 mode is very important and will be greatly appreciated. OldAl. -- Algis http://akabaila.pcug.org.au/StructuralAnalysis.pdf From shaohong86 at gmail.com Thu Feb 24 21:14:31 2011 From: shaohong86 at gmail.com (Shao Hong) Date: Fri, 25 Feb 2011 10:14:31 +0800 Subject: [Numpy-discussion] Porting numpy to Python3 In-Reply-To: <201102251306.49886.akabaila@pcug.org.au> References: <201102251306.49886.akabaila@pcug.org.au> Message-ID: Hi correct me if I am wrong, I thought there is package ported for python 3.1 already? On 25 Feb 2011 10:07, "Algis Kabaila" wrote: Are there plans to port numpy to Python3? In particular, when will the packages of Linear Algebra (viz matrix inversion) be available in Python 3 compatible modules. Because of the importance of numpy in many scientific endeavours is so great, information of the availability in Python 3 mode is very important and will be greatly appreciated. OldAl. -- Algis http://akabaila.pcug.org.au/StructuralAnalysis.pdf _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From akabaila at pcug.org.au Thu Feb 24 22:03:54 2011 From: akabaila at pcug.org.au (Algis Kabaila) Date: Fri, 25 Feb 2011 14:03:54 +1100 Subject: [Numpy-discussion] Porting numpy to Python3 In-Reply-To: References: <201102251306.49886.akabaila@pcug.org.au> Message-ID: <201102251403.54842.akabaila@pcug.org.au> If there is, then its great news (for me). Where can I check it out? Thanks for responding - Al. On Friday 25 February 2011 13:14:31 Shao Hong wrote: > Hi correct me if I am wrong, I thought there is package > ported for python 3.1 already? > -- Algis http://akabaila.pcug.org.au/StructuralAnalysis.pdf From bsouthey at gmail.com Thu Feb 24 22:22:04 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 24 Feb 2011 21:22:04 -0600 Subject: [Numpy-discussion] Porting numpy to Python3 In-Reply-To: <201102251403.54842.akabaila@pcug.org.au> References: <201102251306.49886.akabaila@pcug.org.au> <201102251403.54842.akabaila@pcug.org.au> Message-ID: On Thu, Feb 24, 2011 at 9:03 PM, Algis Kabaila wrote: > If there is, then its great news (for me). ?Where can I check it > out? > > Thanks for responding - > > Al. > > On Friday 25 February 2011 13:14:31 Shao Hong wrote: >> Hi correct me if I am wrong, I thought there is package >> ported for python 3.1 already? >> > -- > Algis > http://akabaila.pcug.org.au/StructuralAnalysis.pdf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Python 3.1+ support occurred with numpy 1.5 that was released last year (2010-08-31) - 1.5.1 is the current release. scipy 0.9 due very soon (release candidates are available at http://sourceforge.net/projects/scipy/) also supports Python 3 except for weave module. Bruce From akabaila at pcug.org.au Thu Feb 24 22:44:07 2011 From: akabaila at pcug.org.au (Algis Kabaila) Date: Fri, 25 Feb 2011 14:44:07 +1100 Subject: [Numpy-discussion] Porting numpy to Python3 In-Reply-To: References: <201102251306.49886.akabaila@pcug.org.au> <201102251403.54842.akabaila@pcug.org.au> Message-ID: <201102251444.07861.akabaila@pcug.org.au> On Friday 25 February 2011 14:22:04 Bruce Southey wrote: > > Python 3.1+ support occurred with numpy 1.5 that was released > last year (2010-08-31) - 1.5.1 is the current release. > scipy 0.9 due very soon (release candidates are available at > http://sourceforge.net/projects/scipy/) also supports Python > 3 except for weave module. > > Bruce Bruce, The link http://sourceforge.net/projects/scipy/ responds with "Whoops, we can't find that page". I use Linux kubuntu 10.10 ("Maverick Meerkat") OS and installed numpy with lapac from the ubuntu binaries. As ubuntu 10.10 was released on 2010-10-10 10:x the version on numpy is probably earlier than 1.5, so presumably I have to either wait a couple of months for the next ubuntu release or install from source. Bruce, can you tell me how can I check the version of installed numpy? There probably is a __version__ or something like that which would enable to verify. Thanks for the answer - I unsuccessfully googled for the answer; I also recall my much earlier enquiries when the replies did not shed much light on the possible release date. Your reply is very much appreciated - thank you! Al. -- Algis http://akabaila.pcug.org.au/StructuralAnalysis.pdf From akabaila at pcug.org.au Thu Feb 24 23:22:30 2011 From: akabaila at pcug.org.au (Algis Kabaila) Date: Fri, 25 Feb 2011 15:22:30 +1100 Subject: [Numpy-discussion] Porting numpy to Python3 In-Reply-To: <201102251444.07861.akabaila@pcug.org.au> References: <201102251306.49886.akabaila@pcug.org.au> <201102251444.07861.akabaila@pcug.org.au> Message-ID: <201102251522.30537.akabaila@pcug.org.au> On Friday 25 February 2011 14:44:07 Algis Kabaila wrote: > On Friday 25 February 2011 14:22:04 Bruce Southey wrote: > > Python 3.1+ support occurred with numpy 1.5 that was > > released last year (2010-08-31) - 1.5.1 is the current > > release. scipy 0.9 due very soon (release candidates are > > available at http://sourceforge.net/projects/scipy/) also > > supports Python 3 except for weave module. > > > > Bruce > > Bruce, > > The link > http://sourceforge.net/projects/scipy/ > responds with > "Whoops, we can't find that page". > I use Linux kubuntu 10.10 ("Maverick Meerkat") OS and > installed numpy with lapac from the ubuntu binaries. As > ubuntu 10.10 was released on 2010-10-10 10:x the version on > numpy is probably earlier than 1.5, so presumably I have to > either wait a couple of months for the next ubuntu release > or install from source. > > Bruce, can you tell me how can I check the version of > installed numpy? There probably is a __version__ or > something like that which would enable to verify. > > Thanks for the answer - I unsuccessfully googled for the > answer; I also recall my much earlier enquiries when the > replies did not shed much light on the possible release > date. > > Your reply is very much appreciated - thank you! > > Al. PS: a little investigation shows that my version of numpy is 1.3.0 and scipy is 0.7.2 - so ubuntu binaries are way behind the "bleeding edge"... Al. -- Algis http://akabaila.pcug.org.au/StructuralAnalysis.pdf From scott.sinclair.za at gmail.com Fri Feb 25 02:54:13 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Fri, 25 Feb 2011 09:54:13 +0200 Subject: [Numpy-discussion] Porting numpy to Python3 In-Reply-To: <201102251522.30537.akabaila@pcug.org.au> References: <201102251306.49886.akabaila@pcug.org.au> <201102251444.07861.akabaila@pcug.org.au> <201102251522.30537.akabaila@pcug.org.au> Message-ID: On 25 February 2011 06:22, Algis Kabaila wrote: > On Friday 25 February 2011 14:44:07 Algis Kabaila wrote: > PS: a little investigation shows that my version of numpy is > 1.3.0 and scipy is 0.7.2 - so ubuntu binaries are way behind the > "bleeding edge"... ... and built for the system Python (2.6), so even if the Ubuntu binaries were more up to date you'd need to build your own Numpy for Python 3. Cheers, Scott From akabaila at pcug.org.au Fri Feb 25 03:01:33 2011 From: akabaila at pcug.org.au (Algis Kabaila) Date: Fri, 25 Feb 2011 19:01:33 +1100 Subject: [Numpy-discussion] Porting numpy to Python3 In-Reply-To: References: <201102251306.49886.akabaila@pcug.org.au> <201102251522.30537.akabaila@pcug.org.au> Message-ID: <201102251901.33766.akabaila@pcug.org.au> On Friday 25 February 2011 18:54:13 Scott Sinclair wrote: > On 25 February 2011 06:22, Algis Kabaila wrote: > > On Friday 25 February 2011 14:44:07 Algis Kabaila wrote: > > PS: a little investigation shows that my version of numpy > > is 1.3.0 and scipy is 0.7.2 - so ubuntu binaries are way > > behind the "bleeding edge"... > > ... and built for the system Python (2.6), so even if the > Ubuntu binaries were more up to date you'd need to build > your own Numpy for Python 3. > > Cheers, > Scott > _______________________________________________ Scott, Good point! Thanks. Al. -- Algis http://akabaila.pcug.org.au/StructuralAnalysis.pdf From mustapha.bouikhif at idris.fr Fri Feb 25 04:01:21 2011 From: mustapha.bouikhif at idris.fr (Mustapha BOUIKHIF) Date: Fri, 25 Feb 2011 10:01:21 +0100 Subject: [Numpy-discussion] Problems with numpy In-Reply-To: References: <4D668170.20209@idris.fr> <4D6688D8.4060602@idris.fr> Message-ID: <4D676FE1.8060405@idris.fr> Pauli Virtanen wrote: > Thu, 24 Feb 2011 16:47:14 +0000, Pauli Virtanen wrote: >> Another possible reason is that Numpy was installed wrong (as the >> numpy.__config__ module is apparently missing). Numpy needs to be >> installed via "python setup.py install", manually copying the "numpy" >> directory is not enough. > > Or, the permissions of "/usr/local/lib64/python2.6/site-packages/numpy/ > __config__.py" (and possibly other files) are such that only root can > read them. > Numpy was installed correctly. It was a permission problem: many files were not readable by non-root users. It works fine now. Thanks. -- Cordialement Mustapha Bouikhif CNRS - IDRIS Tel: +33 1 69 35 85 27 Fax: +33 1 69 35 37 75 E-mail: mustapha.bouikhif at idris.fr http://www.idris.fr From fredmfp at gmail.com Fri Feb 25 04:36:42 2011 From: fredmfp at gmail.com (Fred) Date: Fri, 25 Feb 2011 10:36:42 +0100 Subject: [Numpy-discussion] Condensing array... In-Reply-To: References: <4D4FE3BB.90405@gmail.com> Message-ID: <4D67782A.3000205@gmail.com> Hi there, I have a big array (44 GB) I want to decimate. But this array has a lot of NaN (only 1/3 has value, in fact, so 2/3 of NaN). If I "basically" decimate it (a la NumPy, ie data[::nx, ::ny, ::nz], for instance), the decimated array will also have a lot of NaN. What I would like to have in one cell of the decimated array is the nearest (for instance) value in the big array. This is what I call a "condensated array". How could I do that, a la NumPy? TIA. Cheers, -- Fred From gael.varoquaux at normalesup.org Fri Feb 25 04:42:39 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 25 Feb 2011 10:42:39 +0100 Subject: [Numpy-discussion] Condensing array... In-Reply-To: <4D67782A.3000205@gmail.com> References: <4D4FE3BB.90405@gmail.com> <4D67782A.3000205@gmail.com> Message-ID: <20110225094239.GF24960@phare.normalesup.org> On Fri, Feb 25, 2011 at 10:36:42AM +0100, Fred wrote: > I have a big array (44 GB) I want to decimate. > But this array has a lot of NaN (only 1/3 has value, in fact, so 2/3 of > NaN). > If I "basically" decimate it (a la NumPy, ie data[::nx, ::ny, ::nz], for > instance), the decimated array will also have a lot of NaN. > What I would like to have in one cell of the decimated array is the > nearest (for instance) value in the big array. This is what I call a > "condensated array". What exactly do you mean by 'decimating'. To me is seems that you are looking for matrix factorization or matrix completion techniques, which are trendy topics in machine learning currently. They however are a bit challenging, and I fear that you will have read the papers and do some implementation, unless you have a clear application in mind that enables for simple tricks to solve it. G From fredmfp at gmail.com Fri Feb 25 04:52:09 2011 From: fredmfp at gmail.com (Fred) Date: Fri, 25 Feb 2011 10:52:09 +0100 Subject: [Numpy-discussion] Condensing array... In-Reply-To: <20110225094239.GF24960@phare.normalesup.org> References: <4D4FE3BB.90405@gmail.com> <4D67782A.3000205@gmail.com> <20110225094239.GF24960@phare.normalesup.org> Message-ID: <4D677BC9.90406@gmail.com> Le 25/02/2011 10:42, Gael Varoquaux a ?crit : > What exactly do you mean by 'decimating'. To me is seems that you are > looking for matrix factorization or matrix completion techniques, which > are trendy topics in machine learning currently. By decimating, I mean this: input array data.shape = (nx, ny, nz) -> data[::ax, ::ay, ::az], ie output array data[::ax, ::ay, ::az].shape = (nx/ax, ny/ay, nz/az). -- Fred From gael.varoquaux at normalesup.org Fri Feb 25 05:22:06 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 25 Feb 2011 11:22:06 +0100 Subject: [Numpy-discussion] Condensing array... In-Reply-To: <4D677BC9.90406@gmail.com> References: <4D4FE3BB.90405@gmail.com> <4D67782A.3000205@gmail.com> <20110225094239.GF24960@phare.normalesup.org> <4D677BC9.90406@gmail.com> Message-ID: <20110225102206.GG24960@phare.normalesup.org> On Fri, Feb 25, 2011 at 10:52:09AM +0100, Fred wrote: > > What exactly do you mean by 'decimating'. To me is seems that you are > > looking for matrix factorization or matrix completion techniques, which > > are trendy topics in machine learning currently. > By decimating, I mean this: > input array data.shape = (nx, ny, nz) -> data[::ax, ::ay, ::az], ie > output array data[::ax, ::ay, ::az].shape = (nx/ax, ny/ay, nz/az). OK, this can be seen as an interpolation on a grid with a nearest neighbor interpolator. What I am unsure about is whether you want to interpolate your NaN, or whether they just mean missing data. I would do this by representing the matrix as a sparse matrix in COO, this would give you a list of row and col positions for your data points. Then I would use a nearest neighbor (such as scipy's KDTree, or the scikit-learn's BallTree for even better performance http://scikit-learn.sourceforge.net/modules/neighbors.html) to find, for each grid point which data point is closest and fill in your grid. I suspect that your problem is that you can't fit the whole matrix in memory. If your data points are reasonnably homogeneously distributed in the matrix, I would simply process the problem using sub matrices, and making sure that I train the nearest neighbor on a sub matrix that is largest than the sampling grid by a factor of more than the inter-point distance. HTH, Gael From olivier.grisel at ensta.org Fri Feb 25 05:52:29 2011 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Fri, 25 Feb 2011 11:52:29 +0100 Subject: [Numpy-discussion] Condensing array... In-Reply-To: <20110225094239.GF24960@phare.normalesup.org> References: <4D4FE3BB.90405@gmail.com> <4D67782A.3000205@gmail.com> <20110225094239.GF24960@phare.normalesup.org> Message-ID: 2011/2/25 Gael Varoquaux : > On Fri, Feb 25, 2011 at 10:36:42AM +0100, Fred wrote: >> I have a big array (44 GB) I want to decimate. > >> But this array has a lot of NaN (only 1/3 has value, in fact, so 2/3 of >> NaN). > >> If I "basically" decimate it (a la NumPy, ie data[::nx, ::ny, ::nz], for >> instance), the decimated array will also have a lot of NaN. > >> What I would like to have in one cell of the decimated array is the >> nearest (for instance) value in the big array. This is what I call a >> "condensated array". > > What exactly do you mean by 'decimating'. To me is seems that you are > looking for matrix factorization or matrix completion techniques, which > are trendy topics in machine learning currently. > > They however are a bit challenging, and I fear that you will have read > the papers and do some implementation, unless you have a clear > application in mind that enables for simple tricks to solve it. Indeed the following paper by G. Martinsson from there is also a section on matrix summarization: http://arxiv.org/abs/0909.4061 http://www.stanford.edu/group/mmds/slides2010/Martinsson.pdf The scikit-learn randomized SVD implementation is coming this paper. It's pretty useful in practice. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From fredmfp at gmail.com Fri Feb 25 07:15:32 2011 From: fredmfp at gmail.com (Fred) Date: Fri, 25 Feb 2011 13:15:32 +0100 Subject: [Numpy-discussion] Condensing array... In-Reply-To: References: <4D4FE3BB.90405@gmail.com> <4D67782A.3000205@gmail.com> <20110225094239.GF24960@phare.normalesup.org> Message-ID: <4D679D64.60203@gmail.com> Ga?l, Olivier, I finally got working it. I don't compute the nearest value but the mean. Works like a charm ;-) Thanks anyway. Cheers, -- Fred From deshpande.jaidev at gmail.com Fri Feb 25 10:37:25 2011 From: deshpande.jaidev at gmail.com (Jaidev Deshpande) Date: Fri, 25 Feb 2011 21:07:25 +0530 Subject: [Numpy-discussion] Memory error with numpy.loadtxt() Message-ID: Hi Is it possible to load a text file 664 MB large with integer values and about 98% sparse? numpy.loadtxt() shows a memory error. If it's not possible, what alternatives could I have? The usable RAM on my machine running Windows 7 is 3.24 GB. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Fri Feb 25 10:58:19 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 25 Feb 2011 09:58:19 -0600 Subject: [Numpy-discussion] Porting numpy to Python3 In-Reply-To: <201102251901.33766.akabaila@pcug.org.au> References: <201102251306.49886.akabaila@pcug.org.au> <201102251522.30537.akabaila@pcug.org.au> <201102251901.33766.akabaila@pcug.org.au> Message-ID: <4D67D19B.7080707@gmail.com> On 02/25/2011 02:01 AM, Algis Kabaila wrote: > On Friday 25 February 2011 18:54:13 Scott Sinclair wrote: >> On 25 February 2011 06:22, Algis Kabaila > wrote: >>> On Friday 25 February 2011 14:44:07 Algis Kabaila wrote: >>> PS: a little investigation shows that my version of numpy >>> is 1.3.0 and scipy is 0.7.2 - so ubuntu binaries are way >>> behind the "bleeding edge"... >> ... and built for the system Python (2.6), so even if the >> Ubuntu binaries were more up to date you'd need to build >> your own Numpy for Python 3. >> >> Cheers, >> Scott >> _______________________________________________ > Scott, > > Good point! Thanks. > > Al. I just build numpy and scipy from source so I do not know how you get Python 3 or which Ubuntu versions include recent numpy versions (there is a upcoming release that will probably contain a more recent numpy). It is very easy to install numpy and scipy from source on Linux although it is important that Blas/Lapack/Altas are built and installed correctly (I just use my distro's package). Please see the following link for more details: http://www.scipy.org/Installing_SciPy/Linux There was also a recent post on the list as well. Bruce From kwgoodman at gmail.com Fri Feb 25 12:30:05 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 25 Feb 2011 09:30:05 -0800 Subject: [Numpy-discussion] When memory access is a bottleneck Message-ID: A topic that often comes up on the list is that arr.sum(axis=-1) is faster than arr.sum(axis=0). For C ordered arrays, moving along the last axis moves the smallest amount in memory. And moving small amounts in memory keeps the data in cache longer. Can I use that fact to speed up calculations along axis=0? As a test I used the nanmean function from the Bottleneck package. I created a second version of the function that calculates the nanmean of two columns at a time (code below). I figured that would be a more efficient use of cache. It is faster for many array sizes but not for all. Is there anything I can do about that? Is the speed up really due to cache? The smallest arrays I tried should fit entirely in cache, so perhaps unrolling the loop is helping the compiler be more efficient? Yeah, so obviously, I don't understand what is going on. Faster: >> a = np.random.rand(10,10) >> timeit nanmean_2d_float64_axis0(a) 1000000 loops, best of 3: 1.24 us per loop >> timeit nanmean_2d_float64_axis0_double(a) 1000000 loops, best of 3: 1.17 us per loop >> a = np.random.rand(100,100) >> timeit nanmean_2d_float64_axis0(a) 100000 loops, best of 3: 16 us per loop >> timeit nanmean_2d_float64_axis0_double(a) 100000 loops, best of 3: 15.7 us per loop >> a = np.random.rand(1000,1000) >> timeit nanmean_2d_float64_axis0(a) 100 loops, best of 3: 8.57 ms per loop >> timeit nanmean_2d_float64_axis0_double(a) 100 loops, best of 3: 5.02 ms per loop >> a = np.random.rand(10000,100) >> timeit nanmean_2d_float64_axis0(a) 100 loops, best of 3: 3.52 ms per loop >> timeit nanmean_2d_float64_axis0_double(a) 100 loops, best of 3: 3.35 ms per loop Slower: >> a = np.random.rand(100,10000) >> timeit nanmean_2d_float64_axis0(a) 100 loops, best of 3: 2.22 ms per loop >> timeit nanmean_2d_float64_axis0_double(a) 100 loops, best of 3: 2.57 ms per loop Code (Bottleneck simplified BSD license): @cython.boundscheck(False) @cython.wraparound(False) def nanmean_2d_float64_axis0(np.ndarray[np.float64_t, ndim=2] a): "Mean of 2d array with dtype=float64 along axis=0 ignoring NaNs." cdef int count = 0 cdef np.float64_t asum = 0, ai cdef Py_ssize_t i0, i1 cdef np.npy_intp *dim dim = PyArray_DIMS(a) cdef int n0 = dim[0] cdef int n1 = dim[1] cdef np.npy_intp *dims = [n1] cdef np.ndarray[np.float64_t, ndim=1] y = PyArray_EMPTY(1, dims, NPY_float64, 0) for i1 in range(n1): asum = 0 count = 0 for i0 in range(n0): ai = a[i0, i1] if ai == ai: asum += ai count += 1 if count > 0: y[i1] = asum / count else: y[i1] = NAN return y @cython.boundscheck(False) @cython.wraparound(False) def nanmean_2d_float64_axis0_double(np.ndarray[np.float64_t, ndim=2] a): "Mean of 2d array with dtype=float64 along axis=0 ignoring NaNs." cdef int count = 0, count2 = 0 cdef np.float64_t asum = 0, asum2 = 0, ai cdef Py_ssize_t i0, i1, i11 cdef np.npy_intp *dim dim = PyArray_DIMS(a) cdef int n0 = dim[0] cdef int n1 = dim[1] cdef np.npy_intp *dims = [n1] cdef np.ndarray[np.float64_t, ndim=1] y = PyArray_EMPTY(1, dims, NPY_float64, 0) for i1 in range(0,n1,2): asum = 0 count = 0 asum2 = 0 count2 = 0 i11 = i1 + 1 for i0 in range(n0): ai = a[i0, i1] if ai == ai: asum += ai count += 1 ai = a[i0, i11] if ai == ai: asum2 += ai count2 += 1 if count > 0: y[i1] = asum / count else: y[i1] = NAN if count2 > 0: y[i11] = asum / count else: y[i11] = NAN return y From jkington at wisc.edu Fri Feb 25 12:52:27 2011 From: jkington at wisc.edu (Joe Kington) Date: Fri, 25 Feb 2011 11:52:27 -0600 Subject: [Numpy-discussion] Memory error with numpy.loadtxt() In-Reply-To: References: Message-ID: Do you expect to have very large integer values, or only values over a limited range? If your integer values will fit in into 16-bit range (or even 32-bit, if you're on a 64-bit machine, the default dtype is float64...) you can potentially halve your memory usage. I.e. Something like: data = numpy.loadtxt(filename, dtype=numpy.int16) Alternately, if you're already planning on using a (scipy) sparse array anyway, it's easy to do something like this: import numpy as np import scipy.sparse I, J, V = [], [], [] with open('infile.txt') as infile: for i, line in enumerate(infile): line = np.array(line.strip().split(), dtype=np.int) nonzeros, = line.nonzero() I.extend([i]*nonzeros.size) J.extend(nonzeros) V.extend(line[nonzeros]) data = scipy.sparse.coo_matrix((V,(I,J)), dtype=np.int, shape=(i+1, line.size)) This will be much slower than numpy.loadtxt(...), but if you're just converting the output of loadtxt to a sparse array, regardless, this would avoid memory usage problems (assuming the array is mostly sparse, of course). Hope that helps, -Joe On Fri, Feb 25, 2011 at 9:37 AM, Jaidev Deshpande < deshpande.jaidev at gmail.com> wrote: > Hi > > Is it possible to load a text file 664 MB large with integer values and > about 98% sparse? numpy.loadtxt() shows a memory error. > > If it's not possible, what alternatives could I have? > > The usable RAM on my machine running Windows 7 is 3.24 GB. > > Thanks. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Fri Feb 25 13:38:34 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 25 Feb 2011 13:38:34 -0500 Subject: [Numpy-discussion] Memory error with numpy.loadtxt() In-Reply-To: References: Message-ID: On Fri, Feb 25, 2011 at 12:52 PM, Joe Kington wrote: > Do you expect to have very large integer values, or only values over a > limited range? > > If your integer values will fit in into 16-bit range (or even 32-bit, if > you're on a 64-bit machine, the default dtype is float64...) you can > potentially halve your memory usage. > > I.e. Something like: > data = numpy.loadtxt(filename, dtype=numpy.int16) > > Alternately, if you're already planning on using a (scipy) sparse array > anyway, it's easy to do something like this: > > import numpy as np > import scipy.sparse > I, J, V = [], [], [] > with open('infile.txt') as infile: > for i, line in enumerate(infile): > line = np.array(line.strip().split(), dtype=np.int) > nonzeros, = line.nonzero() > I.extend([i]*nonzeros.size) > J.extend(nonzeros) > V.extend(line[nonzeros]) > data = scipy.sparse.coo_matrix((V,(I,J)), dtype=np.int, shape=(i+1, > line.size)) > > This will be much slower than numpy.loadtxt(...), but if you're just > converting the output of loadtxt to a sparse array, regardless, this would > avoid memory usage problems (assuming the array is mostly sparse, of > course). > > Hope that helps, > -Joe > > > > On Fri, Feb 25, 2011 at 9:37 AM, Jaidev Deshpande < > deshpande.jaidev at gmail.com> wrote: > >> Hi >> >> Is it possible to load a text file 664 MB large with integer values and >> about 98% sparse? numpy.loadtxt() shows a memory error. >> >> If it's not possible, what alternatives could I have? >> >> The usable RAM on my machine running Windows 7 is 3.24 GB. >> >> Thanks. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > In addition to this is helpful to remember that just becuase you have 3.24 GB available, doesn't mean that 664MB of that is contiguous, which is what NumPy would need to hold it all in memory. -------------- next part -------------- An HTML attachment was scrubbed... URL: From akabaila at pcug.org.au Fri Feb 25 17:12:47 2011 From: akabaila at pcug.org.au (Algis Kabaila) Date: Sat, 26 Feb 2011 09:12:47 +1100 Subject: [Numpy-discussion] Porting numpy to Python3 In-Reply-To: <4D67D19B.7080707@gmail.com> References: <201102251306.49886.akabaila@pcug.org.au> <201102251901.33766.akabaila@pcug.org.au> <4D67D19B.7080707@gmail.com> Message-ID: <201102260912.48413.akabaila@pcug.org.au> On Saturday 26 February 2011 02:58:19 Bruce Southey wrote: > On 02/25/2011 02:01 AM, Algis Kabaila wrote: > > I just build numpy and scipy from source so I do not know how > you get Python 3 or which Ubuntu versions include recent > numpy versions (there is a upcoming release that will > probably contain a more recent numpy). It is very easy to > install numpy and scipy from source on Linux although it is > important that Blas/Lapack/Altas are built and installed > correctly (I just use my distro's package). > > Please see the following link for more details: > http://www.scipy.org/Installing_SciPy/Linux > There was also a recent post on the list as well. > > Bruce > Bruce, Thank you for the information. I had a look at the URL that you gave (Actually, I had already seen it in my wanderings on internet.) For me it is very important to hear from the users' perspective how easy or hard the installation was. Python 3 is readily available in most distros, certainly in ubuntu, so that is not a problem. In fact in ubuntu Python 2.6 happily coexists with Python 3.1 for one and the same user. Thank you for your help - greatly appreciated. OldAl. -- Algis http://akabaila.pcug.org.au/StructuralAnalysis.pdf From wkerzendorf at googlemail.com Sat Feb 26 03:36:48 2011 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Sat, 26 Feb 2011 19:36:48 +1100 Subject: [Numpy-discussion] Grid indexing Message-ID: <4D68BBA0.4060009@gmail.com> Hello, I have a n dimensional grid. The grids axes are linear but not intergers. Let's say I want the value in gridcell [3.2,-5.6,0.01]. Is there an easy way to transform the index? Do I have to write my own class to accomplish this. Is there an easy way in numpy to accomplish this. Can I give functions to transform the index somehow to the numpy array? Cheers Wolfgang From moura.mario at gmail.com Sat Feb 26 08:41:49 2011 From: moura.mario at gmail.com (Mario Moura) Date: Sat, 26 Feb 2011 10:41:49 -0300 Subject: [Numpy-discussion] How recreated a integer list with a generator? Message-ID: Hi Folks How recreated a integer list with generator? import random tmp = [] for x in range(0,600): tmp.append(random.randint(0,5000)) tmp = list(set(tmp)) tmp.sort() Should be possible recreated this list using same magic math formula or numpy array? Can I create a generator? How? What is my target? I need store this list im my database, the list itself get big size (resources) so I want save resources and save the generator object with pickle. Is it possible? What I have: size = len(tmp) min = tmp[1] max = tmp[-1] All numbers are integer and it is a random serie so each list will have your own generator. Again tmp is a static list and must be recreated with a generator the target is pickle the generator to be smaller than the list itself. Best Regards Mario From deshpande.jaidev at gmail.com Sat Feb 26 10:09:03 2011 From: deshpande.jaidev at gmail.com (Jaidev Deshpande) Date: Sat, 26 Feb 2011 20:39:03 +0530 Subject: [Numpy-discussion] Largest possible numpy array Message-ID: Hi How can I know the size of the largest possible 2-D array in numpy, given a specific 'dtype' and my system memory? How can one play around with this? Would it help to note that the array might be say *m* megabytes on disk, say, *n* % sparse? Also, is there some good literature about the largest possible 2-D arrays on different computing platforms? Google isn't helping much. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sat Feb 26 13:44:03 2011 From: sturla at molden.no (Sturla Molden) Date: Sat, 26 Feb 2011 19:44:03 +0100 Subject: [Numpy-discussion] Largest possible numpy array In-Reply-To: References: Message-ID: <4D6949F3.5000205@molden.no> Den 26.02.2011 16:09, skrev Jaidev Deshpande: > Hi > > How can I know the size of the largest possible 2-D array in numpy, > given a specific 'dtype' and my system memory? > The largest array size is not dicatated by system memory but virtual memory. On Windws the available virtual address space is 2 GB with x86 and 16 TB with AMD64. If you have less physical memory than the array size, the OS will swap some of it to disk. Sturla From bbelson at princeton.edu Sat Feb 26 20:15:11 2011 From: bbelson at princeton.edu (Brandt Belson) Date: Sat, 26 Feb 2011 20:15:11 -0500 Subject: [Numpy-discussion] Where can I download numpy.i? Message-ID: I just want to know exactly how to get the file numpy.i (for SWIG), I've come across links that seem to take me nowhere. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wfspotz at sandia.gov Sat Feb 26 23:30:14 2011 From: wfspotz at sandia.gov (Bill Spotz) Date: Sat, 26 Feb 2011 21:30:14 -0700 Subject: [Numpy-discussion] Where can I download numpy.i? In-Reply-To: References: Message-ID: When you download numpy, it should be in doc/swig/numpy.i On Feb 26, 2011, at 6:15 PM, Brandt Belson wrote: > I just want to know exactly how to get the file numpy.i (for SWIG), I've come across links that seem to take me nowhere. > Thank you. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From bbelson at princeton.edu Sun Feb 27 13:26:09 2011 From: bbelson at princeton.edu (Brandt Belson) Date: Sun, 27 Feb 2011 13:26:09 -0500 Subject: [Numpy-discussion] Where can I download numpy.i? Message-ID: Thanks, but I just downloaded the source for versions 1.5.1 and 1.4.1 and neither appear to have a swig directory in doc or numpy/doc. Are you sure that it is included with the standard download from http://sourceforge.net/projects/numpy/files/NumPy/ ? Is there another way to get the file? Thanks On Sun, Feb 27, 2011 at 1:00 PM, wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion at scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request at scipy.org > > You can reach the person managing the list at > numpy-discussion-owner at scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: Largest possible numpy array (Sturla Molden) > 2. Where can I download numpy.i? (Brandt Belson) > 3. Re: Where can I download numpy.i? (Bill Spotz) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 26 Feb 2011 19:44:03 +0100 > From: Sturla Molden > Subject: Re: [Numpy-discussion] Largest possible numpy array > To: Discussion of Numerical Python > Message-ID: <4D6949F3.5000205 at molden.no> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Den 26.02.2011 16:09, skrev Jaidev Deshpande: > > Hi > > > > How can I know the size of the largest possible 2-D array in numpy, > > given a specific 'dtype' and my system memory? > > > > The largest array size is not dicatated by system memory but virtual > memory. > > On Windws the available virtual address space is 2 GB with x86 and 16 TB > with AMD64. > > If you have less physical memory than the array size, the OS will swap > some of it to disk. > > Sturla > > > ------------------------------ > > Message: 2 > Date: Sat, 26 Feb 2011 20:15:11 -0500 > From: Brandt Belson > Subject: [Numpy-discussion] Where can I download numpy.i? > To: numpy-discussion at scipy.org > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > I just want to know exactly how to get the file numpy.i (for SWIG), I've > come across links that seem to take me nowhere. > Thank you. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110226/f4679a53/attachment-0001.html > > ------------------------------ > > Message: 3 > Date: Sat, 26 Feb 2011 21:30:14 -0700 > From: "Bill Spotz" > Subject: Re: [Numpy-discussion] Where can I download numpy.i? > To: "Discussion of Numerical Python" > Message-ID: > Content-Type: text/plain; charset=us-ascii > > When you download numpy, it should be in doc/swig/numpy.i > > On Feb 26, 2011, at 6:15 PM, Brandt Belson wrote: > > > I just want to know exactly how to get the file numpy.i (for SWIG), I've > come across links that seem to take me nowhere. > > Thank you. > > ** Bill Spotz ** > ** Sandia National Laboratories Voice: (505)845-0170 ** > ** P.O. Box 5800 Fax: (505)284-0154 ** > ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** > > > > > ------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > End of NumPy-Discussion Digest, Vol 53, Issue 62 > ************************************************ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Sun Feb 27 13:31:51 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sun, 27 Feb 2011 12:31:51 -0600 Subject: [Numpy-discussion] Where can I download numpy.i? In-Reply-To: References: Message-ID: On Sun, Feb 27, 2011 at 12:26 PM, Brandt Belson wrote: > Thanks, but I just downloaded the source for versions 1.5.1 and 1.4.1 and > neither appear to have a swig directory in doc or numpy/doc. Are you sure > that it is included with the standard download from > http://sourceforge.net/projects/numpy/files/NumPy/ > ? > Is there another way to get the file? > It is in the source code repository on github: https://github.com/numpy/numpy/tree/master/doc/swig Warren Thanks > > On Sun, Feb 27, 2011 at 1:00 PM, wrote: > >> Send NumPy-Discussion mailing list submissions to >> numpy-discussion at scipy.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> or, via email, send a message with subject or body 'help' to >> numpy-discussion-request at scipy.org >> >> You can reach the person managing the list at >> numpy-discussion-owner at scipy.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of NumPy-Discussion digest..." >> >> >> Today's Topics: >> >> 1. Re: Largest possible numpy array (Sturla Molden) >> 2. Where can I download numpy.i? (Brandt Belson) >> 3. Re: Where can I download numpy.i? (Bill Spotz) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Sat, 26 Feb 2011 19:44:03 +0100 >> From: Sturla Molden >> Subject: Re: [Numpy-discussion] Largest possible numpy array >> To: Discussion of Numerical Python >> Message-ID: <4D6949F3.5000205 at molden.no> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> Den 26.02.2011 16:09, skrev Jaidev Deshpande: >> > Hi >> > >> > How can I know the size of the largest possible 2-D array in numpy, >> > given a specific 'dtype' and my system memory? >> > >> >> The largest array size is not dicatated by system memory but virtual >> memory. >> >> On Windws the available virtual address space is 2 GB with x86 and 16 TB >> with AMD64. >> >> If you have less physical memory than the array size, the OS will swap >> some of it to disk. >> >> Sturla >> >> >> ------------------------------ >> >> Message: 2 >> Date: Sat, 26 Feb 2011 20:15:11 -0500 >> From: Brandt Belson >> Subject: [Numpy-discussion] Where can I download numpy.i? >> To: numpy-discussion at scipy.org >> Message-ID: >> >> Content-Type: text/plain; charset="iso-8859-1" >> >> >> I just want to know exactly how to get the file numpy.i (for SWIG), I've >> come across links that seem to take me nowhere. >> Thank you. >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110226/f4679a53/attachment-0001.html >> >> ------------------------------ >> >> Message: 3 >> Date: Sat, 26 Feb 2011 21:30:14 -0700 >> From: "Bill Spotz" >> Subject: Re: [Numpy-discussion] Where can I download numpy.i? >> To: "Discussion of Numerical Python" >> Message-ID: >> Content-Type: text/plain; charset=us-ascii >> >> >> When you download numpy, it should be in doc/swig/numpy.i >> >> On Feb 26, 2011, at 6:15 PM, Brandt Belson wrote: >> >> > I just want to know exactly how to get the file numpy.i (for SWIG), I've >> come across links that seem to take me nowhere. >> > Thank you. >> >> ** Bill Spotz ** >> ** Sandia National Laboratories Voice: (505)845-0170 ** >> ** P.O. Box 5800 Fax: (505)284-0154 ** >> ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** >> >> >> >> >> ------------------------------ >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> End of NumPy-Discussion Digest, Vol 53, Issue 62 >> ************************************************ >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.raybaut at gmail.com Sun Feb 27 13:34:53 2011 From: pierre.raybaut at gmail.com (Pierre Raybaut) Date: Sun, 27 Feb 2011 19:34:53 +0100 Subject: [Numpy-discussion] ANN: Spyder v2.0.8 Message-ID: Hi all, I am pleased to announced that Spyder v2.0.8 has just been released. As this is mostly a maintenance release, a lot of bugs were fixed (see below) and some minor features were added. Integration Spyder's GUI-based Python shell in your application: Note that this version includes an example of application using the Spyder's internal shell as a debugging console which also demonstrates the py2exe deployment procedure. Spyder (previously known as Pydee) is a free open-source Python development environment providing MATLAB-like features in a simple and light-weighted software, available for Windows XP/Vista/7, GNU/Linux and MacOS X: http://spyderlib.googlecode.com/ Spyder is part of spyderlib, a Python module based on PyQt4, pyflakes and rope (QScintilla's dependency has been removed in version 2.0 and rope features have been integrated since this version as well). Some of Spyder basic features: * Python, C/C++, Fortran source editor with class/function browser, code completion and calltips * consoles: o open as many Python interpreters, IPython consoles or command windows as you need o code completion and calltips o variable explorer with GUI-based editors for a lot of data types (numbers, strings, lists, arrays, dictionaries, ...) * object inspector: provide documentation or source code on any Python object (class, function, module, ...) * online documentation: automatically generated html documentation on installed Python modules * find in files * file explorer * project manager * MATLAB-like PYTHONPATH management dialog box (works with all consoles) * Windows only: current user environment variables editor * direct links to documentation (Python, Qt, Matplotlib, NumPy, Scipy, etc.) * direct link to Python(x,y) launcher * direct links to QtDesigner, QtLinguist and QtAssistant (Qt documentation) Bug fixes (since v2.0.6): Consoles/bugfix: saving history log (see context menu) was not working following a recent code cleaning/refactoring On non-Windows platforms, the file selection dialog "All files (*.*)" filter was not matching files without extension dochelpers.isdefined/bugfix: ignoring syntax errors while evaluating object Preferences Dialog (dialog box + keyboard shortcut page): improved size/resize behavior Editor: when cursor was on the very last line, Duplicate/Delete line features were getting stuck in an infinite loop Editor/duplicate line feature - fixed unexpected behavior: when duplicating selected text, text selection was extended to duplicated part Editor/bugfix with multiple editor windows: when opening file on one editor window, the top-left corner menu (file list) was not updated correctly in other editor windows Editor/fixed unexpected behavior: when clicking on the main window's outline explorer while a separate editor window had focus, the latter was used to show the associated line of code Project explorer: added new debugging options (profiling 'rope' calls) Console/Advanced settings/UMD module list: removing all entries (empty module list) was not working Editor/File list management dialog (Ctrl+E): double-clicking/pressing Return on a listwidget item will switch to the associated file Editor/Tab bar: fixed missing tooltips issue (and missing file switch menu entries) Code completion/bugfix: list widget was not hiding as expected when pressing ':' Editor/fixed unexpected behavior: when some text was selected, "Ctrl+Left mouse click" was trying to "go to definition" instead of doing the standard drag n'drop feature Editor/bugfix: disabling code completion/calltips for non-Python source code (was not working -as expected- but was taking time to simply not work...) Editor/go to line: fixed unicode error Code editor/bugfix: cursor position was not restored when undoing an indent operation with "tab always indent" feature turned on *and* the cursor at the end of the line Tab behavior when "tab always indents" is turned off: inserting 4-(len(leading_text) % 4) spaces (instead of 4) Object inspector/bugfix: ignoring unknown objects when called automatically from editor/console, i.e. do not log, do not show 'no doc available' Other changes (since v2.0.6): Code editor syntax highlighting: added more keywords to Cython syntax highlighter (cpdef, inline, cimport and DEF) Added example of application using the Spyder's internal shell as a debugging console (demonstrates also the py2exe deployment procedure) Object inspector: added "Source" combo box (Console/Editor) -> object inspected from editor are now analyzed only with rope (if available) and then shown in object inspector Added keyboard shortcut to open Preferences Dialog (default: Ctrl+Alt+Shift+P) Editor: added "Copy line" feature (Ctrl+Alt+Down), similar to "Duplicate line" (Ctrl+Alt+Up) but paste text before the current line/selected text (instead of after) Array editor: added option to set row/col labels (resp. ylabels and xlabels) Editor/rope: improved performance for calltips/doc feature Cheers, Pierre From jsalvati at u.washington.edu Mon Feb 28 01:37:14 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Sun, 27 Feb 2011 22:37:14 -0800 Subject: [Numpy-discussion] Example of PyArray_RemoveSmallest use? Message-ID: Does anyone have or know of an example of how to use PyArray_RemoveSmallest(multiiterator) to make the inner loop fast (how to get the right strides etc.)? I am finding it a bit confusing. Best Regards, John -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Feb 28 02:15:48 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 28 Feb 2011 15:15:48 +0800 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: References: Message-ID: Hi Jon, Thanks for the patch, and sorry for the slow reply. On Thu, Feb 24, 2011 at 11:49 PM, Jon Olav Vik wrote: > https://github.com/jonovik/numpy/compare/master...offset_memmap > > The `offset` argument to np.memmap enables memory-mapping a portion of a file > on disk to a memory-mapped Numpy array. Memory-mapping can also be done with > np.load, but it uses np.lib.format.open_memmap, which has no offset argument. > > I have added an offset argument to np.lib.format.open_memmap and np.load as > detailed in the link above, and humbly submit the changes for review. This is > my first time using git, apologies for any mistakes. My first question after looking at this is why we would want three very similar ways to load memory-mapped arrays (np.memmap, np.load, np.lib.format.open_memmap)? They already exist but your changes make those three even more similar. I'd think we want one simple version (load) and a full-featured one. So imho changing open_memmap but leaving np.load as-is would be the way to go. > Note that the offset is in terms of array elements, not bytes (which is what > np.memmap uses), because that was what I had use for. This should be kept in bytes like in np.memmap I think, it's just confusing if those two functions differ like that. Another thing to change: you should not use an assert statement, use "if ....: raise ...." instead. > My use case was to preallocate a big record array on disk, then start many > processes writing to their separate, memory-mapped segments of the file. The > end result was one big array on disk, with the correct shape and data type > information. Using a record array makes the data structure more self- > documenting. Using open_memmap with mode="w+" is the fastest way I've found to > preallocate an array on disk; it does not create the huge array in memory. > Letting multiple processes memory-map and read/write to non-overlapping > portions without interfering with each other allows for fast, simple parallel I/ > O. > > I've used this extensively on Numpy 1.4.0, but based my Git checkout on the > current Numpy trunk. There have been some rearrangements in np.load since then > (it used to be in np.lib.io and is now in np.lib.npyio), but as far as I can > see, my modifications carry over fine. I haven't had a chance to test with > Numpy trunk, though. (What is the best way to set up a test version without > affecting my working 1.4.0 setup?) You can use an in-place build (http://projects.scipy.org/numpy/wiki/DevelopmentTips) and add that dir to your PYTHONPATH. Cheers, Ralf From ralf.gommers at googlemail.com Mon Feb 28 03:00:39 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 28 Feb 2011 16:00:39 +0800 Subject: [Numpy-discussion] Numpy 2.0 schedule In-Reply-To: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> Message-ID: Hi, On Fri, Jan 28, 2011 at 7:15 AM, Travis Oliphant wrote: > The reason for a NumPy 1.6 suggestion, is that Mark (and others it would > seem) have additional work and features that do not need to wait for the > NumPy 2.0 ABI design to finalize in order to get out there. > If someone is willing to manage the release of NumPy 1.6, then it sounds > like a great idea to me. This thread ended without a conclusion a month ago. Now I think master is in a better state than a month ago for a release (py 2.4/2.5/3.x issues and segfault on OS X fixed, more testing of changes), and I have a better idea of my free time for March/April. Basically, I have a good amount of time for the next couple of weeks, and not so much at the end of March / first half of April due to an inter-continental move. But I think we can get out a beta by mid-March, and I can manage the release. I've had a look at the bug tracker, here's a list of tickets for 1.6: #1748 (blocker: regression for astype('str')) #1619 (issue with dtypes, with patch) #1749 (distutils, py 3.2) #1601 (distutils, py 3.2) #1622 (Solaris segfault, with patch) #1713 (Solaris segfault) #1631 (Solaris segfault) I can look at the distutils tickets. The other thing that needs to be done is some (more) documentation of new features. Einsum and the new iterator seem to be well documented, but not described in the release notes. Datetime has no docs as far as I can see except for two similar NEPs. Proposed schedule: March 15: beta 1 March 28: rc 1 April 17: rc 2 (if needed) April 24: final release Let me know what you think. Bonus points for volunteering to fix some of those tickets:) Cheers, Ralf From ralf.gommers at googlemail.com Mon Feb 28 03:50:59 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 28 Feb 2011 16:50:59 +0800 Subject: [Numpy-discussion] Numpy 2.0 schedule In-Reply-To: References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> Message-ID: On Mon, Feb 28, 2011 at 4:00 PM, Ralf Gommers wrote: > The other thing that needs to be done is some (more) documentation of > new features. Einsum and the new iterator seem to be well documented, > but not described in the release notes. Hmm, I take that back. Just saw that np.newiter and its methods don't have any docstrings at all. From pav at iki.fi Mon Feb 28 05:49:13 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 28 Feb 2011 10:49:13 +0000 (UTC) Subject: [Numpy-discussion] Numpy 2.0 schedule References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> Message-ID: Mon, 28 Feb 2011 16:50:59 +0800, Ralf Gommers wrote: > On Mon, Feb 28, 2011 at 4:00 PM, Ralf Gommers > wrote: > >> The other thing that needs to be done is some (more) documentation of >> new features. Einsum and the new iterator seem to be well documented, >> but not described in the release notes. > > Hmm, I take that back. Just saw that np.newiter and its methods don't > have any docstrings at all. The new iterator C-API should also be documented (if it is supposed to be public), which is not AFAIK done yet. From ijstokes at hkl.hms.harvard.edu Mon Feb 28 09:21:55 2011 From: ijstokes at hkl.hms.harvard.edu (Ian Stokes-Rees) Date: Mon, 28 Feb 2011 09:21:55 -0500 Subject: [Numpy-discussion] Grid indexing In-Reply-To: <4D68BBA0.4060009@gmail.com> References: <4D68BBA0.4060009@gmail.com> Message-ID: <4D6BAF83.1010100@hkl.hms.harvard.edu> On 2/26/11 3:36 AM, Wolfgang Kerzendorf wrote: > I have a n dimensional grid. The grids axes are linear but not > intergers. Let's say I want the value in gridcell [3.2,-5.6,0.01]. Is > there an easy way to transform the index? Do I have to write my own > class to accomplish this. Is there an easy way in numpy to accomplish > this. Can I give functions to transform the index somehow to the numpy > array? I can't answer this definitively, but this sounds like a tough one to find builtin -- the idea of array indexing is very tightly tied to integers starting from zero. I think you'll need to map from your grid coordinate system to an integer system (and vice-versa). If you can come up with a linear transformation for each dimension that map your floats to non-negative ints, that is probably pretty fast and easy for single cell lookups. Sparse arrays could be helpful if the data isn't "dense" on your grid. Ian From wkerzendorf at googlemail.com Mon Feb 28 09:34:16 2011 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Tue, 01 Mar 2011 01:34:16 +1100 Subject: [Numpy-discussion] Grid indexing In-Reply-To: <4D6BAF83.1010100@hkl.hms.harvard.edu> References: <4D68BBA0.4060009@gmail.com> <4D6BAF83.1010100@hkl.hms.harvard.edu> Message-ID: <4D6BB268.90508@gmail.com> I think it is possible to construct a class that takes as init function index converter functions and stores an array. That do for example linear mappings from the floats to the int. I am wondering if the numpy community would be interested in such a thing? I think the code would be very short, but might help a few people. Cheers W On 1/03/11 1:21 AM, Ian Stokes-Rees wrote: > > On 2/26/11 3:36 AM, Wolfgang Kerzendorf wrote: >> I have a n dimensional grid. The grids axes are linear but not >> intergers. Let's say I want the value in gridcell [3.2,-5.6,0.01]. Is >> there an easy way to transform the index? Do I have to write my own >> class to accomplish this. Is there an easy way in numpy to accomplish >> this. Can I give functions to transform the index somehow to the numpy >> array? > I can't answer this definitively, but this sounds like a tough one to > find builtin -- the idea of array indexing is very tightly tied to > integers starting from zero. I think you'll need to map from your grid > coordinate system to an integer system (and vice-versa). If you can > come up with a linear transformation for each dimension that map your > floats to non-negative ints, that is probably pretty fast and easy for > single cell lookups. Sparse arrays could be helpful if the data isn't > "dense" on your grid. > > Ian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From jsalvati at u.washington.edu Mon Feb 28 09:41:44 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Mon, 28 Feb 2011 06:41:44 -0800 Subject: [Numpy-discussion] Numpy 2.0 schedule In-Reply-To: References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> Message-ID: I just want to say that I am looking forward to np.newiter, and I am impressed at how quickly it's being released. On Mon, Feb 28, 2011 at 2:49 AM, Pauli Virtanen wrote: > Mon, 28 Feb 2011 16:50:59 +0800, Ralf Gommers wrote: > > On Mon, Feb 28, 2011 at 4:00 PM, Ralf Gommers > > wrote: > > > >> The other thing that needs to be done is some (more) documentation of > >> new features. Einsum and the new iterator seem to be well documented, > >> but not described in the release notes. > > > > Hmm, I take that back. Just saw that np.newiter and its methods don't > > have any docstrings at all. > > The new iterator C-API should also be documented (if it is supposed to be > public), which is not AFAIK done yet. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Mon Feb 28 09:36:05 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 28 Feb 2011 08:36:05 -0600 Subject: [Numpy-discussion] Numpy 2.0 schedule In-Reply-To: References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> Message-ID: <4D6BB2D5.6010809@gmail.com> On 02/28/2011 02:00 AM, Ralf Gommers wrote: > Hi, > > On Fri, Jan 28, 2011 at 7:15 AM, Travis Oliphant wrote: >> The reason for a NumPy 1.6 suggestion, is that Mark (and others it would >> seem) have additional work and features that do not need to wait for the >> NumPy 2.0 ABI design to finalize in order to get out there. >> If someone is willing to manage the release of NumPy 1.6, then it sounds >> like a great idea to me. > This thread ended without a conclusion a month ago. Now I think master > is in a better state than a month ago for a release (py 2.4/2.5/3.x > issues and segfault on OS X fixed, more testing of changes), and I > have a better idea of my free time for March/April. Basically, I have > a good amount of time for the next couple of weeks, and not so much at > the end of March / first half of April due to an inter-continental > move. But I think we can get out a beta by mid-March, and I can manage > the release. > > I've had a look at the bug tracker, here's a list of tickets for 1.6: > #1748 (blocker: regression for astype('str')) > #1619 (issue with dtypes, with patch) > #1749 (distutils, py 3.2) > #1601 (distutils, py 3.2) > #1622 (Solaris segfault, with patch) > #1713 (Solaris segfault) > #1631 (Solaris segfault) > > I can look at the distutils tickets. > > The other thing that needs to be done is some (more) documentation of > new features. Einsum and the new iterator seem to be well documented, > but not described in the release notes. Datetime has no docs as far as > I can see except for two similar NEPs. > > Proposed schedule: > March 15: beta 1 > March 28: rc 1 > April 17: rc 2 (if needed) > April 24: final release > > Let me know what you think. Bonus points for volunteering to fix some > of those tickets:) > > Cheers, > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Is this 1.6 or 2.0? The title is 2.0 but you talk about 1.6 so some tickets listed as 2.0 may apply to 1.6. It would be great to do some 'housekeeping' and try to address some of the old tickets dealt with before numpy 2.0. For example, I think ticket 225 (bincount does not accept input of type > N.uint16) has been addressed but it needs to be checked from windows and 32-bit systems. Bruce Created 2006: #38 strides accepted as an argument to records.array #57 ufunc methods need improved BUFFER loop #213 SharedLibrary builder for numpy.distutils #225 bincount does not accept input of type > N.uint16 #236 reduceat cornercase #237 reduceat should handle outlier indices gracefully #244 Build fails with Intel Visual Fortran compiler #260 Add mechanism for registering objects to be deallocated and memory-to-be freed at Python exit #274 Speed up N-D Boolean indexing #301 power with negative argument returns 0 #333 Creating an array from a n-dim dtype type fails #338 Valgrind warning when calling scipy.interpolate.interp1d #349 Improve unit tests in linalg #354 Possible inconsistency in 0-dim and scalar empty array types #398 Compatibility loader for old Numeric pickles #400 C API access to fft for C scipy extension ? #402 newaxis incompatible with array indexing Numpy 1.0 #450 Make a.min() not copy data #417 Numpy 1.0.1 compilation fails on IRIX 6.5 #527 fortran linking flag option... #1176 deepcopy turns ndarry into string_ #1143 Improve performance of PyUFunc_Reduce #931 Records containing zero-length items pickle just fine, but cannot be unpickled #803 Assignment problem on matrix advanced selection Numpy 1.1 #1266 Extremely long runtimes in numpy.fft.fft #963 Object array comparisons eat exceptions #929 empty_like and zeros_like behave differently from ones_like #934 Documentation error in site.cfg.example Numpy 1.2 #1374 Ticket 628 not fixed for Solaris (polyfit uses 100% CPU and does not stop) #1209 Docstring for numpy.numarray.random_array.multinomial is out of date. #1192 integer dot product #1172 abs does not work with -maxint #1163 Incorrect conversion to Int64 by loadtxt (traced to _getconv in numpy.lib.io) #1161 Errors and/or wrong result with reverse slicing in numpy.delete #1094 masked array autotest fails with bus error #1085 Surprising results from in-place operations involving views #1071 loadtxt fails if the last column contains empty value From ben.root at ou.edu Mon Feb 28 10:02:14 2011 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 28 Feb 2011 09:02:14 -0600 Subject: [Numpy-discussion] Numpy 2.0 schedule In-Reply-To: <4D6BB2D5.6010809@gmail.com> References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> <4D6BB2D5.6010809@gmail.com> Message-ID: On Mon, Feb 28, 2011 at 8:36 AM, Bruce Southey wrote: > On 02/28/2011 02:00 AM, Ralf Gommers wrote: > > Hi, > > > > On Fri, Jan 28, 2011 at 7:15 AM, Travis Oliphant > wrote: > >> The reason for a NumPy 1.6 suggestion, is that Mark (and others it would > >> seem) have additional work and features that do not need to wait for the > >> NumPy 2.0 ABI design to finalize in order to get out there. > >> If someone is willing to manage the release of NumPy 1.6, then it sounds > >> like a great idea to me. > > This thread ended without a conclusion a month ago. Now I think master > > is in a better state than a month ago for a release (py 2.4/2.5/3.x > > issues and segfault on OS X fixed, more testing of changes), and I > > have a better idea of my free time for March/April. Basically, I have > > a good amount of time for the next couple of weeks, and not so much at > > the end of March / first half of April due to an inter-continental > > move. But I think we can get out a beta by mid-March, and I can manage > > the release. > > > > I've had a look at the bug tracker, here's a list of tickets for 1.6: > > #1748 (blocker: regression for astype('str')) > > #1619 (issue with dtypes, with patch) > > #1749 (distutils, py 3.2) > > #1601 (distutils, py 3.2) > > #1622 (Solaris segfault, with patch) > > #1713 (Solaris segfault) > > #1631 (Solaris segfault) > > > > I can look at the distutils tickets. > > > > The other thing that needs to be done is some (more) documentation of > > new features. Einsum and the new iterator seem to be well documented, > > but not described in the release notes. Datetime has no docs as far as > > I can see except for two similar NEPs. > > > > Proposed schedule: > > March 15: beta 1 > > March 28: rc 1 > > April 17: rc 2 (if needed) > > April 24: final release > > > > Let me know what you think. Bonus points for volunteering to fix some > > of those tickets:) > > > > Cheers, > > Ralf > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Is this 1.6 or 2.0? > The title is 2.0 but you talk about 1.6 so some tickets listed as 2.0 > may apply to 1.6. > > It would be great to do some 'housekeeping' and try to address some of > the old tickets dealt with before numpy 2.0. For example, I think ticket > 225 (bincount does not accept input of type > N.uint16) has been > addressed but it needs to be checked from windows and 32-bit systems. > > Bruce > > Created 2006: > #38 strides accepted as an argument to records.array > #57 ufunc methods need improved BUFFER loop > #213 SharedLibrary builder for numpy.distutils > #225 bincount does not accept input of type > N.uint16 > #236 reduceat cornercase > #237 reduceat should handle outlier indices gracefully > #244 Build fails with Intel Visual Fortran compiler > #260 Add mechanism for registering objects to be deallocated and > memory-to-be freed at Python exit > #274 Speed up N-D Boolean indexing > #301 power with negative argument returns 0 > #333 Creating an array from a n-dim dtype type fails > #338 Valgrind warning when calling scipy.interpolate.interp1d > #349 Improve unit tests in linalg > #354 Possible inconsistency in 0-dim and scalar empty array types > #398 Compatibility loader for old Numeric pickles > #400 C API access to fft for C scipy extension ? > #402 newaxis incompatible with array indexing > > > Numpy 1.0 > #450 Make a.min() not copy data > #417 Numpy 1.0.1 compilation fails on IRIX 6.5 > #527 fortran linking flag option... > #1176 deepcopy turns ndarry into string_ > #1143 Improve performance of PyUFunc_Reduce > #931 Records containing zero-length items pickle just fine, but > cannot be unpickled > #803 Assignment problem on matrix advanced selection > > Numpy 1.1 > #1266 Extremely long runtimes in numpy.fft.fft > #963 Object array comparisons eat exceptions > #929 empty_like and zeros_like behave differently from ones_like > #934 Documentation error in site.cfg.example > > Numpy 1.2 > #1374 Ticket 628 not fixed for Solaris (polyfit uses 100% CPU and > does not stop) > #1209 Docstring for numpy.numarray.random_array.multinomial is out of > date. > #1192 integer dot product > #1172 abs does not work with -maxint > #1163 Incorrect conversion to Int64 by loadtxt (traced to _getconv in > numpy.lib.io) > #1161 Errors and/or wrong result with reverse slicing in numpy.delete > #1094 masked array autotest fails with bus error > #1085 Surprising results from in-place operations involving views > #1071 loadtxt fails if the last column contains empty value > > > So, is there still no hope in addressing this old bug report of mine? http://projects.scipy.org/numpy/ticket/1562 Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Mon Feb 28 10:25:52 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 28 Feb 2011 09:25:52 -0600 Subject: [Numpy-discussion] Numpy 2.0 schedule In-Reply-To: References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> <4D6BB2D5.6010809@gmail.com> Message-ID: <4D6BBE80.608@gmail.com> On 02/28/2011 09:02 AM, Benjamin Root wrote: [snip] > > > So, is there still no hope in addressing this old bug report of mine? > > http://projects.scipy.org/numpy/ticket/1562 > > Ben Root > I think you need to add more details to this. So do you have an example of the problem that includes code and expected output? Perhaps genfromtxt is probably more appropriate than loadtxt for what you want: from StringIO import StringIO import numpy as np t = StringIO("1,1.3,abcde\n2,2.3,wxyz\n1\n3,3.3,mnop") data = np.genfromtxt(t, [('myint','i8'),('myfloat','f8'),('mystring','S5')], names = ['myint','myfloat','mystring'], delimiter=",", invalid_raise=False) print 'Bad data raise\n',data This gives the output that skips the incomplete 3rd line: /usr/lib64/python2.7/site-packages/numpy/lib/npyio.py:1507: ConversionWarning: Some errors were detected ! Line #3 (got 1 columns instead of 3) warnings.warn(errmsg, ConversionWarning) Bad data raise [(1, 1.3, 'abcde') (2, 2.3, 'wxyz') (3, 3.3, 'mnop')] Bruce From ralf.gommers at googlemail.com Mon Feb 28 10:31:59 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 28 Feb 2011 23:31:59 +0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) Message-ID: On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey wrote: > On 02/28/2011 02:00 AM, Ralf Gommers wrote: >> Hi, >> >> On Fri, Jan 28, 2011 at 7:15 AM, Travis Oliphant ?wrote: >>> The reason for a NumPy 1.6 suggestion, is that Mark (and others it would >>> seem) have additional work and features that do not need to wait for the >>> NumPy 2.0 ABI design to finalize in order to get out there. >>> If someone is willing to manage the release of NumPy 1.6, then it sounds >>> like a great idea to me. >> This thread ended without a conclusion a month ago. Now I think master >> is in a better state than a month ago for a release (py 2.4/2.5/3.x >> issues and segfault on OS X fixed, more testing of changes), and I >> have a better idea of my free time for March/April. Basically, I have >> a good amount of time for the next couple of weeks, and not so much at >> the end of March / first half of April due to an inter-continental >> move. But I think we can get out a beta by mid-March, and I can manage >> the release. >> >> I've had a look at the bug tracker, here's a list of tickets for 1.6: >> #1748 (blocker: regression for astype('str')) >> #1619 (issue with dtypes, with patch) >> #1749 (distutils, py 3.2) >> #1601 (distutils, py 3.2) >> #1622 (Solaris segfault, with patch) >> #1713 (Solaris segfault) >> #1631 (Solaris segfault) >> >> I can look at the distutils tickets. >> >> The other thing that needs to be done is some (more) documentation of >> new features. Einsum and the new iterator seem to be well documented, >> but not described in the release notes. Datetime has no docs as far as >> I can see except for two similar NEPs. >> >> Proposed schedule: >> March 15: beta 1 >> March 28: rc 1 >> April 17: rc 2 (if needed) >> April 24: final release >> >> Let me know what you think. Bonus points for volunteering to fix some >> of those tickets:) >> >> Cheers, >> Ralf >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > Is this 1.6 or 2.0? > The title is 2.0 but you talk about 1.6 so some tickets listed as 2.0 > may apply to 1.6. 1.6, and renamed the thread to make it clearer. > It would be great to do some 'housekeeping' and try to address some of > the old tickets dealt with before numpy 2.0. For example, I think ticket > 225 (bincount does not accept input of type > N.uint16) has been > addressed but it needs to be checked from windows and 32-bit systems. That would be useful of course, it just requires someone with spare time and the knowledge and motivation to close those tickets. I listed only the ones that I think are essential to fix before a release can be made. Ralf > Created 2006: > #38 ? ?strides accepted as an argument to records.array > #57 ? ?ufunc methods need improved BUFFER loop > #213 ? ?SharedLibrary builder for numpy.distutils > #225 ? ?bincount does not accept input of type > N.uint16 > #236 ? ?reduceat cornercase > #237 ? ?reduceat should handle outlier indices gracefully > #244 ? ?Build fails with Intel Visual Fortran compiler > #260 ? ?Add mechanism for registering objects to be deallocated and > memory-to-be freed at Python exit > #274 ? ?Speed up N-D Boolean indexing > #301 ? ?power with negative argument returns 0 > #333 ? ?Creating an array from a n-dim dtype type fails > #338 ? ?Valgrind warning when calling scipy.interpolate.interp1d > #349 ? ?Improve unit tests in linalg > #354 ? ?Possible inconsistency in 0-dim and scalar empty array types > #398 ? ?Compatibility loader for old Numeric pickles > #400 ? ?C API access to fft for C scipy extension ? > #402 ? ?newaxis incompatible with array indexing > > > Numpy 1.0 > #450 ? ?Make a.min() not copy data > #417 ? ?Numpy 1.0.1 compilation fails on IRIX 6.5 > #527 ? ?fortran linking flag option... > #1176 ? ?deepcopy turns ndarry into string_ > #1143 ? ?Improve performance of PyUFunc_Reduce > #931 ? ?Records containing zero-length items pickle just fine, but > cannot be unpickled > #803 ? ?Assignment problem on matrix advanced selection > > Numpy 1.1 > #1266 ? ?Extremely long runtimes in numpy.fft.fft > #963 ? ?Object array comparisons eat exceptions > #929 ? ?empty_like and zeros_like behave differently from ones_like > #934 ? ?Documentation error in site.cfg.example > > Numpy 1.2 > #1374 ? ?Ticket 628 not fixed for Solaris (polyfit uses 100% CPU and > does not stop) > #1209 ? ?Docstring for numpy.numarray.random_array.multinomial is out of > date. > #1192 ? ?integer dot product > #1172 ? ?abs does not work with -maxint > #1163 ? ?Incorrect conversion to Int64 by loadtxt (traced to _getconv in > numpy.lib.io) > #1161 ? ?Errors and/or wrong result with reverse slicing in numpy.delete > #1094 ? ?masked array autotest fails with bus error > #1085 ? ?Surprising results from in-place operations involving views > #1071 ? ?loadtxt fails if the last column contains empty value > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ben.root at ou.edu Mon Feb 28 10:47:54 2011 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 28 Feb 2011 09:47:54 -0600 Subject: [Numpy-discussion] Numpy 2.0 schedule In-Reply-To: <4D6BBE80.608@gmail.com> References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> <4D6BB2D5.6010809@gmail.com> <4D6BBE80.608@gmail.com> Message-ID: On Mon, Feb 28, 2011 at 9:25 AM, Bruce Southey wrote: > On 02/28/2011 09:02 AM, Benjamin Root wrote: > [snip] > > > > > > So, is there still no hope in addressing this old bug report of mine? > > > > http://projects.scipy.org/numpy/ticket/1562 > > > > Ben Root > > > I think you need to add more details to this. So do you have an example > of the problem that includes code and expected output? > > Perhaps genfromtxt is probably more appropriate than loadtxt for what > you want: > > from StringIO import StringIO > import numpy as np > t = StringIO("1,1.3,abcde\n2,2.3,wxyz\n1\n3,3.3,mnop") > data = np.genfromtxt(t, > [('myint','i8'),('myfloat','f8'),('mystring','S5')], names = > ['myint','myfloat','mystring'], delimiter=",", invalid_raise=False) > print 'Bad data raise\n',data > > This gives the output that skips the incomplete 3rd line: > > /usr/lib64/python2.7/site-packages/numpy/lib/npyio.py:1507: > ConversionWarning: Some errors were detected ! > Line #3 (got 1 columns instead of 3) > warnings.warn(errmsg, ConversionWarning) > Bad data raise > [(1, 1.3, 'abcde') (2, 2.3, 'wxyz') (3, 3.3, 'mnop')] > > > Bruce > > Bruce, I think you mis-understood the problem I was reporting. You can find the discussion thread here: http://www.mail-archive.com/numpy-discussion at scipy.org/msg26235.html I have proposed that at the very least, an example of this problem is added to the documentation of loadtxt so that users know to be aware of this possibility. In addition, loadtxt fails on empty files even when provided with a dtype. I believe genfromtxt also fails as well in this case. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Mon Feb 28 10:51:15 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 28 Feb 2011 16:51:15 +0100 Subject: [Numpy-discussion] Small typo in fromregex Message-ID: Hi, I'm sorry I didn't file a bug, I have some troubles getting my old trac account back :| In lib/npyio.py, there is a mistake line 1029. Instead on fh.close(), it should have been file.close(). If fromregex opens the file, it will crash because the name of the file is not correct. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Mon Feb 28 12:45:26 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 28 Feb 2011 11:45:26 -0600 Subject: [Numpy-discussion] ticket 1562 on loadtxt (was Re: Numpy 2.0 schedule) In-Reply-To: References: <0A6387EF-1227-46C0-BB60-C5AA7A024D68@enthought.com> <4D6BB2D5.6010809@gmail.com> <4D6BBE80.608@gmail.com> Message-ID: <4D6BDF36.3080903@gmail.com> On 02/28/2011 09:47 AM, Benjamin Root wrote: > On Mon, Feb 28, 2011 at 9:25 AM, Bruce Southey > wrote: > > On 02/28/2011 09:02 AM, Benjamin Root wrote: > [snip] > > > > > > So, is there still no hope in addressing this old bug report of > mine? > > > > http://projects.scipy.org/numpy/ticket/1562 > > > > Ben Root > > > I think you need to add more details to this. So do you have an > example > of the problem that includes code and expected output? > > Perhaps genfromtxt is probably more appropriate than loadtxt for what > you want: > > from StringIO import StringIO > import numpy as np > t = StringIO("1,1.3,abcde\n2,2.3,wxyz\n1\n3,3.3,mnop") > data = np.genfromtxt(t, > [('myint','i8'),('myfloat','f8'),('mystring','S5')], names = > ['myint','myfloat','mystring'], delimiter=",", invalid_raise=False) > print 'Bad data raise\n',data > > This gives the output that skips the incomplete 3rd line: > > /usr/lib64/python2.7/site-packages/numpy/lib/npyio.py:1507: > ConversionWarning: Some errors were detected ! > Line #3 (got 1 columns instead of 3) > warnings.warn(errmsg, ConversionWarning) > Bad data raise > [(1, 1.3, 'abcde') (2, 2.3, 'wxyz') (3, 3.3, 'mnop')] > > > Bruce > > > Bruce, > > I think you mis-understood the problem I was reporting. Probably - which is why I asked for more details. > You can find the discussion thread here: > > http://www.mail-archive.com/numpy-discussion at scipy.org/msg26235.html > > I have proposed that at the very least, an example of this problem is > added to the documentation of loadtxt so that users know to be aware > of this possibility. I did not connect the ticket to that email thread. Removing the structured array part of your email, I think essentially the argument is which should be the output of: np.loadtxt(StringIO("89.23")) np.arange(5)[1] These return an 0-d array and an rather old argument about that (which may address the other part of the ticket). Really I see this behavior as standard so you add an example to the documentation to reflect that. > In addition, loadtxt fails on empty files even when provided with a > dtype. I believe genfromtxt also fails as well in this case. > > Ben Root > Errors on empty files probably should be a new bug report as that was not in the ticket. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From wesmckinn at gmail.com Mon Feb 28 17:52:14 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 28 Feb 2011 17:52:14 -0500 Subject: [Numpy-discussion] {zeros, empty, ...}_like functions behavior with subclasses Message-ID: I'm having some trouble with the zeros_like function via np.fix: def zeros_like(a): if isinstance(a, ndarray): res = ndarray.__new__(type(a), a.shape, a.dtype, order=a.flags.fnc) res.fill(0) return res try: wrap = a.__array_wrap__ except AttributeError: wrap = None a = asarray(a) res = zeros(a.shape, a.dtype) if wrap: res = wrap(res) return res As you can see this is going to discard any metadata stored in a subtype. I'm not sure whether this is a bug or a feature but wanted to bring it up. Thanks, Wes From jonovik at gmail.com Mon Feb 28 18:35:22 2011 From: jonovik at gmail.com (Jon Olav Vik) Date: Mon, 28 Feb 2011 23:35:22 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Adding_=60offset=60_argument_to_np?= =?utf-8?q?=2Elib=2Eformat=2Eopen=5Fmemmap_and_np=2Eload?= References: Message-ID: Ralf Gommers googlemail.com> writes: > My first question after looking at this is why we would want three > very similar ways to load memory-mapped arrays (np.memmap, np.load, > np.lib.format.open_memmap)? They already exist but your changes make > those three even more similar. If I understand correctly, np.memmap requires you to specify the dtype. It cannot of itself memory-map a file that has been saved with e.g. np.save. A file with (structured, in my case) dtype information is much more self- documenting than a plain binary file with no dtype. Functions for further processing of the data need then only read the file to know how to interpret it. > I'd think we want one simple version (load) and a full-featured one. > So imho changing open_memmap but leaving np.load as-is would be the > way to go. np.load calls open_memmap if mmap_mode is specified. The only change I made to np.load was add offset and shape parameters that are passed through to open_memmap. (A shape argument is required for offset to be really useful, at least for my use case of multiple processes memory-mapping their own portion of a file.) > > Note that the offset is in terms of array elements, not bytes (which is what > > np.memmap uses), because that was what I had use for. > > This should be kept in bytes like in np.memmap I think, it's just > confusing if those two functions differ like that. I agree that there is room for confusion, but it is quite inconvenient having to access the file once for the dtype, then compute the offset based on the item size of the dtype, then access the file again for the real memory-mapping. The single most common scenario for me is "I am process n out of N, and I will memory-map my fair share of this file". Given n and N, I can compute the offset and shape in terms of array elements, but converting it to bytes means a couple extra lines of code every time I do it. (If anything, I'd prefer the offset argument for np.memmap to be in elements too, in accordance with how indexing and striding works with Numpy arrays.) > Another thing to change: you should not use an assert statement, use > "if ....: raise ...." instead. Will do if this gets support. Thanks for the feedback 8-) > > My use case was to preallocate a big record array on disk, then start many > > processes writing to their separate, memory-mapped segments of the file. The > > end result was one big array on disk, with the correct shape and data type > > information. Using a record array makes the data structure more self- > > documenting. Using open_memmap with mode="w+" is the fastest way I've found to > > preallocate an array on disk; it does not create the huge array in memory. > > Letting multiple processes memory-map and read/write to non-overlapping > > portions without interfering with each other allows for fast, simple parallel I/ > > O. > > > > I've used this extensively on Numpy 1.4.0, but based my Git checkout on the > > current Numpy trunk. There have been some rearrangements in np.load since then > > (it used to be in np.lib.io and is now in np.lib.npyio), but as far as I can > > see, my modifications carry over fine. I haven't had a chance to test with > > Numpy trunk, though. (What is the best way to set up a test version without > > affecting my working 1.4.0 setup?) > > You can use an in-place build > (http://projects.scipy.org/numpy/wiki/DevelopmentTips) and add that > dir to your PYTHONPATH. Very helpful. Thanks again! Regards, Jon Olav From amcmorl at gmail.com Mon Feb 28 18:50:24 2011 From: amcmorl at gmail.com (Angus McMorland) Date: Mon, 28 Feb 2011 18:50:24 -0500 Subject: [Numpy-discussion] permute and sum? Message-ID: Hi all, I want to create a transpose of a vector, such that if the same index is given in the 'axes' list (as per np.transpose), then the sum of the original values sharing the same index is placed in the corresponding output index. For example: In: data = np.array([5, 7, 9, 6, 2]) In: order = np.array([0, 2, 0, 3, 1]) In: permute_and_sum(data, order) Out: array([14, 2, 7, 6]) I can obviously do this using the following, def permute_and_sum(data, order): result = np.zeros(np.max(order) + 1) for val, dest in zip(data, order): result[dest] += val return result but it seems like there's bound to be a much more elegant method I'm not currently seeing. Can anyone point me in the right direction? Thanks, Angus. -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From bsouthey at gmail.com Mon Feb 28 19:05:06 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 28 Feb 2011 18:05:06 -0600 Subject: [Numpy-discussion] {zeros, empty, ...}_like functions behavior with subclasses In-Reply-To: References: Message-ID: On Mon, Feb 28, 2011 at 4:52 PM, Wes McKinney wrote: > I'm having some trouble with the zeros_like function via np.fix: > > def zeros_like(a): > ? ?if isinstance(a, ndarray): > ? ? ? ?res = ndarray.__new__(type(a), a.shape, a.dtype, order=a.flags.fnc) > ? ? ? ?res.fill(0) > ? ? ? ?return res > ? ?try: > ? ? ? ?wrap = a.__array_wrap__ > ? ?except AttributeError: > ? ? ? ?wrap = None > ? ?a = asarray(a) > ? ?res = zeros(a.shape, a.dtype) > ? ?if wrap: > ? ? ? ?res = wrap(res) > ? ?return res > > As you can see this is going to discard any metadata stored in a > subtype. I'm not sure whether this is a bug or a feature but wanted to > bring it up. > > Thanks, > Wes > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I guess this is ticket 929. http://projects.scipy.org/numpy/ticket/929 I was looking at it today but was not sure what is really desired here. I considered that this just meant shape and dtype but not sure about masked or record arrays behavior. So: What is the value of having the metadata? What is the meaning of 'like' here? Bruce From robert.kern at gmail.com Mon Feb 28 19:15:42 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 28 Feb 2011 18:15:42 -0600 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: References: Message-ID: On Thu, Feb 24, 2011 at 09:49, Jon Olav Vik wrote: > My use case was to preallocate a big record array on disk, then start many > processes writing to their separate, memory-mapped segments of the file. The > end result was one big array on disk, with the correct shape and data type > information. Using a record array makes the data structure more self- > documenting. Using open_memmap with mode="w+" is the fastest way I've found to > preallocate an array on disk; it does not create the huge array in memory. > Letting multiple processes memory-map and read/write to non-overlapping > portions without interfering with each other allows for fast, simple parallel I/ > O. You can have each of those processes memory-map the whole file and just operate on their own slices. Your operating system's virtual memory manager should handle all of the details for you. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pgmdevlist at gmail.com Mon Feb 28 19:24:07 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 1 Mar 2011 01:24:07 +0100 Subject: [Numpy-discussion] {zeros, empty, ...}_like functions behavior with subclasses In-Reply-To: References: Message-ID: <28A39E36-034E-4A62-AB24-3D545769C7F6@gmail.com> On Mar 1, 2011, at 1:05 AM, Bruce Southey wrote: > On Mon, Feb 28, 2011 at 4:52 PM, Wes McKinney wrote: >> I'm having some trouble with the zeros_like function via np.fix: >> >> def zeros_like(a): >> if isinstance(a, ndarray): >> res = ndarray.__new__(type(a), a.shape, a.dtype, order=a.flags.fnc) >> res.fill(0) >> return res >> try: >> wrap = a.__array_wrap__ >> except AttributeError: >> wrap = None >> a = asarray(a) >> res = zeros(a.shape, a.dtype) >> if wrap: >> res = wrap(res) >> return res >> >> As you can see this is going to discard any metadata stored in a >> subtype. I'm not sure whether this is a bug or a feature but wanted to >> bring it up. >> >> Thanks, >> Wes >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > I guess this is ticket 929. > http://projects.scipy.org/numpy/ticket/929 > > I was looking at it today but was not sure what is really desired > here. I considered that this just meant shape and dtype but not sure > about masked or record arrays behavior. So: > What is the value of having the metadata? > What is the meaning of 'like' here? Well, that depends on what you wanna do, of course. To handle metadata, I use some kind of dictionary updated in the __array_finalize__. Check numpy.ma.MaskedArray and its subclasses (like scikits.timeseries.TimeSeries) for the details. Now that you could store some extra data in the dtype (if I remmbr and understand correctly), it might be worth considering a proper way to deal with that. From sturla at molden.no Mon Feb 28 19:50:00 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 01 Mar 2011 01:50:00 +0100 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: References: Message-ID: <4D6C42B8.7040902@molden.no> Den 01.03.2011 01:15, skrev Robert Kern: > You can have each of those processes memory-map the whole file and > just operate on their own slices. Your operating system's virtual > memory manager should handle all of the details for you. Mapping large files from the start will not always work on 32-bit systems. That is why mmap.mmap take an offset argument now (Python 2.7 and 3.1.) Making a view np.memmap with slices is useful on 64-bit but not 32-bit systems. Sturla From josef.pktd at gmail.com Mon Feb 28 19:50:53 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 28 Feb 2011 19:50:53 -0500 Subject: [Numpy-discussion] permute and sum? In-Reply-To: References: Message-ID: On Mon, Feb 28, 2011 at 6:50 PM, Angus McMorland wrote: > Hi all, > > I want to create a transpose of a vector, such that if the same index > is given in the 'axes' list (as per np.transpose), then the sum of the > original values sharing the same index is placed in the corresponding > output index. > > For example: > > In: data ?= np.array([5, 7, 9, 6, 2]) > In: order = np.array([0, 2, 0, 3, 1]) > In: permute_and_sum(data, order) > Out: array([14, 2, 7, 6]) Not sure I fully understand, but isn't this np.bincount(order, weights=data) Josef > I can obviously do this using the following, > > def permute_and_sum(data, order): > ? result = np.zeros(np.max(order) + 1) > ? for val, dest in zip(data, order): > ? ? ? result[dest] += val > ? return result > > but it seems like there's bound to be a much more elegant method I'm > not currently seeing. Can anyone point me in the right direction? > > Thanks, > > Angus. > -- > AJC McMorland > Post-doctoral research fellow > Neurobiology, University of Pittsburgh > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sturla at molden.no Mon Feb 28 19:58:08 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 01 Mar 2011 01:58:08 +0100 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: <4D6C42B8.7040902@molden.no> References: <4D6C42B8.7040902@molden.no> Message-ID: <4D6C44A0.3000406@molden.no> Den 01.03.2011 01:50, skrev Sturla Molden: > Mapping large files from the start will not always work on 32-bit > systems. That is why mmap.mmap take an offset argument now (Python 2.7 > and 3.1.) > Also, numpy.memmap is stricty speaking not needed. One can just use mmap.mmap and pass it to np.frombuffer. Sturla From robert.kern at gmail.com Mon Feb 28 20:10:17 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 28 Feb 2011 19:10:17 -0600 Subject: [Numpy-discussion] Adding `offset` argument to np.lib.format.open_memmap and np.load In-Reply-To: <4D6C42B8.7040902@molden.no> References: <4D6C42B8.7040902@molden.no> Message-ID: On Mon, Feb 28, 2011 at 18:50, Sturla Molden wrote: > Den 01.03.2011 01:15, skrev Robert Kern: >> You can have each of those processes memory-map the whole file and >> just operate on their own slices. Your operating system's virtual >> memory manager should handle all of the details for you. > > Mapping large files from the start will not always work on 32-bit > systems. That is why mmap.mmap take an offset argument now (Python 2.7 > and 3.1.) > > Making a view np.memmap with slices is useful on 64-bit but not 32-bit > systems. I'm talking about the OP's stated use case where he generates the file via memory-mapping the whole thing on the same machine. The whole file does fit into the address space in his use case. I'd like to see a real use case where this does not hold. I suspect that this is not the API we would want for such use cases. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Mon Feb 28 21:26:39 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 1 Mar 2011 10:26:39 +0800 Subject: [Numpy-discussion] Numpy 1.6 schedule (was: Numpy 2.0 schedule) In-Reply-To: References: Message-ID: On Mon, Feb 28, 2011 at 11:31 PM, Ralf Gommers wrote: > On Mon, Feb 28, 2011 at 10:36 PM, Bruce Southey wrote: >> It would be great to do some 'housekeeping' and try to address some of >> the old tickets dealt with before numpy 2.0. For example, I think ticket >> 225 (bincount does not accept input of type > N.uint16) has been >> addressed but it needs to be checked from windows and 32-bit systems. I checked #225, that's still not working. But yes, quite a few of the below tickets can be closed, some that are not applicable anymore, enhancement requests that obviously won't be implemented, numeric/numarray related ones (do we still actively support those modules?). Ralf >> Created 2006: >> #38 ? ?strides accepted as an argument to records.array >> #57 ? ?ufunc methods need improved BUFFER loop >> #213 ? ?SharedLibrary builder for numpy.distutils >> #225 ? ?bincount does not accept input of type > N.uint16 >> #236 ? ?reduceat cornercase >> #237 ? ?reduceat should handle outlier indices gracefully >> #244 ? ?Build fails with Intel Visual Fortran compiler >> #260 ? ?Add mechanism for registering objects to be deallocated and >> memory-to-be freed at Python exit >> #274 ? ?Speed up N-D Boolean indexing >> #301 ? ?power with negative argument returns 0 >> #333 ? ?Creating an array from a n-dim dtype type fails >> #338 ? ?Valgrind warning when calling scipy.interpolate.interp1d >> #349 ? ?Improve unit tests in linalg >> #354 ? ?Possible inconsistency in 0-dim and scalar empty array types >> #398 ? ?Compatibility loader for old Numeric pickles >> #400 ? ?C API access to fft for C scipy extension ? >> #402 ? ?newaxis incompatible with array indexing >> >> >> Numpy 1.0 >> #450 ? ?Make a.min() not copy data >> #417 ? ?Numpy 1.0.1 compilation fails on IRIX 6.5 >> #527 ? ?fortran linking flag option... >> #1176 ? ?deepcopy turns ndarry into string_ >> #1143 ? ?Improve performance of PyUFunc_Reduce >> #931 ? ?Records containing zero-length items pickle just fine, but >> cannot be unpickled >> #803 ? ?Assignment problem on matrix advanced selection >> >> Numpy 1.1 >> #1266 ? ?Extremely long runtimes in numpy.fft.fft >> #963 ? ?Object array comparisons eat exceptions >> #929 ? ?empty_like and zeros_like behave differently from ones_like >> #934 ? ?Documentation error in site.cfg.example >> >> Numpy 1.2 >> #1374 ? ?Ticket 628 not fixed for Solaris (polyfit uses 100% CPU and >> does not stop) >> #1209 ? ?Docstring for numpy.numarray.random_array.multinomial is out of >> date. >> #1192 ? ?integer dot product >> #1172 ? ?abs does not work with -maxint >> #1163 ? ?Incorrect conversion to Int64 by loadtxt (traced to _getconv in >> numpy.lib.io) >> #1161 ? ?Errors and/or wrong result with reverse slicing in numpy.delete >> #1094 ? ?masked array autotest fails with bus error >> #1085 ? ?Surprising results from in-place operations involving views >> #1071 ? ?loadtxt fails if the last column contains empty value >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From ralf.gommers at googlemail.com Mon Feb 28 21:37:35 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 1 Mar 2011 10:37:35 +0800 Subject: [Numpy-discussion] Small typo in fromregex In-Reply-To: References: Message-ID: On Mon, Feb 28, 2011 at 11:51 PM, Matthieu Brucher wrote: > Hi, > I'm sorry I didn't file a bug, I have some troubles getting my old trac > account back :| > In lib/npyio.py, there is a mistake line 1029. > Instead on fh.close(), it should have been file.close(). If fromregex opens > the file, it will crash because the name of the file is not correct. Fixed in 4eebdf9, thanks. Ralf From wesmckinn at gmail.com Mon Feb 28 22:52:41 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 28 Feb 2011 22:52:41 -0500 Subject: [Numpy-discussion] {zeros, empty, ...}_like functions behavior with subclasses In-Reply-To: <28A39E36-034E-4A62-AB24-3D545769C7F6@gmail.com> References: <28A39E36-034E-4A62-AB24-3D545769C7F6@gmail.com> Message-ID: On Mon, Feb 28, 2011 at 7:24 PM, Pierre GM wrote: > > On Mar 1, 2011, at 1:05 AM, Bruce Southey wrote: > >> On Mon, Feb 28, 2011 at 4:52 PM, Wes McKinney wrote: >>> I'm having some trouble with the zeros_like function via np.fix: >>> >>> def zeros_like(a): >>> ? ?if isinstance(a, ndarray): >>> ? ? ? ?res = ndarray.__new__(type(a), a.shape, a.dtype, order=a.flags.fnc) >>> ? ? ? ?res.fill(0) >>> ? ? ? ?return res >>> ? ?try: >>> ? ? ? ?wrap = a.__array_wrap__ >>> ? ?except AttributeError: >>> ? ? ? ?wrap = None >>> ? ?a = asarray(a) >>> ? ?res = zeros(a.shape, a.dtype) >>> ? ?if wrap: >>> ? ? ? ?res = wrap(res) >>> ? ?return res >>> >>> As you can see this is going to discard any metadata stored in a >>> subtype. I'm not sure whether this is a bug or a feature but wanted to >>> bring it up. >>> >>> Thanks, >>> Wes >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> I guess this is ticket 929. >> http://projects.scipy.org/numpy/ticket/929 >> >> I was looking at it today but was not sure what is really desired >> here. I considered that this just meant shape and dtype but not sure >> about masked or record arrays behavior. So: >> What is the value of having the metadata? >> What is the meaning of 'like' here? > > Well, that depends on what you wanna do, of course. To handle metadata, I use some kind of dictionary updated in the __array_finalize__. Check numpy.ma.MaskedArray and its subclasses (like scikits.timeseries.TimeSeries) for the details. > Now that you could store some extra data in the dtype (if I remmbr and understand correctly), it might be worth considering a proper way to deal with that. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > The ticket is exactly related to the problem at hand-- having __array_finalize__ defined won't help you as it never gets called.