From markflorisson88 at gmail.com Tue Nov 1 05:48:46 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 1 Nov 2011 09:48:46 +0000 Subject: [Numpy-discussion] NumPy nogil API In-Reply-To: References: <4EAE800D.3030006@astro.uio.no> <4EAE83D5.5020604@astro.uio.no> Message-ID: 2011/10/31 St?fan van der Walt : > On Mon, Oct 31, 2011 at 11:28 AM, Zachary Pincus > wrote: >>> As an example, it'd be nice to have scipy.ndimage available without the GIL: >>> http://docs.scipy.org/doc/scipy/reference/ndimage.html >>> >>> Now, this *can* easily be done as the core is written in C++. I'm just >>> pointing out that some people may wish more for calling scipy.ndimage >>> inside their prange than for some parts of NumPy. >> >> Not exactly to your larger point wrt the GIL, but I *think* some skimage (n?e scikits.image) folks are trying to rewrite most of ndimage's functionality in cython. I don't know what the status of this effort is though... > > We still rely on scipy.ndimage in some places, but since we don't have > to support N-dimensional arrays, we can often do things in a slightly > simpler and faster way. ?Almost all the performance code in the scikit > is written in Cython, which makes it trivial to release the GIL on > internal loops. That's good to hear. However, as was the point of my inquiry, it would be a lot nicer if it didn't release the GIL itself and require the GIL to be called, but if it wouldn't require the GIL to be called in the first place. Such an API (especially if written in Cython), can be easily exposed to both C and Cython. You then want a wrapper function that is callable from Python (i.e., requires the GIL), which decides whether or not it should release the GIL, and which calls the nogil equivalent. In the simplest of cases, one may use a cpdef nogil function, but I expect that often you'd have to use cdef nogil with a def wrapper. It does mean that the nogil API part would not take any kind of objects, but only C structures and such (which could be encapsulated by the respective objects). Of course, it may not always be possible, feasible or useful to have such an API, but when it comes to arrays and images it probably will be. > I am actively soliciting feedback from current or prospective users, > so that we can drive the scikit in the right direction. ?Already, it's > an entirely different project from what is was a couple of months ago. > ?We stopped trying to duplicate the MATLAB toolbox functionality, and > have been adding some exciting new algorithms. ?The number of pull > requests have tripled since the 0.3 release, and we're aiming to have > 0.4 done this week. > > Regards > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From morph at debian.org Tue Nov 1 06:36:01 2011 From: morph at debian.org (Sandro Tosi) Date: Tue, 1 Nov 2011 11:36:01 +0100 Subject: [Numpy-discussion] Reason behind C_ABI_VERSION so high In-Reply-To: References: Message-ID: Hi St?fan, thanks for your reply. 2011/11/1 St?fan van der Walt : >?But nowadays the ABI number is simply bumped > by one after every change, so for a good couple of million releases > you should be safe ignoring the "1" perfect, that's good for us > (but is the value really large enough to cause problems?). well, we plan to have that number into a package name, so having python-numpy-abi01000009 is kinda ugly (not that python-numpy-abi9 is that better , but at least is shorter). Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From Chris.Barker at noaa.gov Tue Nov 1 11:39:54 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Tue, 01 Nov 2011 08:39:54 -0700 Subject: [Numpy-discussion] float64 / int comparison different from float / int comparison In-Reply-To: References: Message-ID: <4EB012CA.5010209@noaa.gov> On 10/31/11 6:38 PM, St?fan van der Walt wrote: > On Mon, Oct 31, 2011 at 6:25 PM, Matthew Brett wrote: >> Oh, dear, I'm suffering now: >> In [12]: res> 2**31-1 >> Out[12]: array([False], dtype=bool) > I'm seeing: ... > Your result seems very strange, because the numpy scalars should > perform exactly the same inside and outside an array. I get what St?fan gets: In [32]: res = np.array((2**31,), dtype=np.float32) In [33]: res > 2**31-1 Out[33]: array([ True], dtype=bool) In [34]: res[0] > 2**31-1 Out[34]: True In [35]: res[0].dtype Out[35]: dtype('float32') In [36]: np.__version__ Out[36]: '1.6.1' (OS-X, Intel, Python2.7) Something is very odd with your build! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From e.piasini at ucl.ac.uk Tue Nov 1 12:03:11 2011 From: e.piasini at ucl.ac.uk (Eugenio Piasini) Date: Tue, 1 Nov 2011 16:03:11 +0000 Subject: [Numpy-discussion] einsum performance with many operands Message-ID: <4EB0183F.1080803@ucl.ac.uk> Hi, I was playing around with einsum (which is really cool, by the way!), and I noticed something strange (or unexpected at least) performance-wise. Here is an example: Let x and y be two NxM arrays, and W be a NxN array. I want to compute f = x^{ik} W_i^j y_{jk} (I hope the notation is clear enough) What surprises me is that it's much faster to compute the two products separately - as in ((xW)y) or (x(Wy)), rather than directly passing the three-operands expression to einsum. I did a quick benchmark, with the following script #======================================================== import numpy as np def f1(x,W,y): return np.einsum('ik,ij,jk', x,W,y) def f2(x,W,y): return np.einsum('jk,jk', np.einsum('ik,ij->jk', x, W), y) n = 30 m = 150 x=np.random.random(size=(n, m)) y=np.random.random(size=(n, m)) W=np.random.random(size=(n, n)) setup = """\ from __main__ import f1,f2,x,y,W """ if __name__ == '__main__': import timeit min_f1_time = min(timeit.repeat(stmt='f1(x,W,y)', setup=setup, repeat=10, number=10000)) min_f2_time = min(timeit.repeat(stmt='f2(x,W,y)', setup=setup, repeat=10, number=10000)) print('f1: {time}'.format(time=min_f1_time)) print('f2: {time}'.format(time=min_f2_time)) #============================================================ and I get (on a particular trial on my intel 64 bit machine running linux and numpy 1.6.1) the following: f1: 2.86719584465 f2: 0.891730070114 Of course, I know nothing of einsum's internals and there's probably a good reason for this. But still, it's quite counterintuitive for me; I just thought I'd mention it because a quick search didn't bring up anything on this topic, and AFAIK einsum's docs/examples don't cover the case with more than two operands. Anyway: I hope I've been clear enough, and that I didn't bring up some already-debated topic. Thanks in advance for any clarification, Eugenio From matthew.brett at gmail.com Tue Nov 1 20:55:36 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 1 Nov 2011 17:55:36 -0700 Subject: [Numpy-discussion] float64 / int comparison different from float / int comparison In-Reply-To: <4EB012CA.5010209@noaa.gov> References: <4EB012CA.5010209@noaa.gov> Message-ID: Hi, On Tue, Nov 1, 2011 at 8:39 AM, Chris.Barker wrote: > On 10/31/11 6:38 PM, St?fan van der Walt wrote: >> On Mon, Oct 31, 2011 at 6:25 PM, Matthew Brett ?wrote: >>> Oh, dear, I'm suffering now: > >>> In [12]: res> ?2**31-1 >>> Out[12]: array([False], dtype=bool) > >> I'm seeing: > ... > >> Your result seems very strange, because the numpy scalars should >> perform exactly the same inside and outside an array. > > I get what St?fan ?gets: > > In [32]: res = np.array((2**31,), dtype=np.float32) > > In [33]: res > 2**31-1 > Out[33]: array([ True], dtype=bool) > > In [34]: res[0] > 2**31-1 > Out[34]: True > > In [35]: res[0].dtype > Out[35]: dtype('float32') > > > In [36]: np.__version__ > Out[36]: '1.6.1' > > (OS-X, Intel, Python2.7) > > > Something is very odd with your build! Well - numpy 1.4.1 on Debian squeeze. I get the same as you with current numpy trunk. Stefan and I explored the issue a bit further and concluded that, in numpy trunk, the current behavior is explicable by upcasting to float64 during the comparison: In [86]: np.array(2**63, dtype=np.float) > 2**63 - 1 Out[86]: False In [87]: np.array(2**31, dtype=np.float) > 2**31 - 1 Out[87]: True because 2**31 and 2**31-1 are both exactly representable in float64, but 2**31-1 is not exactly representable in float32. Maybe this: In [88]: np.promote_types('f4', int) Out[88]: dtype('float64') tells us this information. The command is not available for numpy 1.4.1. I suppose it's possible that the upcasting rules were different in 1.4.1 and that is the cause of the different behavior. Best, Matthew From matthew.brett at gmail.com Tue Nov 1 21:47:49 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 1 Nov 2011 18:47:49 -0700 Subject: [Numpy-discussion] Float128 integer comparison In-Reply-To: <35AAFB94-08D2-47C1-A066-7C8B2CFD2961@astro.physik.uni-goettingen.de> References: <35AAFB94-08D2-47C1-A066-7C8B2CFD2961@astro.physik.uni-goettingen.de> Message-ID: Hi, On Sat, Oct 15, 2011 at 1:34 PM, Derek Homeier wrote: > On 15.10.2011, at 9:42PM, Aronne Merrelli wrote: > >> >> On Sat, Oct 15, 2011 at 1:12 PM, Matthew Brett wrote: >> Hi, >> >> Continuing the exploration of float128 - can anyone explain this behavior? >> >> >>> np.float64(9223372036854775808.0) == 9223372036854775808L >> True >> >>> np.float128(9223372036854775808.0) == 9223372036854775808L >> False >> >>> int(np.float128(9223372036854775808.0)) == 9223372036854775808L >> True >> >>> np.round(np.float128(9223372036854775808.0)) == np.float128(9223372036854775808.0) >> True >> >> >> I know little about numpy internals, but while fiddling with this, I noticed a possible clue: >> >> >>> np.float128(9223372036854775808.0) == 9223372036854775808L >> False >> >>> np.float128(4611686018427387904.0) == 4611686018427387904L >> True >> >>> np.float128(9223372036854775808.0) - 9223372036854775808L >> Traceback (most recent call last): >> ? File "", line 1, in >> TypeError: unsupported operand type(s) for -: 'numpy.float128' and 'long' >> >>> np.float128(4611686018427387904.0) - 4611686018427387904L >> 0.0 >> >> >> My speculation - 9223372036854775808L is the first integer that is too big to fit into a signed 64 bit integer. Python is OK with this but that means it must be containing that value in some more complicated object. Since you don't get the type error between float64() and long: >> >> >>> np.float64(9223372036854775808.0) - 9223372036854775808L >> 0.0 >> >> Maybe there are some unimplemented pieces in numpy for dealing with operations between float128 and python "arbitrary longs"? I could see the == test just producing false in that case, because it defaults back to some object equality test which isn't actually looking at the numbers. > > That seems to make sense, since even upcasting from a np.float64 still lets the test fail: >>>> np.float128(np.float64(9223372036854775808.0)) == 9223372036854775808L > False > while >>>> np.float128(9223372036854775808.0) == np.uint64(9223372036854775808L) > True > > and >>>> np.float128(9223372036854775809) == np.uint64(9223372036854775809L) > False >>>> np.float128(np.uint(9223372036854775809L) == np.uint64(9223372036854775809L) > True > > Showing again that the normal casting to, or reading in of, a np.float128 internally inevitably > calls the python float(), as already suggested in one of the parallel threads (I think this > also came up with some of the tests for precision) - leading to different results than > when you can convert from a np.int64 - this makes the outcome look even weirder: > >>>> np.float128(9223372036854775807.0) - np.float128(np.int64(9223372036854775807)) > 1.0 >>>> np.float128(9223372036854775296.0) - np.float128(np.int64(9223372036854775807)) > 1.0 >>>> np.float128(9223372036854775295.0) - np.float128(np.int64(9223372036854775807)) > -1023.0 >>>> np.float128(np.int64(9223372036854775296)) - np.float128(np.int64(9223372036854775807)) > -511.0 > > simply due to the nearest np.float64 always being equal to MAX_INT64 in the two first cases > above (or anything in between)... Right - just for the record, I think there are four relevant problems. 1: values being cast to float128 appear to go through float64 -------------------------------------------------------------------------------------- In [119]: np.float128(2**54-1) Out[119]: 18014398509481984.0 In [120]: np.float128(2**54)-1 Out[120]: 18014398509481983.0 2: values being cast from float128 to int appear to go through float64 again ----------------------------------------------------------------------------------------------------------- In [121]: int(np.float128(2**54-1)) Out[121]: 18014398509481984 http://projects.scipy.org/numpy/ticket/1395 3: comparison to python long ints is always unequal --------------------------------------------------------------------------- In [139]: 2**63 # 2*63 correctly represented in float128 Out[139]: 9223372036854775808L In [140]: int(np.float64(2**63)) Out[140]: 9223372036854775808L In [141]: int(np.float128(2**63)) Out[141]: 9223372036854775808L In [142]: np.float128(2**63) == 2**63 Out[142]: False In [143]: np.float128(2**63)-1 == 2**63-1 Out[143]: True In [144]: np.float128(2**63) == np.float128(2**63) Out[144]: True Probably because, as y'all are saying, numpy tries to convert to np.int64, fails, and falls back to an object array: In [145]: np.array(2**63) Out[145]: array(9223372036854775808L, dtype=object) In [146]: np.array(2**63-1) Out[146]: array(9223372036854775807L) 4 : any other operation of float128 with python long ints fails -------------------------------------------------------------------------------------- In [148]: np.float128(0) + 2**63 --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/mb312/ in () ----> 1 np.float128(0) + 2**63 TypeError: unsupported operand type(s) for +: 'numpy.float128' and 'long' In [149]: np.float128(0) - 2**63 --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/mb312/ in () ----> 1 np.float128(0) - 2**63 TypeError: unsupported operand type(s) for -: 'numpy.float128' and 'long' In [150]: np.float128(0) * 2**63 --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/mb312/ in () ----> 1 np.float128(0) * 2**63 TypeError: unsupported operand type(s) for *: 'numpy.float128' and 'long' In [151]: np.float128(0) / 2**63 --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/mb312/ in () ----> 1 np.float128(0) / 2**63 TypeError: unsupported operand type(s) for /: 'numpy.float128' and 'long' Thanks for the feedback, Best, Matthew From matthew.brett at gmail.com Wed Nov 2 00:02:12 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 1 Nov 2011 21:02:12 -0700 Subject: [Numpy-discussion] Nice float -> integer conversion? In-Reply-To: References: Message-ID: Hi, On Sat, Oct 15, 2011 at 12:20 PM, Matthew Brett wrote: > Hi, > > On Tue, Oct 11, 2011 at 7:32 PM, Benjamin Root wrote: >> On Tue, Oct 11, 2011 at 2:06 PM, Derek Homeier >> wrote: >>> >>> On 11 Oct 2011, at 20:06, Matthew Brett wrote: >>> >>> > Have I missed a fast way of doing nice float to integer conversion? >>> > >>> > By nice I mean, rounding to the nearest integer, converting NaN to 0, >>> > inf, -inf to the max and min of the integer range? ?The astype method >>> > and cast functions don't do what I need here: >>> > >>> > In [40]: np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) >>> > Out[40]: array([1, 0, 0, 0], dtype=int16) >>> > >>> > In [41]: np.cast[np.int16](np.array([1.6, np.nan, np.inf, -np.inf])) >>> > Out[41]: array([1, 0, 0, 0], dtype=int16) >>> > >>> > Have I missed something obvious? >>> >>> np.[a]round comes closer to what you wish (is there consensus >>> that NaN should map to 0?), but not quite there, and it's not really >>> consistent either! >>> >> >> In a way, there is already consensus in the code.? np.nan_to_num() by >> default converts nans to zero, and the infinities go to very large and very >> small. >> >> ??? >>> np.set_printoptions(precision=8) >> ??? >>> x = np.array([np.inf, -np.inf, np.nan, -128, 128]) >> ??? >>> np.nan_to_num(x) >> ??? array([? 1.79769313e+308,? -1.79769313e+308,?? 0.00000000e+000, >> ??????????? -1.28000000e+002,?? 1.28000000e+002]) > > Right - but - we'd still need to round, and take care of the nasty > issue of thresholding: > >>>> x = np.array([np.inf, -np.inf, np.nan, -128, 128]) >>>> x > array([ ?inf, ?-inf, ? nan, -128., ?128.]) >>>> nnx = np.nan_to_num(x) >>>> nnx > > array([ ?1.79769313e+308, ?-1.79769313e+308, ? 0.00000000e+000, > ? ? ? ?-1.28000000e+002, ? 1.28000000e+002]) >>>> np.rint(nnx).astype(np.int8) > array([ ? 0, ? ?0, ? ?0, -128, -128], dtype=int8) > > So, I think nice_round would look something like: > > def nice_round(arr, out_type): > ? ?in_type = arr.dtype.type > ? ?mx = floor_exact(np.iinfo(out_type).max, in_type) > ? ?mn = floor_exact(np.iinfo(out_type).max, in_type) > ? ?nans = np.isnan(arr) > ? ?out = np.rint(np.clip(arr, mn, mx)).astype(out_type) > ? ?out[nans] = 0 > ? ?return out > > with floor_exact being something like: > > https://github.com/matthew-brett/nibabel/blob/range-dtype-conversions/nibabel/floating.py In case anyone is interested or for the sake of anyone later googling this thread - I made a working version of nice_round: https://github.com/matthew-brett/nibabel/blob/floating-stash/nibabel/casting.py Docstring: def nice_round(arr, int_type, nan2zero=True, infmax=False): """ Round floating point array `arr` to type `int_type` Parameters ---------- arr : array-like Array of floating point type int_type : object Numpy integer type nan2zero : {True, False} Whether to convert NaN value to zero. Default is True. If False, and NaNs are present, raise CastingError infmax : {False, True} If True, set np.inf values in `arr` to be `int_type` integer maximum value, -np.inf as `int_type` integer minimum. If False, merely set infs to be numbers at or near the maximum / minumum number in `arr` that can be contained in `int_type`. Therefore False gives faster conversion at the expense of infs that are further from infinity. Returns ------- iarr : ndarray of type `int_type` Examples -------- >>> nice_round([np.nan, np.inf, -np.inf, 1.1, 6.6], np.int16) array([ 0, 32767, -32768, 1, 7], dtype=int16) It wasn't straightforward to find the right place to clip the array to stop overflow on casting, but I think it's working and tested now. See y'all, Matthew From akshar.bhosale at gmail.com Wed Nov 2 12:20:18 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Wed, 2 Nov 2011 21:50:18 +0530 Subject: [Numpy-discussion] numpy error with mkl 10.1 Message-ID: Hi, i am getting following error. python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], numpy.complex128).T.I.H' MKL FATAL ERROR: Cannot load libmkl_lapack.so have installed numpy 1.6.0 with python 2.6. i have intel cluster toolkit installed on my system. (11/069 version and mlk=10.1). i have machine having intel xeon processor and rhel 5.2 x86_64 platform. Kindly help -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 2 12:31:59 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 2 Nov 2011 12:31:59 -0400 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: Locate your libmkl_lapack.so and try to add the directory that contains it to your LD_LIBRARY_PATH environment variable. -=- Olivier 2011/11/2 akshar bhosale > Hi, > > i am getting following error. > python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], > numpy.complex128).T.I.H' > MKL FATAL ERROR: Cannot load libmkl_lapack.so > > have installed numpy 1.6.0 with python 2.6. > i have intel cluster toolkit installed on my system. (11/069 version > and mlk=10.1). i have machine having intel xeon processor and rhel 5.2 > x86_64 > platform. > Kindly help > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 12:36:25 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Wed, 2 Nov 2011 22:06:25 +0530 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: HI, It is already added in the LD_LIBRARY_PATH, thenalso it is generating the same error. On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: > Locate your libmkl_lapack.so and try to add the directory that contains it > to your LD_LIBRARY_PATH environment variable. > > -=- Olivier > > 2011/11/2 akshar bhosale > >> Hi, >> >> i am getting following error. >> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], >> numpy.complex128).T.I.H' >> MKL FATAL ERROR: Cannot load libmkl_lapack.so >> >> have installed numpy 1.6.0 with python 2.6. >> i have intel cluster toolkit installed on my system. (11/069 version >> and mlk=10.1). i have machine having intel xeon processor and rhel 5.2 >> x86_64 >> platform. >> Kindly help >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 2 12:44:20 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 2 Nov 2011 12:44:20 -0400 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: Ok, can you print the output of ldd numpy/core/_dotblas.so? -=- Olivier 2011/11/2 akshar bhosale > HI, > It is already added in the LD_LIBRARY_PATH, thenalso it is generating the > same error. > > > On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: > >> Locate your libmkl_lapack.so and try to add the directory that contains >> it to your LD_LIBRARY_PATH environment variable. >> >> -=- Olivier >> >> 2011/11/2 akshar bhosale >> >>> Hi, >>> >>> i am getting following error. >>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], >>> numpy.complex128).T.I.H' >>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>> >>> have installed numpy 1.6.0 with python 2.6. >>> i have intel cluster toolkit installed on my system. (11/069 version >>> and mlk=10.1). i have machine having intel xeon processor and rhel 5.2 >>> x86_64 >>> platform. >>> Kindly help >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 13:52:26 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Wed, 2 Nov 2011 23:22:26 +0530 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: Hi, ldd _dotblas.so libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) libmkl_def.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so (0x00002b12f099c000) libmkl_intel_lp64.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so (0x00002b12f14f1000) libmkl_intel_thread.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so (0x00002b12f184c000) libmkl_core.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so (0x00002b12f2575000) libmkl_mc.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) libintlc.so.5 => /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: > Ok, can you print the output of ldd numpy/core/_dotblas.so? > > > -=- Olivier > > 2011/11/2 akshar bhosale > >> HI, >> It is already added in the LD_LIBRARY_PATH, thenalso it is generating the >> same error. >> >> >> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: >> >>> Locate your libmkl_lapack.so and try to add the directory that contains >>> it to your LD_LIBRARY_PATH environment variable. >>> >>> -=- Olivier >>> >>> 2011/11/2 akshar bhosale >>> >>>> Hi, >>>> >>>> i am getting following error. >>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], >>>> numpy.complex128).T.I.H' >>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>> >>>> have installed numpy 1.6.0 with python 2.6. >>>> i have intel cluster toolkit installed on my system. (11/069 version >>>> and mlk=10.1). i have machine having intel xeon processor and rhel 5.2 >>>> x86_64 >>>> platform. >>>> Kindly help >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 2 13:56:50 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 2 Nov 2011 13:56:50 -0400 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: Doh I'm sorry, I forgot the problem was with lapack, what about ldd numpy/linalg/lapack_lite.so? 2011/11/2 akshar bhosale > Hi, > ldd _dotblas.so > libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) > libmkl_def.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so > (0x00002b12f099c000) > libmkl_intel_lp64.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so > (0x00002b12f14f1000) > libmkl_intel_thread.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so > (0x00002b12f184c000) > libmkl_core.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so > (0x00002b12f2575000) > libmkl_mc.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) > libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so > (0x00002b12f36db000) > libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so > (0x00002b12f3a32000) > libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) > libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so > (0x00002b12f3e74000) > libintlc.so.5 => > /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) > libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) > libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) > /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) > > > > On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: > >> Ok, can you print the output of ldd numpy/core/_dotblas.so? >> >> >> -=- Olivier >> >> 2011/11/2 akshar bhosale >> >>> HI, >>> It is already added in the LD_LIBRARY_PATH, thenalso it is generating >>> the same error. >>> >>> >>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: >>> >>>> Locate your libmkl_lapack.so and try to add the directory that contains >>>> it to your LD_LIBRARY_PATH environment variable. >>>> >>>> -=- Olivier >>>> >>>> 2011/11/2 akshar bhosale >>>> >>>>> Hi, >>>>> >>>>> i am getting following error. >>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], >>>>> numpy.complex128).T.I.H' >>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>> >>>>> have installed numpy 1.6.0 with python 2.6. >>>>> i have intel cluster toolkit installed on my system. (11/069 >>>>> version and mlk=10.1). i have machine having intel xeon processor and rhel >>>>> 5.2 x86_64 >>>>> platform. >>>>> Kindly help >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 14:03:58 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Wed, 2 Nov 2011 23:33:58 +0530 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: Hi, ldd Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) libmkl_def.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so (0x00002b1199653000) libmkl_intel_lp64.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so (0x00002b119a1a8000) libmkl_intel_thread.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so (0x00002b119a503000) libmkl_core.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so (0x00002b119b22c000) libmkl_mc.so => /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b119c392000) libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) libintlc.so.5 => /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: > Doh I'm sorry, I forgot the problem was with lapack, what about ldd > numpy/linalg/lapack_lite.so? > > > 2011/11/2 akshar bhosale > >> Hi, >> ldd _dotblas.so >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >> libmkl_def.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >> (0x00002b12f099c000) >> libmkl_intel_lp64.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >> (0x00002b12f14f1000) >> libmkl_intel_thread.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >> (0x00002b12f184c000) >> libmkl_core.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >> (0x00002b12f2575000) >> libmkl_mc.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >> (0x00002b12f36db000) >> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >> (0x00002b12f3a32000) >> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >> libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so >> (0x00002b12f3e74000) >> libintlc.so.5 => >> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >> >> >> >> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: >> >>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>> >>> >>> -=- Olivier >>> >>> 2011/11/2 akshar bhosale >>> >>>> HI, >>>> It is already added in the LD_LIBRARY_PATH, thenalso it is generating >>>> the same error. >>>> >>>> >>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: >>>> >>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>> >>>>> -=- Olivier >>>>> >>>>> 2011/11/2 akshar bhosale >>>>> >>>>>> Hi, >>>>>> >>>>>> i am getting following error. >>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], >>>>>> numpy.complex128).T.I.H' >>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>> >>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>> i have intel cluster toolkit installed on my system. (11/069 >>>>>> version and mlk=10.1). i have machine having intel xeon processor and rhel >>>>>> 5.2 x86_64 >>>>>> platform. >>>>>> Kindly help >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 2 14:21:48 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 2 Nov 2011 14:21:48 -0400 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: Hmm that's interesting, it's not linked against libmkl_lapack. On my system with MKL it says: ldd linalg/lapack_lite.so libmkl_lapack.so => /opt/intel/mkl/ 10.1.3.027/lib/em64t/libmkl_lapack.so (0x00002acf0e25a000) libmkl_intel_lp64.so => /opt/intel/mkl/ 10.1.3.027/lib/em64t/libmkl_intel_lp64.so (0x00002acf0ebb8000) libmkl_intel_thread.so => /opt/intel/mkl/ 10.1.3.027/lib/em64t/libmkl_intel_thread.so (0x00002acf0ef2e000) libmkl_def.so => /opt/intel/mkl/10.1.3.027/lib/em64t/libmkl_def.so(0x00002acf0fc93000) libmkl_core.so => /opt/intel/mkl/10.1.3.027/lib/em64t/libmkl_core.so(0x00002acf1080e000) libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so (0x00002acf10a03000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002acf10ba6000) libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) libpython2.6.so.1.0 => /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) So to clarify, what I was trying to achieve is find which of numpy's shared libraries was trying to load libmkl_lapack.so, and if it could be found properly with ldd. Just to make sure it's not another .so, can you run from within your numpy directory: find -name "*.so" -exec ldd \{} \; | grep lapack If the output is not empty, try to run ldd on all .so within numpy to see which one is linked against libmkl_lapack.so. Then hopefully something will stand out (I admit that's not guaranteed though ;). -=- Olivier 2011/11/2 akshar bhosale > Hi, > > ldd Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so > libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) > libmkl_def.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so > (0x00002b1199653000) > libmkl_intel_lp64.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so > (0x00002b119a1a8000) > libmkl_intel_thread.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so > (0x00002b119a503000) > libmkl_core.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so > (0x00002b119b22c000) > libmkl_mc.so => > /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) > libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so > (0x00002b119c392000) > libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so > (0x00002b119c6e9000) > libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) > libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so > (0x00002b119cb2b000) > libintlc.so.5 => > /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) > libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) > libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) > /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) > > > > On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: > >> Doh I'm sorry, I forgot the problem was with lapack, what about ldd >> numpy/linalg/lapack_lite.so? >> >> >> 2011/11/2 akshar bhosale >> >>> Hi, >>> ldd _dotblas.so >>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>> libmkl_def.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>> (0x00002b12f099c000) >>> libmkl_intel_lp64.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>> (0x00002b12f14f1000) >>> libmkl_intel_thread.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>> (0x00002b12f184c000) >>> libmkl_core.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>> (0x00002b12f2575000) >>> libmkl_mc.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>> (0x00002b12f36db000) >>> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >>> (0x00002b12f3a32000) >>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>> libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so >>> (0x00002b12f3e74000) >>> libintlc.so.5 => >>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>> >>> >>> >>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: >>> >>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>> >>>> >>>> -=- Olivier >>>> >>>> 2011/11/2 akshar bhosale >>>> >>>>> HI, >>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is generating >>>>> the same error. >>>>> >>>>> >>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: >>>>> >>>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>>> >>>>>> -=- Olivier >>>>>> >>>>>> 2011/11/2 akshar bhosale >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> i am getting following error. >>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], >>>>>>> numpy.complex128).T.I.H' >>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>> >>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>> i have intel cluster toolkit installed on my system. (11/069 >>>>>>> version and mlk=10.1). i have machine having intel xeon processor and rhel >>>>>>> 5.2 x86_64 >>>>>>> platform. >>>>>>> Kindly help >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 14:44:18 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 00:14:18 +0530 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: hi, thanks for the reply none of them is linked with libmkl_lapack.so. what can be the problem? libmkl_lapack.so is present in its location. if possible, can you send your site.cfg and other related files like intelccompilers.py / system_config.py etc...and configure/build options, i will try the same? On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: > Hmm that's interesting, it's not linked against libmkl_lapack. On my > system with MKL it says: > > ldd linalg/lapack_lite.so > libmkl_lapack.so => /opt/intel/mkl/ > 10.1.3.027/lib/em64t/libmkl_lapack.so (0x00002acf0e25a000) > libmkl_intel_lp64.so => /opt/intel/mkl/ > 10.1.3.027/lib/em64t/libmkl_intel_lp64.so (0x00002acf0ebb8000) > libmkl_intel_thread.so => /opt/intel/mkl/ > 10.1.3.027/lib/em64t/libmkl_intel_thread.so (0x00002acf0ef2e000) > libmkl_def.so => /opt/intel/mkl/10.1.3.027/lib/em64t/libmkl_def.so(0x00002acf0fc93000) > libmkl_core.so => /opt/intel/mkl/ > 10.1.3.027/lib/em64t/libmkl_core.so (0x00002acf1080e000) > libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so > (0x00002acf10a03000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00002acf10ba6000) > libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) > libpython2.6.so.1.0 => /opt/python64/2.6.4/lib/libpython2.6.so.1.0 > (0x00002acf11044000) > libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) > libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) > /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) > libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) > > So to clarify, what I was trying to achieve is find which of numpy's > shared libraries was trying to load libmkl_lapack.so, and if it could be > found properly with ldd. Just to make sure it's not another .so, can you > run from within your numpy directory: > find -name "*.so" -exec ldd \{} \; | grep lapack > > If the output is not empty, try to run ldd on all .so within numpy to see > which one is linked against libmkl_lapack.so. Then hopefully something will > stand out (I admit that's not guaranteed though ;). > > > -=- Olivier > > 2011/11/2 akshar bhosale > >> Hi, >> >> ldd Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) >> libmkl_def.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >> (0x00002b1199653000) >> libmkl_intel_lp64.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >> (0x00002b119a1a8000) >> libmkl_intel_thread.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >> (0x00002b119a503000) >> libmkl_core.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >> (0x00002b119b22c000) >> libmkl_mc.so => >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >> (0x00002b119c392000) >> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >> (0x00002b119c6e9000) >> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >> libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so >> (0x00002b119cb2b000) >> libintlc.so.5 => >> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >> >> >> >> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: >> >>> Doh I'm sorry, I forgot the problem was with lapack, what about ldd >>> numpy/linalg/lapack_lite.so? >>> >>> >>> 2011/11/2 akshar bhosale >>> >>>> Hi, >>>> ldd _dotblas.so >>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>> libmkl_def.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>> (0x00002b12f099c000) >>>> libmkl_intel_lp64.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>> (0x00002b12f14f1000) >>>> libmkl_intel_thread.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>> (0x00002b12f184c000) >>>> libmkl_core.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>> (0x00002b12f2575000) >>>> libmkl_mc.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>> (0x00002b12f36db000) >>>> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >>>> (0x00002b12f3a32000) >>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>> libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so >>>> (0x00002b12f3e74000) >>>> libintlc.so.5 => >>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>> >>>> >>>> >>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: >>>> >>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>> >>>>> >>>>> -=- Olivier >>>>> >>>>> 2011/11/2 akshar bhosale >>>>> >>>>>> HI, >>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is generating >>>>>> the same error. >>>>>> >>>>>> >>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: >>>>>> >>>>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>>>> >>>>>>> -=- Olivier >>>>>>> >>>>>>> 2011/11/2 akshar bhosale >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> i am getting following error. >>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], >>>>>>>> numpy.complex128).T.I.H' >>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>> >>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>> i have intel cluster toolkit installed on my system. (11/069 >>>>>>>> version and mlk=10.1). i have machine having intel xeon processor and rhel >>>>>>>> 5.2 x86_64 >>>>>>>> platform. >>>>>>>> Kindly help >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 2 15:57:29 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 2 Nov 2011 15:57:29 -0400 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: Sorry, no clue :/ I made a tarball with my distutils folder here: http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 Hope this helps... -=- Olivier 2011/11/2 akshar bhosale > hi, > thanks for the reply > none of them is linked with libmkl_lapack.so. > what can be the problem? > libmkl_lapack.so is present in its location. > > if possible, can you send your site.cfg and other related files like > intelccompilers.py / system_config.py etc...and configure/build options, i > will try the same? > > On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: > >> Hmm that's interesting, it's not linked against libmkl_lapack. On my >> system with MKL it says: >> >> ldd linalg/lapack_lite.so >> libmkl_lapack.so => /opt/intel/mkl/*MailScanner warning: >> numerical links are often malicious:*10.1.3.027/lib/em64t/libmkl_lapack.so(0x00002acf0e25a000) >> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner warning: >> numerical links are often malicious:*10.1.3.027/lib/em64t/libmkl_intel_lp64.so(0x00002acf0ebb8000) >> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner warning: >> numerical links are often malicious:*10.1.3.027/lib/em64t/libmkl_intel_thread.so(0x00002acf0ef2e000) >> libmkl_def.so => /opt/intel/mkl/*MailScanner warning: numerical >> links are often malicious:* 10.1.3.027/lib/em64t/libmkl_def.so(0x00002acf0fc93000) >> libmkl_core.so => /opt/intel/mkl/*MailScanner warning: numerical >> links are often malicious:* 10.1.3.027/lib/em64t/libmkl_core.so(0x00002acf1080e000) >> >> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >> (0x00002acf10a03000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002acf10ba6000) >> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >> libpython2.6.so.1.0 => >> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >> >> So to clarify, what I was trying to achieve is find which of numpy's >> shared libraries was trying to load libmkl_lapack.so, and if it could be >> found properly with ldd. Just to make sure it's not another .so, can you >> run from within your numpy directory: >> find -name "*.so" -exec ldd \{} \; | grep lapack >> >> If the output is not empty, try to run ldd on all .so within numpy to see >> which one is linked against libmkl_lapack.so. Then hopefully something will >> stand out (I admit that's not guaranteed though ;). >> >> >> -=- Olivier >> >> 2011/11/2 akshar bhosale >> >>> Hi, >>> >>> ldd Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) >>> libmkl_def.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>> (0x00002b1199653000) >>> libmkl_intel_lp64.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>> (0x00002b119a1a8000) >>> libmkl_intel_thread.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>> (0x00002b119a503000) >>> libmkl_core.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>> (0x00002b119b22c000) >>> libmkl_mc.so => >>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>> (0x00002b119c392000) >>> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >>> (0x00002b119c6e9000) >>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>> libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so >>> (0x00002b119cb2b000) >>> libintlc.so.5 => >>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>> >>> >>> >>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: >>> >>>> Doh I'm sorry, I forgot the problem was with lapack, what about ldd >>>> numpy/linalg/lapack_lite.so? >>>> >>>> >>>> 2011/11/2 akshar bhosale >>>> >>>>> Hi, >>>>> ldd _dotblas.so >>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>> libmkl_def.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>> (0x00002b12f099c000) >>>>> libmkl_intel_lp64.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>> (0x00002b12f14f1000) >>>>> libmkl_intel_thread.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>> (0x00002b12f184c000) >>>>> libmkl_core.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>> (0x00002b12f2575000) >>>>> libmkl_mc.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >>>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>>> (0x00002b12f36db000) >>>>> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >>>>> (0x00002b12f3a32000) >>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>> libiomp5.so => >>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>> libintlc.so.5 => >>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>> >>>>> >>>>> >>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: >>>>> >>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>> >>>>>> >>>>>> -=- Olivier >>>>>> >>>>>> 2011/11/2 akshar bhosale >>>>>> >>>>>>> HI, >>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>> generating the same error. >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: >>>>>>> >>>>>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>> >>>>>>>> -=- Olivier >>>>>>>> >>>>>>>> 2011/11/2 akshar bhosale >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> i am getting following error. >>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], >>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>> >>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>> i have intel cluster toolkit installed on my system. (11/069 >>>>>>>>> version and mlk=10.1). i have machine having intel xeon processor and rhel >>>>>>>>> 5.2 x86_64 >>>>>>>>> platform. >>>>>>>>> Kindly help >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Nov 2 19:37:56 2011 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 2 Nov 2011 16:37:56 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? Message-ID: Hi again, Okay, here's my attempt at an *uncontroversial* email! Specifically, I think it'll be easier to talk about this NA stuff if we can establish some common ground, and easier for people to follow if the basic points of agreement are laid out in one place. So I'm going to try and summarize just the things that we can agree about. Note that right now I'm *only* talking about what kind of tools we want to give the user -- i.e., what kind of problems we are trying to solve. AFAICT we don't have as much consensus on implementation matters, and anyway it's hard to make implementation decisions without knowing what we're trying to accomplish. 1) I think we have consensus that there are (at least) two different possible ways of thinking about this problem, with somewhat different constituencies. Let's call these two concepts "MISSING data" and "IGNORED data". 2) I also think we have at least a rough consensus on what these concepts mean, and what their supporters want from them: MISSING data: - Conceptually, MISSINGness acts like a property of a datum -- assigning MISSING to a location is like assigning any other value to that location - Ufuncs and other operations must propagate these values by default, and there must be an option to cause them to be ignored - Must be competitive with NaNs in terms of speed and memory usage (or else people will just use NaNs) - Compatibility with R is valuable - To avoid user confusion, ideally it should *not* be possible to 'unmask' a missing value, since this is inconsistent with the "missing value" metaphor (e.g., see Wes's comment about "leaky abstractions") - Possible useful extension: having different classes of missing values (similar to Stata) - Target audience: data analysis with missing data, neuroimaging, econometrics, former R users, ... IGNORED data: - Conceptually, IGNOREDness acts like a property of the array -- toggling a location to be IGNORED is kind of vaguely similar to changing an array's shape - Ufuncs and other operations must ignore these values by default, and there doesn't really need to be a way to propagate them, even as an option (though it probably wouldn't hurt either) - Some memory overhead is inevitable and acceptable - Compatibility with R neither possible nor valuable - Ability to toggle the IGNORED state of a location is critical, and should be as convenient as possible - Possible useful extension: having not just different types of ignored values, but richer ways to combine them -- e.g., the example of combining astronomical images with some kind of associated per-pixel quality scores, where one might want the 'mask' to be not just a boolean IGNORED/not-IGNORED flag, but an integer (perhaps a multi-byte integer) or even a float, and to allow these 'masks' to be combined in some more complex way than just logical_and. - Target audience: anyone who's already doing this kind of thing by hand using a second mask array + boolean indexing, former numpy.ma users, matplotlib, ... 3) And perhaps we can all agree that the biggest *un*resolved question is whether we want to: - emphasize the similarities between these two use cases and build a single interface that can handle both concepts, with some compromises - or, treat these at two mostly-separate features that can each become exactly what the respective constituency wants without compromise -- but with some potential redundancy and extra code. Each approach has advantages and disadvantages. Does that seem like a fair summary? Anything more we can add? Most importantly, anything here that you disagree with? Did I summarize your needs well? Do you have a use case that you feel doesn't fit naturally into either category? [Also, I thought this might make the start of a good wiki page for people to reference during these discussions, but I don't seem to have edit rights. If other people agree, maybe someone could put it up, or give me access? My trac id is njs at pobox.com.] Thanks, -- Nathaniel From ben.root at ou.edu Wed Nov 2 20:25:15 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 2 Nov 2011 19:25:15 -0500 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: Message-ID: On Wed, Nov 2, 2011 at 6:37 PM, Nathaniel Smith wrote: > Hi again, > > Okay, here's my attempt at an *uncontroversial* email! > > Specifically, I think it'll be easier to talk about this NA stuff if > we can establish some common ground, and easier for people to follow > if the basic points of agreement are laid out in one place. So I'm > going to try and summarize just the things that we can agree about. > > Note that right now I'm *only* talking about what kind of tools we > want to give the user -- i.e., what kind of problems we are trying to > solve. AFAICT we don't have as much consensus on implementation > matters, and anyway it's hard to make implementation decisions without > knowing what we're trying to accomplish. > > 1) I think we have consensus that there are (at least) two different > possible ways of thinking about this problem, with somewhat different > constituencies. Let's call these two concepts "MISSING data" and > "IGNORED data". > > 2) I also think we have at least a rough consensus on what these > concepts mean, and what their supporters want from them: > > MISSING data: > - Conceptually, MISSINGness acts like a property of a datum -- > assigning MISSING to a location is like assigning any other value to > that location > - Ufuncs and other operations must propagate these values by default, > and there must be an option to cause them to be ignored > - Must be competitive with NaNs in terms of speed and memory usage (or > else people will just use NaNs) > - Compatibility with R is valuable > - To avoid user confusion, ideally it should *not* be possible to > 'unmask' a missing value, since this is inconsistent with the "missing > value" metaphor (e.g., see Wes's comment about "leaky abstractions") > - Possible useful extension: having different classes of missing > values (similar to Stata) > - Target audience: data analysis with missing data, neuroimaging, > econometrics, former R users, ... > > IGNORED data: > - Conceptually, IGNOREDness acts like a property of the array -- > toggling a location to be IGNORED is kind of vaguely similar to > changing an array's shape > - Ufuncs and other operations must ignore these values by default, and > there doesn't really need to be a way to propagate them, even as an > option (though it probably wouldn't hurt either) > - Some memory overhead is inevitable and acceptable > - Compatibility with R neither possible nor valuable > - Ability to toggle the IGNORED state of a location is critical, and > should be as convenient as possible > - Possible useful extension: having not just different types of > ignored values, but richer ways to combine them -- e.g., the example > of combining astronomical images with some kind of associated > per-pixel quality scores, where one might want the 'mask' to be not > just a boolean IGNORED/not-IGNORED flag, but an integer (perhaps a > multi-byte integer) or even a float, and to allow these 'masks' to be > combined in some more complex way than just logical_and. > - Target audience: anyone who's already doing this kind of thing by > hand using a second mask array + boolean indexing, former numpy.ma > users, matplotlib, ... > > 3) And perhaps we can all agree that the biggest *un*resolved question > is whether we want to: > - emphasize the similarities between these two use cases and build a > single interface that can handle both concepts, with some compromises > - or, treat these at two mostly-separate features that can each become > exactly what the respective constituency wants without compromise -- > but with some potential redundancy and extra code. > Each approach has advantages and disadvantages. > > Does that seem like a fair summary? Anything more we can add? Most > importantly, anything here that you disagree with? Did I summarize > your needs well? Do you have a use case that you feel doesn't fit > naturally into either category? > > [Also, I thought this might make the start of a good wiki page for > people to reference during these discussions, but I don't seem to have > edit rights. If other people agree, maybe someone could put it up, or > give me access? My trac id is njs at pobox.com.] > > Thanks, > -- Nathaniel > I want to pare this down even more. I think the above lists makes too many unneeded extrapolations. MISSING data: - Conceptually, MISSINGness acts like a property of a datum -- assigning MISSING to a location is like assigning any other value to that location - Ufuncs and other operations must propagate these values by default, and there must be an option to cause them to be ignored - Assigning MISSING is destructive - Must be competitive with NaNs in terms of speed and memory usage (or else people will just use NaNs) - Target audience: data analysis with missing data, neuroimaging, econometrics, former R users, ... - Possible useful extension: having different classes of missing values (similar to Stata) IGNORED data: - Conceptually, IGNOREDness acts like a property of the array -- toggling a location to be IGNORED is kind of vaguely similar to changing an array's shape - Ufuncs and other operations must ignore these values by default, and there doesn't really need to be a way to propagate them, even as an option (though it probably wouldn't hurt either) - Assigning IGNORE is non-destructive - Must be competitive with np.ma for speed and memory (or else users would just use np.ma) - Target audience: anyone who's already doing this kind of thing by hand using a second mask array + boolean indexing, former numpy.ma users, matplotlib, ... - Possible useful extension: having not just different types of ignored values, but richer ways to combine them -- e.g., the example of combining astronomical images with some kind of associated per-pixel quality scores, where one might want the 'mask' to be not just a boolean IGNORED/not-IGNORED flag, but an integer (perhaps a multi-byte integer) or even a float, and to allow these 'masks' to be combined in some more complex way than just logical_and. Then, as a third-party module developer, I can tell you that having separate and independent ways to detect "MISSING"/"IGNORED" would likely make support more difficult and would greatly benefit from a common (or easily combinable) method of identification. Ben Root P.S. - I took out the phrase "compatibility with R" not as a slight against R, but because of the vagueness of the statement. Does it mean raw binary data format compatibility? Some sort of ABI compatibility (does R or python have the ability to call and pass data to each other?). Rather, I find the declaration of R-users being the target audience *much* more important and allows for more flexibility in achieving that goal for both forms of data. -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 20:49:03 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 06:19:03 +0530 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: thanks..what about site.cfg? On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: > Sorry, no clue :/ > > I made a tarball with my distutils folder here: > http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 > Hope this helps... > > -=- Olivier > > 2011/11/2 akshar bhosale > >> hi, >> thanks for the reply >> none of them is linked with libmkl_lapack.so. >> what can be the problem? >> libmkl_lapack.so is present in its location. >> >> if possible, can you send your site.cfg and other related files like >> intelccompilers.py / system_config.py etc...and configure/build options, i >> will try the same? >> >> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >> >>> Hmm that's interesting, it's not linked against libmkl_lapack. On my >>> system with MKL it says: >>> >>> ldd linalg/lapack_lite.so >>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner warning: >>> numerical links are often malicious:*10.1.3.027/lib/em64t/libmkl_lapack.so(0x00002acf0e25a000) >>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner warning: >>> numerical links are often malicious:*10.1.3.027/lib/em64t/libmkl_intel_lp64.so(0x00002acf0ebb8000) >>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner warning: >>> numerical links are often malicious:*10.1.3.027/lib/em64t/libmkl_intel_thread.so(0x00002acf0ef2e000) >>> libmkl_def.so => /opt/intel/mkl/*MailScanner warning: numerical >>> links are often malicious:* 10.1.3.027/lib/em64t/libmkl_def.so(0x00002acf0fc93000) >>> libmkl_core.so => /opt/intel/mkl/*MailScanner warning: >>> numerical links are often malicious:*10.1.3.027/lib/em64t/libmkl_core.so(0x00002acf1080e000) >>> >>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>> (0x00002acf10a03000) >>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002acf10ba6000) >>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>> libpython2.6.so.1.0 => >>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>> >>> So to clarify, what I was trying to achieve is find which of numpy's >>> shared libraries was trying to load libmkl_lapack.so, and if it could be >>> found properly with ldd. Just to make sure it's not another .so, can you >>> run from within your numpy directory: >>> find -name "*.so" -exec ldd \{} \; | grep lapack >>> >>> If the output is not empty, try to run ldd on all .so within numpy to >>> see which one is linked against libmkl_lapack.so. Then hopefully something >>> will stand out (I admit that's not guaranteed though ;). >>> >>> >>> -=- Olivier >>> >>> 2011/11/2 akshar bhosale >>> >>>> Hi, >>>> >>>> ldd Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) >>>> libmkl_def.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>> (0x00002b1199653000) >>>> libmkl_intel_lp64.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>> (0x00002b119a1a8000) >>>> libmkl_intel_thread.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>> (0x00002b119a503000) >>>> libmkl_core.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>> (0x00002b119b22c000) >>>> libmkl_mc.so => >>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>> (0x00002b119c392000) >>>> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >>>> (0x00002b119c6e9000) >>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>> libiomp5.so => /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so >>>> (0x00002b119cb2b000) >>>> libintlc.so.5 => >>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>> >>>> >>>> >>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: >>>> >>>>> Doh I'm sorry, I forgot the problem was with lapack, what about ldd >>>>> numpy/linalg/lapack_lite.so? >>>>> >>>>> >>>>> 2011/11/2 akshar bhosale >>>>> >>>>>> Hi, >>>>>> ldd _dotblas.so >>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>>> libmkl_def.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>> (0x00002b12f099c000) >>>>>> libmkl_intel_lp64.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>> (0x00002b12f14f1000) >>>>>> libmkl_intel_thread.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>> (0x00002b12f184c000) >>>>>> libmkl_core.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>> (0x00002b12f2575000) >>>>>> libmkl_mc.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >>>>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>>>> (0x00002b12f36db000) >>>>>> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >>>>>> (0x00002b12f3a32000) >>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>> libiomp5.so => >>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>> libintlc.so.5 => >>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: >>>>>> >>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>> >>>>>>> >>>>>>> -=- Olivier >>>>>>> >>>>>>> 2011/11/2 akshar bhosale >>>>>>> >>>>>>>> HI, >>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>> generating the same error. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: >>>>>>>> >>>>>>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>>>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>> >>>>>>>>> -=- Olivier >>>>>>>>> >>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> i am getting following error. >>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, >>>>>>>>>> 4]], >>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>> >>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>> i have intel cluster toolkit installed on my system. (11/069 >>>>>>>>>> version and mlk=10.1). i have machine having intel xeon processor and rhel >>>>>>>>>> 5.2 x86_64 >>>>>>>>>> platform. >>>>>>>>>> Kindly help >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 2 21:00:03 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 2 Nov 2011 21:00:03 -0400 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: It's inside the distutils folder. -=- Olivier 2011/11/2 akshar bhosale > thanks..what about site.cfg? > > On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: > >> Sorry, no clue :/ >> >> I made a tarball with my distutils folder here: >> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >> Hope this helps... >> >> -=- Olivier >> >> 2011/11/2 akshar bhosale >> >>> hi, >>> thanks for the reply >>> none of them is linked with libmkl_lapack.so. >>> what can be the problem? >>> libmkl_lapack.so is present in its location. >>> >>> if possible, can you send your site.cfg and other related files like >>> intelccompilers.py / system_config.py etc...and configure/build options, i >>> will try the same? >>> >>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >>> >>>> Hmm that's interesting, it's not linked against libmkl_lapack. On my >>>> system with MKL it says: >>>> >>>> ldd linalg/lapack_lite.so >>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner soup?onne le >>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>> * *MailScanner warning: numerical links are often malicious: >>>> 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner soup?onne >>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>> * *MailScanner warning: numerical links are often malicious: >>>> 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>> "10.1.3.027" * *MailScanner warning: numerical links are often >>>> malicious: 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne le lien >>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" * *MailScanner >>>> warning: numerical links are often malicious: >>>> 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne le >>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>> * *MailScanner warning: numerical links are often malicious: >>>> 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>> >>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>> (0x00002acf10a03000) >>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002acf10ba6000) >>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>> libpython2.6.so.1.0 => >>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>> >>>> So to clarify, what I was trying to achieve is find which of numpy's >>>> shared libraries was trying to load libmkl_lapack.so, and if it could be >>>> found properly with ldd. Just to make sure it's not another .so, can you >>>> run from within your numpy directory: >>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>> >>>> If the output is not empty, try to run ldd on all .so within numpy to >>>> see which one is linked against libmkl_lapack.so. Then hopefully something >>>> will stand out (I admit that's not guaranteed though ;). >>>> >>>> >>>> -=- Olivier >>>> >>>> 2011/11/2 akshar bhosale >>>> >>>>> Hi, >>>>> >>>>> ldd Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) >>>>> libmkl_def.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>> (0x00002b1199653000) >>>>> libmkl_intel_lp64.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>> (0x00002b119a1a8000) >>>>> libmkl_intel_thread.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>> (0x00002b119a503000) >>>>> libmkl_core.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>> (0x00002b119b22c000) >>>>> libmkl_mc.so => >>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>>> (0x00002b119c392000) >>>>> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >>>>> (0x00002b119c6e9000) >>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>> libiomp5.so => >>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>> libintlc.so.5 => >>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>> >>>>> >>>>> >>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: >>>>> >>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about ldd >>>>>> numpy/linalg/lapack_lite.so? >>>>>> >>>>>> >>>>>> 2011/11/2 akshar bhosale >>>>>> >>>>>>> Hi, >>>>>>> ldd _dotblas.so >>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>>>> libmkl_def.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>> (0x00002b12f099c000) >>>>>>> libmkl_intel_lp64.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>> (0x00002b12f14f1000) >>>>>>> libmkl_intel_thread.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>> (0x00002b12f184c000) >>>>>>> libmkl_core.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>> (0x00002b12f2575000) >>>>>>> libmkl_mc.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >>>>>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>>>>> (0x00002b12f36db000) >>>>>>> libsvml.so => >>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>> libiomp5.so => >>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>> libintlc.so.5 => >>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: >>>>>>> >>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>> >>>>>>>> >>>>>>>> -=- Olivier >>>>>>>> >>>>>>>> 2011/11/2 akshar bhosale >>>>>>>> >>>>>>>>> HI, >>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>> generating the same error. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau wrote: >>>>>>>>> >>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>>>>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>> >>>>>>>>>> -=- Olivier >>>>>>>>>> >>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> i am getting following error. >>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, >>>>>>>>>>> 4]], >>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>> >>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>> i have intel cluster toolkit installed on my system. (11/069 >>>>>>>>>>> version and mlk=10.1). i have machine having intel xeon processor and rhel >>>>>>>>>>> 5.2 x86_64 >>>>>>>>>>> platform. >>>>>>>>>>> Kindly help >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 21:11:03 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 06:41:03 +0530 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: hi, are amd,fftw and umfpack required? my architecture is : intel xeon x_86_64. On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau wrote: > It's inside the distutils folder. > > -=- Olivier > > 2011/11/2 akshar bhosale > >> thanks..what about site.cfg? >> >> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: >> >>> Sorry, no clue :/ >>> >>> I made a tarball with my distutils folder here: >>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>> Hope this helps... >>> >>> -=- Olivier >>> >>> 2011/11/2 akshar bhosale >>> >>>> hi, >>>> thanks for the reply >>>> none of them is linked with libmkl_lapack.so. >>>> what can be the problem? >>>> libmkl_lapack.so is present in its location. >>>> >>>> if possible, can you send your site.cfg and other related files like >>>> intelccompilers.py / system_config.py etc...and configure/build options, i >>>> will try the same? >>>> >>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >>>> >>>>> Hmm that's interesting, it's not linked against libmkl_lapack. On my >>>>> system with MKL it says: >>>>> >>>>> ldd linalg/lapack_lite.so >>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>> * *MailScanner warning: numerical links are often malicious: >>>>> 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner soup?onne >>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>> * *MailScanner warning: numerical links are often malicious: >>>>> 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>> "10.1.3.027" * *MailScanner warning: numerical links are often >>>>> malicious: 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>> * *MailScanner warning: numerical links are often malicious: >>>>> 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>> * *MailScanner warning: numerical links are often malicious: >>>>> 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>> >>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>> (0x00002acf10a03000) >>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002acf10ba6000) >>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>> libpython2.6.so.1.0 => >>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>> >>>>> So to clarify, what I was trying to achieve is find which of numpy's >>>>> shared libraries was trying to load libmkl_lapack.so, and if it could be >>>>> found properly with ldd. Just to make sure it's not another .so, can you >>>>> run from within your numpy directory: >>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>> >>>>> If the output is not empty, try to run ldd on all .so within numpy to >>>>> see which one is linked against libmkl_lapack.so. Then hopefully something >>>>> will stand out (I admit that's not guaranteed though ;). >>>>> >>>>> >>>>> -=- Olivier >>>>> >>>>> 2011/11/2 akshar bhosale >>>>> >>>>>> Hi, >>>>>> >>>>>> ldd >>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) >>>>>> libmkl_def.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>> (0x00002b1199653000) >>>>>> libmkl_intel_lp64.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>> (0x00002b119a1a8000) >>>>>> libmkl_intel_thread.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>> (0x00002b119a503000) >>>>>> libmkl_core.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>> (0x00002b119b22c000) >>>>>> libmkl_mc.so => >>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>>>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>>>> (0x00002b119c392000) >>>>>> libsvml.so => /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so >>>>>> (0x00002b119c6e9000) >>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>> libiomp5.so => >>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>> libintlc.so.5 => >>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: >>>>>> >>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about ldd >>>>>>> numpy/linalg/lapack_lite.so? >>>>>>> >>>>>>> >>>>>>> 2011/11/2 akshar bhosale >>>>>>> >>>>>>>> Hi, >>>>>>>> ldd _dotblas.so >>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>>>>> libmkl_def.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>> (0x00002b12f099c000) >>>>>>>> libmkl_intel_lp64.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>> (0x00002b12f14f1000) >>>>>>>> libmkl_intel_thread.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>> (0x00002b12f184c000) >>>>>>>> libmkl_core.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>> (0x00002b12f2575000) >>>>>>>> libmkl_mc.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >>>>>>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>>>>>> (0x00002b12f36db000) >>>>>>>> libsvml.so => >>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>> libiomp5.so => >>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>> libintlc.so.5 => >>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: >>>>>>>> >>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>> >>>>>>>>> >>>>>>>>> -=- Olivier >>>>>>>>> >>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>> >>>>>>>>>> HI, >>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>> generating the same error. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau >>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>>>>>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>> >>>>>>>>>>> -=- Olivier >>>>>>>>>>> >>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> i am getting following error. >>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, >>>>>>>>>>>> 4]], >>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>> >>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>> i have intel cluster toolkit installed on my system. (11/069 >>>>>>>>>>>> version and mlk=10.1). i have machine having intel xeon processor and rhel >>>>>>>>>>>> 5.2 x86_64 >>>>>>>>>>>> platform. >>>>>>>>>>>> Kindly help >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 2 21:24:50 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 2 Nov 2011 21:24:50 -0400 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: I believe they are optional, but I'm not sure. Note that this system I copied the distutils from is not one I setup myself so I'm not sure how numpy was installed exactly. I just know it works and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462 @ 2.80GHz (x86_64 as well). -=- Olivier 2011/11/2 akshar bhosale > hi, > are amd,fftw and umfpack required? my architecture is : intel xeon x_86_64. > > On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau wrote: > >> It's inside the distutils folder. >> >> -=- Olivier >> >> 2011/11/2 akshar bhosale >> >>> thanks..what about site.cfg? >>> >>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: >>> >>>> Sorry, no clue :/ >>>> >>>> I made a tarball with my distutils folder here: >>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>> Hope this helps... >>>> >>>> -=- Olivier >>>> >>>> 2011/11/2 akshar bhosale >>>> >>>>> hi, >>>>> thanks for the reply >>>>> none of them is linked with libmkl_lapack.so. >>>>> what can be the problem? >>>>> libmkl_lapack.so is present in its location. >>>>> >>>>> if possible, can you send your site.cfg and other related files like >>>>> intelccompilers.py / system_config.py etc...and configure/build options, i >>>>> will try the same? >>>>> >>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >>>>> >>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. On my >>>>>> system with MKL it says: >>>>>> >>>>>> ldd linalg/lapack_lite.so >>>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>> fraude de la part de "10.1.3.027" MailScanner warning: numerical >>>>>> links are often malicious: 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner >>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner warning: >>>>>> numerical links are often malicious: >>>>>> 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner warning: >>>>>> numerical links are often malicious: >>>>>> 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>> fraude de la part de "10.1.3.027" MailScanner warning: numerical >>>>>> links are often malicious: 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>> fraude de la part de "10.1.3.027" MailScanner warning: numerical >>>>>> links are often malicious: 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>>> >>>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>> (0x00002acf10a03000) >>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002acf10ba6000) >>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>> libpython2.6.so.1.0 => >>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>> >>>>>> So to clarify, what I was trying to achieve is find which of numpy's >>>>>> shared libraries was trying to load libmkl_lapack.so, and if it could be >>>>>> found properly with ldd. Just to make sure it's not another .so, can you >>>>>> run from within your numpy directory: >>>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>> >>>>>> If the output is not empty, try to run ldd on all .so within numpy to >>>>>> see which one is linked against libmkl_lapack.so. Then hopefully something >>>>>> will stand out (I admit that's not guaranteed though ;). >>>>>> >>>>>> >>>>>> -=- Olivier >>>>>> >>>>>> 2011/11/2 akshar bhosale >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> ldd >>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) >>>>>>> libmkl_def.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>> (0x00002b1199653000) >>>>>>> libmkl_intel_lp64.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>> (0x00002b119a1a8000) >>>>>>> libmkl_intel_thread.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>> (0x00002b119a503000) >>>>>>> libmkl_core.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>> (0x00002b119b22c000) >>>>>>> libmkl_mc.so => >>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>>>>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>>>>> (0x00002b119c392000) >>>>>>> libsvml.so => >>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>> libiomp5.so => >>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>> libintlc.so.5 => >>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: >>>>>>> >>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about ldd >>>>>>>> numpy/linalg/lapack_lite.so? >>>>>>>> >>>>>>>> >>>>>>>> 2011/11/2 akshar bhosale >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> ldd _dotblas.so >>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>>>>>> libmkl_def.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>> (0x00002b12f099c000) >>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>> (0x00002b12f14f1000) >>>>>>>>> libmkl_intel_thread.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>> (0x00002b12f184c000) >>>>>>>>> libmkl_core.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>> (0x00002b12f2575000) >>>>>>>>> libmkl_mc.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >>>>>>>>> libimf.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>> libsvml.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>> libiomp5.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>> libintlc.so.5 => >>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau wrote: >>>>>>>>> >>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -=- Olivier >>>>>>>>>> >>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>> >>>>>>>>>>> HI, >>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>> generating the same error. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau < >>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>> >>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>>>>>>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>> >>>>>>>>>>>> -=- Olivier >>>>>>>>>>>> >>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, >>>>>>>>>>>>> 4]], >>>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>> >>>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>> i have intel cluster toolkit installed on my system. >>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>> platform. >>>>>>>>>>>>> Kindly help >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Nov 2 22:16:39 2011 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 2 Nov 2011 19:16:39 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: Message-ID: Hi Benjamin, On Wed, Nov 2, 2011 at 5:25 PM, Benjamin Root wrote: > I want to pare this down even more.? I think the above lists makes too many > unneeded extrapolations. Okay. I found your formatting a little confusing, so I want to make sure I understood the changes you're suggesting: For the description of what MISSING means, you removed the lines: - Compatibility with R is valuable - To avoid user confusion, ideally it should *not* be possible to 'unmask' a missing value, since this is inconsistent with the "missing value" metaphor (e.g., see Wes's comment about "leaky abstractions") And you added the line: + Assigning MISSING is destructive And for the description of what IGNORED means, you removed the lines: - Some memory overhead is inevitable and acceptable - Compatibility with R neither possible nor valuable - Ability to toggle the IGNORED state of a location is critical, and should be as convenient as possible And you added the lines: + IGNORE is non-destructive + Must be competitive with np.ma for speed and memory (or else users would just use np.ma) Is that right? Assuming it is, my thoughts are: By R compatibility, I specifically had in mind in-memory compatibility. rpy2 provides a more-or-less seamless within-process interface between R and Python (and specifically lets you get numpy views on arrays returned by R functions), so if we can make this work for R arrays containing NA too then that'd be handy. (The rpy2 author requested this in the last discussion here: http://mail.scipy.org/pipermail/numpy-discussion/2011-June/057084.html) When it comes to disk formats, then this doesn't matter so much, since IO routines have to translate between different representations all the time anyway. I take the replacement of my line about MISSING disallowing unmasking and your line about MISSING assignment being destructive as basically expressing the same idea. Is that fair, or did you mean something else? Finally, do you think that people who want IGNORED support care about having a convenient API for masking/unmasking values? You removed that line, but I don't know if that was because you disagreed with it, or were just trying to simplify. > Then, as a third-party module developer, I can tell you that having separate > and independent ways to detect "MISSING"/"IGNORED" would likely make support > more difficult and would greatly benefit from a common (or easily > combinable) method of identification. Right, sorry... I didn't forget, and that's part of what I was thinking when I described the second approach as keeping them as *mostly*-separate interfaces... but I should have made it more explicit! Anyway, yes: 4) There is consensus that whatever approach is taken, there should be a quick and convenient way to identify values that are MISSING, IGNORED, or both. (E.g., functions is_MISSING, is_IGNORED, is_MISSING_or_IGNORED, or some equivalent.) -- Nathaniel From akshar.bhosale at gmail.com Wed Nov 2 22:30:52 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 08:00:52 +0530 Subject: [Numpy-discussion] numpy error with mkl 10.1 In-Reply-To: References: Message-ID: hi, on your machine, is numpy.test passed? On Thu, Nov 3, 2011 at 6:54 AM, Olivier Delalleau wrote: > I believe they are optional, but I'm not sure. > > Note that this system I copied the distutils from is not one I setup > myself so I'm not sure how numpy was installed exactly. I just know it > works and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462 @ > 2.80GHz (x86_64 as well). > > -=- Olivier > > 2011/11/2 akshar bhosale > >> hi, >> are amd,fftw and umfpack required? my architecture is : intel xeon >> x_86_64. >> >> On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau wrote: >> >>> It's inside the distutils folder. >>> >>> -=- Olivier >>> >>> 2011/11/2 akshar bhosale >>> >>>> thanks..what about site.cfg? >>>> >>>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: >>>> >>>>> Sorry, no clue :/ >>>>> >>>>> I made a tarball with my distutils folder here: >>>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>>> Hope this helps... >>>>> >>>>> -=- Olivier >>>>> >>>>> 2011/11/2 akshar bhosale >>>>> >>>>>> hi, >>>>>> thanks for the reply >>>>>> none of them is linked with libmkl_lapack.so. >>>>>> what can be the problem? >>>>>> libmkl_lapack.so is present in its location. >>>>>> >>>>>> if possible, can you send your site.cfg and other related files like >>>>>> intelccompilers.py / system_config.py etc...and configure/build options, i >>>>>> will try the same? >>>>>> >>>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >>>>>> >>>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. On >>>>>>> my system with MKL it says: >>>>>>> >>>>>>> ldd linalg/lapack_lite.so >>>>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>> fraude de la part de "10.1.3.027" MailScanner warning: numerical >>>>>>> links are often malicious: 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner >>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner warning: >>>>>>> numerical links are often malicious: >>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner warning: >>>>>>> numerical links are often malicious: >>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>> fraude de la part de "10.1.3.027" MailScanner warning: numerical >>>>>>> links are often malicious: 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>> fraude de la part de "10.1.3.027" MailScanner warning: numerical >>>>>>> links are often malicious: 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>>>> >>>>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>>> (0x00002acf10a03000) >>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>> (0x00002acf10ba6000) >>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>>> libpython2.6.so.1.0 => >>>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>>> >>>>>>> So to clarify, what I was trying to achieve is find which of numpy's >>>>>>> shared libraries was trying to load libmkl_lapack.so, and if it could be >>>>>>> found properly with ldd. Just to make sure it's not another .so, can you >>>>>>> run from within your numpy directory: >>>>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>>> >>>>>>> If the output is not empty, try to run ldd on all .so within numpy >>>>>>> to see which one is linked against libmkl_lapack.so. Then hopefully >>>>>>> something will stand out (I admit that's not guaranteed though ;). >>>>>>> >>>>>>> >>>>>>> -=- Olivier >>>>>>> >>>>>>> 2011/11/2 akshar bhosale >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> ldd >>>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) >>>>>>>> libmkl_def.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>> (0x00002b1199653000) >>>>>>>> libmkl_intel_lp64.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>> (0x00002b119a1a8000) >>>>>>>> libmkl_intel_thread.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>> (0x00002b119a503000) >>>>>>>> libmkl_core.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>> (0x00002b119b22c000) >>>>>>>> libmkl_mc.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>>>>>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>>>>>> (0x00002b119c392000) >>>>>>>> libsvml.so => >>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>>> libiomp5.so => >>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>>> libintlc.so.5 => >>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: >>>>>>>> >>>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about >>>>>>>>> ldd numpy/linalg/lapack_lite.so? >>>>>>>>> >>>>>>>>> >>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> ldd _dotblas.so >>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>>>>>>> libmkl_def.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>> (0x00002b12f099c000) >>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>> (0x00002b12f14f1000) >>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>> (0x00002b12f184c000) >>>>>>>>>> libmkl_core.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>> (0x00002b12f2575000) >>>>>>>>>> libmkl_mc.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >>>>>>>>>> libimf.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>>> libsvml.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>>> libiomp5.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>>> libintlc.so.5 => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau >>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -=- Olivier >>>>>>>>>>> >>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>> >>>>>>>>>>>> HI, >>>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>>> generating the same error. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau < >>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>>>>>>>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>>> >>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>> >>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, >>>>>>>>>>>>>> 3j, 4]], >>>>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>>> >>>>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>>> i have intel cluster toolkit installed on my system. >>>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>>> platform. >>>>>>>>>>>>>> Kindly help >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 22:32:43 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 08:02:43 +0530 Subject: [Numpy-discussion] Fwd: numpy error with mkl 10.1 In-Reply-To: References: Message-ID: my numpy.test hangs here Test whether equivalent subarray dtypes hash the same. ... ok Test whether different subarray dtypes hash differently. ... ok Test some data types that are equal ... ok Test some more complicated cases that shouldn't be equal ... ok Test some simple cases that shouldn't be equal ... ok test_single_subarray (test_dtype.TestSubarray) ... ok test_einsum_errors (test_einsum.TestEinSum) ... ok test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... ---------- Forwarded message ---------- From: akshar bhosale Date: Thu, Nov 3, 2011 at 8:00 AM Subject: Re: [Numpy-discussion] numpy error with mkl 10.1 To: Discussion of Numerical Python hi, on your machine, is numpy.test passed? On Thu, Nov 3, 2011 at 6:54 AM, Olivier Delalleau wrote: > I believe they are optional, but I'm not sure. > > Note that this system I copied the distutils from is not one I setup > myself so I'm not sure how numpy was installed exactly. I just know it > works and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462 @ > 2.80GHz (x86_64 as well). > > -=- Olivier > > 2011/11/2 akshar bhosale > >> hi, >> are amd,fftw and umfpack required? my architecture is : intel xeon >> x_86_64. >> >> On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau wrote: >> >>> It's inside the distutils folder. >>> >>> -=- Olivier >>> >>> 2011/11/2 akshar bhosale >>> >>>> thanks..what about site.cfg? >>>> >>>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: >>>> >>>>> Sorry, no clue :/ >>>>> >>>>> I made a tarball with my distutils folder here: >>>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>>> Hope this helps... >>>>> >>>>> -=- Olivier >>>>> >>>>> 2011/11/2 akshar bhosale >>>>> >>>>>> hi, >>>>>> thanks for the reply >>>>>> none of them is linked with libmkl_lapack.so. >>>>>> what can be the problem? >>>>>> libmkl_lapack.so is present in its location. >>>>>> >>>>>> if possible, can you send your site.cfg and other related files like >>>>>> intelccompilers.py / system_config.py etc...and configure/build options, i >>>>>> will try the same? >>>>>> >>>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >>>>>> >>>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. On >>>>>>> my system with MKL it says: >>>>>>> >>>>>>> ldd linalg/lapack_lite.so >>>>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>> fraude de la part de "10.1.3.027" MailScanner warning: numerical >>>>>>> links are often malicious: 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner >>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner warning: >>>>>>> numerical links are often malicious: >>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner warning: >>>>>>> numerical links are often malicious: >>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>> fraude de la part de "10.1.3.027" MailScanner warning: numerical >>>>>>> links are often malicious: 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>> fraude de la part de "10.1.3.027" MailScanner warning: numerical >>>>>>> links are often malicious: 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>>>> >>>>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>>> (0x00002acf10a03000) >>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>> (0x00002acf10ba6000) >>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>>> libpython2.6.so.1.0 => >>>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>>> >>>>>>> So to clarify, what I was trying to achieve is find which of numpy's >>>>>>> shared libraries was trying to load libmkl_lapack.so, and if it could be >>>>>>> found properly with ldd. Just to make sure it's not another .so, can you >>>>>>> run from within your numpy directory: >>>>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>>> >>>>>>> If the output is not empty, try to run ldd on all .so within numpy >>>>>>> to see which one is linked against libmkl_lapack.so. Then hopefully >>>>>>> something will stand out (I admit that's not guaranteed though ;). >>>>>>> >>>>>>> >>>>>>> -=- Olivier >>>>>>> >>>>>>> 2011/11/2 akshar bhosale >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> ldd >>>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b1199349000) >>>>>>>> libmkl_def.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>> (0x00002b1199653000) >>>>>>>> libmkl_intel_lp64.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>> (0x00002b119a1a8000) >>>>>>>> libmkl_intel_thread.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>> (0x00002b119a503000) >>>>>>>> libmkl_core.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>> (0x00002b119b22c000) >>>>>>>> libmkl_mc.so => >>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>>>>>>> libimf.so => /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so >>>>>>>> (0x00002b119c392000) >>>>>>>> libsvml.so => >>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>>> libiomp5.so => >>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>>> libintlc.so.5 => >>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: >>>>>>>> >>>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about >>>>>>>>> ldd numpy/linalg/lapack_lite.so? >>>>>>>>> >>>>>>>>> >>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> ldd _dotblas.so >>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>>>>>>> libmkl_def.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>> (0x00002b12f099c000) >>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>> (0x00002b12f14f1000) >>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>> (0x00002b12f184c000) >>>>>>>>>> libmkl_core.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>> (0x00002b12f2575000) >>>>>>>>>> libmkl_mc.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b12f34bf000) >>>>>>>>>> libimf.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>>> libsvml.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>>> libiomp5.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>>> libintlc.so.5 => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau >>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -=- Olivier >>>>>>>>>>> >>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>> >>>>>>>>>>>> HI, >>>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>>> generating the same error. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau < >>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory that >>>>>>>>>>>>> contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>>> >>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>> >>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, >>>>>>>>>>>>>> 3j, 4]], >>>>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>>> >>>>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>>> i have intel cluster toolkit installed on my system. >>>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>>> platform. >>>>>>>>>>>>>> Kindly help >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 2 22:45:23 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 2 Nov 2011 22:45:23 -0400 Subject: [Numpy-discussion] Fwd: numpy error with mkl 10.1 In-Reply-To: References: Message-ID: Yes they pass: Ran 2030 tests in 6.672s OK (KNOWNFAIL=1) Google "einsum hang".... I know you're not the first one with this bug (but I'm not sure exactly how to fix it). -=- Olivier 2011/11/2 akshar bhosale > my numpy.test hangs here > > Test whether equivalent subarray dtypes hash the same. ... ok > Test whether different subarray dtypes hash differently. ... ok > Test some data types that are equal ... ok > Test some more complicated cases that shouldn't be equal ... ok > Test some simple cases that shouldn't be equal ... ok > test_single_subarray (test_dtype.TestSubarray) ... ok > test_einsum_errors (test_einsum.TestEinSum) ... ok > test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... > > > ---------- Forwarded message ---------- > From: akshar bhosale > Date: Thu, Nov 3, 2011 at 8:00 AM > Subject: Re: [Numpy-discussion] numpy error with mkl 10.1 > To: Discussion of Numerical Python > > > hi, > on your machine, is numpy.test passed? > > > On Thu, Nov 3, 2011 at 6:54 AM, Olivier Delalleau wrote: > >> I believe they are optional, but I'm not sure. >> >> Note that this system I copied the distutils from is not one I setup >> myself so I'm not sure how numpy was installed exactly. I just know it >> works and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462 @ >> 2.80GHz (x86_64 as well). >> >> -=- Olivier >> >> 2011/11/2 akshar bhosale >> >>> hi, >>> are amd,fftw and umfpack required? my architecture is : intel xeon >>> x_86_64. >>> >>> On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau wrote: >>> >>>> It's inside the distutils folder. >>>> >>>> -=- Olivier >>>> >>>> 2011/11/2 akshar bhosale >>>> >>>>> thanks..what about site.cfg? >>>>> >>>>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: >>>>> >>>>>> Sorry, no clue :/ >>>>>> >>>>>> I made a tarball with my distutils folder here: >>>>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>>>> Hope this helps... >>>>>> >>>>>> -=- Olivier >>>>>> >>>>>> 2011/11/2 akshar bhosale >>>>>> >>>>>>> hi, >>>>>>> thanks for the reply >>>>>>> none of them is linked with libmkl_lapack.so. >>>>>>> what can be the problem? >>>>>>> libmkl_lapack.so is present in its location. >>>>>>> >>>>>>> if possible, can you send your site.cfg and other related files like >>>>>>> intelccompilers.py / system_config.py etc...and configure/build options, i >>>>>>> will try the same? >>>>>>> >>>>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >>>>>>> >>>>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. On >>>>>>>> my system with MKL it says: >>>>>>>> >>>>>>>> ldd linalg/lapack_lite.so >>>>>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>> warning: numerical links are often malicious: >>>>>>>> 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner >>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>> warning: numerical links are often malicious: >>>>>>>> 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>> warning: numerical links are often malicious: >>>>>>>> 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>>>>> >>>>>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>>>> (0x00002acf10a03000) >>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>> (0x00002acf10ba6000) >>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>>>> libpython2.6.so.1.0 => >>>>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>>>> >>>>>>>> So to clarify, what I was trying to achieve is find which of >>>>>>>> numpy's shared libraries was trying to load libmkl_lapack.so, and if it >>>>>>>> could be found properly with ldd. Just to make sure it's not another .so, >>>>>>>> can you run from within your numpy directory: >>>>>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>>>> >>>>>>>> If the output is not empty, try to run ldd on all .so within numpy >>>>>>>> to see which one is linked against libmkl_lapack.so. Then hopefully >>>>>>>> something will stand out (I admit that's not guaranteed though ;). >>>>>>>> >>>>>>>> >>>>>>>> -=- Olivier >>>>>>>> >>>>>>>> 2011/11/2 akshar bhosale >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> ldd >>>>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>> (0x00002b1199349000) >>>>>>>>> libmkl_def.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>> (0x00002b1199653000) >>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>> (0x00002b119a1a8000) >>>>>>>>> libmkl_intel_thread.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>> (0x00002b119a503000) >>>>>>>>> libmkl_core.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>> (0x00002b119b22c000) >>>>>>>>> libmkl_mc.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>>>>>>>> libimf.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b119c392000) >>>>>>>>> libsvml.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>>>> libiomp5.so => >>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>>>> libintlc.so.5 => >>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau wrote: >>>>>>>>> >>>>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about >>>>>>>>>> ldd numpy/linalg/lapack_lite.so? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> ldd _dotblas.so >>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>>>>>>>> libmkl_def.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>> (0x00002b12f099c000) >>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>> (0x00002b12f14f1000) >>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>> (0x00002b12f184c000) >>>>>>>>>>> libmkl_core.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>> (0x00002b12f2575000) >>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>> (0x00002b12f34bf000) >>>>>>>>>>> libimf.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>>>> libsvml.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>>>> libiomp5.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau < >>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>> >>>>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -=- Olivier >>>>>>>>>>>> >>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>> >>>>>>>>>>>>> HI, >>>>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>>>> generating the same error. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau < >>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory >>>>>>>>>>>>>> that contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, >>>>>>>>>>>>>>> 3j, 4]], >>>>>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>>>> i have intel cluster toolkit installed on my system. >>>>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>>>> platform. >>>>>>>>>>>>>>> Kindly help >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Wed Nov 2 23:20:15 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 2 Nov 2011 22:20:15 -0500 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: Message-ID: On Wednesday, November 2, 2011, Nathaniel Smith wrote: > Hi Benjamin, > > On Wed, Nov 2, 2011 at 5:25 PM, Benjamin Root wrote: >> I want to pare this down even more. I think the above lists makes too many >> unneeded extrapolations. > > Okay. I found your formatting a little confusing, so I want to make > sure I understood the changes you're suggesting: > > For the description of what MISSING means, you removed the lines: > - Compatibility with R is valuable > - To avoid user confusion, ideally it should *not* be possible to > 'unmask' a missing value, since this is inconsistent with the "missing > value" metaphor (e.g., see Wes's comment about "leaky abstractions") > > And you added the line: > + Assigning MISSING is destructive > > And for the description of what IGNORED means, you removed the lines: > - Some memory overhead is inevitable and acceptable > - Compatibility with R neither possible nor valuable > - Ability to toggle the IGNORED state of a location is critical, and > should be as convenient as possible > > And you added the lines: > + IGNORE is non-destructive > + Must be competitive with np.ma for speed and memory (or else users > would just use np.ma) > > Is that right? Correct. > > Assuming it is, my thoughts are: > > By R compatibility, I specifically had in mind in-memory > compatibility. rpy2 provides a more-or-less seamless within-process > interface between R and Python (and specifically lets you get numpy > views on arrays returned by R functions), so if we can make this work > for R arrays containing NA too then that'd be handy. (The rpy2 author > requested this in the last discussion here: > http://mail.scipy.org/pipermail/numpy-discussion/2011-June/057084.html) > When it comes to disk formats, then this doesn't matter so much, since > IO routines have to translate between different representations all > the time anyway. > Interesting, but I still have to wonder if that should be on the wishlist for MISSING. I guess it would matter by knowing whether people would be fully converting from R or gradually transitioning from it? That is something that I can't answer. > I take the replacement of my line about MISSING disallowing unmasking > and your line about MISSING assignment being destructive as basically > expressing the same idea. Is that fair, or did you mean something > else? I am someone who wants to get to the absolute core of ideas. Also, this expression cleanly delineates the differences as binary. By expressing it this way, we also shy away from implementation details. For example, Unmasking can be programmatically prevented for MISSING while it could be implemented by other indirect means for IGNORE. Not that those are the preferred ways, only that the phrasing is more flexible and exacting. > > Finally, do you think that people who want IGNORED support care about > having a convenient API for masking/unmasking values? You removed that > line, but I don't know if that was because you disagreed with it, or > were just trying to simplify. See previous. > >> Then, as a third-party module developer, I can tell you that having separate >> and independent ways to detect "MISSING"/"IGNORED" would likely make support >> more difficult and would greatly benefit from a common (or easily >> combinable) method of identification. > > Right, sorry... I didn't forget, and that's part of what I was > thinking when I described the second approach as keeping them as > *mostly*-separate interfaces... but I should have made it more > explicit! Anyway, yes: > > 4) There is consensus that whatever approach is taken, there should be > a quick and convenient way to identify values that are MISSING, > IGNORED, or both. (E.g., functions is_MISSING, is_IGNORED, > is_MISSING_or_IGNORED, or some equivalent.) > Good. Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 23:23:26 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 08:53:26 +0530 Subject: [Numpy-discussion] Fwd: numpy error with mkl 10.1 In-Reply-To: References: Message-ID: thanks once again. how can i use ur distutils directly (ur supplied files etc) On Thu, Nov 3, 2011 at 8:15 AM, Olivier Delalleau wrote: > Yes they pass: > > Ran 2030 tests in 6.672s > > OK (KNOWNFAIL=1) > > > Google "einsum hang".... I know you're not the first one with this bug > (but I'm not sure exactly how to fix it). > > -=- Olivier > > 2011/11/2 akshar bhosale > >> my numpy.test hangs here >> >> Test whether equivalent subarray dtypes hash the same. ... ok >> Test whether different subarray dtypes hash differently. ... ok >> Test some data types that are equal ... ok >> Test some more complicated cases that shouldn't be equal ... ok >> Test some simple cases that shouldn't be equal ... ok >> test_single_subarray (test_dtype.TestSubarray) ... ok >> test_einsum_errors (test_einsum.TestEinSum) ... ok >> test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... >> >> >> ---------- Forwarded message ---------- >> From: akshar bhosale >> Date: Thu, Nov 3, 2011 at 8:00 AM >> Subject: Re: [Numpy-discussion] numpy error with mkl 10.1 >> To: Discussion of Numerical Python >> >> >> hi, >> on your machine, is numpy.test passed? >> >> >> On Thu, Nov 3, 2011 at 6:54 AM, Olivier Delalleau wrote: >> >>> I believe they are optional, but I'm not sure. >>> >>> Note that this system I copied the distutils from is not one I setup >>> myself so I'm not sure how numpy was installed exactly. I just know it >>> works and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462 @ >>> 2.80GHz (x86_64 as well). >>> >>> -=- Olivier >>> >>> 2011/11/2 akshar bhosale >>> >>>> hi, >>>> are amd,fftw and umfpack required? my architecture is : intel xeon >>>> x_86_64. >>>> >>>> On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau wrote: >>>> >>>>> It's inside the distutils folder. >>>>> >>>>> -=- Olivier >>>>> >>>>> 2011/11/2 akshar bhosale >>>>> >>>>>> thanks..what about site.cfg? >>>>>> >>>>>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: >>>>>> >>>>>>> Sorry, no clue :/ >>>>>>> >>>>>>> I made a tarball with my distutils folder here: >>>>>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>>>>> Hope this helps... >>>>>>> >>>>>>> -=- Olivier >>>>>>> >>>>>>> 2011/11/2 akshar bhosale >>>>>>> >>>>>>>> hi, >>>>>>>> thanks for the reply >>>>>>>> none of them is linked with libmkl_lapack.so. >>>>>>>> what can be the problem? >>>>>>>> libmkl_lapack.so is present in its location. >>>>>>>> >>>>>>>> if possible, can you send your site.cfg and other related files >>>>>>>> like intelccompilers.py / system_config.py etc...and configure/build >>>>>>>> options, i will try the same? >>>>>>>> >>>>>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >>>>>>>> >>>>>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. On >>>>>>>>> my system with MKL it says: >>>>>>>>> >>>>>>>>> ldd linalg/lapack_lite.so >>>>>>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>> warning: numerical links are often malicious: >>>>>>>>> 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>>>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner >>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>>>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>>>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>> warning: numerical links are often malicious: >>>>>>>>> 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>>>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>> warning: numerical links are often malicious: >>>>>>>>> 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>>>>>> >>>>>>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>>>>> (0x00002acf10a03000) >>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>> (0x00002acf10ba6000) >>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>>>>> libpython2.6.so.1.0 => >>>>>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>>>>> >>>>>>>>> So to clarify, what I was trying to achieve is find which of >>>>>>>>> numpy's shared libraries was trying to load libmkl_lapack.so, and if it >>>>>>>>> could be found properly with ldd. Just to make sure it's not another .so, >>>>>>>>> can you run from within your numpy directory: >>>>>>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>>>>> >>>>>>>>> If the output is not empty, try to run ldd on all .so within numpy >>>>>>>>> to see which one is linked against libmkl_lapack.so. Then hopefully >>>>>>>>> something will stand out (I admit that's not guaranteed though ;). >>>>>>>>> >>>>>>>>> >>>>>>>>> -=- Olivier >>>>>>>>> >>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> ldd >>>>>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>> (0x00002b1199349000) >>>>>>>>>> libmkl_def.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>> (0x00002b1199653000) >>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>> (0x00002b119a1a8000) >>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>> (0x00002b119a503000) >>>>>>>>>> libmkl_core.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>> (0x00002b119b22c000) >>>>>>>>>> libmkl_mc.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>>>>>>>>> libimf.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b119c392000) >>>>>>>>>> libsvml.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>>>>> libiomp5.so => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>>>>> libintlc.so.5 => >>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau >>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about >>>>>>>>>>> ldd numpy/linalg/lapack_lite.so? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> ldd _dotblas.so >>>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>>>>>>>>> libmkl_def.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>>> (0x00002b12f099c000) >>>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>>> (0x00002b12f14f1000) >>>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>>> (0x00002b12f184c000) >>>>>>>>>>>> libmkl_core.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>>> (0x00002b12f2575000) >>>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>>> (0x00002b12f34bf000) >>>>>>>>>>>> libimf.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>>>>> libsvml.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>>>>> libiomp5.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau < >>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>> >>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>> >>>>>>>>>>>>>> HI, >>>>>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>>>>> generating the same error. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau < >>>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory >>>>>>>>>>>>>>> that contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, >>>>>>>>>>>>>>>> 3j, 4]], >>>>>>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>>>>> i have intel cluster toolkit installed on my system. >>>>>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>>>>> platform. >>>>>>>>>>>>>>>> Kindly help >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 2 23:28:34 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 2 Nov 2011 23:28:34 -0400 Subject: [Numpy-discussion] Fwd: numpy error with mkl 10.1 In-Reply-To: References: Message-ID: I'm sorry, I don't know. I'm not at all an expert about numpy installation, I just happened to have access to a computer with a working install of numpy with MKL, so I thought it might help... but I'm afraid that's all I can do for you :/ -=- Olivier 2011/11/2 akshar bhosale > thanks once again. how can i use ur distutils directly (ur supplied files > etc) > > On Thu, Nov 3, 2011 at 8:15 AM, Olivier Delalleau wrote: > >> Yes they pass: >> >> Ran 2030 tests in 6.672s >> >> OK (KNOWNFAIL=1) >> >> >> Google "einsum hang".... I know you're not the first one with this bug >> (but I'm not sure exactly how to fix it). >> >> -=- Olivier >> >> 2011/11/2 akshar bhosale >> >>> my numpy.test hangs here >>> >>> Test whether equivalent subarray dtypes hash the same. ... ok >>> Test whether different subarray dtypes hash differently. ... ok >>> Test some data types that are equal ... ok >>> Test some more complicated cases that shouldn't be equal ... ok >>> Test some simple cases that shouldn't be equal ... ok >>> test_single_subarray (test_dtype.TestSubarray) ... ok >>> test_einsum_errors (test_einsum.TestEinSum) ... ok >>> test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... >>> >>> >>> ---------- Forwarded message ---------- >>> From: akshar bhosale >>> Date: Thu, Nov 3, 2011 at 8:00 AM >>> Subject: Re: [Numpy-discussion] numpy error with mkl 10.1 >>> To: Discussion of Numerical Python >>> >>> >>> hi, >>> on your machine, is numpy.test passed? >>> >>> >>> On Thu, Nov 3, 2011 at 6:54 AM, Olivier Delalleau wrote: >>> >>>> I believe they are optional, but I'm not sure. >>>> >>>> Note that this system I copied the distutils from is not one I setup >>>> myself so I'm not sure how numpy was installed exactly. I just know it >>>> works and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462 @ >>>> 2.80GHz (x86_64 as well). >>>> >>>> -=- Olivier >>>> >>>> 2011/11/2 akshar bhosale >>>> >>>>> hi, >>>>> are amd,fftw and umfpack required? my architecture is : intel xeon >>>>> x_86_64. >>>>> >>>>> On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau wrote: >>>>> >>>>>> It's inside the distutils folder. >>>>>> >>>>>> -=- Olivier >>>>>> >>>>>> 2011/11/2 akshar bhosale >>>>>> >>>>>>> thanks..what about site.cfg? >>>>>>> >>>>>>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: >>>>>>> >>>>>>>> Sorry, no clue :/ >>>>>>>> >>>>>>>> I made a tarball with my distutils folder here: >>>>>>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>>>>>> Hope this helps... >>>>>>>> >>>>>>>> -=- Olivier >>>>>>>> >>>>>>>> 2011/11/2 akshar bhosale >>>>>>>> >>>>>>>>> hi, >>>>>>>>> thanks for the reply >>>>>>>>> none of them is linked with libmkl_lapack.so. >>>>>>>>> what can be the problem? >>>>>>>>> libmkl_lapack.so is present in its location. >>>>>>>>> >>>>>>>>> if possible, can you send your site.cfg and other related files >>>>>>>>> like intelccompilers.py / system_config.py etc...and configure/build >>>>>>>>> options, i will try the same? >>>>>>>>> >>>>>>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >>>>>>>>> >>>>>>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. >>>>>>>>>> On my system with MKL it says: >>>>>>>>>> >>>>>>>>>> ldd linalg/lapack_lite.so >>>>>>>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>>>>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>>>>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>>>>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>>>>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>>>>>>> >>>>>>>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>>>>>> (0x00002acf10a03000) >>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>> (0x00002acf10ba6000) >>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>>>>>> libpython2.6.so.1.0 => >>>>>>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>>>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>>>>>> >>>>>>>>>> So to clarify, what I was trying to achieve is find which of >>>>>>>>>> numpy's shared libraries was trying to load libmkl_lapack.so, and if it >>>>>>>>>> could be found properly with ldd. Just to make sure it's not another .so, >>>>>>>>>> can you run from within your numpy directory: >>>>>>>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>>>>>> >>>>>>>>>> If the output is not empty, try to run ldd on all .so within >>>>>>>>>> numpy to see which one is linked against libmkl_lapack.so. Then hopefully >>>>>>>>>> something will stand out (I admit that's not guaranteed though ;). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -=- Olivier >>>>>>>>>> >>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> ldd >>>>>>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>>> (0x00002b1199349000) >>>>>>>>>>> libmkl_def.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>> (0x00002b1199653000) >>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>> (0x00002b119a1a8000) >>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>> (0x00002b119a503000) >>>>>>>>>>> libmkl_core.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>> (0x00002b119b22c000) >>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>> (0x00002b119c176000) >>>>>>>>>>> libimf.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b119c392000) >>>>>>>>>>> libsvml.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>>>>>> libiomp5.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau < >>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>> >>>>>>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about >>>>>>>>>>>> ldd numpy/linalg/lapack_lite.so? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> ldd _dotblas.so >>>>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>>>>> (0x00002b12f0692000) >>>>>>>>>>>>> libmkl_def.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>>>> (0x00002b12f099c000) >>>>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>>>> (0x00002b12f14f1000) >>>>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>>>> (0x00002b12f184c000) >>>>>>>>>>>>> libmkl_core.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>>>> (0x00002b12f2575000) >>>>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>>>> (0x00002b12f34bf000) >>>>>>>>>>>>> libimf.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>>>>>> libsvml.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>>>>>> libiomp5.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau < >>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>> >>>>>>>>>>>>>>> HI, >>>>>>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>>>>>> generating the same error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau < >>>>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory >>>>>>>>>>>>>>>> that contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, >>>>>>>>>>>>>>>>> 3j, 4]], >>>>>>>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>>>>>> i have intel cluster toolkit installed on my system. >>>>>>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>>>>>> platform. >>>>>>>>>>>>>>>>> Kindly help >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 23:32:45 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 09:02:45 +0530 Subject: [Numpy-discussion] Fwd: numpy error with mkl 10.1 In-Reply-To: References: Message-ID: thank you very much for your kind help. On Thu, Nov 3, 2011 at 8:58 AM, Olivier Delalleau wrote: > I'm sorry, I don't know. I'm not at all an expert about numpy > installation, I just happened to have access to a computer with a working > install of numpy with MKL, so I thought it might help... but I'm afraid > that's all I can do for you :/ > > -=- Olivier > > 2011/11/2 akshar bhosale > >> thanks once again. how can i use ur distutils directly (ur supplied files >> etc) >> >> On Thu, Nov 3, 2011 at 8:15 AM, Olivier Delalleau wrote: >> >>> Yes they pass: >>> >>> Ran 2030 tests in 6.672s >>> >>> OK (KNOWNFAIL=1) >>> >>> >>> Google "einsum hang".... I know you're not the first one with this bug >>> (but I'm not sure exactly how to fix it). >>> >>> -=- Olivier >>> >>> 2011/11/2 akshar bhosale >>> >>>> my numpy.test hangs here >>>> >>>> Test whether equivalent subarray dtypes hash the same. ... ok >>>> Test whether different subarray dtypes hash differently. ... ok >>>> Test some data types that are equal ... ok >>>> Test some more complicated cases that shouldn't be equal ... ok >>>> Test some simple cases that shouldn't be equal ... ok >>>> test_single_subarray (test_dtype.TestSubarray) ... ok >>>> test_einsum_errors (test_einsum.TestEinSum) ... ok >>>> test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... >>>> >>>> >>>> ---------- Forwarded message ---------- >>>> From: akshar bhosale >>>> Date: Thu, Nov 3, 2011 at 8:00 AM >>>> Subject: Re: [Numpy-discussion] numpy error with mkl 10.1 >>>> To: Discussion of Numerical Python >>>> >>>> >>>> hi, >>>> on your machine, is numpy.test passed? >>>> >>>> >>>> On Thu, Nov 3, 2011 at 6:54 AM, Olivier Delalleau wrote: >>>> >>>>> I believe they are optional, but I'm not sure. >>>>> >>>>> Note that this system I copied the distutils from is not one I setup >>>>> myself so I'm not sure how numpy was installed exactly. I just know it >>>>> works and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462 @ >>>>> 2.80GHz (x86_64 as well). >>>>> >>>>> -=- Olivier >>>>> >>>>> 2011/11/2 akshar bhosale >>>>> >>>>>> hi, >>>>>> are amd,fftw and umfpack required? my architecture is : intel xeon >>>>>> x_86_64. >>>>>> >>>>>> On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau wrote: >>>>>> >>>>>>> It's inside the distutils folder. >>>>>>> >>>>>>> -=- Olivier >>>>>>> >>>>>>> 2011/11/2 akshar bhosale >>>>>>> >>>>>>>> thanks..what about site.cfg? >>>>>>>> >>>>>>>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: >>>>>>>> >>>>>>>>> Sorry, no clue :/ >>>>>>>>> >>>>>>>>> I made a tarball with my distutils folder here: >>>>>>>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>>>>>>> Hope this helps... >>>>>>>>> >>>>>>>>> -=- Olivier >>>>>>>>> >>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>> >>>>>>>>>> hi, >>>>>>>>>> thanks for the reply >>>>>>>>>> none of them is linked with libmkl_lapack.so. >>>>>>>>>> what can be the problem? >>>>>>>>>> libmkl_lapack.so is present in its location. >>>>>>>>>> >>>>>>>>>> if possible, can you send your site.cfg and other related files >>>>>>>>>> like intelccompilers.py / system_config.py etc...and configure/build >>>>>>>>>> options, i will try the same? >>>>>>>>>> >>>>>>>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau >>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. >>>>>>>>>>> On my system with MKL it says: >>>>>>>>>>> >>>>>>>>>>> ldd linalg/lapack_lite.so >>>>>>>>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre >>>>>>>>>>> une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>>>>>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre >>>>>>>>>>> une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>>>>>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre >>>>>>>>>>> une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>>>>>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative >>>>>>>>>>> de fraude de la part de "10.1.3.027" MailScanner soup?onne le >>>>>>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>>>>>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative >>>>>>>>>>> de fraude de la part de "10.1.3.027" MailScanner soup?onne le >>>>>>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>>>>>>>> >>>>>>>>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>>>>>>> (0x00002acf10a03000) >>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>> (0x00002acf10ba6000) >>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>>>>>>> libpython2.6.so.1.0 => >>>>>>>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>>>>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>>>>>>> >>>>>>>>>>> So to clarify, what I was trying to achieve is find which of >>>>>>>>>>> numpy's shared libraries was trying to load libmkl_lapack.so, and if it >>>>>>>>>>> could be found properly with ldd. Just to make sure it's not another .so, >>>>>>>>>>> can you run from within your numpy directory: >>>>>>>>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>>>>>>> >>>>>>>>>>> If the output is not empty, try to run ldd on all .so within >>>>>>>>>>> numpy to see which one is linked against libmkl_lapack.so. Then hopefully >>>>>>>>>>> something will stand out (I admit that's not guaranteed though ;). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -=- Olivier >>>>>>>>>>> >>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> ldd >>>>>>>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>>>> (0x00002b1199349000) >>>>>>>>>>>> libmkl_def.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>>> (0x00002b1199653000) >>>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>>> (0x00002b119a1a8000) >>>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>>> (0x00002b119a503000) >>>>>>>>>>>> libmkl_core.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>>> (0x00002b119b22c000) >>>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>>> (0x00002b119c176000) >>>>>>>>>>>> libimf.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b119c392000) >>>>>>>>>>>> libsvml.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>>>>>>> libiomp5.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau < >>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what >>>>>>>>>>>>> about ldd numpy/linalg/lapack_lite.so? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> ldd _dotblas.so >>>>>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>>>>>> (0x00002b12f0692000) >>>>>>>>>>>>>> libmkl_def.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>>>>> (0x00002b12f099c000) >>>>>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>>>>> (0x00002b12f14f1000) >>>>>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>>>>> (0x00002b12f184c000) >>>>>>>>>>>>>> libmkl_core.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>>>>> (0x00002b12f2575000) >>>>>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>>>>> (0x00002b12f34bf000) >>>>>>>>>>>>>> libimf.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>>>>>>> libsvml.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>>>>>>> libiomp5.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau < >>>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> HI, >>>>>>>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>>>>>>> generating the same error. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau < >>>>>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory >>>>>>>>>>>>>>>>> that contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], >>>>>>>>>>>>>>>>>> [1.0, 3j, 4]], >>>>>>>>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>>>>>>> i have intel cluster toolkit installed on my system. >>>>>>>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>>>>>>> platform. >>>>>>>>>>>>>>>>>> Kindly help >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Wed Nov 2 23:33:44 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 09:03:44 +0530 Subject: [Numpy-discussion] Fwd: Fwd: numpy error with mkl 10.1 In-Reply-To: References: Message-ID: any other means of getting it fixed? ---------- Forwarded message ---------- From: Olivier Delalleau Date: Thu, Nov 3, 2011 at 8:58 AM Subject: Re: [Numpy-discussion] Fwd: numpy error with mkl 10.1 To: Discussion of Numerical Python I'm sorry, I don't know. I'm not at all an expert about numpy installation, I just happened to have access to a computer with a working install of numpy with MKL, so I thought it might help... but I'm afraid that's all I can do for you :/ -=- Olivier 2011/11/2 akshar bhosale > thanks once again. how can i use ur distutils directly (ur supplied files > etc) > > On Thu, Nov 3, 2011 at 8:15 AM, Olivier Delalleau wrote: > >> Yes they pass: >> >> Ran 2030 tests in 6.672s >> >> OK (KNOWNFAIL=1) >> >> >> Google "einsum hang".... I know you're not the first one with this bug >> (but I'm not sure exactly how to fix it). >> >> -=- Olivier >> >> 2011/11/2 akshar bhosale >> >>> my numpy.test hangs here >>> >>> Test whether equivalent subarray dtypes hash the same. ... ok >>> Test whether different subarray dtypes hash differently. ... ok >>> Test some data types that are equal ... ok >>> Test some more complicated cases that shouldn't be equal ... ok >>> Test some simple cases that shouldn't be equal ... ok >>> test_single_subarray (test_dtype.TestSubarray) ... ok >>> test_einsum_errors (test_einsum.TestEinSum) ... ok >>> test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... >>> >>> >>> ---------- Forwarded message ---------- >>> From: akshar bhosale >>> Date: Thu, Nov 3, 2011 at 8:00 AM >>> Subject: Re: [Numpy-discussion] numpy error with mkl 10.1 >>> To: Discussion of Numerical Python >>> >>> >>> hi, >>> on your machine, is numpy.test passed? >>> >>> >>> On Thu, Nov 3, 2011 at 6:54 AM, Olivier Delalleau wrote: >>> >>>> I believe they are optional, but I'm not sure. >>>> >>>> Note that this system I copied the distutils from is not one I setup >>>> myself so I'm not sure how numpy was installed exactly. I just know it >>>> works and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462 @ >>>> 2.80GHz (x86_64 as well). >>>> >>>> -=- Olivier >>>> >>>> 2011/11/2 akshar bhosale >>>> >>>>> hi, >>>>> are amd,fftw and umfpack required? my architecture is : intel xeon >>>>> x_86_64. >>>>> >>>>> On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau wrote: >>>>> >>>>>> It's inside the distutils folder. >>>>>> >>>>>> -=- Olivier >>>>>> >>>>>> 2011/11/2 akshar bhosale >>>>>> >>>>>>> thanks..what about site.cfg? >>>>>>> >>>>>>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau wrote: >>>>>>> >>>>>>>> Sorry, no clue :/ >>>>>>>> >>>>>>>> I made a tarball with my distutils folder here: >>>>>>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>>>>>> Hope this helps... >>>>>>>> >>>>>>>> -=- Olivier >>>>>>>> >>>>>>>> 2011/11/2 akshar bhosale >>>>>>>> >>>>>>>>> hi, >>>>>>>>> thanks for the reply >>>>>>>>> none of them is linked with libmkl_lapack.so. >>>>>>>>> what can be the problem? >>>>>>>>> libmkl_lapack.so is present in its location. >>>>>>>>> >>>>>>>>> if possible, can you send your site.cfg and other related files >>>>>>>>> like intelccompilers.py / system_config.py etc...and configure/build >>>>>>>>> options, i will try the same? >>>>>>>>> >>>>>>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau wrote: >>>>>>>>> >>>>>>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. >>>>>>>>>> On my system with MKL it says: >>>>>>>>>> >>>>>>>>>> ldd linalg/lapack_lite.so >>>>>>>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>>>>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>>>>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>>>>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>>>>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>>>>>>> >>>>>>>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>>>>>> (0x00002acf10a03000) >>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>> (0x00002acf10ba6000) >>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>>>>>> libpython2.6.so.1.0 => >>>>>>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>>>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>>>>>> >>>>>>>>>> So to clarify, what I was trying to achieve is find which of >>>>>>>>>> numpy's shared libraries was trying to load libmkl_lapack.so, and if it >>>>>>>>>> could be found properly with ldd. Just to make sure it's not another .so, >>>>>>>>>> can you run from within your numpy directory: >>>>>>>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>>>>>> >>>>>>>>>> If the output is not empty, try to run ldd on all .so within >>>>>>>>>> numpy to see which one is linked against libmkl_lapack.so. Then hopefully >>>>>>>>>> something will stand out (I admit that's not guaranteed though ;). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -=- Olivier >>>>>>>>>> >>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> ldd >>>>>>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>>> (0x00002b1199349000) >>>>>>>>>>> libmkl_def.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>> (0x00002b1199653000) >>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>> (0x00002b119a1a8000) >>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>> (0x00002b119a503000) >>>>>>>>>>> libmkl_core.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>> (0x00002b119b22c000) >>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>> (0x00002b119c176000) >>>>>>>>>>> libimf.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b119c392000) >>>>>>>>>>> libsvml.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>>>>>> libiomp5.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau < >>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>> >>>>>>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about >>>>>>>>>>>> ldd numpy/linalg/lapack_lite.so? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> ldd _dotblas.so >>>>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>>>>> (0x00002b12f0692000) >>>>>>>>>>>>> libmkl_def.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>>>> (0x00002b12f099c000) >>>>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>>>> (0x00002b12f14f1000) >>>>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>>>> (0x00002b12f184c000) >>>>>>>>>>>>> libmkl_core.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>>>> (0x00002b12f2575000) >>>>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>>>> (0x00002b12f34bf000) >>>>>>>>>>>>> libimf.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>>>>>> libsvml.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>>>>>> libiomp5.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau < >>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>> >>>>>>>>>>>>>>> HI, >>>>>>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>>>>>> generating the same error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau < >>>>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory >>>>>>>>>>>>>>>> that contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, >>>>>>>>>>>>>>>>> 3j, 4]], >>>>>>>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>>>>>> i have intel cluster toolkit installed on my system. >>>>>>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>>>>>> platform. >>>>>>>>>>>>>>>>> Kindly help >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Nov 3 02:25:17 2011 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 2 Nov 2011 23:25:17 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: Message-ID: On Wed, Nov 2, 2011 at 8:20 PM, Benjamin Root wrote: > On Wednesday, November 2, 2011, Nathaniel Smith wrote: >> By R compatibility, I specifically had in mind in-memory >> compatibility. rpy2 provides a more-or-less seamless within-process >> interface between R and Python (and specifically lets you get numpy >> views on arrays returned by R functions), so if we can make this work >> for R arrays containing NA too then that'd be handy. (The rpy2 author >> requested this in the last discussion here: >> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/057084.html) >> When it comes to disk formats, then this doesn't matter so much, since >> IO routines have to translate between different representations all >> the time anyway. > > Interesting, but I still have to wonder if that should be on the wishlist > for MISSING. ?I guess it would matter by knowing whether people would be > fully converting from R or gradually transitioning from it? ?That is > something that I can't answer. Well, I'm one of the people who would use it, so yeah :-). I've been trying to standardize my code on Python for a while now, but there's a ton of statistical tools that are only really available through R, and that will remain true for a while yet. So I use rpy2 when I have to. >> I take the replacement of my line about MISSING disallowing unmasking >> and your line about MISSING assignment being destructive as basically >> expressing the same idea. Is that fair, or did you mean something >> else? > > I am someone who wants to get to the absolute core of ideas. Also, this > expression cleanly delineates the differences as binary. > > By expressing it this way, we also shy away from implementation details. For > example, Unmasking can be programmatically prevented for MISSING while it > could be implemented by other indirect means for IGNORE. Not that those are > the preferred ways, only that the phrasing is more flexible and exacting. > >> >> Finally, do you think that people who want IGNORED support care about >> having a convenient API for masking/unmasking values? You removed that >> line, but I don't know if that was because you disagreed with it, or >> were just trying to simplify. > > See previous. I like getting to the core of things too, but unless there's actual disagreement, then I think even less central points are still worth noting :-). I've tried editing things a bit to make the compare/contrast clearer based on your comments, and put it up here: https://github.com/njsmith/numpy/wiki/NA-discussion-status Maybe it would be better to split each list into core idea versus extra niceties or something? I'm not sure. -- Nathaniel From njs at pobox.com Thu Nov 3 02:27:20 2011 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 2 Nov 2011 23:27:20 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: Message-ID: I also mentioned this at the bottom of a reply to Benjamin, but to make sure people joining the thread see it: I went ahead and put this up on a github wiki page that everyone should be able to edit https://github.com/njsmith/numpy/wiki/NA-discussion-status We could move it to the numpy wiki or whatever if people prefer, this just seemed like the easiest way to get something up there that everyone would have write access to. -- Nathaniel On Wed, Nov 2, 2011 at 4:37 PM, Nathaniel Smith wrote: > Hi again, > > Okay, here's my attempt at an *uncontroversial* email! > > Specifically, I think it'll be easier to talk about this NA stuff if > we can establish some common ground, and easier for people to follow > if the basic points of agreement are laid out in one place. So I'm > going to try and summarize just the things that we can agree about. > > Note that right now I'm *only* talking about what kind of tools we > want to give the user -- i.e., what kind of problems we are trying to > solve. AFAICT we don't have as much consensus on implementation > matters, and anyway it's hard to make implementation decisions without > knowing what we're trying to accomplish. > > 1) I think we have consensus that there are (at least) two different > possible ways of thinking about this problem, with somewhat different > constituencies. Let's call these two concepts "MISSING data" and > "IGNORED data". > > 2) I also think we have at least a rough consensus on what these > concepts mean, and what their supporters want from them: > > MISSING data: > - Conceptually, MISSINGness acts like a property of a datum -- > assigning MISSING to a location is like assigning any other value to > that location > - Ufuncs and other operations must propagate these values by default, > and there must be an option to cause them to be ignored > - Must be competitive with NaNs in terms of speed and memory usage (or > else people will just use NaNs) > - Compatibility with R is valuable > - To avoid user confusion, ideally it should *not* be possible to > 'unmask' a missing value, since this is inconsistent with the "missing > value" metaphor (e.g., see Wes's comment about "leaky abstractions") > - Possible useful extension: having different classes of missing > values (similar to Stata) > - Target audience: data analysis with missing data, neuroimaging, > econometrics, former R users, ... > > IGNORED data: > - Conceptually, IGNOREDness acts like a property of the array -- > toggling a location to be IGNORED is kind of vaguely similar to > changing an array's shape > - Ufuncs and other operations must ignore these values by default, and > there doesn't really need to be a way to propagate them, even as an > option (though it probably wouldn't hurt either) > - Some memory overhead is inevitable and acceptable > - Compatibility with R neither possible nor valuable > - Ability to toggle the IGNORED state of a location is critical, and > should be as convenient as possible > - Possible useful extension: having not just different types of > ignored values, but richer ways to combine them -- e.g., the example > of combining astronomical images with some kind of associated > per-pixel quality scores, where one might want the 'mask' to be not > just a boolean IGNORED/not-IGNORED flag, but an integer (perhaps a > multi-byte integer) or even a float, and to allow these 'masks' to be > combined in some more complex way than just logical_and. > - Target audience: anyone who's already doing this kind of thing by > hand using a second mask array + boolean indexing, former numpy.ma > users, matplotlib, ... > > 3) And perhaps we can all agree that the biggest *un*resolved question > is whether we want to: > - emphasize the similarities between these two use cases and build a > single interface that can handle both concepts, with some compromises > - or, treat these at two mostly-separate features that can each become > exactly what the respective constituency wants without compromise -- > but with some potential redundancy and extra code. > Each approach has advantages and disadvantages. > > Does that seem like a fair summary? Anything more we can add? Most > importantly, anything here that you disagree with? Did I summarize > your needs well? Do you have a use case that you feel doesn't fit > naturally into either category? > > [Also, I thought this might make the start of a good wiki page for > people to reference during these discussions, but I don't seem to have > edit rights. If other people agree, maybe someone could put it up, or > give me access? My trac id is njs at pobox.com.] > > Thanks, > -- Nathaniel > From lgautier at gmail.com Thu Nov 3 04:03:29 2011 From: lgautier at gmail.com (Laurent Gautier) Date: Thu, 03 Nov 2011 09:03:29 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: Message-ID: <4EB24AD1.9050208@gmail.com> On 2011-11-03 04:22, numpy-discussion-request at scipy.org wrote: > > Message: 1 > Date: Wed, 2 Nov 2011 22:20:15 -0500 > From: Benjamin Root > Subject: Re: [Numpy-discussion] in the NA discussion, what can we > agree on? > To: Discussion of Numerical Python > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > On Wednesday, November 2, 2011, Nathaniel Smith wrote: >> Hi Benjamin, >> >> On Wed, Nov 2, 2011 at 5:25 PM, Benjamin Root wrote: >>> I want to pare this down even more. I think the above lists makes too > many >>> unneeded extrapolations. >> Okay. I found your formatting a little confusing, so I want to make >> sure I understood the changes you're suggesting: >> >> For the description of what MISSING means, you removed the lines: >> - Compatibility with R is valuable >> - To avoid user confusion, ideally it should *not* be possible to >> 'unmask' a missing value, since this is inconsistent with the "missing >> value" metaphor (e.g., see Wes's comment about "leaky abstractions") >> >> And you added the line: >> + Assigning MISSING is destructive >> >> And for the description of what IGNORED means, you removed the lines: >> - Some memory overhead is inevitable and acceptable >> - Compatibility with R neither possible nor valuable >> - Ability to toggle the IGNORED state of a location is critical, and >> should be as convenient as possible >> >> And you added the lines: >> + IGNORE is non-destructive >> + Must be competitive with np.ma for speed and memory (or else users >> would just use np.ma) >> >> Is that right? > Correct. > >> Assuming it is, my thoughts are: >> >> By R compatibility, I specifically had in mind in-memory >> compatibility. rpy2 provides a more-or-less seamless within-process >> interface between R and Python (and specifically lets you get numpy >> views on arrays returned by R functions), so if we can make this work >> for R arrays containing NA too then that'd be handy. (The rpy2 author >> requested this in the last discussion here: >> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/057084.html) >> When it comes to disk formats, then this doesn't matter so much, since >> IO routines have to translate between different representations all >> the time anyway. >> > Interesting, but I still have to wonder if that should be on the wishlist > for MISSING. I guess it would matter by knowing whether people would be > fully converting from R or gradually transitioning from it? That is > something that I can't answer. I probably do not have all possible use-cases but what I'd think of as the most common is: use R stuff just straight out of R from Python. Say that you are doing your work in Python and read about some statistical method for which an implementation in R exists (but not in Python/numpy). You can just pass your numpy arrays or vectors to the relevant R function(s) and retrieve the results in a form directly usable by numpy (without having the data copied around). Should performances become an issue, and that method be of crucial importance, you will probably want to reimplement it (C, or Cython, for example). Otherwise you could pick R's phenomenal toolbox without much effort and keep those calls to R as part of your code. In my experience, the later would be the most frequent. Get some compatibility for the NA "magic" values and that possible coupling between R and numpy becomes even better by preventing one side or the other to understand them as non-NA values. > >> I take the replacement of my line about MISSING disallowing unmasking >> and your line about MISSING assignment being destructive as basically >> expressing the same idea. Is that fair, or did you mean something >> else? > I am someone who wants to get to the absolute core of ideas. Also, this > expression cleanly delineates the differences as binary. > > By expressing it this way, we also shy away from implementation details. > For example, Unmasking can be programmatically prevented for MISSING while > it could be implemented by other indirect means for IGNORE. Not that those > are the preferred ways, only that the phrasing is more flexible and > exacting. > >> Finally, do you think that people who want IGNORED support care about >> having a convenient API for masking/unmasking values? You removed that >> line, but I don't know if that was because you disagreed with it, or >> were just trying to simplify. > See previous. > >>> Then, as a third-party module developer, I can tell you that having > separate >>> and independent ways to detect "MISSING"/"IGNORED" would likely make > support >>> more difficult and would greatly benefit from a common (or easily >>> combinable) method of identification. >> Right, sorry... I didn't forget, and that's part of what I was >> thinking when I described the second approach as keeping them as >> *mostly*-separate interfaces... but I should have made it more >> explicit! Anyway, yes: >> >> 4) There is consensus that whatever approach is taken, there should be >> a quick and convenient way to identify values that are MISSING, >> IGNORED, or both. (E.g., functions is_MISSING, is_IGNORED, >> is_MISSING_or_IGNORED, or some equivalent.) >> > Good. > > Cheers! > Ben Root > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20111102/e4be49be/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Thu, 3 Nov 2011 08:53:26 +0530 > From: akshar bhosale > Subject: Re: [Numpy-discussion] Fwd: numpy error with mkl 10.1 > To: Discussion of Numerical Python > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > thanks once again. how can i use ur distutils directly (ur supplied files > etc) > > On Thu, Nov 3, 2011 at 8:15 AM, Olivier Delalleau wrote: > >> Yes they pass: >> >> Ran 2030 tests in 6.672s >> >> OK (KNOWNFAIL=1) >> >> >> Google "einsum hang".... I know you're not the first one with this bug >> (but I'm not sure exactly how to fix it). >> >> -=- Olivier >> >> 2011/11/2 akshar bhosale >> >>> my numpy.test hangs here >>> >>> Test whether equivalent subarray dtypes hash the same. ... ok >>> Test whether different subarray dtypes hash differently. ... ok >>> Test some data types that are equal ... ok >>> Test some more complicated cases that shouldn't be equal ... ok >>> Test some simple cases that shouldn't be equal ... ok >>> test_single_subarray (test_dtype.TestSubarray) ... ok >>> test_einsum_errors (test_einsum.TestEinSum) ... ok >>> test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... >>> >>> >>> ---------- Forwarded message ---------- >>> From: akshar bhosale >>> Date: Thu, Nov 3, 2011 at 8:00 AM >>> Subject: Re: [Numpy-discussion] numpy error with mkl 10.1 >>> To: Discussion of Numerical Python >>> >>> >>> hi, >>> on your machine, is numpy.test passed? >>> >>> >>> On Thu, Nov 3, 2011 at 6:54 AM, Olivier Delalleau wrote: >>> >>>> I believe they are optional, but I'm not sure. >>>> >>>> Note that this system I copied the distutils from is not one I setup >>>> myself so I'm not sure how numpy was installed exactly. I just know it >>>> works and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462 @ >>>> 2.80GHz (x86_64 as well). >>>> >>>> -=- Olivier >>>> >>>> 2011/11/2 akshar bhosale >>>> >>>>> hi, >>>>> are amd,fftw and umfpack required? my architecture is : intel xeon >>>>> x_86_64. >>>>> >>>>> On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleauwrote: >>>>> >>>>>> It's inside the distutils folder. >>>>>> >>>>>> -=- Olivier >>>>>> >>>>>> 2011/11/2 akshar bhosale >>>>>> >>>>>>> thanks..what about site.cfg? >>>>>>> >>>>>>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleauwrote: >>>>>>> >>>>>>>> Sorry, no clue :/ >>>>>>>> >>>>>>>> I made a tarball with my distutils folder here: >>>>>>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>>>>>> Hope this helps... >>>>>>>> >>>>>>>> -=- Olivier >>>>>>>> >>>>>>>> 2011/11/2 akshar bhosale >>>>>>>> >>>>>>>>> hi, >>>>>>>>> thanks for the reply >>>>>>>>> none of them is linked with libmkl_lapack.so. >>>>>>>>> what can be the problem? >>>>>>>>> libmkl_lapack.so is present in its location. >>>>>>>>> >>>>>>>>> if possible, can you send your site.cfg and other related files >>>>>>>>> like intelccompilers.py / system_config.py etc...and configure/build >>>>>>>>> options, i will try the same? >>>>>>>>> >>>>>>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleauwrote: >>>>>>>>> >>>>>>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. On >>>>>>>>>> my system with MKL it says: >>>>>>>>>> >>>>>>>>>> ldd linalg/lapack_lite.so >>>>>>>>>> libmkl_lapack.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_lapack.so*(0x00002acf0e25a000) >>>>>>>>>> libmkl_intel_lp64.so => /opt/intel/mkl/*MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_intel_lp64.so*(0x00002acf0ebb8000) >>>>>>>>>> libmkl_intel_thread.so => /opt/intel/mkl/*MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" * *MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often >>>>>>>>>> malicious: 10.1.3.027/lib/em64t/libmkl_intel_thread.so*(0x00002acf0ef2e000) >>>>>>>>>> libmkl_def.so => /opt/intel/mkl/*MailScanner soup?onne le >>>>>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_def.so*(0x00002acf0fc93000) >>>>>>>>>> libmkl_core.so => /opt/intel/mkl/*MailScanner soup?onne >>>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>> * *MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>> warning: numerical links are often malicious: >>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_core.so*(0x00002acf1080e000) >>>>>>>>>> >>>>>>>>>> libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>>>>>> (0x00002acf10a03000) >>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>> (0x00002acf10ba6000) >>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>>>>>> libpython2.6.so.1.0 => >>>>>>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>>>>>> libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>>>>>> >>>>>>>>>> So to clarify, what I was trying to achieve is find which of >>>>>>>>>> numpy's shared libraries was trying to load libmkl_lapack.so, and if it >>>>>>>>>> could be found properly with ldd. Just to make sure it's not another .so, >>>>>>>>>> can you run from within your numpy directory: >>>>>>>>>> find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>>>>>> >>>>>>>>>> If the output is not empty, try to run ldd on all .so within numpy >>>>>>>>>> to see which one is linked against libmkl_lapack.so. Then hopefully >>>>>>>>>> something will stand out (I admit that's not guaranteed though ;). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -=- Olivier >>>>>>>>>> >>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> ldd >>>>>>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>>> (0x00002b1199349000) >>>>>>>>>>> libmkl_def.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>> (0x00002b1199653000) >>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>> (0x00002b119a1a8000) >>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>> (0x00002b119a503000) >>>>>>>>>>> libmkl_core.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>> (0x00002b119b22c000) >>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b119c176000) >>>>>>>>>>> libimf.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b119c392000) >>>>>>>>>>> libsvml.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>>>>>> libiomp5.so => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau>>>>>>>>>>> wrote: >>>>>>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about >>>>>>>>>>>> ldd numpy/linalg/lapack_lite.so? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> ldd _dotblas.so >>>>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b12f0692000) >>>>>>>>>>>>> libmkl_def.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>>>> (0x00002b12f099c000) >>>>>>>>>>>>> libmkl_intel_lp64.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>>>> (0x00002b12f14f1000) >>>>>>>>>>>>> libmkl_intel_thread.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>>>> (0x00002b12f184c000) >>>>>>>>>>>>> libmkl_core.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>>>> (0x00002b12f2575000) >>>>>>>>>>>>> libmkl_mc.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>>>> (0x00002b12f34bf000) >>>>>>>>>>>>> libimf.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>>>>>> libsvml.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>>>>>> libiomp5.so => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>>>>>> libintlc.so.5 => >>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau< >>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>> >>>>>>>>>>>>>>> HI, >>>>>>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>>>>>> generating the same error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau< >>>>>>>>>>>>>>> shish at keba.be> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory >>>>>>>>>>>>>>>> that contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>>>>>> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, >>>>>>>>>>>>>>>>> 3j, 4]], >>>>>>>>>>>>>>>>> numpy.complex128).T.I.H' >>>>>>>>>>>>>>>>> MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>>>>>> i have intel cluster toolkit installed on my system. >>>>>>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>>>>>> platform. >>>>>>>>>>>>>>>>> Kindly help >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> NumPy-Discussion mailing list >>>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>>> >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> NumPy-Discussion mailing list >>>>>>>>> NumPy-Discussion at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>>> >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> NumPy-Discussion mailing list >>>>>>>> NumPy-Discussion at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20111103/8b0cffc0/attachment.html > > ------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > End of NumPy-Discussion Digest, Vol 62, Issue 15 > ************************************************ From pierre.raybaut at gmail.com Thu Nov 3 05:36:00 2011 From: pierre.raybaut at gmail.com (Pierre Raybaut) Date: Thu, 3 Nov 2011 10:36:00 +0100 Subject: [Numpy-discussion] ANN: Spyder v2.1 Message-ID: Hi all, On the behalf of Spyder's development team (http://code.google.com/p/spyderlib/people/list), I'm pleased to announce that Spyder v2.1 has been released and is available for Windows XP/Vista/7, GNU/Linux and MacOS X: http://code.google.com/p/spyderlib/ Spyder is a free, open-source (MIT license) interactive development environment for the Python language with advanced editing, interactive testing, debugging and introspection features. Originally designed to provide MATLAB-like features (integrated help, interactive console, variable explorer with GUI-based editors for dictionaries, NumPy arrays, ...), it is strongly oriented towards scientific computing and software development. Thanks to the `spyderlib` library, Spyder also provides powerful ready-to-use widgets: embedded Python console (example: http://packages.python.org/guiqwt/_images/sift3.png), NumPy array editor (example: http://packages.python.org/guiqwt/_images/sift2.png), dictionary editor, source code editor, etc. Description of key features with tasty screenshots can be found at: http://code.google.com/p/spyderlib/wiki/Features This release represents a year of development since v2.0 and introduces major enhancements and new features: * Large performance and stability improvements * PySide support (PyQt is no longer exclusively required) * New profiler plugin (thanks to Santiago Jaramillo, a new contributor) * Experimental support for IPython v0.11+ * And many other changes: http://code.google.com/p/spyderlib/wiki/ChangeLog On Windows platforms, Spyder is also available as a stand-alone executable (don't forget to disable UAC on Vista/7). This all-in-one portable version is still experimental (for example, it does not embed sphinx -- meaning no rich text mode for the object inspector) but it should provide a working version of Spyder for Windows platforms without having to install anything else (except Python 2.x itself, of course). Don't forget to follow Spyder updates/news: * on the project website: http://code.google.com/p/spyderlib/ * and on our official blog: http://spyder-ide.blogspot.com/ Last, but not least, we welcome any contribution that helps making Spyder an efficient scientific development/computing environment. Join us to help creating your favourite environment! (http://code.google.com/p/spyderlib/wiki/NoteForContributors) Enjoy! -Pierre From alan.isaac at gmail.com Thu Nov 3 08:01:55 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 03 Nov 2011 08:01:55 -0400 Subject: [Numpy-discussion] ANN: Spyder v2.1 In-Reply-To: References: Message-ID: <4EB282B3.1010608@gmail.com> On 11/3/2011 5:36 AM, Pierre Raybaut wrote: > * PySide support Congratulations! I hope this means that Spyder will be included in the Enthought Python Distribution (so that I can have my students use it). Alan Isaac From bsouthey at gmail.com Thu Nov 3 08:09:08 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 3 Nov 2011 07:09:08 -0500 Subject: [Numpy-discussion] Fwd: Fwd: numpy error with mkl 10.1 In-Reply-To: References: Message-ID: On Wed, Nov 2, 2011 at 10:33 PM, akshar bhosale wrote: > any other means of getting it fixed? > > ---------- Forwarded message ---------- > From: Olivier Delalleau > Date: Thu, Nov 3, 2011 at 8:58 AM > Subject: Re: [Numpy-discussion] Fwd: numpy error with mkl 10.1 > To: Discussion of Numerical Python > > > I'm sorry, I don't know. I'm not at all an expert about numpy installation, > I just happened to have access to a computer with a working install of numpy > with MKL, so I thought it might help... but I'm afraid that's all I can do > for you :/ > > -=- Olivier > > 2011/11/2 akshar bhosale >> >> thanks once again. how can i use ur distutils directly (ur supplied files >> etc) >> >> On Thu, Nov 3, 2011 at 8:15 AM, Olivier Delalleau wrote: >>> >>> Yes they pass: >>> >>> Ran 2030 tests in 6.672s >>> >>> OK (KNOWNFAIL=1) >>> >>> >>> Google "einsum hang".... I know you're not the first one with this bug >>> (but I'm not sure exactly how to fix it). >>> >>> -=- Olivier >>> >>> 2011/11/2 akshar bhosale >>>> >>>> my numpy.test hangs here >>>> >>>> Test whether equivalent subarray dtypes hash the same. ... ok >>>> Test whether different subarray dtypes hash differently. ... ok >>>> Test some data types that are equal ... ok >>>> Test some more complicated cases that shouldn't be equal ... ok >>>> Test some simple cases that shouldn't be equal ... ok >>>> test_single_subarray (test_dtype.TestSubarray) ... ok >>>> test_einsum_errors (test_einsum.TestEinSum) ... ok >>>> test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... >>>> >>>> >>>> ---------- Forwarded message ---------- >>>> From: akshar bhosale >>>> Date: Thu, Nov 3, 2011 at 8:00 AM >>>> Subject: Re: [Numpy-discussion] numpy error with mkl 10.1 >>>> To: Discussion of Numerical Python >>>> >>>> >>>> hi, >>>> on your machine, is numpy.test passed? >>>> >>>> On Thu, Nov 3, 2011 at 6:54 AM, Olivier Delalleau wrote: >>>>> >>>>> I believe they are optional, but I'm not sure. >>>>> >>>>> Note that this system I copied the distutils from is not one I setup >>>>> myself so I'm not sure how numpy was installed exactly. I just know it works >>>>> and it's using MKL :) It's an 8 core Intel(R) Xeon(R) CPU E5462? @ 2.80GHz >>>>> (x86_64 as well). >>>>> >>>>> -=- Olivier >>>>> >>>>> 2011/11/2 akshar bhosale >>>>>> >>>>>> hi, >>>>>> are amd,fftw and umfpack required? my architecture is : intel xeon >>>>>> x_86_64. >>>>>> >>>>>> On Thu, Nov 3, 2011 at 6:30 AM, Olivier Delalleau >>>>>> wrote: >>>>>>> >>>>>>> It's inside the distutils folder. >>>>>>> >>>>>>> -=- Olivier >>>>>>> >>>>>>> 2011/11/2 akshar bhosale >>>>>>>> >>>>>>>> thanks..what about site.cfg? >>>>>>>> >>>>>>>> On Thu, Nov 3, 2011 at 1:27 AM, Olivier Delalleau >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Sorry, no clue :/ >>>>>>>>> >>>>>>>>> I made a tarball with my distutils folder here: >>>>>>>>> http://www.iro.umontreal.ca/~delallea/tmp/distutils.tar.bz2 >>>>>>>>> Hope this helps... >>>>>>>>> >>>>>>>>> -=- Olivier >>>>>>>>> >>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>> >>>>>>>>>> hi, >>>>>>>>>> thanks for the reply >>>>>>>>>> none of them is linked with libmkl_lapack.so. >>>>>>>>>> what can be the problem? >>>>>>>>>> libmkl_lapack.so is present in its location. >>>>>>>>>> >>>>>>>>>> if possible, can you send your site.cfg and other related files >>>>>>>>>> like intelccompilers.py / system_config.py etc...and configure/build >>>>>>>>>> options, i will try the same? >>>>>>>>>> >>>>>>>>>> On Wed, Nov 2, 2011 at 11:51 PM, Olivier Delalleau >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hmm that's interesting, it's not linked against libmkl_lapack. On >>>>>>>>>>> my system with MKL it says: >>>>>>>>>>> >>>>>>>>>>> ldd linalg/lapack_lite.so >>>>>>>>>>> ??????? libmkl_lapack.so => /opt/intel/mkl/MailScanner soup?onne >>>>>>>>>>> le lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>>> MailScanner soup?onne le lien suivant d'?tre une tentative de fraude de la >>>>>>>>>>> part de "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>>> MailScanner warning: numerical links are often malicious: >>>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_lapack.so (0x00002acf0e25a000) >>>>>>>>>>> ??????? libmkl_intel_lp64.so => /opt/intel/mkl/MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien suivant >>>>>>>>>>> d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often malicious: >>>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_lp64.so (0x00002acf0ebb8000) >>>>>>>>>>> ??????? libmkl_intel_thread.so => /opt/intel/mkl/MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une tentative de >>>>>>>>>>> fraude de la part de "10.1.3.027" MailScanner soup?onne le lien suivant >>>>>>>>>>> d'?tre une tentative de fraude de la part de "10.1.3.027" MailScanner >>>>>>>>>>> soup?onne le lien suivant d'?tre une tentative de fraude de la part de >>>>>>>>>>> "10.1.3.027" MailScanner warning: numerical links are often malicious: >>>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_intel_thread.so (0x00002acf0ef2e000) >>>>>>>>>>> ??????? libmkl_def.so => /opt/intel/mkl/MailScanner soup?onne le >>>>>>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>>> MailScanner soup?onne le lien suivant d'?tre une tentative de fraude de la >>>>>>>>>>> part de "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>>> MailScanner warning: numerical links are often malicious: >>>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_def.so (0x00002acf0fc93000) >>>>>>>>>>> ??????? libmkl_core.so => /opt/intel/mkl/MailScanner soup?onne le >>>>>>>>>>> lien suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>>> MailScanner soup?onne le lien suivant d'?tre une tentative de fraude de la >>>>>>>>>>> part de "10.1.3.027" MailScanner soup?onne le lien suivant d'?tre une >>>>>>>>>>> tentative de fraude de la part de "10.1.3.027" MailScanner soup?onne le lien >>>>>>>>>>> suivant d'?tre une tentative de fraude de la part de "10.1.3.027" >>>>>>>>>>> MailScanner warning: numerical links are often malicious: >>>>>>>>>>> 10.1.3.027/lib/em64t/libmkl_core.so (0x00002acf1080e000) >>>>>>>>>>> ??????? libguide.so => /opt/intel/cce/10.1.026/lib/libguide.so >>>>>>>>>>> (0x00002acf10a03000) >>>>>>>>>>> ??????? libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>> (0x00002acf10ba6000) >>>>>>>>>>> ??????? libm.so.6 => /lib64/libm.so.6 (0x00002acf10dc1000) >>>>>>>>>>> ??????? libpython2.6.so.1.0 => >>>>>>>>>>> /opt/python64/2.6.4/lib/libpython2.6.so.1.0 (0x00002acf11044000) >>>>>>>>>>> ??????? libc.so.6 => /lib64/libc.so.6 (0x00002acf113f3000) >>>>>>>>>>> ??????? libdl.so.2 => /lib64/libdl.so.2 (0x00002acf1174b000) >>>>>>>>>>> ??????? /lib64/ld-linux-x86-64.so.2 (0x0000003d90400000) >>>>>>>>>>> ??????? libutil.so.1 => /lib64/libutil.so.1 (0x00002acf1194f000) >>>>>>>>>>> >>>>>>>>>>> So to clarify, what I was trying to achieve is find which of >>>>>>>>>>> numpy's shared libraries was trying to load libmkl_lapack.so, and if it >>>>>>>>>>> could be found properly with ldd. Just to make sure it's not another .so, >>>>>>>>>>> can you run from within your numpy directory: >>>>>>>>>>> ?? find -name "*.so" -exec ldd \{} \; | grep lapack >>>>>>>>>>> >>>>>>>>>>> If the output is not empty, try to run ldd on all .so within >>>>>>>>>>> numpy to see which one is linked against libmkl_lapack.so. Then hopefully >>>>>>>>>>> something will stand out (I admit that's not guaranteed though ;). >>>>>>>>>>> >>>>>>>>>>> -=- Olivier >>>>>>>>>>> >>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> ldd >>>>>>>>>>>> Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so >>>>>>>>>>>> ??? libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>>>> (0x00002b1199349000) >>>>>>>>>>>> ??? libmkl_def.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>>> (0x00002b1199653000) >>>>>>>>>>>> ??? libmkl_intel_lp64.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>>> (0x00002b119a1a8000) >>>>>>>>>>>> ??? libmkl_intel_thread.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>>> (0x00002b119a503000) >>>>>>>>>>>> ??? libmkl_core.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>>> (0x00002b119b22c000) >>>>>>>>>>>> ??? libmkl_mc.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b119b420000) >>>>>>>>>>>> ??? libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>>> (0x00002b119c176000) >>>>>>>>>>>> ??? libimf.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b119c392000) >>>>>>>>>>>> ??? libsvml.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b119c6e9000) >>>>>>>>>>>> ??? libm.so.6 => /lib64/libm.so.6 (0x00002b119c8a6000) >>>>>>>>>>>> ??? libiomp5.so => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b119cb2b000) >>>>>>>>>>>> ??? libintlc.so.5 => >>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b119ccbc000) >>>>>>>>>>>> ??? libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b119cdf9000) >>>>>>>>>>>> ??? libc.so.6 => /lib64/libc.so.6 (0x00002b119d007000) >>>>>>>>>>>> ??? libdl.so.2 => /lib64/libdl.so.2 (0x00002b119d37e000) >>>>>>>>>>>> ??? /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Nov 2, 2011 at 11:26 PM, Olivier Delalleau >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Doh I'm sorry, I forgot the problem was with lapack, what about >>>>>>>>>>>>> ldd numpy/linalg/lapack_lite.so? >>>>>>>>>>>>> >>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> ldd _dotblas.so >>>>>>>>>>>>>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 >>>>>>>>>>>>>> (0x00002b12f0692000) >>>>>>>>>>>>>> ??? libmkl_def.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_def.so >>>>>>>>>>>>>> (0x00002b12f099c000) >>>>>>>>>>>>>> ??? libmkl_intel_lp64.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so >>>>>>>>>>>>>> (0x00002b12f14f1000) >>>>>>>>>>>>>> ??? libmkl_intel_thread.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so >>>>>>>>>>>>>> (0x00002b12f184c000) >>>>>>>>>>>>>> ??? libmkl_core.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so >>>>>>>>>>>>>> (0x00002b12f2575000) >>>>>>>>>>>>>> ??? libmkl_mc.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so (0x00002b12f2769000) >>>>>>>>>>>>>> ??? libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>>>>> (0x00002b12f34bf000) >>>>>>>>>>>>>> ??? libimf.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so (0x00002b12f36db000) >>>>>>>>>>>>>> ??? libsvml.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so (0x00002b12f3a32000) >>>>>>>>>>>>>> ??? libm.so.6 => /lib64/libm.so.6 (0x00002b12f3bef000) >>>>>>>>>>>>>> ??? libiomp5.so => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so (0x00002b12f3e74000) >>>>>>>>>>>>>> ??? libintlc.so.5 => >>>>>>>>>>>>>> /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5 (0x00002b12f4005000) >>>>>>>>>>>>>> ??? libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b12f4142000) >>>>>>>>>>>>>> ??? libc.so.6 => /lib64/libc.so.6 (0x00002b12f4350000) >>>>>>>>>>>>>> ??? libdl.so.2 => /lib64/libdl.so.2 (0x00002b12f46c7000) >>>>>>>>>>>>>> ??? /lib64/ld-linux-x86-64.so.2 (0x0000003fb8000000) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:14 PM, Olivier Delalleau >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ok, can you print the output of ldd numpy/core/_dotblas.so? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> HI, >>>>>>>>>>>>>>>> It is already added in the LD_LIBRARY_PATH, thenalso it is >>>>>>>>>>>>>>>> generating the same error. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Nov 2, 2011 at 10:01 PM, Olivier Delalleau >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Locate your libmkl_lapack.so and try to add the directory >>>>>>>>>>>>>>>>> that contains it to your LD_LIBRARY_PATH environment variable. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -=- Olivier >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2011/11/2 akshar bhosale >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> i am getting following error. >>>>>>>>>>>>>>>>>> ? ?python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, >>>>>>>>>>>>>>>>>> 3j, 4]], >>>>>>>>>>>>>>>>>> ? ?numpy.complex128).T.I.H' >>>>>>>>>>>>>>>>>> ? ?MKL FATAL ERROR: Cannot load libmkl_lapack.so >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ? ?have installed numpy 1.6.0 with python 2.6. >>>>>>>>>>>>>>>>>> ? ?i have intel cluster toolkit installed on my system. >>>>>>>>>>>>>>>>>> (11/069 version and mlk=10.1). i have machine having intel xeon processor >>>>>>>>>>>>>>>>>> and rhel 5.2 x86_64 >>>>>>>>>>>>>>>>>> ? ?platform. >>>>>>>>>>>>>>>>>> Kindly help >>>>>>>>>>>>>>>>>> You have to isolate the specific source yourself if you want further help. For example: 1) Does Numpy install and pass tests with gcc? 2) Does Numpy install and pass tests with gcc and Atlas? 3) Have you correctly installed or reinstalled the latest Python and Intel compiler/libraries? This specific thread suggests that it was not. 4) Which part of the test_einsum_sums_cfloat128 fails? (Previously asked.) Bruce From akshar.bhosale at gmail.com Thu Nov 3 05:27:13 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 14:57:13 +0530 Subject: [Numpy-discussion] numpy with nose Message-ID: Hi, i am using mkl 10.1, intel cluster toolkit 11/069, os rhel 5.2 x86_64, python 2.6, processor is intel xeon numpy version is 1.6.0 my numpy.test hanging at below point : Test whether equivalent subarray dtypes hash the same. ... ok Test whether different subarray dtypes hash differently. ... ok Test some data types that are equal ... ok Test some more complicated cases that shouldn't be equal ... ok Test some simple cases that shouldn't be equal ... ok test_single_subarray (test_dtype.TestSubarray) ... ok test_einsum_errors (test_einsum.TestEinSum) ... ok test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... any pointers for this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Thu Nov 3 09:42:28 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 03 Nov 2011 08:42:28 -0500 Subject: [Numpy-discussion] numpy with nose In-Reply-To: References: Message-ID: <4EB29A44.3020603@gmail.com> On 11/03/2011 04:27 AM, akshar bhosale wrote: > Hi, > i am using mkl 10.1, intel cluster toolkit 11/069, os rhel 5.2 x86_64, > python 2.6, processor is intel xeon > > numpy version is 1.6.0 > > my numpy.test hanging at below point : > Test whether equivalent subarray dtypes hash the same. ... ok > Test whether different subarray dtypes hash differently. ... ok > Test some data types that are equal ... ok > Test some more complicated cases that shouldn't be equal ... ok > Test some simple cases that shouldn't be equal ... ok > test_single_subarray (test_dtype.TestSubarray) ... ok > test_einsum_errors (test_einsum.TestEinSum) ... ok > test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... > > any pointers for this? YES Stop repeating the exact same problem in different threads and answer the previous questions! For example, how much of the 'test_einsum_sums_cfloat128' test is completed before it hangs? Really there is something unusual about your setup so you need to start at the basics because you appear to be the only one that has this problem. Clear out ALL previous installations including personal and system ones and any build directories. 1) Build and test numpy without any numerical libraries including the Intel ones (do not know that) such as: $ BLAS=None LAPACK=None ATLAS=None python setup.py build 2) Build and test numpy without the intel cluster toolkit but with MKL after removing all previous installations and build directories. 3) Build and test numpy with the intel cluster toolkit removing all previous installations and build directories. Bruce From shish at keba.be Thu Nov 3 10:18:36 2011 From: shish at keba.be (Olivier Delalleau) Date: Thu, 3 Nov 2011 10:18:36 -0400 Subject: [Numpy-discussion] numpy with nose In-Reply-To: References: Message-ID: Have you tried to use -O2 instead of -O3 in compilation? (been mentioned by someone else having the same issue). -=- Olivier 2011/11/3 akshar bhosale > Hi, > i am using mkl 10.1, intel cluster toolkit 11/069, os rhel 5.2 x86_64, > python 2.6, processor is intel xeon > > numpy version is 1.6.0 > > my numpy.test hanging at below point : > Test whether equivalent subarray dtypes hash the same. ... ok > Test whether different subarray dtypes hash differently. ... ok > Test some data types that are equal ... ok > Test some more complicated cases that shouldn't be equal ... ok > Test some simple cases that shouldn't be equal ... ok > test_single_subarray (test_dtype.TestSubarray) ... ok > test_einsum_errors (test_einsum.TestEinSum) ... ok > test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... > > any pointers for this? > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xscript at gmx.net Thu Nov 3 10:28:14 2011 From: xscript at gmx.net (=?utf-8?Q?Llu=C3=ADs?=) Date: Thu, 03 Nov 2011 15:28:14 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: (Nathaniel Smith's message of "Wed, 2 Nov 2011 19:16:39 -0700") References: Message-ID: <877h3hqo01.fsf@ginnungagap.bsc.es> Nathaniel Smith writes: > 4) There is consensus that whatever approach is taken, there should be > a quick and convenient way to identify values that are MISSING, > IGNORED, or both. (E.g., functions is_MISSING, is_IGNORED, > is_MISSING_or_IGNORED, or some equivalent.) Well, maybe it's too low level, but I'd rather decouple the two concepts into two orthogonal properties that can be composed: * Destructiveness: whether the previous data value is lost whenever you assign a "special" value. * Propagation: whether any of these "special" values is propagated or just skipped when performing computations. I think we can all agree on the definition of these two properties (where bit-patters are destructive and masks are non-destructive), so I'd say that the first discussion is establishing whether to expose them as separate properties or just expose specific combinations of them: * MISSING: destructive + propagating * IGNORED: non-destructive + non-propagating For example, it makes sense to me to have non-destructive + propagating. If we take this road, then the next points to discuss should probably be how these combinations are expressed: * At the array level: all special values behave the same in a specific array, given its properties (e.g., all of them are destructive+propagating). * At the value level: each special value conveys a specific combination of the aforementioned properties (e.g., assigning A is destructive+propagating and assigning B is non-destructive+non-propagating). * Hybrid: e.g., all special values are destructive, but propagation depends on the specific special value. I think this last decision is crucial, as it will have a direct impact on performance, numpy code maintainability and 3rd party interface simplicity. Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From ben.root at ou.edu Thu Nov 3 10:47:53 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 3 Nov 2011 09:47:53 -0500 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: <877h3hqo01.fsf@ginnungagap.bsc.es> References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: On Thu, Nov 3, 2011 at 9:28 AM, Llu?s wrote: > Nathaniel Smith writes: > > > 4) There is consensus that whatever approach is taken, there should be > > a quick and convenient way to identify values that are MISSING, > > IGNORED, or both. (E.g., functions is_MISSING, is_IGNORED, > > is_MISSING_or_IGNORED, or some equivalent.) > > Well, maybe it's too low level, but I'd rather decouple the two concepts > into > two orthogonal properties that can be composed: > > * Destructiveness: whether the previous data value is lost whenever you > assign a > "special" value. > > * Propagation: whether any of these "special" values is propagated or just > skipped when performing computations. > > I think we can all agree on the definition of these two properties (where > bit-patters are destructive and masks are non-destructive), so I'd say > that the > first discussion is establishing whether to expose them as separate > properties > or just expose specific combinations of them: > > * MISSING: destructive + propagating > * IGNORED: non-destructive + non-propagating > > For example, it makes sense to me to have non-destructive + propagating. > > This is sort of how it is currently implemented. By default, NA propagates, but it is possible to override these defaults on an operation-by-operation basis using the skipna kwarg, and a subclassed array could implement a __ufunc_wrap__() to default the skipna kwarg to True. > > If we take this road, then the next points to discuss should probably be > how > these combinations are expressed: > > * At the array level: all special values behave the same in a specific > array, > given its properties (e.g., all of them are destructive+propagating). > > * At the value level: each special value conveys a specific combination of > the > aforementioned properties (e.g., assigning A is destructive+propagating > and > assigning B is non-destructive+non-propagating). > > * Hybrid: e.g., all special values are destructive, but propagation > depends on > the specific special value. > > I think this last decision is crucial, as it will have a direct impact on > performance, numpy code maintainability and 3rd party interface simplicity. > > This is actually a very good point, and plays directly on the types of implementations that can be done. Currently, Mark's implementation is the first one. The others are not possible with the current design. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Nov 3 12:45:17 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Thu, 03 Nov 2011 09:45:17 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: Message-ID: <4EB2C51D.1000708@noaa.gov> On 11/2/11 7:16 PM, Nathaniel Smith wrote: > By R compatibility, I specifically had in mind in-memory > compatibility. The R crowd has had a big voice in this discussion, and I understand that there are some nice lessons to be learned from it with regard to the NA issues. However, I think making R compatibility a priority is a mistake -- numpy is numpy, it is NOT, nor should it be, an emulation of anything else. NA functionality is useful to virtually everyone -- not just folks doing R-like stuff, and even less so folks directly working with R. > rpy2 provides a more-or-less seamless within-process > interface between R and Python Perhaps rpy2 will need to do some translating -- so be it, better than crippling numpy for other uses. That being said, if the R binary format is a good one for numpy, no harm in using it, but I think that should be a secondary, at best, concern. So should emulating the R API. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From akshar.bhosale at gmail.com Thu Nov 3 14:24:06 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 23:54:06 +0530 Subject: [Numpy-discussion] numpy with nose In-Reply-To: <4EB29A44.3020603@gmail.com> References: <4EB29A44.3020603@gmail.com> Message-ID: hi, extremely sorry for inconvenience caused. i will check with my system. On Thu, Nov 3, 2011 at 7:12 PM, Bruce Southey wrote: > On 11/03/2011 04:27 AM, akshar bhosale wrote: > > Hi, > > i am using mkl 10.1, intel cluster toolkit 11/069, os rhel 5.2 x86_64, > > python 2.6, processor is intel xeon > > > > numpy version is 1.6.0 > > > > my numpy.test hanging at below point : > > Test whether equivalent subarray dtypes hash the same. ... ok > > Test whether different subarray dtypes hash differently. ... ok > > Test some data types that are equal ... ok > > Test some more complicated cases that shouldn't be equal ... ok > > Test some simple cases that shouldn't be equal ... ok > > test_single_subarray (test_dtype.TestSubarray) ... ok > > test_einsum_errors (test_einsum.TestEinSum) ... ok > > test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... > > > > any pointers for this? > YES > Stop repeating the exact same problem in different threads and answer > the previous questions! > For example, how much of the 'test_einsum_sums_cfloat128' test is > completed before it hangs? > > Really there is something unusual about your setup so you need to start > at the basics because you appear to be the only one that has this problem. > > Clear out ALL previous installations including personal and system ones > and any build directories. > > 1) Build and test numpy without any numerical libraries including the > Intel ones (do not know that) such as: > $ BLAS=None LAPACK=None ATLAS=None python setup.py build > > 2) Build and test numpy without the intel cluster toolkit but with MKL > after removing all previous installations and build directories. > > 3) Build and test numpy with the intel cluster toolkit removing all > previous installations and build directories. > > Bruce > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Thu Nov 3 14:25:22 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 23:55:22 +0530 Subject: [Numpy-discussion] numpy with nose In-Reply-To: References: <4EB29A44.3020603@gmail.com> Message-ID: On Thu, Nov 3, 2011 at 11:54 PM, akshar bhosale wrote: > hi, > extremely sorry for inconvenience caused. i will check with my system as > directed. thank you for help. > > On Thu, Nov 3, 2011 at 7:12 PM, Bruce Southey wrote: > >> On 11/03/2011 04:27 AM, akshar bhosale wrote: >> > Hi, >> > i am using mkl 10.1, intel cluster toolkit 11/069, os rhel 5.2 x86_64, >> > python 2.6, processor is intel xeon >> > >> > numpy version is 1.6.0 >> > >> > my numpy.test hanging at below point : >> > Test whether equivalent subarray dtypes hash the same. ... ok >> > Test whether different subarray dtypes hash differently. ... ok >> > Test some data types that are equal ... ok >> > Test some more complicated cases that shouldn't be equal ... ok >> > Test some simple cases that shouldn't be equal ... ok >> > test_single_subarray (test_dtype.TestSubarray) ... ok >> > test_einsum_errors (test_einsum.TestEinSum) ... ok >> > test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... >> > >> > any pointers for this? >> YES >> Stop repeating the exact same problem in different threads and answer >> the previous questions! >> For example, how much of the 'test_einsum_sums_cfloat128' test is >> completed before it hangs? >> >> Really there is something unusual about your setup so you need to start >> at the basics because you appear to be the only one that has this problem. >> >> Clear out ALL previous installations including personal and system ones >> and any build directories. >> >> 1) Build and test numpy without any numerical libraries including the >> Intel ones (do not know that) such as: >> $ BLAS=None LAPACK=None ATLAS=None python setup.py build >> >> 2) Build and test numpy without the intel cluster toolkit but with MKL >> after removing all previous installations and build directories. >> >> 3) Build and test numpy with the intel cluster toolkit removing all >> previous installations and build directories. >> >> Bruce >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akshar.bhosale at gmail.com Thu Nov 3 14:26:22 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Thu, 3 Nov 2011 23:56:22 +0530 Subject: [Numpy-discussion] numpy with nose In-Reply-To: References: Message-ID: hi Olivier, i will check that too. thanks for bearing me. On Thu, Nov 3, 2011 at 7:48 PM, Olivier Delalleau wrote: > Have you tried to use -O2 instead of -O3 in compilation? (been mentioned > by someone else having the same issue). > > -=- Olivier > > 2011/11/3 akshar bhosale > >> Hi, >> i am using mkl 10.1, intel cluster toolkit 11/069, os rhel 5.2 x86_64, >> python 2.6, processor is intel xeon >> >> numpy version is 1.6.0 >> >> my numpy.test hanging at below point : >> Test whether equivalent subarray dtypes hash the same. ... ok >> Test whether different subarray dtypes hash differently. ... ok >> Test some data types that are equal ... ok >> Test some more complicated cases that shouldn't be equal ... ok >> Test some simple cases that shouldn't be equal ... ok >> test_single_subarray (test_dtype.TestSubarray) ... ok >> test_einsum_errors (test_einsum.TestEinSum) ... ok >> test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... >> >> any pointers for this? >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Nov 3 15:02:38 2011 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 3 Nov 2011 12:02:38 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: <4EB2C51D.1000708@noaa.gov> References: <4EB2C51D.1000708@noaa.gov> Message-ID: Hi Chris, On Thu, Nov 3, 2011 at 9:45 AM, Chris.Barker wrote: > On 11/2/11 7:16 PM, Nathaniel Smith wrote: >> By R compatibility, I specifically had in mind in-memory >> compatibility. > > The R crowd has had a big voice in this discussion, and I understand > that there are some nice lessons to be learned from it with regard to > the NA issues. > > However, I think making R compatibility a priority is a mistake -- numpy > is numpy, it is NOT, nor should it be, an emulation of anything else. NA > functionality is useful to virtually everyone -- not just folks doing > R-like stuff, and even less so folks directly working with R. I think we agree, actually. What I currently have written on the wiki page is "In-memory compatibility with R would be handy", which is intended to convey that all else being equal this is a desirable feature, but that it's not worth crippling numpy (as you put it) to get. Do you have a suggestion about how I could make this clearer? or am I misunderstanding your point? -- Nathaniel From njs at pobox.com Thu Nov 3 15:44:06 2011 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 3 Nov 2011 12:44:06 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: <877h3hqo01.fsf@ginnungagap.bsc.es> References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: Hi Llu?s, On Thu, Nov 3, 2011 at 7:28 AM, Llu?s wrote: > Well, maybe it's too low level, but I'd rather decouple the two concepts into > two orthogonal properties that can be composed: > > * Destructiveness: whether the previous data value is lost whenever you assign a > ?"special" value. > > * Propagation: whether any of these "special" values is propagated or just > ?skipped when performing computations. > > I think we can all agree on the definition of these two properties (where > bit-patters are destructive and masks are non-destructive), so I'd say that the > first discussion is establishing whether to expose them as separate properties > or just expose specific combinations of them: > > * MISSING: destructive + propagating > * IGNORED: non-destructive + non-propagating Thanks, that's an interesting idea that I'd forgotten about. I added a link to your message to the "proposals" section, and to the list of proposed solutions in point (3). I'm tempted to respond in more depth, but I'm worried that if we start digging into specific proposals like this right now then we'll start going in circles again -- that's why I'm trying to establish some common ground on what our goals are, so we have more of a basis for comparing different ideas. So obviously your suggestion of breaking things down into finer-grained orthogonal features has merits in terms of simplicity, elegance, etc. Before we get into those, though, I want to ask: do you feel that the extra ability to have values that are destructive+non-propagating and non-destructive+propagating is a major *practical* benefit, or are you more motivated by the simplicity and elegance, and the extra flexibility is just something kind of cool that we'd get for free? Or put another way, do you think that the MISSING and IGNORED concepts are adequate to cover practical use cases, or do you have an example where what's really wanted is say non-destructive + propagating? I can see how it would work, but I don't think I'd ever use it, so I'm curious... -- Nathaniel From jkington at wisc.edu Thu Nov 3 18:07:07 2011 From: jkington at wisc.edu (Joe Kington) Date: Thu, 03 Nov 2011 17:07:07 -0500 Subject: [Numpy-discussion] Indexing a masked array with another masked array leads to unexpected results Message-ID: Forgive me if this is already a well-know oddity of masked arrays. I hadn't seen it before, though. I'm not sure if this is exactly a bug, per se, but it's a very confusing consequence of the current design of masked arrays... Consider the following example: import numpy as np x = np.ma.masked_all(10, dtype=np.float32) print x x[x > 0] = 5 print x The exact results will vary depending the contents of the empty memory the array was initialized from. This wreaks havoc when filtering the contents of masked arrays (and leads to hard-to-find bugs!). The mask of the array in question is altered at random (or, rather, based on the masked values as well as the masked ones). Of course, once you're aware of this, there are a number of workarounds (namely, filling the array or explicitly operating on "x.data" instead of x). I can see the reasoning behind the way it works. It makes sense that "x > 0" returns a masked boolean array with potentially several elements masked, as well as the unmasked elements greater than 0. However, wouldn't it make more sense to have MaskedArray.__setitem__ only operate on the unmasked elements of the "indx" passed in (at least in the case where the assigned "value" isn't a masked array)? Cheers, -Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From strang at nmr.mgh.harvard.edu Thu Nov 3 22:54:18 2011 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Thu, 3 Nov 2011 22:54:18 -0400 (EDT) Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: For the non-destructive+propagating case, do I understand correctly that this would mean I (as a user) could temporarily decide to IGNORE certain portions of my data, perform a series of computation on that data, and the IGNORED flag (or however it is implemented) would be propagated from computation to computation? If that's the case, I suspect I'd use it all the time ... to effectively perform data subsetting without generating (partial) copies of large datasets. But maybe I misunderstand the intended notion of propagation ... Gary > Or put another way, do you think that the MISSING and IGNORED concepts > are adequate to cover practical use cases, or do you have an example > where what's really wanted is say non-destructive + propagating? I can > see how it would work, but I don't think I'd ever use it, so I'm > curious... > > -- Nathaniel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. From ben.root at ou.edu Thu Nov 3 23:20:14 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 3 Nov 2011 22:20:14 -0500 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: On Thursday, November 3, 2011, Gary Strangman wrote: > > For the non-destructive+propagating case, do I understand correctly that > this would mean I (as a user) could temporarily decide to IGNORE certain > portions of my data, perform a series of computation on that data, and the > IGNORED flag (or however it is implemented) would be propagated from > computation to computation? If that's the case, I suspect I'd use it all > the time ... to effectively perform data subsetting without generating > (partial) copies of large datasets. But maybe I misunderstand the > intended notion of propagation ... > > Gary > Propagating is default NaN-like behavior when performing a sum with at least one NaN in it. Ignoring is like using nansum on that same array. Masking, one can think of it as very fancy indexing, but the shape and structure of the data is maintained. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Nov 4 02:07:48 2011 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 3 Nov 2011 23:07:48 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: On Thu, Nov 3, 2011 at 7:54 PM, Gary Strangman wrote: > For the non-destructive+propagating case, do I understand correctly that > this would mean I (as a user) could temporarily decide to IGNORE certain > portions of my data, perform a series of computation on that data, and the > IGNORED flag (or however it is implemented) would be propagated from > computation to computation? If that's the case, I suspect I'd use it all > the time ... to effectively perform data subsetting without generating > (partial) copies of large datasets. But maybe I misunderstand the > intended notion of propagation ... I *think* it's more subtle than that, but I admit I'm somewhat confused about how exactly people would want IGNORED to work in various corner cases. (This is another part of why figuring out our audience/use-cases seems like an important first step to me... fortunately the semantics for MISSING are, I think, much more clear.) Say we have >>> a = np.array([1, IGNORED(2), 3]) >>> b = np.array([10, 20, 30]) (Here's I'm using IGNORED(2) to mean a value that is currently ignored, but if you unmasked it it would have the value 2.) Then we have: # non-propagating *or* propagating, doesn't matter: >>> a + 2 [3, IGNORED(2), 5] # non-propagating: >>> a + b One of these, I don't know which: [11, IGNORED(2), 33] # numpy.ma chooses this [11, 20, 33] "Error: shape mismatch" (An error is maybe the most *consistent* option; the suggestion in the alterNEP was that masks had to match on all axes that were *not* broadcast, so a + 2 and a + a are okay, but a + b is an error. I assume the numpy.ma approach is also useful, but note that it has the surprising effect that addition is not commutative: IGNORED(x) + IGNORED(y) = IGNORED(x). Try it: masked1 = np.ma.masked_array([1, 2, 3], mask=[False, True, False]) masked2 = np.ma.masked_array([10, 20, 30], mask=[False, True, False]) np.asarray(masked1 + masked2) # [11, 2, 33] np.asarray(masked2 + masked1) # [11, 20, 33] I don't really know what people would prefer.) # propagating: >>> a + b One of these, I don't know which: [11, IGNORED(2), 33] # same as numpy.ma, again [11, IGNORED(22), 33] # non-propagating: >>> np.sum(a) 4 # propagating: >>> np.sum(a) One of these, I don't know which: IGNORED(4) IGNORED(6) So from your description, I wouldn't say that you necessarily want non-destructive+propagating -- it really depends on exactly what computations you want to perform, and how you expect them to work. The main difference is how reduction operations are treated. I kind of feel like the non-propagating version makes more sense overall, but I don't know if there's any consensus on that. (You also have the option of just using the new where= argument to your ufuncs, which avoids some of this confusion because it gives a single mask that would apply to the whole operation. The ambiguities here arise because it's not clear what to do when applying a binary operation to two arrays that have different masks.) Maybe you could give some examples of the kinds of computations you're thinking of? -- Nathaniel From ben.root at ou.edu Fri Nov 4 02:47:53 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 4 Nov 2011 01:47:53 -0500 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: On Friday, November 4, 2011, Nathaniel Smith wrote: > On Thu, Nov 3, 2011 at 7:54 PM, Gary Strangman > wrote: >> For the non-destructive+propagating case, do I understand correctly that >> this would mean I (as a user) could temporarily decide to IGNORE certain >> portions of my data, perform a series of computation on that data, and the >> IGNORED flag (or however it is implemented) would be propagated from >> computation to computation? If that's the case, I suspect I'd use it all >> the time ... to effectively perform data subsetting without generating >> (partial) copies of large datasets. But maybe I misunderstand the >> intended notion of propagation ... > > I *think* it's more subtle than that, but I admit I'm somewhat > confused about how exactly people would want IGNORED to work in > various corner cases. (This is another part of why figuring out our > audience/use-cases seems like an important first step to me... > fortunately the semantics for MISSING are, I think, much more clear.) > > Say we have > >>> a = np.array([1, IGNORED(2), 3]) > >>> b = np.array([10, 20, 30]) > (Here's I'm using IGNORED(2) to mean a value that is currently > ignored, but if you unmasked it it would have the value 2.) > > Then we have: > > # non-propagating *or* propagating, doesn't matter: >>>> a + 2 > [3, IGNORED(2), 5] > > # non-propagating: >>>> a + b > One of these, I don't know which: > [11, IGNORED(2), 33] # numpy.ma chooses this > [11, 20, 33] > "Error: shape mismatch" > > (An error is maybe the most *consistent* option; the suggestion in the > alterNEP was that masks had to match on all axes that were *not* > broadcast, so a + 2 and a + a are okay, but a + b is an error. I > assume the numpy.ma approach is also useful, but note that it has the > surprising effect that addition is not commutative: IGNORED(x) + > IGNORED(y) = IGNORED(x). Try it: > masked1 = np.ma.masked_array([1, 2, 3], mask=[False, True, False]) > masked2 = np.ma.masked_array([10, 20, 30], mask=[False, True, False]) > np.asarray(masked1 + masked2) # [11, 2, 33] > np.asarray(masked2 + masked1) # [11, 20, 33] > I don't really know what people would prefer.) > > # propagating: >>>> a + b > One of these, I don't know which: > [11, IGNORED(2), 33] # same as numpy.ma, again > [11, IGNORED(22), 33] > > # non-propagating: >>>> np.sum(a) > 4 > > # propagating: >>>> np.sum(a) > One of these, I don't know which: > IGNORED(4) > IGNORED(6) > > So from your description, I wouldn't say that you necessarily want > non-destructive+propagating -- it really depends on exactly what > computations you want to perform, and how you expect them to work. The > main difference is how reduction operations are treated. I kind of > feel like the non-propagating version makes more sense overall, but I > don't know if there's any consensus on that. I think this is further evidence for my idea that a mask should not be undone, but is non destructive. If you want to be able to access the values after masking, have a view, or only apply the mask to a view. Reduction ufuncs make a lot of sense because they have a basis in mathematics when there are no values. Reduction ufuncs are covered in great detail in Mark's NEP. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Fri Nov 4 06:26:15 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 4 Nov 2011 11:26:15 +0100 Subject: [Numpy-discussion] Indexing a masked array with another masked array leads to unexpected results In-Reply-To: References: Message-ID: <9A892A67-4320-4C4D-B993-AF782206A6B0@gmail.com> On Nov 03, 2011, at 23:07 , Joe Kington wrote: > I'm not sure if this is exactly a bug, per se, but it's a very confusing consequence of the current design of masked arrays? I would just add a "I think" between the "but" and "it's" before I could agree. > Consider the following example: > > import numpy as np > > x = np.ma.masked_all(10, dtype=np.float32) > print x > x[x > 0] = 5 > print x > > The exact results will vary depending the contents of the empty memory the array was initialized from. Not a surprise. But isn't mentioned in the doc somewhere that using a masked array as index is a very bad idea ? And that you should always fill it before you use it as an array ? (Actually, using a MaskedArray as index used to raise an IndexError. But I thought it was a bit too harsh, so I dropped it). ma.masked_all is an empty array with all its elements masked. Ie, you have an uninitialized ndarray as data, and a bool array of the same size, full of True. The operative word is here "uninitialized". > This wreaks havoc when filtering the contents of masked arrays (and leads to hard-to-find bugs!). The mask of the array in question is altered at random (or, rather, based on the masked values as well as the masked ones). Once again, you're working on an *uninitialized* array. What you should really do is to initialize it first, e.g. by 0, or whatever would make sense in your field, and then work from that. > I can see the reasoning behind the way it works. It makes sense that "x > 0" returns a masked boolean array with potentially several elements masked, as well as the unmasked elements greater than 0. Well, "x > 0" is also a masked array, with its mask full of True. Not very usable by itself, and especially *not* for indexing. > However, wouldn't it make more sense to have MaskedArray.__setitem__ only operate on the unmasked elements of the "indx" passed in (at least in the case where the assigned "value" isn't a masked array)? Normally, that should be the case. But you're not working in "normal" conditions, here. A bit like trying to boil water on a stove with a plastic pan. From strang at nmr.mgh.harvard.edu Fri Nov 4 09:29:05 2011 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Fri, 4 Nov 2011 09:29:05 -0400 (EDT) Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: > > non-destructive+propagating -- it really depends on exactly what > > computations you want to perform, and how you expect them to work. The > > main difference is how reduction operations are treated. I kind of > > feel like the non-propagating version makes more sense overall, but I > > don't know if there's any consensus on that. > > I think this is further evidence for my idea that a mask should not be > undone, but is non destructive. ?If you want to be able to access the values > after masking, have a view, or only apply the mask to a view. OK, so my understanding of what's meant by propagating is probably incomplete (and is definitely still fuzzy). I'm a little confused by the phrase "a mask should not be undone" though. Say I want to perform a statistical analysis or filtering procedure excluding and (separately) including a handful of outliers? Isn't that a natural case for undoing a mask? Or did you mean something else? I think I understand the "use a view" option above, though I don't see how one could apply a mask only to a view. What if my view is every other row in a 2D array, and I want to mask the last half of this view? What is the state of the original array once the mask has been applied? (If this is derailing the progress of this thread, feel free to ignore it.) -best Gary The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. From xscript at gmx.net Fri Nov 4 10:22:49 2011 From: xscript at gmx.net (=?utf-8?Q?Llu=C3=ADs?=) Date: Fri, 04 Nov 2011 15:22:49 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: (Gary Strangman's message of "Thu, 3 Nov 2011 22:54:18 -0400 (EDT)") References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: <87ipn0j7ba.fsf@ginnungagap.bsc.es> Gary Strangman writes: > For the non-destructive+propagating case, do I understand correctly that > this would mean I (as a user) could temporarily decide to IGNORE certain > portions of my data, perform a series of computation on that data, and the > IGNORED flag (or however it is implemented) would be propagated from > computation to computation? If that's the case, I suspect I'd use it all > the time ... to effectively perform data subsetting without generating > (partial) copies of large datasets. But maybe I misunderstand the > intended notion of propagation ... I *think* you're right. I say "think" because to *me* IGNORE is the opposite of propagate. For example, you could temporarily (non-destructively) decide to assign a propagating special value to array "a". Then do "a+=2; a*=5" and get something like (which I think is the kind of use case you were talking about): # original values >>> a [1, 1, 1] # with special value >>> a [1, SPECIAL, 1] # computations >>> a += 2 >>> a *= 5 # result >>> a [15, SPECIAL, 15] # without special value >>> a [15, 1, 15] So yes, separating both properties is not only a matter of elegance and simplicity, but also has the practical impact of (as Ben said in another mail) making the non-destructive property into a fancy form of indexing. Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From ben.root at ou.edu Fri Nov 4 10:48:56 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 4 Nov 2011 09:48:56 -0500 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: On Friday, November 4, 2011, Gary Strangman wrote: > >> > non-destructive+propagating -- it really depends on exactly what >> > computations you want to perform, and how you expect them to work. The >> > main difference is how reduction operations are treated. I kind of >> > feel like the non-propagating version makes more sense overall, but I >> > don't know if there's any consensus on that. >> >> I think this is further evidence for my idea that a mask should not be >> undone, but is non destructive. If you want to be able to access the values >> after masking, have a view, or only apply the mask to a view. > > OK, so my understanding of what's meant by propagating is probably incomplete (and is definitely still fuzzy). I'm a little confused by the phrase "a mask should not be undone" though. Say I want to perform a statistical analysis or filtering procedure excluding and (separately) including a handful of outliers? Isn't that a natural case for undoing a mask? Or did you mean something else? > > I think I understand the "use a view" option above, though I don't see how one could apply a mask only to a view. What if my view is every other row in a 2D array, and I want to mask the last half of this view? What is the state of the original array once the mask has been applied? > > (If this is derailing the progress of this thread, feel free to ignore it.) > > -best > Gary Ufuncs can be broadly categorized as element-wise (binary ops like +, *, etc) as well as regular functions that return an array with a shape that matches the inputs broadcasted together. And reduction ops (sum, min, mean, etc). For element-wise, things are a bit murky for IGNORE, and I defer to Mark's NEP: https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst#id17, and it probably should be expanded and clarified in the NEP. For reduction ops, propagation means that sum([3 5 NA 6]) == NA, just like if you had a NaN in the array. Non-propagating (or skipping or ignore) would have that operation produce 14. A mean() for the propagating case would be NA, but 4.6666 for non-propagating. The part about undoing a mask is addressing the issue of when an operation produces a new array that has ignored elements in it, then those elements never were initialized with any value at all. Therefore, "unmasking" those elements and accessing their values make no sense. This and more are covered in this section of the NEP: https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst#id11 For your stated case, I would have two views of the data (or at least the original data and a view of it). For the view, I would apply the mask to hide the outliers from the filtering operation and produce a result. The first view (or the original array) sees the same data as it did before the other view took on a mask, so you can perform the filtering operation on the data and have two separate results. You can keep the masked view for subsequent calculations, and/or keep the original view, and/or create new views with new masks for other analyzes, all while keeping the original data intact. Note that I am right now speaking of views in a somewhat more abstract sense that is only loosely tied to numpy's specific behavior with respect to views right now. As for np.view() in specific, that is an implementation detail that probably shouldn't be in this thread yet, so don't hook too much onto it. Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From strang at nmr.mgh.harvard.edu Fri Nov 4 11:19:54 2011 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Fri, 4 Nov 2011 11:19:54 -0400 (EDT) Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: On Fri, 4 Nov 2011, Benjamin Root wrote: > > On Friday, November 4, 2011, Gary Strangman > wrote: > > > >> > non-destructive+propagating -- it really depends on exactly what > >> > computations you want to perform, and how you expect them to work. The > >> > main difference is how reduction operations are treated. I kind of > >> > feel like the non-propagating version makes more sense overall, but I > >> > don't know if there's any consensus on that. > >> > >> I think this is further evidence for my idea that a mask should not be > >> undone, but is non destructive. ?If you want to be able to access the > values > >> after masking, have a view, or only apply the mask to a view. > > > > OK, so my understanding of what's meant by propagating is probably > incomplete (and is definitely still fuzzy). I'm a little confused by the > phrase "a mask should not be undone" though. Say I want to perform a > statistical analysis or filtering procedure excluding and (separately) > including a handful of outliers? Isn't that a natural case for undoing a > mask? Or did you mean something else? > > > > I think I understand the "use a view" option above, though I don't see how > one could apply a mask only to a view. What if my view is every other row in > a 2D array, and I want to mask the last half of this view? What is the state > of the original array once the mask has been applied? > > > > (If this is derailing the progress of this thread, feel free to ignore > it.) > > > > -best > > Gary > > Ufuncs can be broadly categorized as element-wise (binary ops like +, *, > etc) as well as regular functions that return an array with a shape that > matches the inputs broadcasted together. ?And reduction ops (sum, min, mean, > etc). > > For element-wise, things are a bit murky for IGNORE, and I defer to Mark's > NEP: > https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst#id17, > and it probably should be expanded and clarified in the NEP. > > For reduction ops, propagation means that sum([3 5 NA 6]) == NA, just like > if you had a NaN in the array. Non-propagating (or skipping or ignore) would > have that operation produce 14. ?A mean() for the propagating case would be > NA, but 4.6666 for non-propagating. > > The part about undoing a mask is addressing the issue of when an operation > produces a new array that has ignored elements in it, then those elements > never were initialized with any value at all.? Therefore, "unmasking" those > elements and accessing their values make no sense. This and more are covered > in this section of the NEP: > https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst#id11 > > For your stated case, I would have two views of the data (or at least the > original data and a view of it).? For the view, I would apply the mask to > hide the outliers from the filtering operation and produce a result.? The > first view (or the original array) sees the same data as it did before the > other view took on a mask, so you can perform the filtering operation on the > data and have two separate results. You can keep the masked view for > subsequent calculations, and/or keep the original view, and/or create new > views with new masks for other analyzes, all while keeping the original data > intact. > > Note that I am right now speaking of views in a somewhat more abstract sense > that is only loosely tied to numpy's specific behavior with respect to views > right now.? As for np.view() in specific, that is an implementation detail > that probably shouldn't be in this thread yet, so don't hook too much onto > it. Thanks Ben. That's quite helpful. And it also points to my worry (sorry, I already knew enough about views to be dangerous) ... your "conceptual" version of views is great, but I don't think numpy fully and reliably follows it (occasionally giving copies instead of views, for example, when a view is particularly difficult to generate). So I worry that your notion of views will actually collide with core numpy view implementations. But like you said, perhaps this thread shouldn't go there (yet). Given I'm still fuzzy on all the distinctions, perhaps someone could try to help me (and others?) to define all /4/ logical possibilities ... some may be obvious dead-ends. I'll take a stab at them, but these should definitely get edited by others: destructive + propagating = the data point is truly missing (satellite fell into the ocean; dog ate my source datasheet, or whatever), this is the nature of that data point, such missingness should be replicated in elementwise operations, and the missingness SHOULD interfere with reduction operations that involve that datapoint (np.sum([1,MISSING])=MISSING) destructive + non-propagating = the data point is truly missing, this is the nature of that data point, such missingness should be replicated in elementwise operations, but such missingness should NOT interfere with reduction operations that involve that datapoint (np.sum([1,MISSING])=1) non-destructive + propagating = I want to ignore this datapoint for now; element-wise operations should replicate this "ignore" designation, and missingness of this type SHOULD interfere with reduction operations that involve this datapoint (np.sum([1,IGNORE])=IGNORE) non-destructive + non-propagating = I want to ignore this datapoint for now; element-wise operations should replicate this "ignore" designation, but missingness of this type SHOULD NOT interfere with reduction operations that involve this datapoint (np.sum([1,IGNORE])=1) Comments? -best Gary The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. From xscript at gmx.net Fri Nov 4 12:08:16 2011 From: xscript at gmx.net (=?utf-8?Q?Llu=C3=ADs?=) Date: Fri, 04 Nov 2011 17:08:16 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: (Gary Strangman's message of "Fri, 4 Nov 2011 11:19:54 -0400 (EDT)") References: <877h3hqo01.fsf@ginnungagap.bsc.es> Message-ID: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Gary Strangman writes: [...] > Given I'm still fuzzy on all the distinctions, perhaps someone could try to help > me (and others?) to define all /4/ logical possibilities ... some may be obvious > dead-ends. I'll take a stab at them, but these should definitely get edited by > others: > destructive + propagating = the data point is truly missing (satellite fell into > the ocean; dog ate my source datasheet, or whatever), this is the nature of that > data point, such missingness should be replicated in elementwise operations, and > the missingness SHOULD interfere with reduction operations that involve that > datapoint (np.sum([1,MISSING])=MISSING) Right. > destructive + non-propagating = the data point is truly missing, this is the > nature of that data point, such missingness should be replicated in elementwise > operations, but such missingness should NOT interfere with reduction operations > that involve that datapoint (np.sum([1,MISSING])=1) What do you define as element-wise operations? Is a sum on an array an element-wise operation? >>> [1, MISSING]+2 [1, MISSING] Or is it just a form of reduction (after shape broadcasting)? >>> [1, MISSING]+2 [3, 2] For me it's the second, so the only time where special values "propagate" in a non-propagating scenario is when you slice an array. > non-destructive + propagating = I want to ignore this datapoint for now; > element-wise operations should replicate this "ignore" designation, and > missingness of this type SHOULD interfere with reduction operations that involve > this datapoint (np.sum([1,IGNORE])=IGNORE) Right. > non-destructive + non-propagating = I want to ignore this datapoint for now; > element-wise operations should replicate this "ignore" designation, but > missingness of this type SHOULD NOT interfere with reduction operations that > involve this datapoint (np.sum([1,IGNORE])=1) Same concerns as above. Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From jkington at wisc.edu Fri Nov 4 12:28:01 2011 From: jkington at wisc.edu (Joe Kington) Date: Fri, 04 Nov 2011 11:28:01 -0500 Subject: [Numpy-discussion] Indexing a masked array with another masked array leads to unexpected results In-Reply-To: References: <9A892A67-4320-4C4D-B993-AF782206A6B0@gmail.com> Message-ID: On Fri, Nov 4, 2011 at 5:26 AM, Pierre GM wrote: > > On Nov 03, 2011, at 23:07 , Joe Kington wrote: > > > I'm not sure if this is exactly a bug, per se, but it's a very confusing > consequence of the current design of masked arrays? > I would just add a "I think" between the "but" and "it's" before I could > agree. > > > Consider the following example: > > > > import numpy as np > > > > x = np.ma.masked_all(10, dtype=np.float32) > > print x > > x[x > 0] = 5 > > print x > > > > The exact results will vary depending the contents of the empty memory > the array was initialized from. > > Not a surprise. But isn't mentioned in the doc somewhere that using a > masked array as index is a very bad idea ? And that you should always fill > it before you use it as an array ? (Actually, using a MaskedArray as index > used to raise an IndexError. But I thought it was a bit too harsh, so I > dropped it). > Not that I can find in the docs (Perhaps I just missed it?). At any rate, it's not mentioned in the numpy.ma section on indexing: http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html#indexing-and-slicing The only mention of it is a comment in MaskedArray.__setitem__ where the IndexError is commented out. > ma.masked_all is an empty array with all its elements masked. Ie, you have > an uninitialized ndarray as data, and a bool array of the same size, full > of True. The operative word is here "uninitialized". > > > This wreaks havoc when filtering the contents of masked arrays (and > leads to hard-to-find bugs!). The mask of the array in question is altered > at random (or, rather, based on the masked values as well as the masked > ones). > > Once again, you're working on an *uninitialized* array. What you should > really do is to initialize it first, e.g. by 0, or whatever would make > sense in your field, and then work from that. > Sure, I shouldn't have used that as the example. My point was that it's counter-intuitive that something like "x[x > 0] = 0" alters the mask of x based on the values of _masked_ elements. How it's initialized is irrelevant (though, of course, it wouldn't be semi-random if it were initialized in another way). > > I can see the reasoning behind the way it works. It makes sense that "x > > 0" returns a masked boolean array with potentially several elements > masked, as well as the unmasked elements greater than 0. > > Well, "x > 0" is also a masked array, with its mask full of True. Not very > usable by itself, and especially *not* for indexing. > > However, wouldn't it make more sense to have MaskedArray.__setitem__ > only operate on the unmasked elements of the "indx" passed in (at least in > the case where the assigned "value" isn't a masked array)? > > > Normally, that should be the case. But you're not working in "normal" > conditions, here. A bit like trying to boil water on a stove with a plastic > pan. > "x[x > threshold] = something" is a very common idiom for ndarrays. I think most people would find it surprising that this operation doesn't ignore the masked values. I noticed this because one of my coworkers was complaining that a piece of my code was "messing up" their masked arrays. I'd never tested it with masked arrays, but it took me ages to find, just because I wasn't looking in places where I was just using common idioms. In this particular case, they'd initialized it with "masked_all", so it effectively altered the mask of the array at random. Regardless of how it was initialized, though, it is surprising that the mask of "x" is changed based on masked values. I just think it would be useful for it to be documented. Cheers, -Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Fri Nov 4 12:31:01 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 4 Nov 2011 11:31:01 -0500 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: <87pqh7hnv3.fsf@ginnungagap.bsc.es> References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 11:08 AM, Llu?s wrote: > Gary Strangman writes: > [...] > > > destructive + non-propagating = the data point is truly missing, this is > the > > nature of that data point, such missingness should be replicated in > elementwise > > operations, but such missingness should NOT interfere with reduction > operations > > that involve that datapoint (np.sum([1,MISSING])=1) > > What do you define as element-wise operations? > > Is a sum on an array an element-wise operation? > > >>> [1, MISSING]+2 > [1, MISSING] > > did you mean [3, MISSING]? > Or is it just a form of reduction (after shape broadcasting)? > > >>> [1, MISSING]+2 > [3, 2] > > For me it's the second, so the only time where special values "propagate" > in a > non-propagating scenario is when you slice an array. > > Propagation has a very specific meaning here, and I think it is causing confusion elsewhere. Propagation (to me) is the *exact* same behavior that occurs with NaNs, but generalized to any dtype. It seems like you are taking "propagate" to mean whether the mask of the inputs follow on to the mask of the output. This is related, but is possibly a murkier concept and should probably be cleaned up. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From strang at nmr.mgh.harvard.edu Fri Nov 4 12:31:55 2011 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Fri, 4 Nov 2011 12:31:55 -0400 (EDT) Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: <87pqh7hnv3.fsf@ginnungagap.bsc.es> References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: >> destructive + propagating = the data point is truly missing (satellite fell into >> the ocean; dog ate my source datasheet, or whatever), this is the nature of that >> data point, such missingness should be replicated in elementwise operations, and >> the missingness SHOULD interfere with reduction operations that involve that >> datapoint (np.sum([1,MISSING])=MISSING) > > Right. > > >> destructive + non-propagating = the data point is truly missing, this is the >> nature of that data point, such missingness should be replicated in elementwise >> operations, but such missingness should NOT interfere with reduction operations >> that involve that datapoint (np.sum([1,MISSING])=1) > > What do you define as element-wise operations? > > Is a sum on an array an element-wise operation? > > >>> [1, MISSING]+2 > [1, MISSING] > > Or is it just a form of reduction (after shape broadcasting)? > > >>> [1, MISSING]+2 > [3, 2] > > For me it's the second, so the only time where special values "propagate" in a > non-propagating scenario is when you slice an array. Let's say I want to re-scale a column (or remove the mean from a column). I wouldn't want that to change my "missingness". Thus, I'm thinking: >>> x = [1,2,MISSING] >>> x*3 [3, 6, MISSING] >>> x = [1,2,MISSING] >>> x - x.mean() [-0.5, 0.5, MISSING] To me it makes sense to have identical operations for the temporary IGNORE case below (versus the permanent MISSING case here). Note, the reason to independently have separate IGNORE and MISSING is so that I can (for example) temporarily IGNORE entire rows in my 2D array (which may have scattered MISSING elements), and when I undo the IGNORE operation the MISSING elements are still MISSING. The question does still remain what to do when performing operations like those above in IGNORE cases. Perform the operation "underneath"? Or not? >> non-destructive + propagating = I want to ignore this datapoint for now; >> element-wise operations should replicate this "ignore" designation, and >> missingness of this type SHOULD interfere with reduction operations that involve >> this datapoint (np.sum([1,IGNORE])=IGNORE) > > Right. > > >> non-destructive + non-propagating = I want to ignore this datapoint for now; >> element-wise operations should replicate this "ignore" designation, but >> missingness of this type SHOULD NOT interfere with reduction operations that >> involve this datapoint (np.sum([1,IGNORE])=1) > > Same concerns as above. > > > Lluis > > The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. From strang at nmr.mgh.harvard.edu Fri Nov 4 12:37:33 2011 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Fri, 4 Nov 2011 12:37:33 -0400 (EDT) Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: > On Fri, Nov 4, 2011 at 11:08 AM, Llu?s wrote: > Gary Strangman writes: > [...] > > > destructive + non-propagating = the data point is truly > missing, this is the > > nature of that data point, such missingness should be > replicated in elementwise > > operations, but such missingness should NOT interfere with > reduction operations > > that involve that datapoint (np.sum([1,MISSING])=1) > > What do you define as element-wise operations? > > Is a sum on an array an element-wise operation? > > ?>>> [1, MISSING]+2 > ?[1, MISSING] > > > did you mean [3, MISSING]? > ? > Or is it just a form of reduction (after shape broadcasting)? > > ?>>> [1, MISSING]+2 > ?[3, 2] > > For me it's the second, so the only time where special values > "propagate" in a > non-propagating scenario is when you slice an array. > > > Propagation has a very specific meaning here, and I think it is causing > confusion elsewhere.? Propagation (to me) is the *exact* same behavior that > occurs with NaNs, but generalized to any dtype.? It seems like you are > taking "propagate" to mean whether the mask of the inputs follow on to the > mask of the output.? This is related, but is possibly a murkier concept and > should probably be cleaned up. I think different people have different notions of propagation here. Yes, my notion was more related to input masks "propagating" to output masks. It's important to know you define it differently ... and I think the difference in (implicit) definitions is indeed causing confusion. At least it is for me. ;-) -best Gary The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. From xscript at gmx.net Fri Nov 4 12:50:03 2011 From: xscript at gmx.net (=?utf-8?Q?Llu=C3=ADs?=) Date: Fri, 04 Nov 2011 17:50:03 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: (Benjamin Root's message of "Fri, 4 Nov 2011 11:31:01 -0500") References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: <87zkgbg7d0.fsf@ginnungagap.bsc.es> Benjamin Root writes: > On Fri, Nov 4, 2011 at 11:08 AM, Llu?s wrote: > Gary Strangman writes: > [...] >> destructive + non-propagating = the data point is truly missing, this is the >> nature of that data point, such missingness should be replicated in elementwise >> operations, but such missingness should NOT interfere with reduction operations >> that involve that datapoint (np.sum([1,MISSING])=1) > What do you define as element-wise operations? > Is a sum on an array an element-wise operation? > ?>>> [1, MISSING]+2 > ?[1, MISSING] > did you mean [3, MISSING]? Yes, sorry. > Or is it just a form of reduction (after shape broadcasting)? > ?>>> [1, MISSING]+2 > ?[3, 2] > For me it's the second, so the only time where special values "propagate" in a > non-propagating scenario is when you slice an array. > Propagation has a very specific meaning here, and I think it is causing confusion elsewhere.? Propagation (to me) is the *exact* same behavior that occurs > with NaNs, but generalized to any dtype.? It seems like you are taking "propagate" to mean whether the mask of the inputs follow on to the mask of the > output.? This is related, but is possibly a murkier concept and should probably be cleaned up. If you ignore the existence of a mask (as it is a specific mechanism for handling the destructiveness, not the propagation), I think we both think of the same concept of propagation: High-level: x + SPECIAL Propagating (SPECIAL => NaN-like => MISSING): x + SPECIAL = SPECIAL Non-propagating (SPECIAL => ignore this element, similar to nansum => IGNORE): x + SPECIAL = x Is there an agreement on this, or am I missing something else? Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From ben.root at ou.edu Fri Nov 4 14:20:49 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 4 Nov 2011 13:20:49 -0500 Subject: [Numpy-discussion] what is the point of dx for np.gradient()? Message-ID: For np.gradient(), one can specify a sample distance for each axis to apply to the gradient. But, all this does is just divides the gradient by the sample distance. I could easily do that myself with the output from gradient. Wouldn't it be more valuable to be able to specify the width of the central difference (or is there another function that does that)? Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Nov 4 14:59:48 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 04 Nov 2011 19:59:48 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: 04.11.2011 17:31, Gary Strangman kirjoitti: [clip] > The question does still remain what to do when performing operations like > those above in IGNORE cases. Perform the operation "underneath"? Or not? I have a feeling that if you don't start by mathematically defining the scalar operations first, and only after that generalize them to arrays, some conceptual problems may follow. On the other hand, I should note that numpy.ma does not work this way, and many people seem still happy with how it works. But if you go defining scalars first, as far as I see ufuncs (eg. binary operations), and assignment are what needs to be defined. Since the idea seems to be to use these as "masks", let's assume that each special value can also carry a payload. *** There are a two options how to behave with respect to binary/unary operations: (P) Propagating unop(SPECIAL_1) == SPECIAL_new binop(SPECIAL_1, SPECIAL_2) == SPECIAL_new binop(a, SPECIAL) == SPECIAL_new (N) Non-propagating unop(SPECIAL_1) == SPECIAL_new binop(SPECIAL_1, SPECIAL_2) == SPECIAL_new binop(a, SPECIAL) == binop(a, binop.identity) == a *** And three options on what to do on assignment: (d) Destructive a := SPECIAL # -> a == SPECIAL (n) Non-destructive a := SPECIAL # -> a unchanged (s) Self-destructive a := SPECIAL_1 # -> if `a` is SPECIAL-class, then a == SPECIAL_1, # otherwise `a` remains unchanged *** Finally, there is a question whether the value has a payload or not. The payload complicates the scheme, as binary and unary operations need to create new values. For singletons (eg. NaN) this is not a problem. But if it's a non-singleton, desirable behavior would be to retain commutativity (and other similar properties) of binary ops. I see two sensible approaches for this: either raise an error, or do the computation on the payload. This brings in a third choice: (S) singleton, (E) payload, but raise errors on operations only on special values, and (C) payload, but do computations on payload. *** For shorthand, we can refer to the above choices with the nomenclature ::= ::= "P" | "N" ::= "d" | "n" | "s" ::= "S" | "E" | "C" That makes 2 * 3 * 3 = 18 different ways to construct consistent behavior. Some of them might make sense, the problem is to find out which :) NAN and NA apparently fall into the PdS class. If classified this way, behaviour of items in np.ma arrays is different in different operations, but seems roughly PdX, where X stands for returning a masked value with the first argument as the payload in binary ops if either argument is masked. This makes inline binary ops behave like Nn. Reductions are N. (Assignment: dC, reductions: N, binary ops: PX, unary ops: PC, inline binary ops: Nn). Finally, there's a can of worms on specifying the outcome of binary operations on two special values of different kinds, but it's maybe best to first choose one that behaves sensibly by itself. Cheers, Pauli From pav at iki.fi Fri Nov 4 15:12:05 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 04 Nov 2011 20:12:05 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: 04.11.2011 19:59, Pauli Virtanen kirjoitti: [clip] > This makes inline binary ops > behave like Nn. Reductions are N. (Assignment: dC, reductions: N, binary > ops: PX, unary ops: PC, inline binary ops: Nn). Sorry, inline binary ops are also PdX, not Nn. -- Pauli Virtanen From tjhnson at gmail.com Fri Nov 4 15:49:17 2011 From: tjhnson at gmail.com (T J) Date: Fri, 4 Nov 2011 12:49:17 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 11:59 AM, Pauli Virtanen wrote: > > I have a feeling that if you don't start by mathematically defining the > scalar operations first, and only after that generalize them to arrays, > some conceptual problems may follow. > Yes. I was going to mention this point as well. > > For shorthand, we can refer to the above choices with the nomenclature > > ::= > ::= "P" | "N" > ::= "d" | "n" | "s" > ::= "S" | "E" | "C" > > That makes 2 * 3 * 3 = 18 different ways to construct consistent > behavior. Some of them might make sense, the problem is to find out which :) > This is great for the discussion, IMO. The self-destructive assignment hasn't come up at all, so I'm guessing we can probably ignore it. --- Can you be a bit more explicit on the payload types? Let me try, respond with corrections if necessary. "S" is singleton and in the case of "missing" data, we take it to mean that we only care that data is missing and not *how* missing the data is. >>> x = MISSING >>> -x # unary MISSING >>> x + 3 # binary MISSING "E" means that we acknowledge that we want to track the "how", but that we aren't interested in it. So raise an error. In the case of ignored data, we might have: >>> x = 2 >>> ignore(x) >>> x IGNORED(2) >>> -x Error >>> x + 3 Error "C" means that we acknowledge that we want to track the "how", and that we are interested in it. So do the computations. >>> x = 2 >>> ignore(x) >>> -x IGNORED(-2) >>> x + 3 IGNORED(5) Did I get that mostly right? > NAN and NA apparently fall into the PdS class. > Here is where I think we need ot be a bit more careful. It is true that we want NAN and MISSING to propagate, but then we additionally want to ignore it sometimes. This is precisely why we have functions like nansum. Although people are well-aware of this desire, I think this thread has largely conflated the issues when discussing "propagation". To push this forward a bit, can I propose that IGNORE behave as: PnC >>> x = np.array([1, 2, 3]) >>> y = np.array([10, 20, 30]) >>> ignore(x[2]) >>> x [1, IGNORED(2), 3] >>> x + 2 [3, IGNORED(4), 5] >>> x + y [11, IGNORED(22), 33] >>> z = x.sum() >>> z IGNORED(6) >>> unignore(z) >>> z 6 >>> x.sum(skipIGNORED=True) 4 When done in this fashion, I think it is perfectly fine for "masks to be unmasked". -------------- next part -------------- An HTML attachment was scrubbed... URL: From strang at nmr.mgh.harvard.edu Fri Nov 4 16:03:02 2011 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Fri, 4 Nov 2011 16:03:02 -0400 (EDT) Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: > > NAN and NA apparently fall into the PdS class. > > > > Here is where I think we need ot be a bit more careful.? It is true that we want > NAN and MISSING to propagate, but then we additionally want to ignore it > sometimes.? This is precisely why we have functions like nansum.? Although people > are well-aware of this desire, I think this thread has largely conflated the > issues when discussing "propagation". > > To push this forward a bit, can I propose that IGNORE behave as:?? PnC > > >>> x = np.array([1, 2, 3]) > >>> y = np.array([10, 20, 30]) > >>> ignore(x[2]) > >>> x > [1, IGNORED(2), 3] > >>> x + 2 > [3, IGNORED(4), 5] > >>> x + y > [11, IGNORED(22), 33] > >>> z = x.sum() > >>> z > IGNORED(6) > >>> unignore(z) > >>> z > 6 > >>> x.sum(skipIGNORED=True) > 4 > > When done in this fashion, I think it is perfectly fine for "masks to be > unmasked". In my mind, IGNORED items should be skipped by default (i.e., skipIGNORED seems redundant ... isn't that what ignoring is all about?). Thus I might instead suggest the opposite (default) behavior at the end: >>> x = np.array([1, 2, 3]) >>> y = np.array([10, 20, 30]) >>> ignore(x[2]) >>> x [1, IGNORED(2), 3] >>> x + 2 [3, IGNORED(4), 5] >>> x + y [11, IGNORED(22), 33] >>> z = x.sum() >>> z 4 >>> unignore(x).sum() 6 >>> x.sum(keepIGNORED=True) 6 (Obviously all the syntax is totally up for debate.) -best Gary The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. From tjhnson at gmail.com Fri Nov 4 16:22:18 2011 From: tjhnson at gmail.com (T J) Date: Fri, 4 Nov 2011 13:22:18 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 1:03 PM, Gary Strangman wrote: > > To push this forward a bit, can I propose that IGNORE behave as: PnC >> >> >>> x = np.array([1, 2, 3]) >> >>> y = np.array([10, 20, 30]) >> >>> ignore(x[2]) >> >>> x >> [1, IGNORED(2), 3] >> >>> x + 2 >> [3, IGNORED(4), 5] >> >>> x + y >> [11, IGNORED(22), 33] >> >>> z = x.sum() >> >>> z >> IGNORED(6) >> >>> unignore(z) >> >>> z >> 6 >> >>> x.sum(skipIGNORED=True) >> 4 >> >> > In my mind, IGNORED items should be skipped by default (i.e., skipIGNORED > seems redundant ... isn't that what ignoring is all about?). Thus I might > instead suggest the opposite (default) behavior at the end: > > > x = np.array([1, 2, 3]) >>>> y = np.array([10, 20, 30]) >>>> ignore(x[2]) >>>> x >>>> >>> [1, IGNORED(2), 3] > >> x + 2 >>>> >>> [3, IGNORED(4), 5] > >> x + y >>>> >>> [11, IGNORED(22), 33] > >> z = x.sum() >>>> z >>>> >>> 4 > >> unignore(x).sum() >>>> >>> 6 > >> x.sum(keepIGNORED=True) >>>> >>> 6 > > (Obviously all the syntax is totally up for debate.) > > I agree that it would be ideal if the default were to skip IGNORED values, but that behavior seems inconsistent with its propagation properties (such as when adding arrays with IGNORED values). To illustrate, when we did "x+2", we were stating that: IGNORED(2) + 2 == IGNORED(4) which means that we propagated the IGNORED value. If we were to skip them by default, then we'd have: IGNORED(2) + 2 == 2 To be consistent, then it seems we also should have had: >>> x + 2 [3, 2, 5] which I think we can agree is not so desirable. What this seems to come down to is that we tend to want different behavior when we are doing reductions, and that for IGNORED data, we want it to propagate in every situation except for a reduction (where we want to skip over it). I don't know if there is a well-defined way to distinguish reductions from the other operations. Would it hold for generalized ufuncs? Would it hold for other functions which might return arrays instead of scalars? -------------- next part -------------- An HTML attachment was scrubbed... URL: From strang at nmr.mgh.harvard.edu Fri Nov 4 16:38:05 2011 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Fri, 4 Nov 2011 16:38:05 -0400 (EDT) Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: > On Fri, Nov 4, 2011 at 1:03 PM, Gary Strangman > wrote: > > To push this forward a bit, can I propose that IGNORE behave > as:?? PnC > > >>> x = np.array([1, 2, 3]) > >>> y = np.array([10, 20, 30]) > >>> ignore(x[2]) > >>> x > [1, IGNORED(2), 3] > >>> x + 2 > [3, IGNORED(4), 5] > >>> x + y > [11, IGNORED(22), 33] > >>> z = x.sum() > >>> z > IGNORED(6) > >>> unignore(z) > >>> z > 6 > >>> x.sum(skipIGNORED=True) > 4 > > > In my mind, IGNORED items should be skipped by default (i.e., skipIGNORED > seems redundant ... isn't that what ignoring is all about?). Thus I might > instead suggest the opposite (default) behavior at the end: > > x = np.array([1, 2, 3]) > y = np.array([10, 20, 30]) > ignore(x[2]) > x > > [1, IGNORED(2), 3] > x + 2 > > [3, IGNORED(4), 5] > x + y > > [11, IGNORED(22), 33] > z = x.sum() > z > > 4 > unignore(x).sum() > > 6 > x.sum(keepIGNORED=True) > > 6 > > (Obviously all the syntax is totally up for debate.) > > > > I agree that it would be ideal if the default were to skip IGNORED values, but > that behavior seems inconsistent with its propagation properties (such as when > adding arrays with IGNORED values).? To illustrate, when we did "x+2", we were > stating that: > > IGNORED(2) + 2 == IGNORED(4) > > which means that we propagated the IGNORED value.? If we were to skip them by > default, then we'd have: > > IGNORED(2) + 2 == 2 > > To be consistent, then it seems we also should have had: > > >>> x + 2 > [3, 2, 5] > > which I think we can agree is not so desirable.?? What this seems to come down to > is that we tend to want different behavior when we are doing reductions, and that > for IGNORED data, we want it to propagate in every situation except for a > reduction (where we want to skip over it). > > I don't know if there is a well-defined way to distinguish reductions from the > other operations.? Would it hold for generalized ufuncs?? Would it hold for other > functions which might return arrays instead of scalars? Ahhh, yes. That clearly explains the issue hung-up in my mind, and also clarifies what I was getting at with the elementwise vs. reduction distinction I made earlier today. Maybe this is a pickle in a jar with no lid. I'll have to think about it ... -best Gary The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. From ben.root at ou.edu Fri Nov 4 16:42:00 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 4 Nov 2011 15:42:00 -0500 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 1:59 PM, Pauli Virtanen wrote: > > For shorthand, we can refer to the above choices with the nomenclature > > ::= > ::= "P" | "N" > ::= "d" | "n" | "s" > ::= "S" | "E" | "C" > > I really like this problem formulation and description. Can we all agree to use this language/shorthand from this point on? I think it really illuminates the discussion, and I would like to have it added to the wiki page. Thanks! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Nov 4 17:29:03 2011 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 Nov 2011 14:29:03 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 1:22 PM, T J wrote: > I agree that it would be ideal if the default were to skip IGNORED values, > but that behavior seems inconsistent with its propagation properties (such > as when adding arrays with IGNORED values).? To illustrate, when we did > "x+2", we were stating that: > > IGNORED(2) + 2 == IGNORED(4) > > which means that we propagated the IGNORED value.? If we were to skip them > by default, then we'd have: > > IGNORED(2) + 2 == 2 > > To be consistent, then it seems we also should have had: > >>>> x + 2 > [3, 2, 5] > > which I think we can agree is not so desirable.?? What this seems to come > down to is that we tend to want different behavior when we are doing > reductions, and that for IGNORED data, we want it to propagate in every > situation except for a reduction (where we want to skip over it). > > I don't know if there is a well-defined way to distinguish reductions from > the other operations.? Would it hold for generalized ufuncs?? Would it hold > for other functions which might return arrays instead of scalars? Continuing my theme of looking for consensus first... there are obviously a ton of ugly corners in here. But my impression is that at least for some simple cases, it's clear what users want: >>> a = [1, IGNORED(2), 3] # array-with-ignored-values + unignored scalar only affects unignored values >>> a + 2 [3, IGNORED(2), 5] # reduction operations skip ignored values >>> np.sum(a) 4 For example, Gary mentioned the common idiom of wanting to take an array and subtract off its mean, and he wants to do that while leaving the masked-out/ignored values unchanged. As long as the above cases work the way I wrote, we will have >>> np.mean(a) 2 >>> a -= np.mean(a) >>> a [-1, IGNORED(2), 1] Which I'm pretty sure is the result that he wants. (Gary, is that right?) Also numpy.ma follows these rules, so that's some additional evidence that they're reasonable. (And I think part of the confusion between Llu?s and me was that these are the rules that I meant when I said "non-propagating", but he understood that to mean something else.) So before we start exploring the whole vast space of possible ways to handle masked-out data, does anyone see any reason to consider rules that don't have, as a subset, the ones above? Do other rules have any use cases or user demand? (I *love* playing with clever mathematics and making things consistent, but there's not much point unless the end result is something that people will use :-).) -- Nathaniel From pav at iki.fi Fri Nov 4 17:41:51 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 04 Nov 2011 22:41:51 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: 04.11.2011 20:49, T J kirjoitti: [clip] > To push this forward a bit, can I propose that IGNORE behave as: PnC The *n* classes can be a bit confusing in Python: ### PnC >>> x = np.array([1, 2, 3]) >>> y = np.array([4, 5, 6]) >>> ignore(y[1]) >>> z = x + y >>> z np.array([5, IGNORE(7), 9]) >>> x += y # NB: x[1] := x[1] + y[1] >>> x np.array([5, 2, 3]) *** I think I defined the "destructive" and "non-destructive" in a different way than earlier in the thread. Maybe this behavior from np.ma is closer to what was meant earlier: >>> x = np.ma.array([1, 2, 3], mask=[0, 0, 1]) >>> y = np.ma.array([4, 5, 6], mask=[0, 1, 1]) >>> x += y >>> x masked_array(data = [5 -- --], mask = [False True True], fill_value = 999999) >>> x.data array([5, 2, 3]) Let's call this (since I botched and already reserved the letter "n" :) (m) mark-ignored a := SPECIAL_1 # -> a == SPECIAL_a ; the payload of the RHS is neglected, # the assigned value has the original LHS # as the payload -- Pauli Virtanen From tjhnson at gmail.com Fri Nov 4 17:57:42 2011 From: tjhnson at gmail.com (T J) Date: Fri, 4 Nov 2011 14:57:42 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 2:41 PM, Pauli Virtanen wrote: > 04.11.2011 20:49, T J kirjoitti: > [clip] > > To push this forward a bit, can I propose that IGNORE behave as: PnC > > The *n* classes can be a bit confusing in Python: > > ### PnC > > >>> x = np.array([1, 2, 3]) > >>> y = np.array([4, 5, 6]) > >>> ignore(y[1]) > >>> z = x + y > >>> z > np.array([5, IGNORE(7), 9]) > >>> x += y # NB: x[1] := x[1] + y[1] > >>> x > np.array([5, 2, 3]) > > *** > > Interesting. > I think I defined the "destructive" and "non-destructive" in a different > way than earlier in the thread. Maybe this behavior from np.ma is closer > to what was meant earlier: > > >>> x = np.ma.array([1, 2, 3], mask=[0, 0, 1]) > >>> y = np.ma.array([4, 5, 6], mask=[0, 1, 1]) > >>> x += y > >>> x > masked_array(data = [5 -- --], > mask = [False True True], > fill_value = 999999) > >>> x.data > array([5, 2, 3]) > > > Let's call this (since I botched and already reserved the letter "n" :) > > (m) mark-ignored > > a := SPECIAL_1 > # -> a == SPECIAL_a ; the payload of the RHS is neglected, > # the assigned value has the original LHS > # as the payload > > > Does this behave as expected for "x + y" (as opposed to the inplace operation)? >>> z = x + y >>> z np.array([5, IGNORED(2), IGNORED(3)]) >>> x += y np.array([5, IGNORED(2), IGNORED(3)]) However, doesn't this have the issue that Nathaniel brought up earlier: commutativity unignore(x + y) != unignore(y + x) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Nov 4 18:04:49 2011 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 Nov 2011 15:04:49 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 11:59 AM, Pauli Virtanen wrote: > I have a feeling that if you don't start by mathematically defining the > scalar operations first, and only after that generalize them to arrays, > some conceptual problems may follow. > > On the other hand, I should note that numpy.ma does not work this way, > and many people seem still happy with how it works. Yes, my impression is that people who want MISSING just want something that acts like a special scalar value (PdS, in your scheme), but the people who want IGNORED want something that *can't* be defined in this way (see my other recent post). That said... > There are a two options how to behave with respect to binary/unary > operations: > > (P) Propagating > > unop(SPECIAL_1) == SPECIAL_new > binop(SPECIAL_1, SPECIAL_2) == SPECIAL_new > binop(a, SPECIAL) == SPECIAL_new > > (N) Non-propagating > > unop(SPECIAL_1) == SPECIAL_new > binop(SPECIAL_1, SPECIAL_2) == SPECIAL_new > binop(a, SPECIAL) == binop(a, binop.identity) == a SPECIAL_1 means "a special value with payload 1", right? Same thing that some of us have been writing IGNORED(1) in other places? Assuming that, I believe that what people want for IGNORED values is unop(SPECIAL_1) == SPECIAL_1 which doesn't seem to be an option in your taxonomy. There's also the option of binop(a, SPECIAL) -> error. > And three options on what to do on assignment: > > (d) Destructive > > a := SPECIAL ? ? ?# -> a == SPECIAL > > (n) Non-destructive > > a := SPECIAL ? ? ?# -> a unchanged > > (s) Self-destructive > > a := SPECIAL_1 > # -> if `a` is SPECIAL-class, then a == SPECIAL_1, > # otherwise `a` remains unchanged I'm not sure "assignment" is a useful way to think about what we've been calling IGNORED values (for MISSING/NA it's fine). I've been talking about masking/unmasking values or "toggling the IGNORED state", because my impression is that what people want is something like: a[0] = 3 a[0] = SPECIAL # now a[0] == SPECIAL(3) This is pretty confusing when written as an assignment (and note that now I'm assigning into an array, because if I were just assigning to a python variable then these semantics would be impossible to implement!). So we might prefer a syntax like a.visible[0] = False or a.ignore(0) > If classified this way, behaviour of items in np.ma arrays is different > in different operations, but seems roughly PdX, where X stands for > returning a masked value with the first argument as the payload in > binary ops if either argument is masked. No -- np.ma implements the assignment semantics I described above, not "d" semantics. Trimming some output for readability: >>> a = np.ma.masked_array([1, 2, 3]) >>> a[1] = np.ma.masked >>> a [1, --, 3] >>> a.mask[1] = False >>> a [1, 2, 3] So assignment is not destructive -- the old value is retained as the "payload". -- Nathaniel From tjhnson at gmail.com Fri Nov 4 18:08:37 2011 From: tjhnson at gmail.com (T J) Date: Fri, 4 Nov 2011 15:08:37 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 2:29 PM, Nathaniel Smith wrote: > On Fri, Nov 4, 2011 at 1:22 PM, T J wrote: > > I agree that it would be ideal if the default were to skip IGNORED > values, > > but that behavior seems inconsistent with its propagation properties > (such > > as when adding arrays with IGNORED values). To illustrate, when we did > > "x+2", we were stating that: > > > > IGNORED(2) + 2 == IGNORED(4) > > > > which means that we propagated the IGNORED value. If we were to skip > them > > by default, then we'd have: > > > > IGNORED(2) + 2 == 2 > > > > To be consistent, then it seems we also should have had: > > > >>>> x + 2 > > [3, 2, 5] > > > > which I think we can agree is not so desirable. What this seems to come > > down to is that we tend to want different behavior when we are doing > > reductions, and that for IGNORED data, we want it to propagate in every > > situation except for a reduction (where we want to skip over it). > > > > I don't know if there is a well-defined way to distinguish reductions > from > > the other operations. Would it hold for generalized ufuncs? Would it > hold > > for other functions which might return arrays instead of scalars? > > Continuing my theme of looking for consensus first... there are > obviously a ton of ugly corners in here. But my impression is that at > least for some simple cases, it's clear what users want: > > >>> a = [1, IGNORED(2), 3] > # array-with-ignored-values + unignored scalar only affects unignored > values > >>> a + 2 > [3, IGNORED(2), 5] > # reduction operations skip ignored values > >>> np.sum(a) > 4 > > For example, Gary mentioned the common idiom of wanting to take an > array and subtract off its mean, and he wants to do that while leaving > the masked-out/ignored values unchanged. As long as the above cases > work the way I wrote, we will have > > >>> np.mean(a) > 2 > >>> a -= np.mean(a) > >>> a > [-1, IGNORED(2), 1] > > Which I'm pretty sure is the result that he wants. (Gary, is that > right?) Also numpy.ma follows these rules, so that's some additional > evidence that they're reasonable. (And I think part of the confusion > between Llu?s and me was that these are the rules that I meant when I > said "non-propagating", but he understood that to mean something > else.) > > So before we start exploring the whole vast space of possible ways to > handle masked-out data, does anyone see any reason to consider rules > that don't have, as a subset, the ones above? Do other rules have any > use cases or user demand? (I *love* playing with clever mathematics > and making things consistent, but there's not much point unless the > end result is something that people will use :-).) > I guess I'm just confused on how one, in principle, would distinguish the various forms of propagation that you are suggesting (ie for reductions). I also don't think it is good that we lack commutativity. If we disallow unignoring, then yes, I agree that what you wrote above is what people want. But if we are allowed to unignore, then I do not. Also, how does something like this get handled? >>> a = [1, 2, IGNORED(3), NaN] If I were to say, "What is the mean of 'a'?", then I think most of the time people would want 1.5. I guess if we kept nanmean around, then we could do: >>> a -= np.nanmean(a) [-.5, .5, IGNORED(3), NaN] Sorry if this is considered digging deeper than consensus. I'm just curious if arrays having NaNs in them, in addition to IGNORED, causes problems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Nov 4 18:12:26 2011 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 Nov 2011 15:12:26 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 3:04 PM, Nathaniel Smith wrote: > On Fri, Nov 4, 2011 at 11:59 AM, Pauli Virtanen wrote: >> If classified this way, behaviour of items in np.ma arrays is different >> in different operations, but seems roughly PdX, where X stands for >> returning a masked value with the first argument as the payload in >> binary ops if either argument is masked. > > No -- np.ma implements the assignment semantics I described above, not > "d" semantics. Trimming some output for readability: Oops, I see we cross-posted :-). So never-mind that part of my email... Okay, I'm going to follow my own suggestion now and stop talking about these thorny details until I know whether we can simplify the discussion to only considering schemes that are consistent with the rules I posted about here: http://article.gmane.org/gmane.comp.python.numeric.general/46760 -- Nathaniel From pav at iki.fi Fri Nov 4 18:29:03 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 04 Nov 2011 23:29:03 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: 04.11.2011 22:57, T J kirjoitti: [clip] > > (m) mark-ignored > > > > a := SPECIAL_1 > > # -> a == SPECIAL_a ; the payload of the RHS is neglected, > > # the assigned value has the original LHS > > # as the payload [clip] > Does this behave as expected for "x + y" (as opposed to the inplace > operation)? [clip] The definition is for assignment, and does not concern binary ops, so it behaves as expected. The inplace operation >>> x += y is defined as equivalent to binary op followed by assignment >>> x[:] = x + y as far as the missing values are concerned. > However, doesn't this have the issue that Nathaniel brought up earlier: > commutativity > > unignore(x + y) != unignore(y + x) As the definition concerns only what happens on assignment, it does not have problems with commutativity. -- Pauli Virtanen From njs at pobox.com Fri Nov 4 18:38:48 2011 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 Nov 2011 15:38:48 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 3:08 PM, T J wrote: > On Fri, Nov 4, 2011 at 2:29 PM, Nathaniel Smith wrote: >> Continuing my theme of looking for consensus first... there are >> obviously a ton of ugly corners in here. But my impression is that at >> least for some simple cases, it's clear what users want: >> >> >>> a = [1, IGNORED(2), 3] >> # array-with-ignored-values + unignored scalar only affects unignored >> values >> >>> a + 2 >> [3, IGNORED(2), 5] >> # reduction operations skip ignored values >> >>> np.sum(a) >> 4 >> >> For example, Gary mentioned the common idiom of wanting to take an >> array and subtract off its mean, and he wants to do that while leaving >> the masked-out/ignored values unchanged. As long as the above cases >> work the way I wrote, we will have >> >> >>> np.mean(a) >> 2 >> >>> a -= np.mean(a) >> >>> a >> [-1, IGNORED(2), 1] >> >> Which I'm pretty sure is the result that he wants. (Gary, is that >> right?) Also numpy.ma follows these rules, so that's some additional >> evidence that they're reasonable. (And I think part of the confusion >> between Llu?s and me was that these are the rules that I meant when I >> said "non-propagating", but he understood that to mean something >> else.) >> >> So before we start exploring the whole vast space of possible ways to >> handle masked-out data, does anyone see any reason to consider rules >> that don't have, as a subset, the ones above? Do other rules have any >> use cases or user demand? (I *love* playing with clever mathematics >> and making things consistent, but there's not much point unless the >> end result is something that people will use :-).) > > I guess I'm just confused on how one, in principle, would distinguish the > various forms of propagation that you are suggesting (ie for reductions). Well, numpy.ma does work this way, so certainly it's possible to do. At the code level, np.add() and np.add.reduce() are different entry points and can behave differently. OTOH, it might be that it's impossible to do *while still maintaining other things we care about*... but in that case we should just shake our fists at the mathematics and then give up, instead of coming up with an elegant system that isn't actually useful. So that's why I think we should figure out what's useful first. > I also don't think it is good that we lack commutativity.? If we disallow > unignoring, then yes, I agree that what you wrote above is what people > want.? But if we are allowed to unignore, then I do not. I *think* that for the no-unignoring (also known as "MISSING") case, we have a pretty clear consensus that we want something like: >>> a + 2 [3, MISSING, 5] >>> np.sum(a) MISSING >>> np.sum(a, skip_MISSING=True) 4 (Please say if you disagree, but I really hope you don't!) This case is also easier, because we don't even have to allow a skip_MISSING flag in cases where it doesn't make sense (e.g., unary or binary operations) -- it's a convenience feature, so no-one will care if it only works when it's useful ;-). The use case that we're still confused about is specifically the one where people want to *temporarily* hide parts of their data, do some calculations that ignore those parts of their data, and then unhide that data again -- e.g., see Gary's first post in this thread. So for this use case, allowing unignore is definitely important, and having np.sum() return IGNORED seems pretty useless to me. (When an operation involves actually missing data, then you need to stop and think what would be a statistically meaningful way to handle that -- sometimes it's skip_MISSING, sometimes something else. So np.sum returning MISSING is useful - it tells you something you might not have realized. If you just ignored some data because you want to ignore that data, then having np.sum return IGNORED is useless, because it tells you something you already knew perfectly well.) > Also, how does something like this get handled? > >>>> a = [1, 2, IGNORED(3), NaN] > > If I were to say, "What is the mean of 'a'?", then I think most of the time > people would want 1.5. I would want NaN! But that's because the only way I get NaN's is when I do dumb things like compute log(0), and again, I want my code to tell me that I was dumb instead of just quietly making up a meaningless answer. -- Nathaniel From pav at iki.fi Fri Nov 4 19:07:24 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 05 Nov 2011 00:07:24 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: 04.11.2011 23:04, Nathaniel Smith kirjoitti: [clip] > Assuming that, I believe that what people want for IGNORED values is > unop(SPECIAL_1) == SPECIAL_1 > which doesn't seem to be an option in your taxonomy. Well, you can always add a new branch for rules on what to do with unary ops. [clip] > I'm not sure "assignment" is a useful way to think about what we've > been calling IGNORED values (for MISSING/NA it's fine). I've been > talking about masking/unmasking values or "toggling the IGNORED > state", because my impression is that what people want is something > like: > > a[0] = 3 > a[0] = SPECIAL > # now a[0] == SPECIAL(3) That's partly syntax sugar. What I meant above by assignment is what happens on a[:] = b and what should occur in in-place operations, a += b which are equivalent to a[:] = a + b Yeah, it's a different definition for "destructive" and "non-destructive" than what was used earlier in the discussion. [clip] >> If classified this way, behaviour of items in np.ma arrays is different >> in different operations, but seems roughly PdX, where X stands for >> returning a masked value with the first argument as the payload in >> binary ops if either argument is masked. > > No -- np.ma implements the assignment semantics I described above, not > "d" semantics. Trimming some output for readability: Well, np.ma implements "d" semantics, but because of the way binary ops are noncommutative, in-place binary ops behave as if they were not mutating. Assignments do actually change the masked data: >>> a[:] = b which changes also masked values in `a`. That may be a bug. -- Pauli Virtanen From tjhnson at gmail.com Fri Nov 4 19:14:54 2011 From: tjhnson at gmail.com (T J) Date: Fri, 4 Nov 2011 16:14:54 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 3:38 PM, Nathaniel Smith wrote: > On Fri, Nov 4, 2011 at 3:08 PM, T J wrote: > > On Fri, Nov 4, 2011 at 2:29 PM, Nathaniel Smith wrote: > >> Continuing my theme of looking for consensus first... there are > >> obviously a ton of ugly corners in here. But my impression is that at > >> least for some simple cases, it's clear what users want: > >> > >> >>> a = [1, IGNORED(2), 3] > >> # array-with-ignored-values + unignored scalar only affects unignored > >> values > >> >>> a + 2 > >> [3, IGNORED(2), 5] > >> # reduction operations skip ignored values > >> >>> np.sum(a) > >> 4 > >> > >> For example, Gary mentioned the common idiom of wanting to take an > >> array and subtract off its mean, and he wants to do that while leaving > >> the masked-out/ignored values unchanged. As long as the above cases > >> work the way I wrote, we will have > >> > >> >>> np.mean(a) > >> 2 > >> >>> a -= np.mean(a) > >> >>> a > >> [-1, IGNORED(2), 1] > >> > >> Which I'm pretty sure is the result that he wants. (Gary, is that > >> right?) Also numpy.ma follows these rules, so that's some additional > >> evidence that they're reasonable. (And I think part of the confusion > >> between Llu?s and me was that these are the rules that I meant when I > >> said "non-propagating", but he understood that to mean something > >> else.) > >> > >> So before we start exploring the whole vast space of possible ways to > >> handle masked-out data, does anyone see any reason to consider rules > >> that don't have, as a subset, the ones above? Do other rules have any > >> use cases or user demand? (I *love* playing with clever mathematics > >> and making things consistent, but there's not much point unless the > >> end result is something that people will use :-).) > > > > I guess I'm just confused on how one, in principle, would distinguish the > > various forms of propagation that you are suggesting (ie for reductions). > > Well, numpy.ma does work this way, so certainly it's possible to do. > At the code level, np.add() and np.add.reduce() are different entry > points and can behave differently. > I see your point, but that seems like just an API difference with a bad name. reduce() is just calling add() a bunch of times, so it seems like it should behave as add() does. That we can create different behaviors with various assignment rules (like Pauli's 'm' for mark-ignored), only makes it more confusing to me. a = 1 a += 2 a += IGNORE b = 1 + 2 + IGNORE I think having a == b is essential. If they can be different, that will only lead to confusion. On this point alone, does anyone think it is acceptable to have a != b? > > OTOH, it might be that it's impossible to do *while still maintaining > other things we care about*... but in that case we should just shake > our fists at the mathematics and then give up, instead of coming up > with an elegant system that isn't actually useful. So that's why I > think we should figure out what's useful first. > Agreed. I'm on the same page. > > > I also don't think it is good that we lack commutativity. If we disallow > > unignoring, then yes, I agree that what you wrote above is what people > > want. But if we are allowed to unignore, then I do not. > > I *think* that for the no-unignoring (also known as "MISSING") case, > we have a pretty clear consensus that we want something like: > > >>> a + 2 > [3, MISSING, 5] > >>> np.sum(a) > MISSING > >>> np.sum(a, skip_MISSING=True) > 4 > > (Please say if you disagree, but I really hope you don't!) This case > is also easier, because we don't even have to allow a skip_MISSING > flag in cases where it doesn't make sense (e.g., unary or binary > operations) -- it's a convenience feature, so no-one will care if it > only works when it's useful ;-). > > Yes, in agreement. I was talking specifically about the IGNORE case. And my point is that if we allow people to remove the IGNORE flag and see the original data (and if the payloads are computed), then we should care about commutativity: >>> x = [1, IGNORE(2), 3] >>> x2 = x.copy() >>> y = [10, 11, IGNORE(12)] >>> z = x + y >>> a = z.sum() >>> x += y >>> b = x.sum() >>> y += x2 >>> c = y.sum() So, we should have: a == b == c. Additionally, if we allow users to unignore data, then we should have: >>> x = [1, IGNORE(2), 3] >>> x2 = x.copy() >>> y = [10, 11, IGNORE(12)] >>> x += y >>> aa = unignore(x).sum() >>> y += x2 >>> bb = unignore(y).sum() >>> aa == bb True Is there agreement on this? > Also, how does something like this get handled? > > > >>>> a = [1, 2, IGNORED(3), NaN] > > > > If I were to say, "What is the mean of 'a'?", then I think most of the > time > > people would want 1.5. > > > I would want NaN! But that's because the only way I get NaN's is when > I do dumb things like compute log(0), and again, I want my code to > tell me that I was dumb instead of just quietly making up a > meaningless answer. > > That's definitely field specific then. In probability: 0 = 0 log(0) is a common idiom. In NumPy, 0 log(0) gives NaN, so you'd want to ignore then when summing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Nov 4 19:21:03 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 4 Nov 2011 16:21:03 -0700 Subject: [Numpy-discussion] Int casting different across platforms Message-ID: Hi, I noticed this: (Intel Mac): In [2]: np.int32(np.float32(2**31)) Out[2]: -2147483648 (PPC): In [3]: np.int32(np.float32(2**31)) Out[3]: 2147483647 I assume what is happening is that the casting is handing off to the c library, and that behavior of the c library differs on these platforms? Should we expect or hope that this behavior would be the same across platforms? Thanks for any pointers, Matthew From pav at iki.fi Fri Nov 4 19:29:56 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 05 Nov 2011 00:29:56 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: 04.11.2011 23:29, Pauli Virtanen kirjoitti: [clip] > As the definition concerns only what happens on assignment, it does not > have problems with commutativity. This is of course then not really true in a wider sense, as an example from "T J" shows: a = 1 a += IGNORE(3) # -> a := a + IGNORE(3) # -> a := IGNORE(4) # -> a == IGNORE(1) which is different from a = 1 + IGNORE(3) # -> a == IGNORE(4) Damn, it seemed so good. Probably anything expect destructive assignment leads to problems like this with propagating special values. -- Pauli Virtanen From tjhnson at gmail.com Fri Nov 4 20:26:04 2011 From: tjhnson at gmail.com (T J) Date: Fri, 4 Nov 2011 17:26:04 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 4:29 PM, Pauli Virtanen wrote: > 04.11.2011 23:29, Pauli Virtanen kirjoitti: > [clip] > > As the definition concerns only what happens on assignment, it does not > > have problems with commutativity. > > This is of course then not really true in a wider sense, as an example > from "T J" shows: > > a = 1 > a += IGNORE(3) > # -> a := a + IGNORE(3) > # -> a := IGNORE(4) > # -> a == IGNORE(1) > > which is different from > > a = 1 + IGNORE(3) > # -> a == IGNORE(4) > > Damn, it seemed so good. Probably anything expect destructive assignment > leads to problems like this with propagating special values. > > > Ok...with what I understand now, it seems like for almost all operations: MISSING : PdS IGNORED : PdC (this gives commutivity when unignoring data points) When you want some sort of "reduction", we want to change the behavior for IGNORED so that it skips the IGNORED values by default. Personally, I still believe that this non-consistent behavior warrants a new method name. What I mean is: >>> x = np.array([1, IGNORED(2), 3]) >>> y = x.sum() >>> z = x[0] + x[1] + x[2] To say that y != z will only be a source of confusion. To remedy, we force people to be explicit, even if they'll need to be explicit 99% of the time: >>> q = x.sum(skipIGNORED=True) Then we can have y == z and y != q. To make the 99% use case easier, we provide a new method which passings the keyword for us. ---- With PdS and PdC is seems rather clear to me why MISSING should be implemented as a bit pattern and IGNORED implemented using masks. Setting implementation details aside and going back to Nathaniel's original "biggest *un*resolved question", I am now convinced that these (IGNORED and MISSING) should be distinct API concepts and still yet distinct from NaN with floating point dtypes. The NA implementation in NumPy does not seem to match either of these (IGNORED and MISSING) exactly. One cannot, as far as I know, unignore an element marked as NA. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Nov 4 20:37:40 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 05 Nov 2011 01:37:40 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: 04.11.2011 22:29, Nathaniel Smith kirjoitti: [clip] > Continuing my theme of looking for consensus first... there are > obviously a ton of ugly corners in here. But my impression is that at > least for some simple cases, it's clear what users want: > >>>> a = [1, IGNORED(2), 3] > # array-with-ignored-values + unignored scalar only affects unignored values >>>> a + 2 > [3, IGNORED(2), 5] > # reduction operations skip ignored values >>>> np.sum(a) > 4 This can break commutativity: >>> a = [1, IGNORED(2), 3] >>> b = [4, IGNORED(5), 6] >>> x = a + b >>> y = b + a >>> x[1] = ??? >>> y[1] = ??? Defining unop(IGNORED(a)) == IGNORED(a) binop(IGNORED(a), b) == IGNORED(a) binop(a, IGNORED(b)) == IGNORED(b) binop(IGNORED(a), IGNORED(b)) == IGNORED(binop(a, b)) # or NA could however get around that. That seems to be pretty much how NaN works, except that it now carries a "hidden" value with it. > For example, Gary mentioned the common idiom of wanting to take an > array and subtract off its mean, and he wants to do that while leaving > the masked-out/ignored values unchanged.As long as the above cases > work the way I wrote, we will have > >>>> np.mean(a) > 2 >>>> a -= np.mean(a) >>>> a > [-1, IGNORED(2), 1] That would be propagating + the above NaN-like rules for binary operators. Whether the reduction methods have skip_IGNORE=True as default or not is in my opinion more of an API question, rather than a question on how the algebra of ignored values should work. *** If destructive assignment is really needed to avoid problems with commutation, [see T. J. (2011)] is then maybe a problem. So, one would need to have >>> x = [1, IGNORED(2), 3] >>> y = [1, IGNORED(2), 3] >>> z = [4, IGNORED(5), IGNORED(6)] >>> x[:] = z >>> x [4, IGNORED(5), IGNORED(6)] >>> y += z >>> y [4, IGNORED(7), IGNORED(6)] This is not how np.ma works. But if you do otherwise, there doesn't seem to be any guarantee that >>> a += 42 >>> a += b is the same thing as >>> a += b >>> a += 42 [clip] > So before we start exploring the whole vast space of possible ways to > handle masked-out data, does anyone see any reason to consider rules > that don't have, as a subset, the ones above? Do other rules have any > use cases or user demand? (I *love* playing with clever mathematics > and making things consistent, but there's not much point unless the > end result is something that people will use :-).) Yep, it's important to keep in mind what people want. People however tend to implicitly expect that simple arithmetic operations on arrays, containing ignored values or not, operate in a certain way. Actually stating how these operations work with scalars gives valuable insight on how you'd like things to work. Also, if you propose to break the rules of arithmetic, in a fundamental library meant for scientific computation, you should be aware that you do so, and how you do so. I mean, at least for me it was not clear before this formulation that there was a reason why binary ops in np.ma were not commutative! Now I kind of see that there is an asymmetry in assignment into masked arrays, and there is a conflict with commuting operations and with "what you'd expect ignored values to do". I'm not sure if it's possible to get rid of this problem, but it could be possible to restrict it to assignments and in-place operations rather than having it in binary ops. -- Pauli Virtanen From strang at nmr.mgh.harvard.edu Fri Nov 4 21:24:25 2011 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Fri, 4 Nov 2011 21:24:25 -0400 (EDT) Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: >> Also, how does something like this get handled? >> >>>>> a = [1, 2, IGNORED(3), NaN] >> >> If I were to say, "What is the mean of 'a'?", then I think most of the time >> people would want 1.5. > > I would want NaN! But that's because the only way I get NaN's is when > I do dumb things like compute log(0), and again, I want my code to > tell me that I was dumb instead of just quietly making up a > meaningless answer. As another data point, I prefer control over this sort of situation. Sometimes I'm completely in agreement with Nathaniel and want the operation to fail. Other times I am forced to perform operations (e.g. log) on a huge matrix, and I fully expect some 0s may be in there. For a complex enough chain of operations, looking for all the bad apples at each step in the chain can be prohibitive, so in those cases I'm looking for "compute it if you can, and give me a NaN if you can't" ... G The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. From pav at iki.fi Fri Nov 4 21:31:46 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 05 Nov 2011 02:31:46 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: 05.11.2011 00:14, T J kirjoitti: [clip] > a = 1 > a += 2 > a += IGNORE > b = 1 + 2 + IGNORE > > I think having a == b is essential. If they can be different, that will > only lead to confusion. On this point alone, does anyone think it is > acceptable to have a != b? It seems to me that requiring this sort of a thing gives some limitations on how array operations should behave. An acid test for proposed rules: given two arrays `a` and `b`, a = [1, 2, IGNORED(3), IGNORED(4)] b = [10, IGNORED(20), 30, IGNORED(40)] (a) Are the following pieces of code equivalent: print unmask(a + b) a += 42 a += b print unmask(a) and print unmask(b + a) a += b a += 42 print unmask(a) (b) Are the following two statements equivalent (wrt. ignored values): a += b a[:] = a + b For np.ma (a) is false whereas (b) is true. For arrays containing nans, on the other hand (a) and (b) are both true (but of course, in this case values cannot be unmasked). Is there a way to define operations so that (a) is true, while retaining the desired other properties of arrays with ignored values? Is there a real-word need to have (a) be true? -- Pauli Virtanen From tjhnson at gmail.com Fri Nov 4 22:43:25 2011 From: tjhnson at gmail.com (T J) Date: Fri, 4 Nov 2011 19:43:25 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 6:31 PM, Pauli Virtanen wrote: > 05.11.2011 00:14, T J kirjoitti: > [clip] > > a = 1 > > a += 2 > > a += IGNORE > > b = 1 + 2 + IGNORE > > > > I think having a == b is essential. If they can be different, that will > > only lead to confusion. On this point alone, does anyone think it is > > acceptable to have a != b? > > It seems to me that requiring this sort of a thing gives some > limitations on how array operations should behave. > > An acid test for proposed rules: given two arrays `a` and `b`, > > a = [1, 2, IGNORED(3), IGNORED(4)] > b = [10, IGNORED(20), 30, IGNORED(40)] > > (a) Are the following pieces of code equivalent: > > print unmask(a + b) > a += 42 > a += b > print unmask(a) > > and > > print unmask(b + a) > a += b > a += 42 > print unmask(a) > > (b) Are the following two statements equivalent (wrt. ignored values): > > a += b > a[:] = a + b > > For np.ma (a) is false whereas (b) is true. For arrays containing nans, > on the other hand (a) and (b) are both true (but of course, in this case > values cannot be unmasked). > > Is there a way to define operations so that (a) is true, while retaining > the desired other properties of arrays with ignored values? > > I thought that "PdC" satisfied (a) and (b). Let me show you what I thought they were. Perhaps I am not being consistent. If so, point out my mistake. (A) You are making two comparisons. (A1) Does unmask(a+b) == unmask(b + a) ? Yes. They both equal: unmask([11, IGNORED(22), IGNORED(33), IGNORED(44)]) = [11, 22, 33, 44] (A2) Is 'a' the same after "a += 42; a += b" and "a += b; a += 42"? Yes. >>> a = [1, 2, IGNORED(3), IGNORED(4)] >>> b = [10, IGNORED(20), 30, IGNORED(40)] >>> a += 42; a [43, 44, IGNORED(45), IGNORED(46)] >>> a += b; a [53, IGNORED(64), IGNORED(75), IGNORED(86)] vs >>> a = [1, 2, IGNORED(3), IGNORED(4)] >>> b = [10, IGNORED(20), 30, IGNORED(40)] >>> a += b; a [11, IGNORED(22), IGNORED(33), IGNORED(44)] >>> a += 42; a [53, IGNORED(64), IGNORED(75), IGNORED(86)] For part (B), I thought we were in agreement that inplace assignment should be defined so that: a += b is equivalent to: tmp = a + b a = tmp If so, this definitely holds. Have I missed something? Probably. Please spell it out for me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Nov 4 23:03:39 2011 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 Nov 2011 20:03:39 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 7:43 PM, T J wrote: > On Fri, Nov 4, 2011 at 6:31 PM, Pauli Virtanen wrote: >> An acid test for proposed rules: given two arrays `a` and `b`, >> >> ? ? ? ? a = [1, 2, IGNORED(3), IGNORED(4)] >> ? ? ? ?b = [10, IGNORED(20), 30, IGNORED(40)] [...] > (A1)? Does? unmask(a+b) == unmask(b + a) ? > > Yes.? They both equal: > > ?? unmask([11, IGNORED(22), IGNORED(33), IGNORED(44)]) > ???? = > ?? [11, 22, 33, 44] Again, I really don't think you're going to be able to sell an API where [2] + [IGNORED(20)] == [IGNORED(22)] I mean, it's not me you have to convince, it's Gary, Pierre, maybe Benjamin, Llu?s, etc. So I could be wrong. But you might want to figure that out first before making plans based on this... -- Nathaniel From tjhnson at gmail.com Fri Nov 4 23:33:34 2011 From: tjhnson at gmail.com (T J) Date: Fri, 4 Nov 2011 20:33:34 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 8:03 PM, Nathaniel Smith wrote: > On Fri, Nov 4, 2011 at 7:43 PM, T J wrote: > > On Fri, Nov 4, 2011 at 6:31 PM, Pauli Virtanen wrote: > >> An acid test for proposed rules: given two arrays `a` and `b`, > >> > >> a = [1, 2, IGNORED(3), IGNORED(4)] > >> b = [10, IGNORED(20), 30, IGNORED(40)] > [...] > > (A1) Does unmask(a+b) == unmask(b + a) ? > > > > Yes. They both equal: > > > > unmask([11, IGNORED(22), IGNORED(33), IGNORED(44)]) > > = > > [11, 22, 33, 44] > > Again, I really don't think you're going to be able to sell an API where > [2] + [IGNORED(20)] == [IGNORED(22)] > I mean, it's not me you have to convince, it's Gary, Pierre, maybe > Benjamin, Llu?s, etc. So I could be wrong. But you might want to > figure that out first before making plans based on this... > But this is how np.ma currently does it, except that it doesn't compute the payload---it just calls it IGNORED. And it seems that this generalizes the way people want it to: >>> z = [2, 4] + [IGNORED(20), 3] >>> z [IGNORED(24), 7] >>> z.sum(skip_ignored=True) # True could be the default 7 >>> z.sum(skip_ignored=False) IGNORED(31) I guess I am confused because it seems that you implicitly used this same rule here: Say we have >>> a = np.array([1, IGNORED(2), 3]) >>> b = np.array([10, 20, 30]) (Here's I'm using IGNORED(2) to mean a value that is currently ignored, but if you unmasked it it would have the value 2.) Then we have: # non-propagating **or** propagating, doesn't matter: >>> a + 2 [3, IGNORED(2), 5] That is, element-wise, you had to have done: IGNORED(2) + 2 --> IGNORED(2). I said it should be equal to IGNORED(4), but the result is still some form of ignore. Sorry if I am missing the bigger picture at this point....its late and a Fri. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Fri Nov 4 23:45:00 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 4 Nov 2011 22:45:00 -0500 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 10:33 PM, T J wrote: > On Fri, Nov 4, 2011 at 8:03 PM, Nathaniel Smith wrote: > >> On Fri, Nov 4, 2011 at 7:43 PM, T J wrote: >> > On Fri, Nov 4, 2011 at 6:31 PM, Pauli Virtanen wrote: >> >> An acid test for proposed rules: given two arrays `a` and `b`, >> >> >> >> a = [1, 2, IGNORED(3), IGNORED(4)] >> >> b = [10, IGNORED(20), 30, IGNORED(40)] >> [...] >> > (A1) Does unmask(a+b) == unmask(b + a) ? >> > >> > Yes. They both equal: >> > >> > unmask([11, IGNORED(22), IGNORED(33), IGNORED(44)]) >> > = >> > [11, 22, 33, 44] >> >> Again, I really don't think you're going to be able to sell an API where >> [2] + [IGNORED(20)] == [IGNORED(22)] >> I mean, it's not me you have to convince, it's Gary, Pierre, maybe >> Benjamin, Llu?s, etc. So I could be wrong. But you might want to >> figure that out first before making plans based on this... >> > > But this is how np.ma currently does it, except that it doesn't compute > the payload---it just calls it IGNORED. > And it seems that this generalizes the way people want it to: > > >>> z = [2, 4] + [IGNORED(20), 3] > >>> z > [IGNORED(24), 7] > >>> z.sum(skip_ignored=True) # True could be the default > 7 > >>> z.sum(skip_ignored=False) > IGNORED(31) > > I guess I am confused because it seems that you implicitly used this same > rule here: > > Say we have > >>> a = np.array([1, IGNORED(2), 3]) > > >>> b = np.array([10, 20, 30]) > (Here's I'm using IGNORED(2) to mean a value that is currently > ignored, but if you unmasked it it would have the value 2.) > > Then we have: > > # non-propagating **or** propagating, doesn't matter: > > >>> a + 2 > > [3, IGNORED(2), 5] > > > That is, element-wise, you had to have done: > > IGNORED(2) + 2 --> IGNORED(2). > > I said it should be equal to IGNORED(4), but the result is still some form > of ignore. Sorry if I am missing the bigger picture at this point....its > late and a Fri. > > > This scheme is actually somewhat intriguing. Not totally convinced, by intrigued. Unfortunately, I fell behind in the postings by having dinner... We probably should start a new thread soon with a bunch of this stuff solidified and stated to give others a chance to hop back into the game. Maybe a table of some sort with pros/cons (mathematically speaking, deferring implementation details for later). I swear, if we can get this to make sense... we should have a Nobel prize or something. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Nov 5 03:55:13 2011 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 5 Nov 2011 00:55:13 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Fri, Nov 4, 2011 at 8:33 PM, T J wrote: > On Fri, Nov 4, 2011 at 8:03 PM, Nathaniel Smith wrote: >> Again, I really don't think you're going to be able to sell an API where >> ?[2] + [IGNORED(20)] == [IGNORED(22)] >> I mean, it's not me you have to convince, it's Gary, Pierre, maybe >> Benjamin, Llu?s, etc. So I could be wrong. But you might want to >> figure that out first before making plans based on this... > > But this is how np.ma currently does it, except that it doesn't compute the > payload---it just calls it IGNORED. Yes, that's what I mean -- if you're just temporarily masking something out because you want it to be IGNORED, then you don't want it to change around when you do something like a += 2, right? If the operation is changing the payload, then it's weird to say that the operation ignored the payload... Anyway, I think this is another way to think about your suggestion: -- each array gets an extra boolean array called the "mask" that it carries around with it -- Unary ufuncs automatically copy these masks to their results. For binary ufuncs, the input masks get automatically ORed together, and that determines the mask attached to the output array -- these masks have absolutely no effect on any computations, except that ufunc.reduce(a, skip_IGNORED=True) is defined to be a synonym for ufunc.reduce(a, where=a.mask) Is that correct? Also, if can I ask -- is this something you would find useful yourself? -- Nathaniel From pav at iki.fi Sat Nov 5 06:57:58 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 05 Nov 2011 11:57:58 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: Hi, 05.11.2011 03:43, T J kirjoitti: [clip] > I thought that "PdC" satisfied (a) and (b). > Let me show you what I thought they were. Perhaps I am not being > consistent. If so, point out my mistake. Yes, propagating + destructive assigment + do-computations-on-payload should satisfy (a) and (b). (NA also works as it's a singleton.) The question is now that are there other rules, with more desirable behavior of masked values, that also have a += b a += 42 print unmask(a) and a += 42 a += b print unmask(a) as equivalent operations. The rules chosen by np.ma don't satisfy this. If taking a commutative version of np.ma's binary op rules, it seems that it's not clear how to make assignment work exactly in the way you'd expect of masked values while retaining equivalence in the above code. It seems that having `a += b` have `a[j]` unchanged if it's ignored, and having ignored values propagate creates the problem. -- Pauli Virtanen From ben.root at ou.edu Sat Nov 5 12:37:06 2011 From: ben.root at ou.edu (Benjamin Root) Date: Sat, 5 Nov 2011 11:37:06 -0500 Subject: [Numpy-discussion] what is the point of dx for np.gradient()? In-Reply-To: References: Message-ID: On Fri, Nov 4, 2011 at 1:20 PM, Benjamin Root wrote: > For np.gradient(), one can specify a sample distance for each axis to > apply to the gradient. But, all this does is just divides the gradient by > the sample distance. I could easily do that myself with the output from > gradient. Wouldn't it be more valuable to be able to specify the width of > the central difference (or is there another function that does that)? > > Thanks, > Ben Root > Nevermind, I should have realized the difficulty in coordinating the various divisions when dealing with multiple dimensions. My other question remains, though. Is there a function somewhere that allows me to perform central differences of varying widths. Preferably something that works with masks? Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Nov 5 14:24:20 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Nov 2011 12:24:20 -0600 Subject: [Numpy-discussion] Int casting different across platforms In-Reply-To: References: Message-ID: On Fri, Nov 4, 2011 at 5:21 PM, Matthew Brett wrote: > Hi, > > I noticed this: > > (Intel Mac): > > In [2]: np.int32(np.float32(2**31)) > Out[2]: -2147483648 > > (PPC): > > In [3]: np.int32(np.float32(2**31)) > Out[3]: 2147483647 > > I assume what is happening is that the casting is handing off to the c > library, and that behavior of the c library differs on these > platforms? Should we expect or hope that this behavior would be the > same across platforms? > Heh. I think the conversion is basically undefined because 2**31 won't fit in int32. The Intel example just takes the bottom 32 bits of 2**31 expressed as a binary integer, the PPC throws up its hands and returns the maximum value supported by int32. Numpy supports casts from unsigned to signed 32 bit numbers by using the same bits, as does C, and that would comport with the Intel example. It would probably be useful to have a Numpy convention for this so that the behavior was consistent across platforms. Maybe for float types we should raise an error if the value is out of bounds. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Nov 5 14:29:46 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 5 Nov 2011 19:29:46 +0100 Subject: [Numpy-discussion] ANN: scipy 0.10 release candidate 1 Message-ID: Hi all, I am pleased to announce the availability of the first release release of SciPy 0.10.0. For this release over a 100 tickets and pull requests have been closed, and many new features have been added. Some of the highlights are: - support for Bento as a build system for scipy - generalized and shift-invert eigenvalue problems in sparse.linalg - addition of discrete-time linear systems in the signal module Sources and binaries can be found at http://sourceforge.net/projects/scipy/files/scipy/0.10.0rc1/, release notes are copied below. Please try this release and report problems on the mailing list. Note: one problem with Python 2.5 (syntax) was discovered after tagging the release, it's fixed in the 0.10.x branch already so no need to report that one. Cheers, Ralf ========================== SciPy 0.10.0 Release Notes ========================== .. note:: Scipy 0.10.0 is not released yet! .. contents:: SciPy 0.10.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a limited number of deprecations and backwards-incompatible changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.10.x branch, and on adding new features on the development master branch. Release highlights: - Support for Bento as optional build system. - Support for generalized eigenvalue problems, and all shift-invert modes available in ARPACK. This release requires Python 2.4-2.7 or 3.1- and NumPy 1.5 or greater. New features ============ Bento: new optional build system -------------------------------- Scipy can now be built with `Bento `_. Bento has some nice features like parallel builds and partial rebuilds, that are not possible with the default build system (distutils). For usage instructions see BENTO_BUILD.txt in the scipy top-level directory. Currently Scipy has three build systems, distutils, numscons and bento. Numscons is deprecated and is planned and will likely be removed in the next release. Generalized and shift-invert eigenvalue problems in ``scipy.sparse.linalg`` --------------------------------------------------------------------------- The sparse eigenvalue problem solver functions ``scipy.sparse.eigs/eigh`` now support generalized eigenvalue problems, and all shift-invert modes available in ARPACK. Discrete-Time Linear Systems (``scipy.signal``) ----------------------------------------------- Support for simulating discrete-time linear systems, including ``scipy.signal.dlsim``, ``scipy.signal.dimpulse``, and ``scipy.signal.dstep``, has been added to SciPy. Conversion of linear systems from continuous-time to discrete-time representations is also present via the ``scipy.signal.cont2discrete`` function. Enhancements to ``scipy.signal`` -------------------------------- A Lomb-Scargle periodogram can now be computed with the new function ``scipy.signal.lombscargle``. The forward-backward filter function ``scipy.signal.filtfilt`` can now filter the data in a given axis of an n-dimensional numpy array. (Previously it only handled a 1-dimensional array.) Options have been added to allow more control over how the data is extended before filtering. FIR filter design with ``scipy.signal.firwin2`` now has options to create filters of type III (zero at zero and Nyquist frequencies) and IV (zero at zero frequency). Additional decomposition options (``scipy.linalg``) --------------------------------------------------- A sort keyword has been added to the Schur decomposition routine (``scipy.linalg.schur``) to allow the sorting of eigenvalues in the resultant Schur form. Additional special matrices (``scipy.linalg``) ---------------------------------------------- The functions ``hilbert`` and ``invhilbert`` were added to ``scipy.linalg``. Enhancements to ``scipy.stats`` ------------------------------- * The *one-sided form* of Fisher's exact test is now also implemented in ``stats.fisher_exact``. * The function ``stats.chi2_contingency`` for computing the chi-square test of independence of factors in a contingency table has been added, along with the related utility functions ``stats.contingency.margins`` and ``stats.contingency.expected_freq``. Basic support for Harwell-Boeing file format for sparse matrices ---------------------------------------------------------------- Both read and write are support through a simple function-based API, as well as a more complete API to control number format. The functions may be found in scipy.sparse.io. The following features are supported: * Read and write sparse matrices in the CSC format * Only real, symmetric, assembled matrix are supported (RUA format) Deprecated features =================== ``scipy.maxentropy`` -------------------- The maxentropy module is unmaintained, rarely used and has not been functioning well for several releases. Therefore it has been deprecated for this release, and will be removed for scipy 0.11. Logistic regression in scikits.learn is a good alternative for this functionality. The ``scipy.maxentropy.logsumexp`` function has been moved to ``scipy.misc``. ``scipy.lib.blas`` ------------------ There are similar BLAS wrappers in ``scipy.linalg`` and ``scipy.lib``. These have now been consolidated as ``scipy.linalg.blas``, and ``scipy.lib.blas`` is deprecated. Numscons build system --------------------- The numscons build system is being replaced by Bento, and will be removed in one of the next scipy releases. Backwards-incompatible changes ============================== The deprecated name `invnorm` was removed from ``scipy.stats.distributions``, this distribution is available as `invgauss`. The following deprecated nonlinear solvers from ``scipy.optimize`` have been removed:: - ``broyden_modified`` (bad performance) - ``broyden1_modified`` (bad performance) - ``broyden_generalized`` (equivalent to ``anderson``) - ``anderson2`` (equivalent to ``anderson``) - ``broyden3`` (obsoleted by new limited-memory broyden methods) - ``vackar`` (renamed to ``diagbroyden``) Other changes ============= ``scipy.constants`` has been updated with the CODATA 2010 constants. ``__all__`` dicts have been added to all modules, which has cleaned up the namespaces (particularly useful for interactive work). An API section has been added to the documentation, giving recommended import guidelines and specifying which submodules are public and which aren't. Authors ======= This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order): * Jeff Armstrong + * Matthew Brett * Lars Buitinck + * David Cournapeau * FI$H 2000 + * Michael McNeil Forbes + * Matty G + * Christoph Gohlke * Ralf Gommers * Yaroslav Halchenko * Charles Harris * Thouis (Ray) Jones + * Chris Jordan-Squire + * Robert Kern * Chris Lasher + * Wes McKinney + * Travis Oliphant * Fabian Pedregosa * Josef Perktold * Thomas Robitaille + * Pim Schellart + * Anthony Scopatz + * Skipper Seabold + * Fazlul Shahriar + * David Simcha + * Scott Sinclair + * Andrey Smirnov + * Collin RM Stocks + * Martin Teichmann + * Jake Vanderplas + * Ga?l Varoquaux + * Pauli Virtanen * Stefan van der Walt * Warren Weckesser * Mark Wiebe + A total of 35 people contributed to this release. People with a "+" by their names contributed a patch for the first time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjhnson at gmail.com Sat Nov 5 18:22:27 2011 From: tjhnson at gmail.com (T J) Date: Sat, 5 Nov 2011 15:22:27 -0700 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Sat, Nov 5, 2011 at 12:55 AM, Nathaniel Smith wrote: > On Fri, Nov 4, 2011 at 8:33 PM, T J wrote: > > On Fri, Nov 4, 2011 at 8:03 PM, Nathaniel Smith wrote: > >> Again, I really don't think you're going to be able to sell an API where > >> [2] + [IGNORED(20)] == [IGNORED(22)] > >> I mean, it's not me you have to convince, it's Gary, Pierre, maybe > >> Benjamin, Llu?s, etc. So I could be wrong. But you might want to > >> figure that out first before making plans based on this... > > > > But this is how np.ma currently does it, except that it doesn't compute > the > > payload---it just calls it IGNORED. > > Yes, that's what I mean -- if you're just temporarily masking > something out because you want it to be IGNORED, then you don't want > it to change around when you do something like a += 2, right? If the > operation is changing the payload, then it's weird to say that the > operation ignored the payload... > That's a fair critique. > > Anyway, I think this is another way to think about your suggestion: > > -- each array gets an extra boolean array called the "mask" that it > carries around with it > -- Unary ufuncs automatically copy these masks to their results. For > binary ufuncs, the input masks get automatically ORed together, and > that determines the mask attached to the output array > -- these masks have absolutely no effect on any computations, except that > ufunc.reduce(a, skip_IGNORED=True) > is defined to be a synonym for > ufunc.reduce(a, where=a.mask) > > Is that correct? > I believe that is correct. > > Also, if can I ask -- is this something you would find useful yourself? > So I guess this goes back to finding some consensus on what people want out of IGNORED values. With a very naive look at the initial list you provided, it seems that this particular suggestion matches it, and provides a fairly consistent behavior across operations (commutativity and unmasking). However, it doesn't seem to match an unstated expectation you had: which is that ignored values should truly be ignored (and payloads should not be operated on, etc). It seems (see Pauli's email too) that we might have to give up commutativity to achieve that. Maybe that is okay. The suggestion I put forth seems to treat "ignored" more as just another notion of the "where" keyword, as you pointed out. It is not so much a statement that the ignored values should be ignored during computations, just that they should be ignored when we query the valid elements in the array. So it works if you just want to plot certain portions of an array and possibly do calculations on them. But if you want to "double all integers greater than 3 and quadruple all integers less than 3", then this notion of IGNORED will not work as easily. Though this could easily be handled without IGNORED values too: x[x>3] *= 2. So what do people expect out of ignored values? It seems that we might need to extend the list you put forward so that it includes these desires. Since my primary use is with MISSING and not so much IGNORED, I'm not in a very good position to help extend that list. I'd be curious to know if this present suggestion would work with how matplotlib uses masked arrays. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Nov 5 19:07:45 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 5 Nov 2011 23:07:45 +0000 Subject: [Numpy-discussion] Int casting different across platforms In-Reply-To: References: Message-ID: Hi, On Sat, Nov 5, 2011 at 6:24 PM, Charles R Harris wrote: > > > On Fri, Nov 4, 2011 at 5:21 PM, Matthew Brett > wrote: >> >> Hi, >> >> I noticed this: >> >> (Intel Mac): >> >> In [2]: np.int32(np.float32(2**31)) >> Out[2]: -2147483648 >> >> (PPC): >> >> In [3]: np.int32(np.float32(2**31)) >> Out[3]: 2147483647 >> >> I assume what is happening is that the casting is handing off to the c >> library, and that behavior of the c library differs on these >> platforms? ?Should we expect or hope that this behavior would be the >> same across platforms? > > Heh. I think the conversion is basically undefined because 2**31 won't fit > in int32. The Intel example just takes the bottom 32 bits of 2**31 expressed > as a binary integer, the PPC throws up its hands and returns the maximum > value supported by int32. Numpy supports casts from unsigned to signed 32 > bit numbers by using the same bits, as does C, and that would comport with > the Intel example. It would probably be useful to have a Numpy convention > for this so that the behavior was consistent across platforms. Maybe for > float types we should raise an error if the value is out of bounds. Just to see what happens: #include #include int main(int argc, char* argv) { double x; int y; x = pow(2, 31); y = (int)x; printf("%d, %d\n", sizeof(int), y); } Intel, gcc: 4, -2147483648 PPC, gcc: 4, 2147483647 I think that's what you predicted. Is it strange that the same compiler gives different results? It would be good if the behavior was the same across platforms - the unexpected negative overflow caught me out at least. An error sounds sensible to me. Would it cost lots of cycles? Cheers, Matthew From pav at iki.fi Sat Nov 5 20:02:21 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 06 Nov 2011 01:02:21 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: Message-ID: 05.11.2011 23:22, T J kirjoitti: [clip] > So what do people expect out of ignored values? It seems that we might > need to extend the list you put forward so that it includes these > desires. Since my primary use is with MISSING and not so much IGNORED, > I'm not in a very good position to help extend that list. I'd be > curious to know if this present suggestion would work with how > matplotlib uses masked arrays. Another question is if non-propagating (N*C) ignored values would be useful. I'm guessing that for these it would be possible to have the expected behavior on assignment, while retaining commutativity, and they would also behave as expected in reductions. It's a somewhat different concept from np.ma, but I'm wondering if there would be real-world uses for tagging values with "this number is not really here, act as if it wasn't in the array". -- Pauli Virtanen From njs at pobox.com Sat Nov 5 21:35:00 2011 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 5 Nov 2011 18:35:00 -0700 Subject: [Numpy-discussion] Int casting different across platforms In-Reply-To: References: Message-ID: On Sat, Nov 5, 2011 at 4:07 PM, Matthew Brett wrote: > Intel, gcc: > 4, -2147483648 > PPC, gcc: > 4, 2147483647 > > I think that's what you predicted. ?Is it strange that the same > compiler gives different results? > > It would be good if the behavior was the same across platforms - the > unexpected negative overflow caught me out at least. ?An error sounds > sensible to me. ?Would it cost lots of cycles? C99 says (section F.4): "If the floating value is infinite or NaN or if the integral part of the floating value exceeds the range of the integer type, then the ??invalid?? floating-point exception is raised and the resulting value is unspecified. Whether conversion of non-integer floating values whose integral part is within the range of the integer type raises the ??inexact?? floating-point exception is unspecified." So it sounds like the compiler is allowed to return whatever nonsense it likes in this case. But, you should be able to cause this to raise an exception by fiddling with np.seterr. However, that doesn't seem to work for me with numpy 1.5.1 on x86-64 linux :-( >>> np.int32(np.float32(2**31)) -2147483648 >>> np.seterr(all="raise") >>> np.int32(np.float32(2**31)) -2147483648 I think this must be a numpy or compiler bug? -- Nathaniel From charlesr.harris at gmail.com Sat Nov 5 22:39:31 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Nov 2011 20:39:31 -0600 Subject: [Numpy-discussion] Int casting different across platforms In-Reply-To: References: Message-ID: On Sat, Nov 5, 2011 at 7:35 PM, Nathaniel Smith wrote: > On Sat, Nov 5, 2011 at 4:07 PM, Matthew Brett > wrote: > > Intel, gcc: > > 4, -2147483648 > > PPC, gcc: > > 4, 2147483647 > > > > I think that's what you predicted. Is it strange that the same > > compiler gives different results? > > > > It would be good if the behavior was the same across platforms - the > > unexpected negative overflow caught me out at least. An error sounds > > sensible to me. Would it cost lots of cycles? > > C99 says (section F.4): > > "If the floating value is infinite or NaN or if the integral part of > the floating value exceeds the range of the integer type, then the > ??invalid?? floating-point exception is raised and the resulting value > is unspecified. Whether conversion of non-integer floating values > whose integral part is within the range of the integer type raises the > ??inexact?? floating-point exception is unspecified." > > So it sounds like the compiler is allowed to return whatever nonsense > it likes in this case. But, you should be able to cause this to raise > an exception by fiddling with np.seterr. > > However, that doesn't seem to work for me with numpy 1.5.1 on x86-64 linux > :-( > > >>> np.int32(np.float32(2**31)) > -2147483648 > >>> np.seterr(all="raise") > >>> np.int32(np.float32(2**31)) > -2147483648 > > I think this must be a numpy or compiler bug? > > I don't believe the floating point status is checked in the numpy conversion routines. That looks like a nice small project for someone interested in learning the numpy internals. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Nov 5 23:13:20 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 6 Nov 2011 03:13:20 +0000 Subject: [Numpy-discussion] Int casting different across platforms In-Reply-To: References: Message-ID: Hi, On Sun, Nov 6, 2011 at 2:39 AM, Charles R Harris wrote: > > > On Sat, Nov 5, 2011 at 7:35 PM, Nathaniel Smith wrote: >> >> On Sat, Nov 5, 2011 at 4:07 PM, Matthew Brett >> wrote: >> > Intel, gcc: >> > 4, -2147483648 >> > PPC, gcc: >> > 4, 2147483647 >> > >> > I think that's what you predicted. ?Is it strange that the same >> > compiler gives different results? >> > >> > It would be good if the behavior was the same across platforms - the >> > unexpected negative overflow caught me out at least. ?An error sounds >> > sensible to me. ?Would it cost lots of cycles? >> >> C99 says (section F.4): >> >> "If the floating value is infinite or NaN or if the integral part of >> the floating value exceeds the range of the integer type, then the >> ??invalid?? floating-point exception is raised and the resulting value >> is unspecified. Whether conversion of non-integer floating values >> whose integral part is within the range of the integer type raises the >> ??inexact?? floating-point exception is unspecified." >> >> So it sounds like the compiler is allowed to return whatever nonsense >> it likes in this case. But, you should be able to cause this to raise >> an exception by fiddling with np.seterr. >> >> However, that doesn't seem to work for me with numpy 1.5.1 on x86-64 linux >> :-( >> >> >>> np.int32(np.float32(2**31)) >> -2147483648 >> >>> np.seterr(all="raise") >> >>> np.int32(np.float32(2**31)) >> -2147483648 >> >> I think this must be a numpy or compiler bug? >> > > I don't believe the floating point status is checked in the numpy conversion > routines. That looks like a nice small project for someone interested in > learning the numpy - . To my shame I doubt that I will have the time to do this, but just in case I or someone does get time, is there a good place to start to look? Cheers, Matthew From njs at pobox.com Sun Nov 6 18:19:06 2011 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 6 Nov 2011 15:19:06 -0800 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: References: <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: On Sat, Nov 5, 2011 at 3:22 PM, T J wrote: > So what do people expect out of ignored values?? It seems that we might need > to extend the list you put forward so that it includes these desires. Since > my primary use is with MISSING and not so much IGNORED, I'm not in a very > good position to help extend that list.? I'd be curious to know if this > present suggestion would work with how matplotlib uses masked arrays. I'm in a similar position -- I don't have any use cases for IGNORED myself right now, so I'm just trying to guess from what I've seen people say. Just having where= around makes it at least possible, if cumbersome, to handle one of the big problems -- working with subsets of large datasets without having to make a copy. It's possible we should leave IGNORED support alone until people have more experience with where=, and can say what kind of convenience features would be useful. Or maybe we can get some experts to speak up... perhaps this will help: http://thread.gmane.org/gmane.comp.python.matplotlib.devel/10740 -- Nathaniel From xscript at gmx.net Mon Nov 7 06:37:46 2011 From: xscript at gmx.net (=?utf-8?Q?Llu=C3=ADs?=) Date: Mon, 07 Nov 2011 12:37:46 +0100 Subject: [Numpy-discussion] in the NA discussion, what can we agree on? In-Reply-To: (Nathaniel Smith's message of "Fri, 4 Nov 2011 15:04:49 -0700") References: <877h3hqo01.fsf@ginnungagap.bsc.es> <87pqh7hnv3.fsf@ginnungagap.bsc.es> Message-ID: <87lirsgo39.fsf@ginnungagap.bsc.es> Nathaniel Smith writes: > So assignment is not destructive -- the old value is retained as the "payload". I never assumed (and I think it is also the case for others) that the payload was retaining the old value. In fact, AFAIR, the payloads were introduced as a way of having more than one special value that (if wanted by the user) can be handled differently depending on the payload. Note that while you're assuming "IGNORED(x)" means a value that is ignoring the "x" original value, you're never writing "MISSING(x)" to retain the original value that is now missing. Thus I think that decoupling the payload from the "previous value" concept makes it all consistent regardless of the destructiveness property. That's one of the reasons why I used the "special value" concept since the beginning, so that no assumption can be made about its propagation and destructiveness properties. Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From clusher2 at yahoo.com.br Mon Nov 7 13:16:04 2011 From: clusher2 at yahoo.com.br (Carlos Neves) Date: Mon, 07 Nov 2011 18:16:04 +0000 Subject: [Numpy-discussion] Cross-Compiling Numpy for NAO Message-ID: <4EB82064.1040508@yahoo.com.br> Hello everyone, I am a robotics student and I have used numpy to develop a controller for a humanoid robot. Now I am trying to implement the same controller to a different robot, NAO - http://www.aldebaran-robotics.com/ This robot has a Linux based OS with Python 2.6.4. Since the goal is to implement the controller with the least changes possible and make sure the robot remains autonomous, it would be great to install numpy in it. The problem is that it is not possible to build numpy in the system and I was adviced to use the cross-compilation toolchain provided by Aldebaran. However, all their tutorials assume I can build the libraries I need with cmake, which isn't the case. So, is it possible to cross-compile numpy, given a cross-toolchain, with not much effort? The other option is to use openCV, which comes with the NAO OS, but it require quite a few changes in the program... I tried searching the archives, but was unable to find any relevant post. I hope you can help me! Thank you, Carlos Neves -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Nov 7 13:29:57 2011 From: cournape at gmail.com (David Cournapeau) Date: Mon, 7 Nov 2011 18:29:57 +0000 Subject: [Numpy-discussion] Cross-Compiling Numpy for NAO In-Reply-To: <4EB82064.1040508@yahoo.com.br> References: <4EB82064.1040508@yahoo.com.br> Message-ID: On Mon, Nov 7, 2011 at 6:16 PM, Carlos Neves wrote: > Hello everyone, > > I am a robotics student and I have used numpy to develop a controller for a > humanoid robot. Now I am trying to implement the same controller to a > different robot, NAO - http://www.aldebaran-robotics.com/ > This robot has a Linux based OS with Python 2.6.4. Since the goal is to > implement the controller with the least changes possible and make sure the > robot remains autonomous, it would be great to install numpy in it. The > problem is that it is not possible to build numpy in the system and I was > adviced to use the cross-compilation toolchain provided by Aldebaran. > However, all their tutorials assume I can build the libraries I need with > cmake, which isn't the case. > > So, is it possible to cross-compile numpy, given a cross-toolchain, with not > much effort? I don't know about aldebaran, but cross compiling numpy will be a challenge. cheers, David From madsipsen at gmail.com Tue Nov 8 03:40:14 2011 From: madsipsen at gmail.com (Mads Ipsen) Date: Tue, 08 Nov 2011 09:40:14 +0100 Subject: [Numpy-discussion] Failure to build numpy 1.6.1 Message-ID: <4EB8EAEE.90105@gmail.com> Hi, I am trying to build numpy-1.6.1 with the following gcc compiler specs: Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-java-awt=gtk --host=x86_64-redhat-linux Thread model: posix gcc version 3.4.6 20060404 (Red Hat 3.4.6-11) I get the following error (any clues at what goes wrong)? creating build/temp.linux-x86_64-2.7/numpy/core/src/multiarray compile options: '-Inumpy/core/include -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/home/quantum/quantumnotes/qw-control/quantumsource/external-libs/build/include/python2.7 -Ibuild/src.linux-x86_64-2.7/numpy/core/src/multiarray -Ibuild/src.linux-x86_64-2.7/numpy/core/src/umath -c' gcc: numpy/core/src/multiarray/multiarraymodule_onefile.c numpy/core/src/multiarray/descriptor.c: In function `_convert_divisor_to_multiple': numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used uninitialized in this function numpy/core/src/multiarray/einsum.c.src: In function `float_sum_of_products_contig_outstride0_one': numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn: (insn:HI 440 228 481 14 /usr/lib/gcc/x86_64-redhat-linux/3.4.6/include/xmmintrin.h:915 (set (reg:SF 148) (vec_select:SF (reg/v:V4SF 67 [ accum_sse ]) (parallel [ (const_int 0 [0x0]) ]))) -1 (insn_list 213 (nil)) (nil)) numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in extract_insn, at recog.c:2083 Please submit a full bug report, with preprocessed source if appropriate. See for instructions. Preprocessed source stored into /tmp/ccXaPpf8.out file, please attach this to your bugreport. numpy/core/src/multiarray/descriptor.c: In function `_convert_divisor_to_multiple': numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used uninitialized in this function numpy/core/src/multiarray/einsum.c.src: In function `float_sum_of_products_contig_outstride0_one': numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn: (insn:HI 440 228 481 14 /usr/lib/gcc/x86_64-redhat-linux/3.4.6/include/xmmintrin.h:915 (set (reg:SF 148) (vec_select:SF (reg/v:V4SF 67 [ accum_sse ]) (parallel [ (const_int 0 [0x0]) ]))) -1 (insn_list 213 (nil)) (nil)) numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in extract_insn, at recog.c:2083 Please submit a full bug report, with preprocessed source if appropriate. See for instructions. Preprocessed source stored into /tmp/ccXaPpf8.out file, please attach this to your bugreport. error: Command "gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/home/quantum/quantumnotes/qw-control/quantumsource/external-libs/build/include/python2.7 -Ibuild/src.linux-x86_64-2.7/numpy/core/src/multiarray -Ibuild/src.linux-x86_64-2.7/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/multiarraymodule_onefile.o" failed with exit status 1 make: *** [/home/quantum/quantumnotes/qw-control/quantumsource/external-libs/src/numpy-1.6.1/make-stamp] Error 1 -- +-----------------------------------------------------+ | Mads Ipsen | +----------------------+------------------------------+ | G?seb?ksvej 7, 4. tv | | | DK-2500 Valby | phone: +45-29716388 | | Denmark | email: mads.ipsen at gmail.com | +----------------------+------------------------------+ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Nov 8 04:01:54 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 8 Nov 2011 09:01:54 +0000 Subject: [Numpy-discussion] Failure to build numpy 1.6.1 In-Reply-To: <4EB8EAEE.90105@gmail.com> References: <4EB8EAEE.90105@gmail.com> Message-ID: Hi Mads, On Tue, Nov 8, 2011 at 8:40 AM, Mads Ipsen wrote: > Hi, > > I am trying to build numpy-1.6.1 with the following gcc compiler specs: > > Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man > --infodir=/usr/share/info --enable-shared --enable-threads=posix > --disable-checking --with-system-zlib --enable-__cxa_atexit > --disable-libunwind-exceptions --enable-java-awt=gtk > --host=x86_64-redhat-linux > Thread model: posix > gcc version 3.4.6 20060404 (Red Hat 3.4.6-11) > > I get the following error (any clues at what goes wrong)? This looks like a compiler bug (gcc 3.4 is really old). einsum uses SSE intrinsics, and old gcc implementations are quite buggy in that area. Could you try the following, at line 38, to add the following: #define EINSUM_USE_SSE1 0 #define EINSUM_USE_SSE2 0 cheers, David From cournape at gmail.com Tue Nov 8 04:02:34 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 8 Nov 2011 09:02:34 +0000 Subject: [Numpy-discussion] Failure to build numpy 1.6.1 In-Reply-To: References: <4EB8EAEE.90105@gmail.com> Message-ID: On Tue, Nov 8, 2011 at 9:01 AM, David Cournapeau wrote: > Hi Mads, > > On Tue, Nov 8, 2011 at 8:40 AM, Mads Ipsen wrote: >> Hi, >> >> I am trying to build numpy-1.6.1 with the following gcc compiler specs: >> >> Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs >> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man >> --infodir=/usr/share/info --enable-shared --enable-threads=posix >> --disable-checking --with-system-zlib --enable-__cxa_atexit >> --disable-libunwind-exceptions --enable-java-awt=gtk >> --host=x86_64-redhat-linux >> Thread model: posix >> gcc version 3.4.6 20060404 (Red Hat 3.4.6-11) >> >> I get the following error (any clues at what goes wrong)? > > This looks like a compiler bug (gcc 3.4 is really old). einsum uses > SSE intrinsics, and old gcc implementations are quite buggy in that > area. > > Could you try the following, at line 38, to add the following: > > #define EINSUM_USE_SSE1 0 > #define EINSUM_USE_SSE2 0 I meant to add this in the file numpy/core/src/multiarray/einsum.c.src, and then rebuild numpy David From madsipsen at gmail.com Tue Nov 8 04:20:42 2011 From: madsipsen at gmail.com (Mads Ipsen) Date: Tue, 08 Nov 2011 10:20:42 +0100 Subject: [Numpy-discussion] Failure to build numpy 1.6.1 In-Reply-To: References: <4EB8EAEE.90105@gmail.com> Message-ID: <4EB8F46A.9020207@gmail.com> On 11/08/2011 10:01 AM, David Cournapeau wrote: > Hi Mads, > > On Tue, Nov 8, 2011 at 8:40 AM, Mads Ipsen wrote: >> Hi, >> >> I am trying to build numpy-1.6.1 with the following gcc compiler specs: >> >> Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs >> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man >> --infodir=/usr/share/info --enable-shared --enable-threads=posix >> --disable-checking --with-system-zlib --enable-__cxa_atexit >> --disable-libunwind-exceptions --enable-java-awt=gtk >> --host=x86_64-redhat-linux >> Thread model: posix >> gcc version 3.4.6 20060404 (Red Hat 3.4.6-11) >> >> I get the following error (any clues at what goes wrong)? > This looks like a compiler bug (gcc 3.4 is really old). einsum uses > SSE intrinsics, and old gcc implementations are quite buggy in that > area. > > Could you try the following, at line 38, to add the following: > > #define EINSUM_USE_SSE1 0 > #define EINSUM_USE_SSE2 0 > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Yup, that fixes it. For now, we can apply a temporary fix on our build system. Is this something that'll go into, say, 1.6.2? Best regards, Mads -- +-----------------------------------------------------+ | Mads Ipsen | +----------------------+------------------------------+ | G?seb?ksvej 7, 4. tv | | | DK-2500 Valby | phone: +45-29716388 | | Denmark | email: mads.ipsen at gmail.com | +----------------------+------------------------------+ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Nov 8 06:14:45 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 8 Nov 2011 11:14:45 +0000 Subject: [Numpy-discussion] Failure to build numpy 1.6.1 In-Reply-To: <4EB8F46A.9020207@gmail.com> References: <4EB8EAEE.90105@gmail.com> <4EB8F46A.9020207@gmail.com> Message-ID: On Tue, Nov 8, 2011 at 9:20 AM, Mads Ipsen wrote: > Yup, that fixes it. For now, we can apply a temporary fix on our build > system. Is this something that'll go into, say, 1.6.2? That's more of a workaround than a fix. We need to decide whether we disable intrinsics altogether or wether we want to drop support for old compilers. cheers, David From madsipsen at gmail.com Tue Nov 8 08:11:53 2011 From: madsipsen at gmail.com (Mads Ipsen) Date: Tue, 08 Nov 2011 14:11:53 +0100 Subject: [Numpy-discussion] Failure to build numpy 1.6.1 In-Reply-To: References: <4EB8EAEE.90105@gmail.com> <4EB8F46A.9020207@gmail.com> Message-ID: <4EB92A99.3080507@gmail.com> On 11/08/2011 12:14 PM, David Cournapeau wrote: > On Tue, Nov 8, 2011 at 9:20 AM, Mads Ipsen wrote: > >> Yup, that fixes it. For now, we can apply a temporary fix on our build >> system. Is this something that'll go into, say, 1.6.2? > That's more of a workaround than a fix. We need to decide whether we > disable intrinsics altogether or wether we want to drop support for > old compilers. > > cheers, > > David OK. But thanks for helping out. Best regards, Mads -- +-----------------------------------------------------+ | Mads Ipsen | +----------------------+------------------------------+ | G?seb?ksvej 7, 4. tv | | | DK-2500 Valby | phone: +45-29716388 | | Denmark | email: mads.ipsen at gmail.com | +----------------------+------------------------------+ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Nov 8 13:11:21 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Tue, 08 Nov 2011 10:11:21 -0800 Subject: [Numpy-discussion] Cross-Compiling Numpy for NAO In-Reply-To: <4EB82064.1040508@yahoo.com.br> References: <4EB82064.1040508@yahoo.com.br> Message-ID: <4EB970C9.6010007@noaa.gov> On 11/7/11 10:16 AM, Carlos Neves wrote: > I am a robotics student and I have used numpy to develop a controller > for a humanoid robot. Now I am trying to implement the same controller > to a different robot, NAO - http://www.aldebaran-robotics.com/ > This robot has a Linux based OS with Python 2.6.4. Since the goal is to > implement the controller with the least changes possible and make sure > the robot remains autonomous, it would be great to install numpy in it. > The problem is that it is not possible to build numpy in the system and > I was adviced to use the cross-compilation toolchain provided by > Aldebaran. However, all their tutorials assume I can build the libraries > I need with cmake, which isn't the case. Over on the cython-users list, I'm pretty sure someone recently contributed a cmake-based build for cython code -- maybe it could be leveraged for numpy? http://groups.google.com/group/cython-users/browse_thread/thread/c2aa7da05ae8df66 -Chris > > So, is it possible to cross-compile numpy, given a cross-toolchain, with > not much effort? The other option is to use openCV, which comes with the > NAO OS, but it require quite a few changes in the program... I tried > searching the archives, but was unable to find any relevant post. > > I hope you can help me! > Thank you, > Carlos Neves > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mdzz-2008 at 163.com Wed Nov 9 05:09:24 2011 From: mdzz-2008 at 163.com (=?GBK?B?zfW617Tv?=) Date: Wed, 9 Nov 2011 18:09:24 +0800 (CST) Subject: [Numpy-discussion] Who has ever used treecluster in Pycluster? Message-ID: <388fcf09.ce24.13387cdb127.Coremail.mdzz-2008@163.com> hi, Pycluster reports an error when the distancematrix is a matrix 6000*6000: ValueError: Row 0 in the distance matrix has incorrect size (6417 should be 0) Is anyone encountered this problem before? Is that because I should only use a left-lower matrix, and leave all else blank? Best, Heda Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.prettenhofer at gmail.com Wed Nov 9 06:38:56 2011 From: peter.prettenhofer at gmail.com (Peter Prettenhofer) Date: Wed, 9 Nov 2011 12:38:56 +0100 Subject: [Numpy-discussion] np.dot and scipy sparse matrices Message-ID: Hi everybody, I recently got the latest numpy version (2.0.0.dev-7297785) from the git repo and realized that `np.dot` causes a segfault if its operands are scipy sparse matrices. Here's some code to reproduce the problem:: import numpy as np from scipy import sparse as sp A = np.random.rand(10, 10) S = sp.csr_matrix(A) _ = np.dot(A, A) # this works OK _ = np.dot(S, S) # this segfaults! thanks, Peter -- Peter Prettenhofer From joshua.reyes at gmail.com Wed Nov 9 11:40:50 2011 From: joshua.reyes at gmail.com (Joshua Anthony Reyes) Date: Wed, 09 Nov 2011 11:40:50 -0500 Subject: [Numpy-discussion] Sampling from the multivariate normal Message-ID: <4EBAAD12.1050608@gmail.com> Hi List, I'm new to Numpy and I'm a little confused about the behavior of numpy.random.multivariate_normal(). I'm not sure I'm passing the variances correctly. My goal is to sample from a bivariate normal, but the kooky behavior shows up when I sample from a univariate distribution. In short, the multivariate normal function doesn't seem to give me values in the appropriate ranges. Here's an example of what I mean. In[1]: from numpy.random import normal, multivariate_normal In [2]: normal(100, 100, 10) Out[2]: array([ 258.62344586, 70.16378815, 49.46826997, 49.58567724, 182.68652256, 226.67292034, 92.03801549, 18.2686146 , 94.09104313, 80.35697507]) The samples look about right to me. But then when I try to do the same using the multivariate_normal, the values it draws look too close to the mean. In [3]: multivariate_normal([100], [[100]], 10) Out[3]: array([[ 109.10083984], [ 97.43526716], [ 108.43923772], [ 97.87345947], [ 103.405562 ], [ 110.2963736 ], [ 103.96445585], [ 90.58168544], [ 91.20549222], [ 104.4051761 ]]) These values all fall within 10 units of the mean. In [4]: multivariate_normal([100], [[1000]], 10) Out[4]: array([[ 62.04304611], [ 123.29364557], [ 83.53840083], [ 64.67679778], [ 127.82433157], [ 11.3305656 ], [ 95.4101701 ], [ 126.53213908], [ 104.68868736], [ 32.45886112]]) In [5]: normal(100, 1000, 10) Out[5]: array([ 1052.93998938, -1254.12576419, 258.75390045, 688.32715327, -2.36543806, -1570.54842269, 472.90045029, 394.62908014, 137.10320437, 1741.85017871]) And just to exaggerate things a little more: In [6]: multivariate_normal([100], [[10000]], 10).T][0] Out[6]: array([ 274.45446694, 85.79359682, 245.03248721, 120.10912405, -34.76526896, 134.93446664, 47.6768889 , 93.34140984, 80.27244669, 229.64700591]) Whereas In [7]: normal(100, 10000, 10) Out[7]: array([ -554.68666687, 3724.59638363, -14873.55303901, -3111.22162495, -10813.66412901, 4688.81310356, -9510.92470735, -12689.02667559, -10379.01381925, -4534.60480031]) I'm shocked that I don't get some negative values in Out[4]. And Out[6] really ought to have some numbers in the thousands. I'd totally be willing to believe that I don't understand the multivariate normal and/or variance. Can someone tell me whether these numbers look sane? For the bivariate case I do something like this: means = [100, 100] variances = [100, 1000] Sx, Sy = variances sx, sy = map(sqrt, variances) cor = 0.7 cov = [[Sx, cor*sx*sy], [cor*sy*sx, Sy]] draws = 10 samples = multivariate_normal(means, cov, draws) As mentioned before, the samples are shockingly close to their means. Thanks, Josh From shish at keba.be Wed Nov 9 12:04:49 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 9 Nov 2011 12:04:49 -0500 Subject: [Numpy-discussion] Sampling from the multivariate normal In-Reply-To: <4EBAAD12.1050608@gmail.com> References: <4EBAAD12.1050608@gmail.com> Message-ID: The "normal" function takes as input the standard deviation, while "multivariate_normal" takes as input the covariance. -=- Olivier 2011/11/9 Joshua Anthony Reyes > Hi List, > > I'm new to Numpy and I'm a little confused about the behavior of > numpy.random.multivariate_normal(). I'm not sure I'm passing the > variances correctly. My goal is to sample from a bivariate normal, but > the kooky behavior shows up when I sample from a univariate distribution. > > In short, the multivariate normal function doesn't seem to give me > values in the appropriate ranges. Here's an example of what I mean. > > In[1]: from numpy.random import normal, multivariate_normal > > In [2]: normal(100, 100, 10) > Out[2]: > array([ 258.62344586, 70.16378815, 49.46826997, 49.58567724, > 182.68652256, 226.67292034, 92.03801549, 18.2686146 , > 94.09104313, 80.35697507]) > > The samples look about right to me. But then when I try to do the same > using the multivariate_normal, the values it draws look too close to the > mean. > > In [3]: multivariate_normal([100], [[100]], 10) > Out[3]: > array([[ 109.10083984], > [ 97.43526716], > [ 108.43923772], > [ 97.87345947], > [ 103.405562 ], > [ 110.2963736 ], > [ 103.96445585], > [ 90.58168544], > [ 91.20549222], > [ 104.4051761 ]]) > > These values all fall within 10 units of the mean. > > In [4]: multivariate_normal([100], [[1000]], 10) > Out[4]: > array([[ 62.04304611], > [ 123.29364557], > [ 83.53840083], > [ 64.67679778], > [ 127.82433157], > [ 11.3305656 ], > [ 95.4101701 ], > [ 126.53213908], > [ 104.68868736], > [ 32.45886112]]) > > In [5]: normal(100, 1000, 10) > Out[5]: > array([ 1052.93998938, -1254.12576419, 258.75390045, 688.32715327, > -2.36543806, -1570.54842269, 472.90045029, 394.62908014, > 137.10320437, 1741.85017871]) > > And just to exaggerate things a little more: > > In [6]: multivariate_normal([100], [[10000]], 10).T][0] > Out[6]: > array([ 274.45446694, 85.79359682, 245.03248721, 120.10912405, > -34.76526896, 134.93446664, 47.6768889 , 93.34140984, > 80.27244669, 229.64700591]) > > Whereas > In [7]: normal(100, 10000, 10) > Out[7]: > array([ -554.68666687, 3724.59638363, -14873.55303901, -3111.22162495, > -10813.66412901, 4688.81310356, -9510.92470735, -12689.02667559, > -10379.01381925, -4534.60480031]) > > I'm shocked that I don't get some negative values in Out[4]. And Out[6] > really ought to have some numbers in the thousands. > > I'd totally be willing to believe that I don't understand the > multivariate normal and/or variance. Can someone tell me whether these > numbers look sane? > > For the bivariate case I do something like this: > > means = [100, 100] > variances = [100, 1000] > Sx, Sy = variances > sx, sy = map(sqrt, variances) > cor = 0.7 > cov = [[Sx, cor*sx*sy], [cor*sy*sx, Sy]] > draws = 10 > samples = multivariate_normal(means, cov, draws) > > As mentioned before, the samples are shockingly close to their means. > > Thanks, > Josh > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Nov 9 12:14:27 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 9 Nov 2011 17:14:27 +0000 Subject: [Numpy-discussion] Sampling from the multivariate normal In-Reply-To: <4EBAAD12.1050608@gmail.com> References: <4EBAAD12.1050608@gmail.com> Message-ID: On Wed, Nov 9, 2011 at 16:40, Joshua Anthony Reyes wrote: > Hi List, > > I'm new to Numpy and I'm a little confused about the behavior of > numpy.random.multivariate_normal(). I'm not sure I'm passing the > variances correctly. My goal is to sample from a bivariate normal, but > the kooky behavior shows up when I sample from a univariate distribution. > > In short, the multivariate normal function doesn't seem to give me > values in the appropriate ranges. Here's an example of what I mean. > > In[1]: from numpy.random import normal, multivariate_normal > > In [2]: normal(100, 100, 10) > Out[2]: > array([ 258.62344586, ? 70.16378815, ? 49.46826997, ? 49.58567724, > ? ? ? ? 182.68652256, ?226.67292034, ? 92.03801549, ? 18.2686146 , > ? ? ? ? ?94.09104313, ? 80.35697507]) Here, you are asking for a Gaussian distribution with a mean of 100 and a stdev of 100 (or equivalently, a variance of 10000). > The samples look about right to me. But then when I try to do the same > using the multivariate_normal, the values it draws look too close to the > mean. > > In [3]: multivariate_normal([100], [[100]], 10) Here you are asking for a Gaussian with a mean of 100 and a variance of 100 (or equivalently, a stdev of 10). Note the differences between the two signatures: np.random.normal(mean, stdev) np.random.multivariate_normal(mean, covariance) > Out[3]: > array([[ 109.10083984], > ? ? ? ?[ ?97.43526716], > ? ? ? ?[ 108.43923772], > ? ? ? ?[ 97.87345947], > ? ? ? ?[ 103.405562 ?], > ? ? ? ?[ 110.2963736 ], > ? ? ? ?[ 103.96445585], > ? ? ? ?[ 90.58168544], > ? ? ? ?[ ?91.20549222], > ? ? ? ?[ 104.4051761 ]]) > > These values all fall within 10 units of the mean. > > In [4]: multivariate_normal([100], [[1000]], 10) > Out[4]: > array([[ ?62.04304611], > ? ? ? ?[ 123.29364557], > ? ? ? ?[ 83.53840083], > ? ? ? ?[ 64.67679778], > ? ? ? ?[ 127.82433157], > ? ? ? ?[ ?11.3305656 ], > ? ? ? ?[ ?95.4101701 ], > ? ? ? ?[ 126.53213908], > ? ? ? ?[ 104.68868736], > ? ? ? ?[ ?32.45886112]]) > > In [5]: normal(100, 1000, 10) > Out[5]: > array([ 1052.93998938, -1254.12576419, ? 258.75390045, ? 688.32715327, > ? ? ? ? ? -2.36543806, -1570.54842269, ? 472.90045029, ? 394.62908014, > ? ? ? ? ?137.10320437, ?1741.85017871]) > > And just to exaggerate things a little more: > > In [6]: multivariate_normal([100], [[10000]], 10).T][0] > Out[6]: > array([ 274.45446694, ? 85.79359682, ?245.03248721, ?120.10912405, > ? ? ? ? -34.76526896, ?134.93446664, ? 47.6768889 , ? 93.34140984, > ? ? ? ? ?80.27244669, ?229.64700591]) > > Whereas > In [7]: normal(100, 10000, 10) > Out[7]: > array([ ?-554.68666687, ? 3724.59638363, -14873.55303901, ?-3111.22162495, > ? ? ? ?-10813.66412901, ? 4688.81310356, ?-9510.92470735, -12689.02667559, > ? ? ? ?-10379.01381925, ?-4534.60480031]) > > I'm shocked that I don't get some negative values in Out[4]. And Out[6] > really ought to have some numbers in the thousands. > > I'd totally be willing to believe that I don't understand the > multivariate normal and/or variance. Can someone tell me whether these > numbers look sane? > > For the bivariate case I do something like this: > > means = [100, 100] > variances = [100, 1000] > Sx, Sy = variances > sx, sy = map(sqrt, variances) > cor = 0.7 > cov = [[Sx, cor*sx*sy], [cor*sy*sx, Sy]] > draws = 10 > samples = multivariate_normal(means, cov, draws) > > As mentioned before, the samples are shockingly close to their means. They look right to me. [~] |19> means = [100, 100] [~] |20> variances = [100, 1000] [~] |21> Sx, Sy = variances [~] |22> sx, sy = map(sqrt, variances) [~] |23> cor = 0.7 [~] |24> cov = np.array([[Sx, cor*sx*sy], [cor*sy*sx, Sy]]) [~] |26> samples = np.random.multivariate_normal(means, cov, 10000) [~] |27> cov array([[ 100. , 221.35943621], [ 221.35943621, 1000. ]]) [~] |28> np.cov(samples.T) array([[ 101.16844481, 222.00301056], [ 222.00301056, 1001.58403922]]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From joshua.reyes at gmail.com Wed Nov 9 12:39:01 2011 From: joshua.reyes at gmail.com (Joshua Anthony Reyes) Date: Wed, 09 Nov 2011 12:39:01 -0500 Subject: [Numpy-discussion] Sampling from the multivariate normal In-Reply-To: References: <4EBAAD12.1050608@gmail.com> Message-ID: <4EBABAB4.4050409@gmail.com> Hi Oliver, Robert, Thanks to both of you for your speedy and enlightening responses. They really cleared things up. Now my simulations seem to work as intended. All the best, Josh On 11/9/11 12:14 PM, Robert Kern wrote: > On Wed, Nov 9, 2011 at 16:40, Joshua Anthony Reyes > wrote: >> Hi List, >> >> I'm new to Numpy and I'm a little confused about the behavior of >> numpy.random.multivariate_normal(). I'm not sure I'm passing the >> variances correctly. My goal is to sample from a bivariate normal, but >> the kooky behavior shows up when I sample from a univariate distribution. >> >> In short, the multivariate normal function doesn't seem to give me >> values in the appropriate ranges. Here's an example of what I mean. >> >> In[1]: from numpy.random import normal, multivariate_normal >> >> In [2]: normal(100, 100, 10) >> Out[2]: >> array([ 258.62344586, 70.16378815, 49.46826997, 49.58567724, >> 182.68652256, 226.67292034, 92.03801549, 18.2686146 , >> 94.09104313, 80.35697507]) > Here, you are asking for a Gaussian distribution with a mean of 100 > and a stdev of 100 (or equivalently, a variance of 10000). > >> The samples look about right to me. But then when I try to do the same >> using the multivariate_normal, the values it draws look too close to the >> mean. >> >> In [3]: multivariate_normal([100], [[100]], 10) > Here you are asking for a Gaussian with a mean of 100 and a variance > of 100 (or equivalently, a stdev of 10). Note the differences between > the two signatures: > > np.random.normal(mean, stdev) > np.random.multivariate_normal(mean, covariance) > >> Out[3]: >> array([[ 109.10083984], >> [ 97.43526716], >> [ 108.43923772], >> [ 97.87345947], >> [ 103.405562 ], >> [ 110.2963736 ], >> [ 103.96445585], >> [ 90.58168544], >> [ 91.20549222], >> [ 104.4051761 ]]) >> >> These values all fall within 10 units of the mean. >> >> In [4]: multivariate_normal([100], [[1000]], 10) >> Out[4]: >> array([[ 62.04304611], >> [ 123.29364557], >> [ 83.53840083], >> [ 64.67679778], >> [ 127.82433157], >> [ 11.3305656 ], >> [ 95.4101701 ], >> [ 126.53213908], >> [ 104.68868736], >> [ 32.45886112]]) >> >> In [5]: normal(100, 1000, 10) >> Out[5]: >> array([ 1052.93998938, -1254.12576419, 258.75390045, 688.32715327, >> -2.36543806, -1570.54842269, 472.90045029, 394.62908014, >> 137.10320437, 1741.85017871]) >> >> And just to exaggerate things a little more: >> >> In [6]: multivariate_normal([100], [[10000]], 10).T][0] >> Out[6]: >> array([ 274.45446694, 85.79359682, 245.03248721, 120.10912405, >> -34.76526896, 134.93446664, 47.6768889 , 93.34140984, >> 80.27244669, 229.64700591]) >> >> Whereas >> In [7]: normal(100, 10000, 10) >> Out[7]: >> array([ -554.68666687, 3724.59638363, -14873.55303901, -3111.22162495, >> -10813.66412901, 4688.81310356, -9510.92470735, -12689.02667559, >> -10379.01381925, -4534.60480031]) >> >> I'm shocked that I don't get some negative values in Out[4]. And Out[6] >> really ought to have some numbers in the thousands. >> >> I'd totally be willing to believe that I don't understand the >> multivariate normal and/or variance. Can someone tell me whether these >> numbers look sane? >> >> For the bivariate case I do something like this: >> >> means = [100, 100] >> variances = [100, 1000] >> Sx, Sy = variances >> sx, sy = map(sqrt, variances) >> cor = 0.7 >> cov = [[Sx, cor*sx*sy], [cor*sy*sx, Sy]] >> draws = 10 >> samples = multivariate_normal(means, cov, draws) >> >> As mentioned before, the samples are shockingly close to their means. > They look right to me. > > [~] > |19> means = [100, 100] > > [~] > |20> variances = [100, 1000] > > [~] > |21> Sx, Sy = variances > > [~] > |22> sx, sy = map(sqrt, variances) > > [~] > |23> cor = 0.7 > > [~] > |24> cov = np.array([[Sx, cor*sx*sy], [cor*sy*sx, Sy]]) > > [~] > |26> samples = np.random.multivariate_normal(means, cov, 10000) > > [~] > |27> cov > array([[ 100. , 221.35943621], > [ 221.35943621, 1000. ]]) > > [~] > |28> np.cov(samples.T) > array([[ 101.16844481, 222.00301056], > [ 222.00301056, 1001.58403922]]) > From stefan at sun.ac.za Wed Nov 9 14:27:11 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 9 Nov 2011 11:27:11 -0800 Subject: [Numpy-discussion] np.dot and scipy sparse matrices In-Reply-To: References: Message-ID: Hi Peter On Wed, Nov 9, 2011 at 3:38 AM, Peter Prettenhofer wrote: > I recently got the latest numpy version (2.0.0.dev-7297785) from the > git repo and realized that `np.dot` causes a segfault if its operands > are scipy sparse matrices. Here's some code to reproduce the problem:: > > ? import numpy as np > ? from scipy import sparse as sp > ? A = np.random.rand(10, 10) > ? S = sp.csr_matrix(A) > ? _ = np.dot(A, A) ? # this works OK > ? _ = np.dot(S, S) ? # this segfaults! Here's a patch to get rid of the segfault: https://github.com/numpy/numpy/pull/170 I don't know if np.dot is supposed to know anything about sparse matrices; if so, then a fancier change may be needed. Regards St?fan From chaoyuejoy at gmail.com Thu Nov 10 05:17:15 2011 From: chaoyuejoy at gmail.com (Chao YUE) Date: Thu, 10 Nov 2011 11:17:15 +0100 Subject: [Numpy-discussion] how to use the name of a ndarray as a string Message-ID: Hi, Does anyone know how I can quickly use the name of a ndarray as a string? for example, I have In [54]: index=np.arange(100) then I want to use the name 'index' as a key in a new dictionary: d3=dict() d3['index']=index I can do it like the way above, but I have many ndarray variables that need to be included in the dictionary. is there something like: d3[index.name()]=index while index.name() would equal the string 'index'? I hope my question is clear. thanks to all. Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************ -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Thu Nov 10 06:57:25 2011 From: shish at keba.be (Olivier Delalleau) Date: Thu, 10 Nov 2011 06:57:25 -0500 Subject: [Numpy-discussion] how to use the name of a ndarray as a string In-Reply-To: References: Message-ID: In such a situation you should probably use a dictionary from the start, i.e.: d3['index'] = np.arange(100) then use d3['index'] everywhere instead of index. It can be more convenient (notation-wise) to use an object instead, i.e. either work within a class method (self.index = np.arange(100)), or use a class as storage instead of a dictionary: class ArrayHolder(object): pass a = ArrayHolder() a.index = np.arange(100) ... d3 = a.__dict__.copy() Finally, another option I can see is to use a custom class that holds both a numpy array and a name. Maybe with a convenient accessor to the array through the call method: class NamedArray(object): def __init__(self, array, name): self.array = array self.name = name def __call__(self): return self.array index = NamedArray(numpy.arange(100), 'index') ... # use index() in the rest of the code whenever you want the numpy array d3[index.name] = index() -=- Olivier 2011/11/10 Chao YUE > Hi, > > Does anyone know how I can quickly use the name of a ndarray as a string? > for example, I have > In [54]: index=np.arange(100) > > then I want to use the name 'index' as a key in a new dictionary: > d3=dict() > d3['index']=index > > I can do it like the way above, but I have many ndarray variables that > need to be included in the dictionary. > is there something like: > d3[index.name()]=index > while index.name() would equal the string 'index'? > > I hope my question is clear. thanks to all. > > Chao > -- > > *********************************************************************************** > Chao YUE > Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) > UMR 1572 CEA-CNRS-UVSQ > Batiment 712 - Pe 119 > 91191 GIF Sur YVETTE Cedex > Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 > > ************************************************************************************ > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.prettenhofer at gmail.com Thu Nov 10 13:45:41 2011 From: peter.prettenhofer at gmail.com (Peter Prettenhofer) Date: Thu, 10 Nov 2011 19:45:41 +0100 Subject: [Numpy-discussion] np.dot and scipy sparse matrices Message-ID: Hi Stefan, thanks a lot for the pull request! > I don't know if np.dot is supposed to know anything about sparse > matrices; if so, then a fancier change may be needed. Ok, then I'll pass that on to the (sparse) scipy folks. Are there any alternative to np.dot you are aware of? Is ndarray.__mul__ supposed to work with sparse matrices? AFAIK scipy does not provide a scipy.sparse.dot function and spmatrix.dot only allows for "multiplication from the rhs" (ok.. one could directly call __rmul__ but that's ugly). best, Peter -- Peter Prettenhofer From stefan at sun.ac.za Thu Nov 10 13:56:39 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 10 Nov 2011 10:56:39 -0800 Subject: [Numpy-discussion] np.dot and scipy sparse matrices In-Reply-To: References: Message-ID: On Thu, Nov 10, 2011 at 10:45 AM, Peter Prettenhofer wrote: >> I don't know if np.dot is supposed to know anything about sparse >> matrices; if so, then a fancier change may be needed. > > Ok, then I'll pass that on to the (sparse) scipy folks. > Are there any alternative to np.dot you are aware of? With sparse matrices, you can simply do S = ss.csr_matrix(np.arange(9).reshape((3,3))) x = np.array([[1],[1],[1]]) print S * x [[ 6.] [ 15.] [ 24.]] But I assume what you want is an easy way to handle both sparse matrices and dense arrays with the same call? I noticed in your comment on the sklearn list that this used to work on older versions of numpy. If that is the case, this is probably a regression that needs to be fixed. Regards St?fan From zonca at deepspace.ucsb.edu Fri Nov 11 01:41:21 2011 From: zonca at deepspace.ucsb.edu (Andrea Zonca) Date: Thu, 10 Nov 2011 22:41:21 -0800 Subject: [Numpy-discussion] dedicated function for resize with averaging or rebin 2d arrays? Message-ID: hi, I work in astrophysics where the most common programming language is currently IDL. A common request of people switching from IDL to python is the implementation of the REBIN function, which either downsizes a 2d array by averaging or increases its dimension by repeating its elements. In both cases the new shape must be an integer factor of the old shape. I believe it is a very handy function for quick smoothing of 2 dimensional data. I found a discussion about this topic in the archives: http://thread.gmane.org/gmane.comp.python.numeric.general/885/focus=894 Do you think it would be useful to add such function to numpy? I created a simple implementation to help in the discussion: https://gist.github.com/1348792 thanks, Andrea Zonca From zachary.pincus at yale.edu Fri Nov 11 10:07:35 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 11 Nov 2011 10:07:35 -0500 Subject: [Numpy-discussion] dedicated function for resize with averaging or rebin 2d arrays? In-Reply-To: References: Message-ID: Hi Andrea, scipy.ndimage.zoom will do this nicely for magnification. (Just set the spline order to 0 to get nearest-neighbor interpolation; otherwise you can use higher orders for better smoothing.) For decimation (zooming out) scipy.ndimage.zoom also works, but it's not as nice as a dedicated decimation filter that would average properly over the area that's being squeezed into a single output pixel. (You'd have to choose the spline order manually to approximate that.) I'm afraid I don't have enough signal-processing background to know how to write a proper general-purpose decimation filter -- basically, you convolve with whatever bandlimiting filter (e.g. a gaussian, or do it in the Fourier domain), then just do nearest-neighbor downsampling, but I'm never sure how to properly choose the filter parameters! Between this and ndimage.zoom for magnifying, one could get together a much better "rebin" function that in the edge cases of integer magnification/minification should work the same as the IDL one. But the participants in the old discussion you highlighted seemed unhappy with the time/space used for proper decimation, so I'm not sure what really would be best. Zach On Nov 11, 2011, at 1:41 AM, Andrea Zonca wrote: > hi, > I work in astrophysics where the most common programming language is > currently IDL. > A common request of people switching from IDL to python is the > implementation of the REBIN function, which either downsizes a 2d > array by averaging or increases its dimension by repeating its > elements. In both cases the new shape must be an integer factor of the > old shape. > > I believe it is a very handy function for quick smoothing of 2 dimensional data. > > I found a discussion about this topic in the archives: > http://thread.gmane.org/gmane.comp.python.numeric.general/885/focus=894 > > Do you think it would be useful to add such function to numpy? > > I created a simple implementation to help in the discussion: > https://gist.github.com/1348792 > > thanks, > Andrea Zonca > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From zonca at deepspace.ucsb.edu Fri Nov 11 23:22:56 2011 From: zonca at deepspace.ucsb.edu (Andrea Zonca) Date: Fri, 11 Nov 2011 20:22:56 -0800 Subject: [Numpy-discussion] dedicated function for resize with averaging or rebin 2d arrays? In-Reply-To: References: Message-ID: Hi, I think the usefulness of the rebin function is to be simple and fast, and the best would be to implement it in the core numpy, as a simple method for smoothing and reshaping, without the need of using scipy. For anything more sophisticated, the ndimage module as you suggest should be used. cheers, Andrea On Fri, Nov 11, 2011 at 07:07, Zachary Pincus wrote: > Hi Andrea, > > scipy.ndimage.zoom will do this nicely for magnification. (Just set the spline order to 0 to get nearest-neighbor interpolation; otherwise you can use higher orders for better smoothing.) > > For decimation (zooming out) scipy.ndimage.zoom also works, but it's not as nice as a dedicated decimation filter that would average properly over the area that's being squeezed into a single output pixel. (You'd have to choose the spline order manually to approximate that.) I'm afraid I don't have enough signal-processing background to know how to write a proper general-purpose decimation filter -- basically, you convolve with whatever bandlimiting filter (e.g. a gaussian, or do it in the Fourier domain), then just do nearest-neighbor downsampling, but I'm never sure how to properly choose the filter parameters! > > Between this and ndimage.zoom for magnifying, one could get together a much better "rebin" function that in the edge cases of integer magnification/minification should work the same as the IDL one. But the participants in the old discussion you highlighted seemed unhappy with the time/space used for proper decimation, so I'm not sure what really would be best. > > Zach > > > On Nov 11, 2011, at 1:41 AM, Andrea Zonca wrote: > >> hi, >> I work in astrophysics where the most common programming language is >> currently IDL. >> A common request of people switching from IDL to python is the >> implementation of the REBIN function, which either downsizes a 2d >> array by averaging or increases its dimension by repeating its >> elements. In both cases the new shape must be an integer factor of the >> old shape. >> >> I believe it is a very handy function for quick smoothing of 2 dimensional data. >> >> I found a discussion about this topic in the archives: >> http://thread.gmane.org/gmane.comp.python.numeric.general/885/focus=894 >> >> Do you think it would be useful to add such function to numpy? >> >> I created a simple implementation to help in the discussion: >> https://gist.github.com/1348792 >> >> thanks, >> Andrea Zonca >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From craigyk at me.com Fri Nov 11 23:28:15 2011 From: craigyk at me.com (Craig Yoshioka) Date: Fri, 11 Nov 2011 20:28:15 -0800 Subject: [Numpy-discussion] dedicated function for resize with averaging or rebin 2d arrays? In-Reply-To: References: Message-ID: <9DD9103D-D5CB-489A-A1CF-0273367651E1@me.com> I once wrote a generic n-dimensional binning routine in C that I could find if anyone is interested in integrating it into numpy... it didn't do size increases though... and I think I implemented it so that binning by a non-divisible factor trimmed the extras. It was very-very fast though. On Nov 11, 2011, at 8:22 PM, Andrea Zonca wrote: > Hi, > I think the usefulness of the rebin function is to be simple and fast, > and the best would be to implement it in the core numpy, as a simple > method for smoothing and reshaping, without the need of using scipy. > For anything more sophisticated, the ndimage module as you suggest > should be used. > cheers, > Andrea > > On Fri, Nov 11, 2011 at 07:07, Zachary Pincus wrote: >> Hi Andrea, >> >> scipy.ndimage.zoom will do this nicely for magnification. (Just set the spline order to 0 to get nearest-neighbor interpolation; otherwise you can use higher orders for better smoothing.) >> >> For decimation (zooming out) scipy.ndimage.zoom also works, but it's not as nice as a dedicated decimation filter that would average properly over the area that's being squeezed into a single output pixel. (You'd have to choose the spline order manually to approximate that.) I'm afraid I don't have enough signal-processing background to know how to write a proper general-purpose decimation filter -- basically, you convolve with whatever bandlimiting filter (e.g. a gaussian, or do it in the Fourier domain), then just do nearest-neighbor downsampling, but I'm never sure how to properly choose the filter parameters! >> >> Between this and ndimage.zoom for magnifying, one could get together a much better "rebin" function that in the edge cases of integer magnification/minification should work the same as the IDL one. But the participants in the old discussion you highlighted seemed unhappy with the time/space used for proper decimation, so I'm not sure what really would be best. >> >> Zach >> >> >> On Nov 11, 2011, at 1:41 AM, Andrea Zonca wrote: >> >>> hi, >>> I work in astrophysics where the most common programming language is >>> currently IDL. >>> A common request of people switching from IDL to python is the >>> implementation of the REBIN function, which either downsizes a 2d >>> array by averaging or increases its dimension by repeating its >>> elements. In both cases the new shape must be an integer factor of the >>> old shape. >>> >>> I believe it is a very handy function for quick smoothing of 2 dimensional data. >>> >>> I found a discussion about this topic in the archives: >>> http://thread.gmane.org/gmane.comp.python.numeric.general/885/focus=894 >>> >>> Do you think it would be useful to add such function to numpy? >>> >>> I created a simple implementation to help in the discussion: >>> https://gist.github.com/1348792 >>> >>> thanks, >>> Andrea Zonca >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From zyzhu2000 at gmail.com Sat Nov 12 03:36:38 2011 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Sat, 12 Nov 2011 02:36:38 -0600 Subject: [Numpy-discussion] speeding up the following expression Message-ID: Hi, I am playing with multiple ways to speed up the following expression (it is in the inner loop): C[1:(M - 1)]=(a * C[2:] + b * C[1:(M-1)] + c * C[:(M-2)]) where C is an array of about 200-300 elements, M=len(C), a, b, c are scalars. I played with numexpr, but it was way slower than directly using numpy. It would be nice if I could create a Mx3 matrix without copying memory and so I can use dot() to calculate the whole thing. Can anyone help with giving some advices to make this faster? Thanks, G From josef.pktd at gmail.com Sat Nov 12 07:43:57 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 12 Nov 2011 07:43:57 -0500 Subject: [Numpy-discussion] speeding up the following expression In-Reply-To: References: Message-ID: On Sat, Nov 12, 2011 at 3:36 AM, Geoffrey Zhu wrote: > Hi, > > I am playing with multiple ways to speed up the following expression > (it is in the inner loop): > > > C[1:(M - 1)]=(a * C[2:] + b * C[1:(M-1)] + c * C[:(M-2)]) > > where C is an array of about 200-300 elements, M=len(C), a, b, c are scalars. > > I played with numexpr, but it was way slower than directly using > numpy. It would be nice if I could create a Mx3 matrix without copying > memory and so I can use dot() to calculate the whole thing. > > Can anyone help with giving some advices to make this faster? looks like a np.convolve(C, [a,b,c]) to me except for the boundary conditions. Josef > > Thanks, > G > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From warren.weckesser at enthought.com Sat Nov 12 10:31:17 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sat, 12 Nov 2011 09:31:17 -0600 Subject: [Numpy-discussion] speeding up the following expression In-Reply-To: References: Message-ID: On Sat, Nov 12, 2011 at 6:43 AM, wrote: > On Sat, Nov 12, 2011 at 3:36 AM, Geoffrey Zhu wrote: > > Hi, > > > > I am playing with multiple ways to speed up the following expression > > (it is in the inner loop): > > > > > > C[1:(M - 1)]=(a * C[2:] + b * C[1:(M-1)] + c * C[:(M-2)]) > > > > where C is an array of about 200-300 elements, M=len(C), a, b, c are > scalars. > > > > I played with numexpr, but it was way slower than directly using > > numpy. It would be nice if I could create a Mx3 matrix without copying > > memory and so I can use dot() to calculate the whole thing. > > > > Can anyone help with giving some advices to make this faster? > > looks like a np.convolve(C, [a,b,c]) to me except for the boundary > conditions. > As Josef pointed out, this is a convolution. There are (at least) three convolution functions in numpy+scipy that you could use: numpy.convolve, scipy.signal.convolve, and scipy.ndimage.convolve1d. Of these, scipy.ndimage.convolve1d is the fastest. However, it doesn't beat the simple expression a*x[2:] + b*x[1:-1] + c*x[:-2] Your idea of forming a matrix without copying memory can be done using "stride tricks", and for arrays of the size you are interested in, it computes the result faster than the simple expression (see below). Another fast alternative is to use one of the inline code generators. This example is a easy to implement with scipy.weave.blitz, and it gives a big speedup. Here's a test script: #----- convolve1dtest.py ----- import numpy as np from numpy.lib.stride_tricks import as_strided from scipy.ndimage import convolve1d from scipy.weave import blitz # weighting coefficients a = -0.5 b = 1.0 c = -0.25 w = np.array((a,b,c)) # Reversed w: rw = w[::-1] # Length of C n = 250 # The original version of the calculation: # Some dummy data C = np.arange(float(n)) C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] # Save for comparison. C0 = C.copy() # Do it again using a matrix multiplication. C = np.arange(float(n)) # The "virtual" matrix view of C. V = as_strided(C, shape=(C.size-2, 3), strides=(C.strides[0], C.strides[0])) C[1:-1] = np.dot(V, rw) C1 = C.copy() # Again, with convolve1d this time. C = np.arange(float(n)) C[1:-1] = convolve1d(C, w)[1:-1] C2 = C.copy() # scipy.weave.blitz C = np.arange(float(n)) # Must work with a copy, D, in the formula, because blitz does not use # a temporary variable. D = C.copy() expr = "C[1:-1] = a * D[2:] + b * D[1:-1] + c * D[:-2]" blitz(expr, check_size=0) C3 = C.copy() # Verify that all the methods give the same result. print np.all(C0 == C1) print np.all(C0 == C2) print np.all(C0 == C3) #----- And here's a snippet from an ipython session: In [51]: run convolve1dtest.py True True True In [52]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] 100000 loops, best of 3: 16.5 us per loop In [53]: %timeit C[1:-1] = np.dot(V, rw) 100000 loops, best of 3: 9.84 us per loop In [54]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] 100000 loops, best of 3: 18.7 us per loop In [55]: %timeit D = C.copy(); blitz(expr, check_size=0) 100000 loops, best of 3: 4.91 us per loop scipy.weave.blitz is fastest (but note that blitz has already been called once, so the time shown does not include the compilation required in the first call). You could also try scipy.weave.inline, cython.inline, or np_inline (http://pages.cs.wisc.edu/~johnl/np_inline/). Also note that C[-1:1] = np.dot(V, rw) is faster than either the simple expression or convolve1d. However, if you also have to set up V inside your inner loop, the speed gain will be lost. The relative speeds also depend on the size of C. For large C, the simple expression is faster than the matrix multiplication by V (but blitz is still faster). In the following, I have changed n to 2500 before running convolve1dtest.py: In [56]: run convolve1dtest.py True True True In [57]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] 10000 loops, best of 3: 29.5 us per loop In [58]: %timeit C[1:-1] = np.dot(V, rw) 10000 loops, best of 3: 56.4 us per loop In [59]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] 10000 loops, best of 3: 37.3 us per loop In [60]: %timeit D = C.copy(); blitz(expr, check_size=0) 100000 loops, best of 3: 10.3 us per loop blitz wins, the simple numpy expression is a distant second, and now the matrix multiplication is slowest. I hope that helps--I know I learned quite a bit. :) Warren > Josef > > > > > > Thanks, > > G > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Nov 12 10:59:27 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 12 Nov 2011 10:59:27 -0500 Subject: [Numpy-discussion] speeding up the following expression In-Reply-To: References: Message-ID: On Sat, Nov 12, 2011 at 10:31 AM, Warren Weckesser wrote: > > > On Sat, Nov 12, 2011 at 6:43 AM, wrote: >> >> On Sat, Nov 12, 2011 at 3:36 AM, Geoffrey Zhu wrote: >> > Hi, >> > >> > I am playing with multiple ways to speed up the following expression >> > (it is in the inner loop): >> > >> > >> > C[1:(M - 1)]=(a * C[2:] + b * C[1:(M-1)] + c * C[:(M-2)]) >> > >> > where C is an array of about 200-300 elements, M=len(C), a, b, c are >> > scalars. >> > >> > I played with numexpr, but it was way slower than directly using >> > numpy. It would be nice if I could create a Mx3 matrix without copying >> > memory and so I can use dot() to calculate the whole thing. >> > >> > Can anyone help with giving some advices to make this faster? >> >> looks like a np.convolve(C, [a,b,c]) ?to me except for the boundary >> conditions. > > > > As Josef pointed out, this is a convolution.? There are (at least) > three convolution functions in numpy+scipy that you could use: > numpy.convolve, scipy.signal.convolve, and scipy.ndimage.convolve1d. > Of these, scipy.ndimage.convolve1d is the fastest.? However, it doesn't > beat the simple expression > ?? a*x[2:] + b*x[1:-1] + c*x[:-2] > Your idea of forming a matrix without copying memory can be done using > "stride tricks", and for arrays of the size you are interested in, it > computes the result faster than the simple expression (see below). > > Another fast alternative is to use one of the inline code generators. > This example is a easy to implement with scipy.weave.blitz, and it gives > a big speedup. > > Here's a test script: > > #----- convolve1dtest.py ----- > > > import numpy as np > from numpy.lib.stride_tricks import as_strided > from scipy.ndimage import convolve1d > from scipy.weave import blitz > > # weighting coefficients > a = -0.5 > b = 1.0 > c = -0.25 > w = np.array((a,b,c)) > # Reversed w: > rw = w[::-1] > > # Length of C > n = 250 > > # The original version of the calculation: > # Some dummy data > C = np.arange(float(n)) > C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > # Save for comparison. > C0 = C.copy() > > # Do it again using a matrix multiplication. > C = np.arange(float(n)) > # The "virtual" matrix view of C. > V = as_strided(C, shape=(C.size-2, 3), strides=(C.strides[0], C.strides[0])) > C[1:-1] = np.dot(V, rw) > C1 = C.copy() > > # Again, with convolve1d this time. > C = np.arange(float(n)) > C[1:-1] = convolve1d(C, w)[1:-1] > C2 = C.copy() > > # scipy.weave.blitz > C = np.arange(float(n)) > # Must work with a copy, D, in the formula, because blitz does not use > # a temporary variable. > D = C.copy() > expr = "C[1:-1] = a * D[2:] + b * D[1:-1] + c * D[:-2]" > blitz(expr, check_size=0) > C3 = C.copy() > > > # Verify that all the methods give the same result. > print np.all(C0 == C1) > print np.all(C0 == C2) > print np.all(C0 == C3) > > #----- > > And here's a snippet from an ipython session: > > In [51]: run convolve1dtest.py > True > True > True > > In [52]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > 100000 loops, best of 3: 16.5 us per loop > > In [53]: %timeit C[1:-1] = np.dot(V, rw) > 100000 loops, best of 3: 9.84 us per loop > > In [54]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] > 100000 loops, best of 3: 18.7 us per loop > > In [55]: %timeit D = C.copy(); blitz(expr, check_size=0) > 100000 loops, best of 3: 4.91 us per loop > > > > scipy.weave.blitz is fastest (but note that blitz has already been called > once, so the time shown does not include the compilation required in > the first call).? You could also try scipy.weave.inline, cython.inline, > or np_inline (http://pages.cs.wisc.edu/~johnl/np_inline/). > > Also note that C[-1:1] = np.dot(V, rw) is faster than either the simple > expression or convolve1d.? However, if you also have to set up V inside > your inner loop, the speed gain will be lost.? The relative speeds also > depend on the size of C.? For large C, the simple expression is faster > than the matrix multiplication by V (but blitz is still faster).? In > the following, I have changed n to 2500 before running convolve1dtest.py: > > In [56]: run convolve1dtest.py > True > True > True > > In [57]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > 10000 loops, best of 3: 29.5 us per loop > > In [58]: %timeit C[1:-1] = np.dot(V, rw) > 10000 loops, best of 3: 56.4 us per loop > > In [59]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] > 10000 loops, best of 3: 37.3 us per loop > > In [60]: %timeit D = C.copy(); blitz(expr, check_size=0) > 100000 loops, best of 3: 10.3 us per loop > > > blitz wins, the simple numpy expression is a distant second, and now > the matrix multiplication is slowest. > > I hope that helps--I know I learned quite a bit. :) Interesting, two questions does scipy.signal convolve have a similar overhead as np.convolve1d ? memory: the blitz code doesn't include the array copy (D), so the timing might be a bit misleading? I assume the as_strided call doesn't allocate any memory yet, so the timing should be correct. (or is this your comment about setting up V in the inner loop) Josef > > Warren > > >> >> Josef >> >> >> > >> > Thanks, >> > G >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From warren.weckesser at enthought.com Sat Nov 12 11:32:58 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sat, 12 Nov 2011 10:32:58 -0600 Subject: [Numpy-discussion] speeding up the following expression In-Reply-To: References: Message-ID: On Sat, Nov 12, 2011 at 9:59 AM, wrote: > On Sat, Nov 12, 2011 at 10:31 AM, Warren Weckesser > wrote: > > > > > > On Sat, Nov 12, 2011 at 6:43 AM, wrote: > >> > >> On Sat, Nov 12, 2011 at 3:36 AM, Geoffrey Zhu > wrote: > >> > Hi, > >> > > >> > I am playing with multiple ways to speed up the following expression > >> > (it is in the inner loop): > >> > > >> > > >> > C[1:(M - 1)]=(a * C[2:] + b * C[1:(M-1)] + c * C[:(M-2)]) > >> > > >> > where C is an array of about 200-300 elements, M=len(C), a, b, c are > >> > scalars. > >> > > >> > I played with numexpr, but it was way slower than directly using > >> > numpy. It would be nice if I could create a Mx3 matrix without copying > >> > memory and so I can use dot() to calculate the whole thing. > >> > > >> > Can anyone help with giving some advices to make this faster? > >> > >> looks like a np.convolve(C, [a,b,c]) to me except for the boundary > >> conditions. > > > > > > > > As Josef pointed out, this is a convolution. There are (at least) > > three convolution functions in numpy+scipy that you could use: > > numpy.convolve, scipy.signal.convolve, and scipy.ndimage.convolve1d. > > Of these, scipy.ndimage.convolve1d is the fastest. However, it doesn't > > beat the simple expression > > a*x[2:] + b*x[1:-1] + c*x[:-2] > > Your idea of forming a matrix without copying memory can be done using > > "stride tricks", and for arrays of the size you are interested in, it > > computes the result faster than the simple expression (see below). > > > > Another fast alternative is to use one of the inline code generators. > > This example is a easy to implement with scipy.weave.blitz, and it gives > > a big speedup. > > > > Here's a test script: > > > > #----- convolve1dtest.py ----- > > > > > > import numpy as np > > from numpy.lib.stride_tricks import as_strided > > from scipy.ndimage import convolve1d > > from scipy.weave import blitz > > > > # weighting coefficients > > a = -0.5 > > b = 1.0 > > c = -0.25 > > w = np.array((a,b,c)) > > # Reversed w: > > rw = w[::-1] > > > > # Length of C > > n = 250 > > > > # The original version of the calculation: > > # Some dummy data > > C = np.arange(float(n)) > > C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > > # Save for comparison. > > C0 = C.copy() > > > > # Do it again using a matrix multiplication. > > C = np.arange(float(n)) > > # The "virtual" matrix view of C. > > V = as_strided(C, shape=(C.size-2, 3), strides=(C.strides[0], > C.strides[0])) > > C[1:-1] = np.dot(V, rw) > > C1 = C.copy() > > > > # Again, with convolve1d this time. > > C = np.arange(float(n)) > > C[1:-1] = convolve1d(C, w)[1:-1] > > C2 = C.copy() > > > > # scipy.weave.blitz > > C = np.arange(float(n)) > > # Must work with a copy, D, in the formula, because blitz does not use > > # a temporary variable. > > D = C.copy() > > expr = "C[1:-1] = a * D[2:] + b * D[1:-1] + c * D[:-2]" > > blitz(expr, check_size=0) > > C3 = C.copy() > > > > > > # Verify that all the methods give the same result. > > print np.all(C0 == C1) > > print np.all(C0 == C2) > > print np.all(C0 == C3) > > > > #----- > > > > And here's a snippet from an ipython session: > > > > In [51]: run convolve1dtest.py > > True > > True > > True > > > > In [52]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > > 100000 loops, best of 3: 16.5 us per loop > > > > In [53]: %timeit C[1:-1] = np.dot(V, rw) > > 100000 loops, best of 3: 9.84 us per loop > > > > In [54]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] > > 100000 loops, best of 3: 18.7 us per loop > > > > In [55]: %timeit D = C.copy(); blitz(expr, check_size=0) > > 100000 loops, best of 3: 4.91 us per loop > > > > > > > > scipy.weave.blitz is fastest (but note that blitz has already been called > > once, so the time shown does not include the compilation required in > > the first call). You could also try scipy.weave.inline, cython.inline, > > or np_inline (http://pages.cs.wisc.edu/~johnl/np_inline/). > > > > Also note that C[-1:1] = np.dot(V, rw) is faster than either the simple > > expression or convolve1d. However, if you also have to set up V inside > > your inner loop, the speed gain will be lost. The relative speeds also > > depend on the size of C. For large C, the simple expression is faster > > than the matrix multiplication by V (but blitz is still faster). In > > the following, I have changed n to 2500 before running convolve1dtest.py: > > > > In [56]: run convolve1dtest.py > > True > > True > > True > > > > In [57]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > > 10000 loops, best of 3: 29.5 us per loop > > > > In [58]: %timeit C[1:-1] = np.dot(V, rw) > > 10000 loops, best of 3: 56.4 us per loop > > > > In [59]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] > > 10000 loops, best of 3: 37.3 us per loop > > > > In [60]: %timeit D = C.copy(); blitz(expr, check_size=0) > > 100000 loops, best of 3: 10.3 us per loop > > > > > > blitz wins, the simple numpy expression is a distant second, and now > > the matrix multiplication is slowest. > > > > I hope that helps--I know I learned quite a bit. :) > > Interesting, two questions > > does scipy.signal convolve have a similar overhead as np.convolve1d ? > Did you mean np.convolve? There is no np.convolve1d. Some of the tests that I've done with convolution functions are here: http://www.scipy.org/Cookbook/ApplyFIRFilter I should add np.convolve to that page. For the case considered here, np.convolve is a bit slower than scipy.ndimage.convolve1d, but for larger arrays, it looks like np.convolve can be much faster. > memory: > the blitz code doesn't include the array copy (D), so the timing might > be a bit misleading? > Look again at my %timeit calls in the ipython snippets. :) > I assume the as_strided call doesn't allocate any memory yet, so the > timing should be correct. (or is this your comment about setting up V > in the inner loop) > > Yes, that's what I meant; if V has to be created inside the inner loop (so as_strided is called in the loop), the time it takes to create V eliminates the benefit of using the matrix approach. Warren > Josef > > > > > Warren > > > > > >> > >> Josef > >> > >> > >> > > >> > Thanks, > >> > G > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Nov 12 12:16:50 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 12 Nov 2011 12:16:50 -0500 Subject: [Numpy-discussion] speeding up the following expression In-Reply-To: References: Message-ID: On Sat, Nov 12, 2011 at 11:32 AM, Warren Weckesser wrote: > > > On Sat, Nov 12, 2011 at 9:59 AM, wrote: >> >> On Sat, Nov 12, 2011 at 10:31 AM, Warren Weckesser >> wrote: >> > >> > >> > On Sat, Nov 12, 2011 at 6:43 AM, wrote: >> >> >> >> On Sat, Nov 12, 2011 at 3:36 AM, Geoffrey Zhu >> >> wrote: >> >> > Hi, >> >> > >> >> > I am playing with multiple ways to speed up the following expression >> >> > (it is in the inner loop): >> >> > >> >> > >> >> > C[1:(M - 1)]=(a * C[2:] + b * C[1:(M-1)] + c * C[:(M-2)]) >> >> > >> >> > where C is an array of about 200-300 elements, M=len(C), a, b, c are >> >> > scalars. >> >> > >> >> > I played with numexpr, but it was way slower than directly using >> >> > numpy. It would be nice if I could create a Mx3 matrix without >> >> > copying >> >> > memory and so I can use dot() to calculate the whole thing. >> >> > >> >> > Can anyone help with giving some advices to make this faster? >> >> >> >> looks like a np.convolve(C, [a,b,c]) ?to me except for the boundary >> >> conditions. >> > >> > >> > >> > As Josef pointed out, this is a convolution.? There are (at least) >> > three convolution functions in numpy+scipy that you could use: >> > numpy.convolve, scipy.signal.convolve, and scipy.ndimage.convolve1d. >> > Of these, scipy.ndimage.convolve1d is the fastest.? However, it doesn't >> > beat the simple expression >> > ?? a*x[2:] + b*x[1:-1] + c*x[:-2] >> > Your idea of forming a matrix without copying memory can be done using >> > "stride tricks", and for arrays of the size you are interested in, it >> > computes the result faster than the simple expression (see below). >> > >> > Another fast alternative is to use one of the inline code generators. >> > This example is a easy to implement with scipy.weave.blitz, and it gives >> > a big speedup. >> > >> > Here's a test script: >> > >> > #----- convolve1dtest.py ----- >> > >> > >> > import numpy as np >> > from numpy.lib.stride_tricks import as_strided >> > from scipy.ndimage import convolve1d >> > from scipy.weave import blitz >> > >> > # weighting coefficients >> > a = -0.5 >> > b = 1.0 >> > c = -0.25 >> > w = np.array((a,b,c)) >> > # Reversed w: >> > rw = w[::-1] >> > >> > # Length of C >> > n = 250 >> > >> > # The original version of the calculation: >> > # Some dummy data >> > C = np.arange(float(n)) >> > C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] >> > # Save for comparison. >> > C0 = C.copy() >> > >> > # Do it again using a matrix multiplication. >> > C = np.arange(float(n)) >> > # The "virtual" matrix view of C. >> > V = as_strided(C, shape=(C.size-2, 3), strides=(C.strides[0], >> > C.strides[0])) >> > C[1:-1] = np.dot(V, rw) >> > C1 = C.copy() >> > >> > # Again, with convolve1d this time. >> > C = np.arange(float(n)) >> > C[1:-1] = convolve1d(C, w)[1:-1] >> > C2 = C.copy() >> > >> > # scipy.weave.blitz >> > C = np.arange(float(n)) >> > # Must work with a copy, D, in the formula, because blitz does not use >> > # a temporary variable. >> > D = C.copy() >> > expr = "C[1:-1] = a * D[2:] + b * D[1:-1] + c * D[:-2]" >> > blitz(expr, check_size=0) >> > C3 = C.copy() >> > >> > >> > # Verify that all the methods give the same result. >> > print np.all(C0 == C1) >> > print np.all(C0 == C2) >> > print np.all(C0 == C3) >> > >> > #----- >> > >> > And here's a snippet from an ipython session: >> > >> > In [51]: run convolve1dtest.py >> > True >> > True >> > True >> > >> > In [52]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] >> > 100000 loops, best of 3: 16.5 us per loop >> > >> > In [53]: %timeit C[1:-1] = np.dot(V, rw) >> > 100000 loops, best of 3: 9.84 us per loop >> > >> > In [54]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] >> > 100000 loops, best of 3: 18.7 us per loop >> > >> > In [55]: %timeit D = C.copy(); blitz(expr, check_size=0) >> > 100000 loops, best of 3: 4.91 us per loop >> > >> > >> > >> > scipy.weave.blitz is fastest (but note that blitz has already been >> > called >> > once, so the time shown does not include the compilation required in >> > the first call).? You could also try scipy.weave.inline, cython.inline, >> > or np_inline (http://pages.cs.wisc.edu/~johnl/np_inline/). >> > >> > Also note that C[-1:1] = np.dot(V, rw) is faster than either the simple >> > expression or convolve1d.? However, if you also have to set up V inside >> > your inner loop, the speed gain will be lost.? The relative speeds also >> > depend on the size of C.? For large C, the simple expression is faster >> > than the matrix multiplication by V (but blitz is still faster).? In >> > the following, I have changed n to 2500 before running >> > convolve1dtest.py: >> > >> > In [56]: run convolve1dtest.py >> > True >> > True >> > True >> > >> > In [57]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] >> > 10000 loops, best of 3: 29.5 us per loop >> > >> > In [58]: %timeit C[1:-1] = np.dot(V, rw) >> > 10000 loops, best of 3: 56.4 us per loop >> > >> > In [59]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] >> > 10000 loops, best of 3: 37.3 us per loop >> > >> > In [60]: %timeit D = C.copy(); blitz(expr, check_size=0) >> > 100000 loops, best of 3: 10.3 us per loop >> > >> > >> > blitz wins, the simple numpy expression is a distant second, and now >> > the matrix multiplication is slowest. >> > >> > I hope that helps--I know I learned quite a bit. :) >> >> Interesting, two questions >> >> does scipy.signal convolve have a similar overhead as np.convolve1d ? > > > Did you mean np.convolve?? There is no np.convolve1d. ? Some of the tests > that I've done with convolution functions are here: > ??? http://www.scipy.org/Cookbook/ApplyFIRFilter > I should add np.convolve to that page.? For the case considered here, > np.convolve is a bit slower than scipy.ndimage.convolve1d, but for larger > arrays, it looks like np.convolve can be much faster. > > >> >> memory: >> the blitz code doesn't include the array copy (D), so the timing might >> be a bit misleading? > > > Look again at my %timeit calls in the ipython snippets.? :) > Sorry, I was reading way too fast or selectively (skipping the imports, namespaces for example). (I thought convolve1d was numpy not ndimage) I tried out your cookbook script a while ago, it's nice, I had only a rough idea about the timing before. Compared to the current case the x is much longer, (unless I don't remember or read it correctly.) Thanks, Josef > >> >> I assume the as_strided call doesn't allocate any memory yet, so the >> timing should be correct. (or is this your comment about setting up V >> in the inner loop) >> > > > Yes, that's what I meant; if V has to be created inside the inner loop (so > as_strided is called in the loop), the time it takes to create V eliminates > the benefit of using the matrix approach. > > Warren > > >> >> Josef >> >> > >> > Warren >> > >> > >> >> >> >> Josef >> >> >> >> >> >> > >> >> > Thanks, >> >> > G >> >> > _______________________________________________ >> >> > NumPy-Discussion mailing list >> >> > NumPy-Discussion at scipy.org >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From warren.weckesser at enthought.com Sat Nov 12 12:51:10 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sat, 12 Nov 2011 11:51:10 -0600 Subject: [Numpy-discussion] speeding up the following expression In-Reply-To: References: Message-ID: On Sat, Nov 12, 2011 at 11:16 AM, wrote: > On Sat, Nov 12, 2011 at 11:32 AM, Warren Weckesser > wrote: > > > > > > On Sat, Nov 12, 2011 at 9:59 AM, wrote: > >> > >> On Sat, Nov 12, 2011 at 10:31 AM, Warren Weckesser > >> wrote: > >> > > >> > > >> > On Sat, Nov 12, 2011 at 6:43 AM, wrote: > >> >> > >> >> On Sat, Nov 12, 2011 at 3:36 AM, Geoffrey Zhu > >> >> wrote: > >> >> > Hi, > >> >> > > >> >> > I am playing with multiple ways to speed up the following > expression > >> >> > (it is in the inner loop): > >> >> > > >> >> > > >> >> > C[1:(M - 1)]=(a * C[2:] + b * C[1:(M-1)] + c * C[:(M-2)]) > >> >> > > >> >> > where C is an array of about 200-300 elements, M=len(C), a, b, c > are > >> >> > scalars. > >> >> > > >> >> > I played with numexpr, but it was way slower than directly using > >> >> > numpy. It would be nice if I could create a Mx3 matrix without > >> >> > copying > >> >> > memory and so I can use dot() to calculate the whole thing. > >> >> > > >> >> > Can anyone help with giving some advices to make this faster? > >> >> > >> >> looks like a np.convolve(C, [a,b,c]) to me except for the boundary > >> >> conditions. > >> > > >> > > >> > > >> > As Josef pointed out, this is a convolution. There are (at least) > >> > three convolution functions in numpy+scipy that you could use: > >> > numpy.convolve, scipy.signal.convolve, and scipy.ndimage.convolve1d. > >> > Of these, scipy.ndimage.convolve1d is the fastest. However, it > doesn't > >> > beat the simple expression > >> > a*x[2:] + b*x[1:-1] + c*x[:-2] > >> > Your idea of forming a matrix without copying memory can be done using > >> > "stride tricks", and for arrays of the size you are interested in, it > >> > computes the result faster than the simple expression (see below). > >> > > >> > Another fast alternative is to use one of the inline code generators. > >> > This example is a easy to implement with scipy.weave.blitz, and it > gives > >> > a big speedup. > >> > > >> > Here's a test script: > >> > > >> > #----- convolve1dtest.py ----- > >> > > >> > > >> > import numpy as np > >> > from numpy.lib.stride_tricks import as_strided > >> > from scipy.ndimage import convolve1d > >> > from scipy.weave import blitz > >> > > >> > # weighting coefficients > >> > a = -0.5 > >> > b = 1.0 > >> > c = -0.25 > >> > w = np.array((a,b,c)) > >> > # Reversed w: > >> > rw = w[::-1] > >> > > >> > # Length of C > >> > n = 250 > >> > > >> > # The original version of the calculation: > >> > # Some dummy data > >> > C = np.arange(float(n)) > >> > C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > >> > # Save for comparison. > >> > C0 = C.copy() > >> > > >> > # Do it again using a matrix multiplication. > >> > C = np.arange(float(n)) > >> > # The "virtual" matrix view of C. > >> > V = as_strided(C, shape=(C.size-2, 3), strides=(C.strides[0], > >> > C.strides[0])) > >> > C[1:-1] = np.dot(V, rw) > >> > C1 = C.copy() > >> > > >> > # Again, with convolve1d this time. > >> > C = np.arange(float(n)) > >> > C[1:-1] = convolve1d(C, w)[1:-1] > >> > C2 = C.copy() > >> > > >> > # scipy.weave.blitz > >> > C = np.arange(float(n)) > >> > # Must work with a copy, D, in the formula, because blitz does not use > >> > # a temporary variable. > >> > D = C.copy() > >> > expr = "C[1:-1] = a * D[2:] + b * D[1:-1] + c * D[:-2]" > >> > blitz(expr, check_size=0) > >> > C3 = C.copy() > >> > > >> > > >> > # Verify that all the methods give the same result. > >> > print np.all(C0 == C1) > >> > print np.all(C0 == C2) > >> > print np.all(C0 == C3) > >> > > >> > #----- > >> > > >> > And here's a snippet from an ipython session: > >> > > >> > In [51]: run convolve1dtest.py > >> > True > >> > True > >> > True > >> > > >> > In [52]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > >> > 100000 loops, best of 3: 16.5 us per loop > >> > > >> > In [53]: %timeit C[1:-1] = np.dot(V, rw) > >> > 100000 loops, best of 3: 9.84 us per loop > >> > > >> > In [54]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] > >> > 100000 loops, best of 3: 18.7 us per loop > >> > > >> > In [55]: %timeit D = C.copy(); blitz(expr, check_size=0) > >> > 100000 loops, best of 3: 4.91 us per loop > >> > > >> > > >> > > >> > scipy.weave.blitz is fastest (but note that blitz has already been > >> > called > >> > once, so the time shown does not include the compilation required in > >> > the first call). You could also try scipy.weave.inline, > cython.inline, > >> > or np_inline (http://pages.cs.wisc.edu/~johnl/np_inline/). > >> > > >> > Also note that C[-1:1] = np.dot(V, rw) is faster than either the > simple > >> > expression or convolve1d. However, if you also have to set up V > inside > >> > your inner loop, the speed gain will be lost. The relative speeds > also > >> > depend on the size of C. For large C, the simple expression is faster > >> > than the matrix multiplication by V (but blitz is still faster). In > >> > the following, I have changed n to 2500 before running > >> > convolve1dtest.py: > >> > > >> > In [56]: run convolve1dtest.py > >> > True > >> > True > >> > True > >> > > >> > In [57]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > >> > 10000 loops, best of 3: 29.5 us per loop > >> > > >> > In [58]: %timeit C[1:-1] = np.dot(V, rw) > >> > 10000 loops, best of 3: 56.4 us per loop > >> > > >> > In [59]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] > >> > 10000 loops, best of 3: 37.3 us per loop > >> > > >> > In [60]: %timeit D = C.copy(); blitz(expr, check_size=0) > >> > 100000 loops, best of 3: 10.3 us per loop > >> > > >> > > >> > blitz wins, the simple numpy expression is a distant second, and now > >> > the matrix multiplication is slowest. > >> > > >> > I hope that helps--I know I learned quite a bit. :) > >> > >> Interesting, two questions > >> > >> does scipy.signal convolve have a similar overhead as np.convolve1d ? > > > > > > Did you mean np.convolve? There is no np.convolve1d. Some of the tests > > that I've done with convolution functions are here: > > http://www.scipy.org/Cookbook/ApplyFIRFilter > > I should add np.convolve to that page. For the case considered here, > > np.convolve is a bit slower than scipy.ndimage.convolve1d, but for larger > > arrays, it looks like np.convolve can be much faster. > > > > > >> > >> memory: > >> the blitz code doesn't include the array copy (D), so the timing might > >> be a bit misleading? > > > > > > Look again at my %timeit calls in the ipython snippets. :) > > > > Sorry, I was reading way too fast or selectively (skipping the > imports, namespaces for example). > (I thought convolve1d was numpy not ndimage) > > I tried out your cookbook script a while ago, it's nice, I had only a > rough idea about the timing before. > Compared to the current case the x is much longer, (unless I don't > remember or read it correctly.) > I just updated the page with np.convolve: http://www.scipy.org/Cookbook/ApplyFIRFilter A big part of the overhead in using np.convolve in that script is that the function only accepts 1-d arrays, so a python loop is required to apply it along an axis of a 2-d array. Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From zyzhu2000 at gmail.com Sat Nov 12 23:07:44 2011 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Sat, 12 Nov 2011 22:07:44 -0600 Subject: [Numpy-discussion] speeding up the following expression In-Reply-To: References: Message-ID: Hi Warren, On Sat, Nov 12, 2011 at 9:31 AM, Warren Weckesser wrote: > > > On Sat, Nov 12, 2011 at 6:43 AM, wrote: >> >> On Sat, Nov 12, 2011 at 3:36 AM, Geoffrey Zhu wrote: >> > Hi, >> > >> > I am playing with multiple ways to speed up the following expression >> > (it is in the inner loop): >> > >> > >> > C[1:(M - 1)]=(a * C[2:] + b * C[1:(M-1)] + c * C[:(M-2)]) >> > >> > where C is an array of about 200-300 elements, M=len(C), a, b, c are >> > scalars. >> > >> > I played with numexpr, but it was way slower than directly using >> > numpy. It would be nice if I could create a Mx3 matrix without copying >> > memory and so I can use dot() to calculate the whole thing. >> > >> > Can anyone help with giving some advices to make this faster? >> >> looks like a np.convolve(C, [a,b,c]) ?to me except for the boundary >> conditions. > > > > As Josef pointed out, this is a convolution.? There are (at least) > three convolution functions in numpy+scipy that you could use: > numpy.convolve, scipy.signal.convolve, and scipy.ndimage.convolve1d. > Of these, scipy.ndimage.convolve1d is the fastest.? However, it doesn't > beat the simple expression > ?? a*x[2:] + b*x[1:-1] + c*x[:-2] > Your idea of forming a matrix without copying memory can be done using > "stride tricks", and for arrays of the size you are interested in, it > computes the result faster than the simple expression (see below). > > Another fast alternative is to use one of the inline code generators. > This example is a easy to implement with scipy.weave.blitz, and it gives > a big speedup. > > Here's a test script: > > #----- convolve1dtest.py ----- > > > import numpy as np > from numpy.lib.stride_tricks import as_strided > from scipy.ndimage import convolve1d > from scipy.weave import blitz > > # weighting coefficients > a = -0.5 > b = 1.0 > c = -0.25 > w = np.array((a,b,c)) > # Reversed w: > rw = w[::-1] > > # Length of C > n = 250 > > # The original version of the calculation: > # Some dummy data > C = np.arange(float(n)) > C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > # Save for comparison. > C0 = C.copy() > > # Do it again using a matrix multiplication. > C = np.arange(float(n)) > # The "virtual" matrix view of C. > V = as_strided(C, shape=(C.size-2, 3), strides=(C.strides[0], C.strides[0])) > C[1:-1] = np.dot(V, rw) > C1 = C.copy() > > # Again, with convolve1d this time. > C = np.arange(float(n)) > C[1:-1] = convolve1d(C, w)[1:-1] > C2 = C.copy() > > # scipy.weave.blitz > C = np.arange(float(n)) > # Must work with a copy, D, in the formula, because blitz does not use > # a temporary variable. > D = C.copy() > expr = "C[1:-1] = a * D[2:] + b * D[1:-1] + c * D[:-2]" > blitz(expr, check_size=0) > C3 = C.copy() > > > # Verify that all the methods give the same result. > print np.all(C0 == C1) > print np.all(C0 == C2) > print np.all(C0 == C3) > > #----- > > And here's a snippet from an ipython session: > > In [51]: run convolve1dtest.py > True > True > True > > In [52]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > 100000 loops, best of 3: 16.5 us per loop > > In [53]: %timeit C[1:-1] = np.dot(V, rw) > 100000 loops, best of 3: 9.84 us per loop > > In [54]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] > 100000 loops, best of 3: 18.7 us per loop > > In [55]: %timeit D = C.copy(); blitz(expr, check_size=0) > 100000 loops, best of 3: 4.91 us per loop > > > > scipy.weave.blitz is fastest (but note that blitz has already been called > once, so the time shown does not include the compilation required in > the first call).? You could also try scipy.weave.inline, cython.inline, > or np_inline (http://pages.cs.wisc.edu/~johnl/np_inline/). > > Also note that C[-1:1] = np.dot(V, rw) is faster than either the simple > expression or convolve1d.? However, if you also have to set up V inside > your inner loop, the speed gain will be lost.? The relative speeds also > depend on the size of C.? For large C, the simple expression is faster > than the matrix multiplication by V (but blitz is still faster).? In > the following, I have changed n to 2500 before running convolve1dtest.py: > > In [56]: run convolve1dtest.py > True > True > True > > In [57]: %timeit C[1:-1] = a * C[2:] + b * C[1:-1] + c * C[:-2] > 10000 loops, best of 3: 29.5 us per loop > > In [58]: %timeit C[1:-1] = np.dot(V, rw) > 10000 loops, best of 3: 56.4 us per loop > > In [59]: %timeit C[1:-1] = convolve1d(C, w)[1:-1] > 10000 loops, best of 3: 37.3 us per loop > > In [60]: %timeit D = C.copy(); blitz(expr, check_size=0) > 100000 loops, best of 3: 10.3 us per loop > > > blitz wins, the simple numpy expression is a distant second, and now > the matrix multiplication is slowest. > > I hope that helps--I know I learned quite a bit. :) > > > Warren Thanks for the very helpful response. Fortunately I don't have to set up the matrix every time as I can work on the same C over and over again inside the loop. It's counterintuitive to see that the performance of dot() degrades as n goes up. If I didn't see the results, I would have guessed otherwise. Thanks again! Geoffrey From matthew.brett at gmail.com Sun Nov 13 02:35:53 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 12 Nov 2011 23:35:53 -0800 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit Message-ID: Hi, Sorry for my continued confusion here. This is numpy 1.6.1 on windows XP 32 bit. In [2]: np.finfo(np.float96).nmant Out[2]: 52 In [3]: np.finfo(np.float96).nexp Out[3]: 15 In [4]: np.finfo(np.float64).nmant Out[4]: 52 In [5]: np.finfo(np.float64).nexp Out[5]: 11 If there are 52 bits of precision, 2**53+1 should not be representable, and sure enough: In [6]: np.float96(2**53)+1 Out[6]: 9007199254740992.0 In [7]: np.float64(2**53)+1 Out[7]: 9007199254740992.0 If the nexp is right, the max should be higher for the float96 type: In [9]: np.finfo(np.float64).max Out[9]: 1.7976931348623157e+308 In [10]: np.finfo(np.float96).max Out[10]: 1.#INF I see that long double in C is 12 bytes wide, and double is the usual 8 bytes. So - now I am not sure what this float96 is. I was expecting 80 bit extended precision, but it doesn't look right for that... Does anyone know what representation this is? Thanks a lot, Matthew From matthew.brett at gmail.com Sun Nov 13 02:57:12 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 12 Nov 2011 23:57:12 -0800 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: Hi, On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett wrote: > Hi, > > Sorry for my continued confusion here. ?This is numpy 1.6.1 on windows > XP 32 bit. > > In [2]: np.finfo(np.float96).nmant > Out[2]: 52 > > In [3]: np.finfo(np.float96).nexp > Out[3]: 15 > > In [4]: np.finfo(np.float64).nmant > Out[4]: 52 > > In [5]: np.finfo(np.float64).nexp > Out[5]: 11 > > If there are 52 bits of precision, 2**53+1 should not be > representable, and sure enough: > > In [6]: np.float96(2**53)+1 > Out[6]: 9007199254740992.0 > > In [7]: np.float64(2**53)+1 > Out[7]: 9007199254740992.0 > > If the nexp is right, the max should be higher for the float96 type: > > In [9]: np.finfo(np.float64).max > Out[9]: 1.7976931348623157e+308 > > In [10]: np.finfo(np.float96).max > Out[10]: 1.#INF > > I see that long double in C is 12 bytes wide, and double is the usual 8 bytes. Sorry - sizeof(long double) is 12 using mingw. I see that long double is the same as double in MS Visual C++. http://en.wikipedia.org/wiki/Long_double but, as expected from the name: In [11]: np.dtype(np.float96).itemsize Out[11]: 12 Cheers, Matthew From dfnsonfsduifb at gmx.de Sun Nov 13 11:04:07 2011 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Sun, 13 Nov 2011 17:04:07 +0100 Subject: [Numpy-discussion] Rebinning numpy array Message-ID: <4EBFEA77.6020102@gmx.de> Hi group, I have a rather simple problem, or so it would seem. However I cannot seem to find the right solution. Here's the problem: A Geiger counter measures counts in distinct time intervals. The time intervals are not of constant length. Imaging for example that the counter would always create a table entry when the counts reach 10. Then we would have the following bins (made-up data for illustration): Seconds Counts Len CPS 0 - 44 10 44 0.23 44 - 120 10 76 0.13 120 - 140 10 20 0.5 140 - 200 10 60 0.16 So we have n bins (in this example 4), but they're not equidistant. I want to rebin samples to make them equidistant. For example, I would like to rebin into 5 bins of 40 seconds time each. Then the rebinned example (I calculate by hand so this might contain errors): 0-40 9.09 40-80 5.65 80-120 5.26 120-160 13.33 160-200 6.66 That means, if a destination bin completely overlaps a source bin, its complete value is taken. If it overlaps partially, linear interpolation of bin sizes should be used. It is very important that the overall count amount stays the same (in this case 40, so my numbers seem to be correct, I checked that). In this example I increased the bin size, but usually I will want to decrease bin size (even dramatically). Now my pathetic attempts look something like this: interpolation_points = 4000 xpts = [ time.mktime(x.timetuple()) for x in self.getx() ] interpolatedx = numpy.linspace(xpts[0], xpts[-1], interpolation_points) interpolatedy = numpy.interp(interpolatedx, xpts, self.gety()) self._xreformatted = [ datetime.datetime.fromtimestamp(x) for x in interpolatedx ] self._yreformatted = interpolatedy This works somewhat, however I see artifacts depending on the destination sample size: for example when I have a spike in the sample input and reduce the number of interpolation points (i.e. increase destination bin size) slowly, the spike will get smaller and smaller (expected behaviour). After some amount of increasing, the spike however will "magically" reappear. I believe this to be an interpolation artifact. Is there some standard way to get from a non-uniformally distributed bin distribution to a unifomally distributed bin distribution of arbitrary bin width? Best regards, Joe From charlesr.harris at gmail.com Sun Nov 13 11:21:59 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Nov 2011 09:21:59 -0700 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett wrote: > Hi, > > On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett > wrote: > > Hi, > > > > Sorry for my continued confusion here. This is numpy 1.6.1 on windows > > XP 32 bit. > > > > In [2]: np.finfo(np.float96).nmant > > Out[2]: 52 > > > > In [3]: np.finfo(np.float96).nexp > > Out[3]: 15 > > > > In [4]: np.finfo(np.float64).nmant > > Out[4]: 52 > > > > In [5]: np.finfo(np.float64).nexp > > Out[5]: 11 > > > > If there are 52 bits of precision, 2**53+1 should not be > > representable, and sure enough: > > > > In [6]: np.float96(2**53)+1 > > Out[6]: 9007199254740992.0 > > > > In [7]: np.float64(2**53)+1 > > Out[7]: 9007199254740992.0 > > > > If the nexp is right, the max should be higher for the float96 type: > > > > In [9]: np.finfo(np.float64).max > > Out[9]: 1.7976931348623157e+308 > > > > In [10]: np.finfo(np.float96).max > > Out[10]: 1.#INF > > > > I see that long double in C is 12 bytes wide, and double is the usual 8 > bytes. > > Sorry - sizeof(long double) is 12 using mingw. I see that long double > is the same as double in MS Visual C++. > > http://en.wikipedia.org/wiki/Long_double > > but, as expected from the name: > > In [11]: np.dtype(np.float96).itemsize > Out[11]: 12 > > Hmm, good point. There should not be a float96 on Windows using the MSVC compiler, and the longdouble types 'gG' should return float64 and complex128 respectively. OTOH, I believe the mingw compiler has real float96 types but I wonder about library support. This is really a build issue and it would be good to have some feedback on what different platforms are doing so that we know if we are doing things right. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Nov 13 12:27:18 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 13 Nov 2011 17:27:18 +0000 Subject: [Numpy-discussion] Rebinning numpy array In-Reply-To: <4EBFEA77.6020102@gmx.de> References: <4EBFEA77.6020102@gmx.de> Message-ID: On Sun, Nov 13, 2011 at 16:04, Johannes Bauer wrote: > Hi group, > > I have a rather simple problem, or so it would seem. However I cannot > seem to find the right solution. Here's the problem: > > A Geiger counter measures counts in distinct time intervals. The time > intervals are not of constant length. Imaging for example that the > counter would always create a table entry when the counts reach 10. Then > we would have the following bins (made-up data for illustration): > > Seconds ? ? ? ? Counts ?Len ? ? CPS > 0 - 44 ? ? ? ? ?10 ? ? ?44 ? ? ?0.23 > 44 - 120 ? ? ? ?10 ? ? ?76 ? ? ?0.13 > 120 - 140 ? ? ? 10 ? ? ?20 ? ? ?0.5 > 140 - 200 ? ? ? 10 ? ? ?60 ? ? ?0.16 > > So we have n bins (in this example 4), but they're not equidistant. I > want to rebin samples to make them equidistant. For example, I would > like to rebin into 5 bins of 40 seconds time each. Then the rebinned > example (I calculate by hand so this might contain errors): > > 0-40 ? ? ? ? ? ?9.09 > 40-80 ? ? ? ? ? 5.65 > 80-120 ? ? ? ? ?5.26 > 120-160 ? ? ? ? 13.33 > 160-200 ? ? ? ? 6.66 > > That means, if a destination bin completely overlaps a source bin, its > complete value is taken. If it overlaps partially, linear interpolation > of bin sizes should be used. What you want to do is set up a linear interpolation based on the boundaries of the uneven bins. Seconds Value 0 0 44 10 120 20 140 30 200 40 Then evaluate that linear interpolation on the boundaries of the uniform bins. [~] |18> bin_bounds = np.array([0.0, 44.0, 120, 140, 200]) [~] |19> bin_values = np.array([0.0, 10, 10, 10, 10]) [~] |20> cum_bin_values = bin_values.cumsum() [~] |21> new_bounds = np.array([0.0, 40, 80, 120, 160, 200]) [~] |22> ecdf = np.interp(new_bounds, bin_bounds, cum_bin_values) [~] |23> ecdf array([ 0. , 9.09090909, 14.73684211, 20. , 33.33333333, 40. ]) [~] |24> uniform_histogram = np.diff(ecdf) [~] |25> uniform_histogram array([ 9.09090909, 5.64593301, 5.26315789, 13.33333333, 6.66666667]) This may be what you are doing already. I'm not sure what is in your getx() and gety() methods. If so, then I think you are on the right track. If you still have problems, then we might need to see some of the problematic data and results. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From shish at keba.be Sun Nov 13 12:40:17 2011 From: shish at keba.be (Olivier Delalleau) Date: Sun, 13 Nov 2011 12:40:17 -0500 Subject: [Numpy-discussion] Rebinning numpy array In-Reply-To: <4EBFEA77.6020102@gmx.de> References: <4EBFEA77.6020102@gmx.de> Message-ID: Just one thing: numpy.interp says it doesn't check that the x coordinates are increasing, so make sure it's the case. Assuming this is ok, I could still see how you may get some non-smooth behavior: this may be because your spike can either be split between two bins (which "dilutes" it somehow), or be included in a single bin (which would make it stand out more). And as you increase your bin size, you will switch between these two situations. -=- Olivier 2011/11/13 Johannes Bauer > Hi group, > > I have a rather simple problem, or so it would seem. However I cannot > seem to find the right solution. Here's the problem: > > A Geiger counter measures counts in distinct time intervals. The time > intervals are not of constant length. Imaging for example that the > counter would always create a table entry when the counts reach 10. Then > we would have the following bins (made-up data for illustration): > > Seconds Counts Len CPS > 0 - 44 10 44 0.23 > 44 - 120 10 76 0.13 > 120 - 140 10 20 0.5 > 140 - 200 10 60 0.16 > > So we have n bins (in this example 4), but they're not equidistant. I > want to rebin samples to make them equidistant. For example, I would > like to rebin into 5 bins of 40 seconds time each. Then the rebinned > example (I calculate by hand so this might contain errors): > > 0-40 9.09 > 40-80 5.65 > 80-120 5.26 > 120-160 13.33 > 160-200 6.66 > > That means, if a destination bin completely overlaps a source bin, its > complete value is taken. If it overlaps partially, linear interpolation > of bin sizes should be used. > > It is very important that the overall count amount stays the same (in > this case 40, so my numbers seem to be correct, I checked that). In this > example I increased the bin size, but usually I will want to decrease > bin size (even dramatically). > > Now my pathetic attempts look something like this: > > interpolation_points = 4000 > xpts = [ time.mktime(x.timetuple()) for x in self.getx() ] > > interpolatedx = numpy.linspace(xpts[0], xpts[-1], interpolation_points) > interpolatedy = numpy.interp(interpolatedx, xpts, self.gety()) > > self._xreformatted = [ datetime.datetime.fromtimestamp(x) for x in > interpolatedx ] > self._yreformatted = interpolatedy > > This works somewhat, however I see artifacts depending on the > destination sample size: for example when I have a spike in the sample > input and reduce the number of interpolation points (i.e. increase > destination bin size) slowly, the spike will get smaller and smaller > (expected behaviour). After some amount of increasing, the spike however > will "magically" reappear. I believe this to be an interpolation artifact. > > Is there some standard way to get from a non-uniformally distributed bin > distribution to a unifomally distributed bin distribution of arbitrary > bin width? > > Best regards, > Joe > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Sun Nov 13 12:48:10 2011 From: shish at keba.be (Olivier Delalleau) Date: Sun, 13 Nov 2011 12:48:10 -0500 Subject: [Numpy-discussion] Rebinning numpy array In-Reply-To: References: <4EBFEA77.6020102@gmx.de> Message-ID: Also: it seems like you are using values at the boundaries of the bins, while I think it would make more sense to compute interpolated values at the middle point of a bin. I'm not sure it'll make a big difference visually, but it may be more appropriate. -=- Olivier 2011/11/13 Olivier Delalleau > Just one thing: numpy.interp says it doesn't check that the x coordinates > are increasing, so make sure it's the case. > > Assuming this is ok, I could still see how you may get some non-smooth > behavior: this may be because your spike can either be split between two > bins (which "dilutes" it somehow), or be included in a single bin (which > would make it stand out more). And as you increase your bin size, you will > switch between these two situations. > > -=- Olivier > > > 2011/11/13 Johannes Bauer > >> Hi group, >> >> I have a rather simple problem, or so it would seem. However I cannot >> seem to find the right solution. Here's the problem: >> >> A Geiger counter measures counts in distinct time intervals. The time >> intervals are not of constant length. Imaging for example that the >> counter would always create a table entry when the counts reach 10. Then >> we would have the following bins (made-up data for illustration): >> >> Seconds Counts Len CPS >> 0 - 44 10 44 0.23 >> 44 - 120 10 76 0.13 >> 120 - 140 10 20 0.5 >> 140 - 200 10 60 0.16 >> >> So we have n bins (in this example 4), but they're not equidistant. I >> want to rebin samples to make them equidistant. For example, I would >> like to rebin into 5 bins of 40 seconds time each. Then the rebinned >> example (I calculate by hand so this might contain errors): >> >> 0-40 9.09 >> 40-80 5.65 >> 80-120 5.26 >> 120-160 13.33 >> 160-200 6.66 >> >> That means, if a destination bin completely overlaps a source bin, its >> complete value is taken. If it overlaps partially, linear interpolation >> of bin sizes should be used. >> >> It is very important that the overall count amount stays the same (in >> this case 40, so my numbers seem to be correct, I checked that). In this >> example I increased the bin size, but usually I will want to decrease >> bin size (even dramatically). >> >> Now my pathetic attempts look something like this: >> >> interpolation_points = 4000 >> xpts = [ time.mktime(x.timetuple()) for x in self.getx() ] >> >> interpolatedx = numpy.linspace(xpts[0], xpts[-1], interpolation_points) >> interpolatedy = numpy.interp(interpolatedx, xpts, self.gety()) >> >> self._xreformatted = [ datetime.datetime.fromtimestamp(x) for x in >> interpolatedx ] >> self._yreformatted = interpolatedy >> >> This works somewhat, however I see artifacts depending on the >> destination sample size: for example when I have a spike in the sample >> input and reduce the number of interpolation points (i.e. increase >> destination bin size) slowly, the spike will get smaller and smaller >> (expected behaviour). After some amount of increasing, the spike however >> will "magically" reappear. I believe this to be an interpolation artifact. >> >> Is there some standard way to get from a non-uniformally distributed bin >> distribution to a unifomally distributed bin distribution of arbitrary >> bin width? >> >> Best regards, >> Joe >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Sun Nov 13 12:55:12 2011 From: shish at keba.be (Olivier Delalleau) Date: Sun, 13 Nov 2011 12:55:12 -0500 Subject: [Numpy-discussion] Rebinning numpy array In-Reply-To: References: <4EBFEA77.6020102@gmx.de> Message-ID: (Sorry for the spam, I should have given more thought to this before hitting reply). It actually seems to me that using a linear interpolation is not a good idea, since it will throw out a lot of information if you decrease the number of bins: to compute the value at time t, it will only use the closest bins (t_k and t_{k+1} such that t_k < t < t_{k+1}), so that data stored in many of the bins will not be used at all. I haven't looked closely at the suggestion from Robert but it may be a better way to achieve what you want. -=- Olivier 2011/11/13 Olivier Delalleau > Also: it seems like you are using values at the boundaries of the bins, > while I think it would make more sense to compute interpolated values at > the middle point of a bin. I'm not sure it'll make a big difference > visually, but it may be more appropriate. > > -=- Olivier > > > 2011/11/13 Olivier Delalleau > >> Just one thing: numpy.interp says it doesn't check that the x coordinates >> are increasing, so make sure it's the case. >> >> Assuming this is ok, I could still see how you may get some non-smooth >> behavior: this may be because your spike can either be split between two >> bins (which "dilutes" it somehow), or be included in a single bin (which >> would make it stand out more). And as you increase your bin size, you will >> switch between these two situations. >> >> -=- Olivier >> >> >> 2011/11/13 Johannes Bauer >> >>> Hi group, >>> >>> I have a rather simple problem, or so it would seem. However I cannot >>> seem to find the right solution. Here's the problem: >>> >>> A Geiger counter measures counts in distinct time intervals. The time >>> intervals are not of constant length. Imaging for example that the >>> counter would always create a table entry when the counts reach 10. Then >>> we would have the following bins (made-up data for illustration): >>> >>> Seconds Counts Len CPS >>> 0 - 44 10 44 0.23 >>> 44 - 120 10 76 0.13 >>> 120 - 140 10 20 0.5 >>> 140 - 200 10 60 0.16 >>> >>> So we have n bins (in this example 4), but they're not equidistant. I >>> want to rebin samples to make them equidistant. For example, I would >>> like to rebin into 5 bins of 40 seconds time each. Then the rebinned >>> example (I calculate by hand so this might contain errors): >>> >>> 0-40 9.09 >>> 40-80 5.65 >>> 80-120 5.26 >>> 120-160 13.33 >>> 160-200 6.66 >>> >>> That means, if a destination bin completely overlaps a source bin, its >>> complete value is taken. If it overlaps partially, linear interpolation >>> of bin sizes should be used. >>> >>> It is very important that the overall count amount stays the same (in >>> this case 40, so my numbers seem to be correct, I checked that). In this >>> example I increased the bin size, but usually I will want to decrease >>> bin size (even dramatically). >>> >>> Now my pathetic attempts look something like this: >>> >>> interpolation_points = 4000 >>> xpts = [ time.mktime(x.timetuple()) for x in self.getx() ] >>> >>> interpolatedx = numpy.linspace(xpts[0], xpts[-1], interpolation_points) >>> interpolatedy = numpy.interp(interpolatedx, xpts, self.gety()) >>> >>> self._xreformatted = [ datetime.datetime.fromtimestamp(x) for x in >>> interpolatedx ] >>> self._yreformatted = interpolatedy >>> >>> This works somewhat, however I see artifacts depending on the >>> destination sample size: for example when I have a spike in the sample >>> input and reduce the number of interpolation points (i.e. increase >>> destination bin size) slowly, the spike will get smaller and smaller >>> (expected behaviour). After some amount of increasing, the spike however >>> will "magically" reappear. I believe this to be an interpolation >>> artifact. >>> >>> Is there some standard way to get from a non-uniformally distributed bin >>> distribution to a unifomally distributed bin distribution of arbitrary >>> bin width? >>> >>> Best regards, >>> Joe >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Nov 13 12:55:21 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 13 Nov 2011 17:55:21 +0000 Subject: [Numpy-discussion] Rebinning numpy array In-Reply-To: References: <4EBFEA77.6020102@gmx.de> Message-ID: On Sun, Nov 13, 2011 at 17:48, Olivier Delalleau wrote: > Also: it seems like you are using values at the boundaries of the bins, > while I think it would make more sense to compute interpolated values at the > middle point of a bin. I'm not sure it'll make a big difference visually, > but it may be more appropriate. No, you do want to compute the interpolated values at the boundaries of the new bins. Then differencing the values at the boundaries will give you the correct values for the "mass" between the bounds. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From shish at keba.be Sun Nov 13 13:03:55 2011 From: shish at keba.be (Olivier Delalleau) Date: Sun, 13 Nov 2011 13:03:55 -0500 Subject: [Numpy-discussion] Rebinning numpy array In-Reply-To: References: <4EBFEA77.6020102@gmx.de> Message-ID: 2011/11/13 Robert Kern > On Sun, Nov 13, 2011 at 17:48, Olivier Delalleau wrote: > > Also: it seems like you are using values at the boundaries of the bins, > > while I think it would make more sense to compute interpolated values at > the > > middle point of a bin. I'm not sure it'll make a big difference visually, > > but it may be more appropriate. > > No, you do want to compute the interpolated values at the boundaries > of the new bins. Then differencing the values at the boundaries will > give you the correct values for the "mass" between the bounds. > I wrote this with non cumulative data in mind. However I just looked at your suggestion, which is to accumulate data, and I agree it seems a better way to achieve what the OP is trying to do, and in that case I agree that computing interpolated values at the boundaries is the right way to go. Sorry for the confusion, -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Nov 13 14:19:48 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 13 Nov 2011 20:19:48 +0100 Subject: [Numpy-discussion] ANN: SciPy 0.10.0 released Message-ID: Hi all, I am pleased to announce the availability of SciPy 0.10.0. For this release over a 100 tickets and pull requests have been closed, and many new features have been added. Some of the highlights are: - support for Bento as a build system for scipy - generalized and shift-invert eigenvalue problems in sparse.linalg - addition of discrete-time linear systems in the signal module Sources and binaries can be found at , release notes are copied below. Enjoy, The SciPy developers ========================== SciPy 0.10.0 Release Notes ========================== .. contents:: SciPy 0.10.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a limited number of deprecations and backwards-incompatible changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.10.x branch, and on adding new features on the development master branch. Release highlights: - Support for Bento as optional build system. - Support for generalized eigenvalue problems, and all shift-invert modes available in ARPACK. This release requires Python 2.4-2.7 or 3.1- and NumPy 1.5 or greater. New features ============ Bento: new optional build system -------------------------------- Scipy can now be built with `Bento `_. Bento has some nice features like parallel builds and partial rebuilds, that are not possible with the default build system (distutils). For usage instructions see BENTO_BUILD.txt in the scipy top-level directory. Currently Scipy has three build systems, distutils, numscons and bento. Numscons is deprecated and is planned and will likely be removed in the next release. Generalized and shift-invert eigenvalue problems in ``scipy.sparse.linalg`` --------------------------------------------------------------------------- The sparse eigenvalue problem solver functions ``scipy.sparse.eigs/eigh`` now support generalized eigenvalue problems, and all shift-invert modes available in ARPACK. Discrete-Time Linear Systems (``scipy.signal``) ----------------------------------------------- Support for simulating discrete-time linear systems, including ``scipy.signal.dlsim``, ``scipy.signal.dimpulse``, and ``scipy.signal.dstep``, has been added to SciPy. Conversion of linear systems from continuous-time to discrete-time representations is also present via the ``scipy.signal.cont2discrete`` function. Enhancements to ``scipy.signal`` -------------------------------- A Lomb-Scargle periodogram can now be computed with the new function ``scipy.signal.lombscargle``. The forward-backward filter function ``scipy.signal.filtfilt`` can now filter the data in a given axis of an n-dimensional numpy array. (Previously it only handled a 1-dimensional array.) Options have been added to allow more control over how the data is extended before filtering. FIR filter design with ``scipy.signal.firwin2`` now has options to create filters of type III (zero at zero and Nyquist frequencies) and IV (zero at zero frequency). Additional decomposition options (``scipy.linalg``) --------------------------------------------------- A sort keyword has been added to the Schur decomposition routine (``scipy.linalg.schur``) to allow the sorting of eigenvalues in the resultant Schur form. Additional special matrices (``scipy.linalg``) ---------------------------------------------- The functions ``hilbert`` and ``invhilbert`` were added to ``scipy.linalg``. Enhancements to ``scipy.stats`` ------------------------------- * The *one-sided form* of Fisher's exact test is now also implemented in ``stats.fisher_exact``. * The function ``stats.chi2_contingency`` for computing the chi-square test of independence of factors in a contingency table has been added, along with the related utility functions ``stats.contingency.margins`` and ``stats.contingency.expected_freq``. Basic support for Harwell-Boeing file format for sparse matrices ---------------------------------------------------------------- Both read and write are support through a simple function-based API, as well as a more complete API to control number format. The functions may be found in scipy.sparse.io. The following features are supported: * Read and write sparse matrices in the CSC format * Only real, symmetric, assembled matrix are supported (RUA format) Deprecated features =================== ``scipy.maxentropy`` -------------------- The maxentropy module is unmaintained, rarely used and has not been functioning well for several releases. Therefore it has been deprecated for this release, and will be removed for scipy 0.11. Logistic regression in scikits.learn is a good alternative for this functionality. The ``scipy.maxentropy.logsumexp`` function has been moved to ``scipy.misc``. ``scipy.lib.blas`` ------------------ There are similar BLAS wrappers in ``scipy.linalg`` and ``scipy.lib``. These have now been consolidated as ``scipy.linalg.blas``, and ``scipy.lib.blas`` is deprecated. Numscons build system --------------------- The numscons build system is being replaced by Bento, and will be removed in one of the next scipy releases. Backwards-incompatible changes ============================== The deprecated name `invnorm` was removed from ``scipy.stats.distributions``, this distribution is available as `invgauss`. The following deprecated nonlinear solvers from ``scipy.optimize`` have been removed:: - ``broyden_modified`` (bad performance) - ``broyden1_modified`` (bad performance) - ``broyden_generalized`` (equivalent to ``anderson``) - ``anderson2`` (equivalent to ``anderson``) - ``broyden3`` (obsoleted by new limited-memory broyden methods) - ``vackar`` (renamed to ``diagbroyden``) Other changes ============= ``scipy.constants`` has been updated with the CODATA 2010 constants. ``__all__`` dicts have been added to all modules, which has cleaned up the namespaces (particularly useful for interactive work). An API section has been added to the documentation, giving recommended import guidelines and specifying which submodules are public and which aren't. Authors ======= This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order): * Jeff Armstrong + * Matthew Brett * Lars Buitinck + * David Cournapeau * FI$H 2000 + * Michael McNeil Forbes + * Matty G + * Christoph Gohlke * Ralf Gommers * Yaroslav Halchenko * Charles Harris * Thouis (Ray) Jones + * Chris Jordan-Squire + * Robert Kern * Chris Lasher + * Wes McKinney + * Travis Oliphant * Fabian Pedregosa * Josef Perktold * Thomas Robitaille + * Pim Schellart + * Anthony Scopatz + * Skipper Seabold + * Fazlul Shahriar + * David Simcha + * Scott Sinclair + * Andrey Smirnov + * Collin RM Stocks + * Martin Teichmann + * Jake Vanderplas + * Ga?l Varoquaux + * Pauli Virtanen * Stefan van der Walt * Warren Weckesser * Mark Wiebe + A total of 35 people contributed to this release. People with a "+" by their names contributed a patch for the first time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Nov 13 14:32:10 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Nov 2011 12:32:10 -0700 Subject: [Numpy-discussion] ANN: SciPy 0.10.0 released In-Reply-To: References: Message-ID: On Sun, Nov 13, 2011 at 12:19 PM, Ralf Gommers wrote: > Hi all, > > I am pleased to announce the availability of SciPy 0.10.0. For this > release over a 100 tickets and pull requests have been closed, and many new > features have been added. Some of the highlights are: > > - support for Bento as a build system for scipy > - generalized and shift-invert eigenvalue problems in sparse.linalg > - addition of discrete-time linear systems in the signal module > > Sources and binaries can be found at , release notes are copied below. > > Enjoy, > The SciPy developers > > > > ========================== > SciPy 0.10.0 Release Notes > ========================== > > .. contents:: > > SciPy 0.10.0 is the culmination of 8 months of hard work. It contains > many new features, numerous bug-fixes, improved test coverage and > better documentation. There have been a limited number of deprecations > and backwards-incompatible changes in this release, which are documented > below. All users are encouraged to upgrade to this release, as there > are a large number of bug-fixes and optimizations. Moreover, our > development attention will now shift to bug-fix releases on the 0.10.x > branch, and on adding new features on the development master branch. > > Release highlights: > > - Support for Bento as optional build system. > - Support for generalized eigenvalue problems, and all shift-invert modes > available in ARPACK. > > This release requires Python 2.4-2.7 or 3.1- and NumPy 1.5 or greater. > > > New features > ============ > > Bento: new optional build system > -------------------------------- > > Scipy can now be built with `Bento `_. > Bento has some nice features like parallel builds and partial rebuilds, > that > are not possible with the default build system (distutils). For usage > instructions see BENTO_BUILD.txt in the scipy top-level directory. > > Currently Scipy has three build systems, distutils, numscons and bento. > Numscons is deprecated and is planned and will likely be removed in the > next > release. > > > Generalized and shift-invert eigenvalue problems in ``scipy.sparse.linalg`` > --------------------------------------------------------------------------- > > The sparse eigenvalue problem solver functions > ``scipy.sparse.eigs/eigh`` now support generalized eigenvalue > problems, and all shift-invert modes available in ARPACK. > > > Discrete-Time Linear Systems (``scipy.signal``) > ----------------------------------------------- > > Support for simulating discrete-time linear systems, including > ``scipy.signal.dlsim``, ``scipy.signal.dimpulse``, and > ``scipy.signal.dstep``, > has been added to SciPy. Conversion of linear systems from > continuous-time to > discrete-time representations is also present via the > ``scipy.signal.cont2discrete`` function. > > > Enhancements to ``scipy.signal`` > -------------------------------- > > A Lomb-Scargle periodogram can now be computed with the new function > ``scipy.signal.lombscargle``. > > The forward-backward filter function ``scipy.signal.filtfilt`` can now > filter the data in a given axis of an n-dimensional numpy array. > (Previously it only handled a 1-dimensional array.) Options have been > added to allow more control over how the data is extended before filtering. > > FIR filter design with ``scipy.signal.firwin2`` now has options to create > filters of type III (zero at zero and Nyquist frequencies) and IV (zero at > zero > frequency). > > > Additional decomposition options (``scipy.linalg``) > --------------------------------------------------- > > A sort keyword has been added to the Schur decomposition routine > (``scipy.linalg.schur``) to allow the sorting of eigenvalues in > the resultant Schur form. > > Additional special matrices (``scipy.linalg``) > ---------------------------------------------- > > The functions ``hilbert`` and ``invhilbert`` were added to > ``scipy.linalg``. > > > Enhancements to ``scipy.stats`` > ------------------------------- > > * The *one-sided form* of Fisher's exact test is now also implemented in > ``stats.fisher_exact``. > * The function ``stats.chi2_contingency`` for computing the chi-square > test of > independence of factors in a contingency table has been added, along with > the related utility functions ``stats.contingency.margins`` and > ``stats.contingency.expected_freq``. > > > Basic support for Harwell-Boeing file format for sparse matrices > ---------------------------------------------------------------- > > Both read and write are support through a simple function-based API, as > well as > a more complete API to control number format. The functions may be found in > scipy.sparse.io. > > The following features are supported: > > * Read and write sparse matrices in the CSC format > * Only real, symmetric, assembled matrix are supported (RUA format) > > > Deprecated features > =================== > > ``scipy.maxentropy`` > -------------------- > > The maxentropy module is unmaintained, rarely used and has not been > functioning > well for several releases. Therefore it has been deprecated for this > release, > and will be removed for scipy 0.11. Logistic regression in scikits.learn > is a > good alternative for this functionality. The > ``scipy.maxentropy.logsumexp`` > function has been moved to ``scipy.misc``. > > > ``scipy.lib.blas`` > ------------------ > > There are similar BLAS wrappers in ``scipy.linalg`` and ``scipy.lib``. > These > have now been consolidated as ``scipy.linalg.blas``, and > ``scipy.lib.blas`` is > deprecated. > > > Numscons build system > --------------------- > > The numscons build system is being replaced by Bento, and will be removed > in > one of the next scipy releases. > > > Backwards-incompatible changes > ============================== > > The deprecated name `invnorm` was removed from > ``scipy.stats.distributions``, > this distribution is available as `invgauss`. > > The following deprecated nonlinear solvers from ``scipy.optimize`` have > been > removed:: > > - ``broyden_modified`` (bad performance) > - ``broyden1_modified`` (bad performance) > - ``broyden_generalized`` (equivalent to ``anderson``) > - ``anderson2`` (equivalent to ``anderson``) > - ``broyden3`` (obsoleted by new limited-memory broyden methods) > - ``vackar`` (renamed to ``diagbroyden``) > > > Other changes > ============= > > ``scipy.constants`` has been updated with the CODATA 2010 constants. > > ``__all__`` dicts have been added to all modules, which has cleaned up the > namespaces (particularly useful for interactive work). > > An API section has been added to the documentation, giving recommended > import > guidelines and specifying which submodules are public and which aren't. > > > Authors > ======= > > This release contains work by the following people (contributed at least > one patch to this release, names in alphabetical order): > > * Jeff Armstrong + > * Matthew Brett > * Lars Buitinck + > * David Cournapeau > * FI$H 2000 + > * Michael McNeil Forbes + > * Matty G + > * Christoph Gohlke > * Ralf Gommers > * Yaroslav Halchenko > * Charles Harris > * Thouis (Ray) Jones + > * Chris Jordan-Squire + > * Robert Kern > * Chris Lasher + > * Wes McKinney + > * Travis Oliphant > * Fabian Pedregosa > * Josef Perktold > * Thomas Robitaille + > * Pim Schellart + > * Anthony Scopatz + > * Skipper Seabold + > * Fazlul Shahriar + > * David Simcha + > * Scott Sinclair + > * Andrey Smirnov + > * Collin RM Stocks + > * Martin Teichmann + > * Jake Vanderplas + > * Ga?l Varoquaux + > * Pauli Virtanen > * Stefan van der Walt > * Warren Weckesser > * Mark Wiebe + > > A total of 35 people contributed to this release. > People with a "+" by their names contributed a patch for the first time. > > > Congratulations to all. This looks like a nice release. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sun Nov 13 14:33:18 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Nov 2011 20:33:18 +0100 Subject: [Numpy-discussion] ANN: SciPy 0.10.0 released In-Reply-To: References: Message-ID: <20111113193318.GC14269@phare.normalesup.org> On Sun, Nov 13, 2011 at 12:32:10PM -0700, Charles R Harris wrote: > Congratulations to all. This looks like a nice release. And congratulations to Ralf for moving this forward! G From matthew.brett at gmail.com Sun Nov 13 16:25:11 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 13 Nov 2011 13:25:11 -0800 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: Hi, On Sun, Nov 13, 2011 at 8:21 AM, Charles R Harris wrote: > > > On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett > wrote: >> >> Hi, >> >> On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett >> wrote: >> > Hi, >> > >> > Sorry for my continued confusion here. ?This is numpy 1.6.1 on windows >> > XP 32 bit. >> > >> > In [2]: np.finfo(np.float96).nmant >> > Out[2]: 52 >> > >> > In [3]: np.finfo(np.float96).nexp >> > Out[3]: 15 >> > >> > In [4]: np.finfo(np.float64).nmant >> > Out[4]: 52 >> > >> > In [5]: np.finfo(np.float64).nexp >> > Out[5]: 11 >> > >> > If there are 52 bits of precision, 2**53+1 should not be >> > representable, and sure enough: >> > >> > In [6]: np.float96(2**53)+1 >> > Out[6]: 9007199254740992.0 >> > >> > In [7]: np.float64(2**53)+1 >> > Out[7]: 9007199254740992.0 >> > >> > If the nexp is right, the max should be higher for the float96 type: >> > >> > In [9]: np.finfo(np.float64).max >> > Out[9]: 1.7976931348623157e+308 >> > >> > In [10]: np.finfo(np.float96).max >> > Out[10]: 1.#INF >> > >> > I see that long double in C is 12 bytes wide, and double is the usual 8 >> > bytes. >> >> Sorry - sizeof(long double) is 12 using mingw. ?I see that long double >> is the same as double in MS Visual C++. >> >> http://en.wikipedia.org/wiki/Long_double >> >> but, as expected from the name: >> >> In [11]: np.dtype(np.float96).itemsize >> Out[11]: 12 >> > > Hmm, good point. There should not be a float96 on Windows using the MSVC > compiler, and the longdouble types 'gG' should return float64 and complex128 > respectively. OTOH, I believe the mingw compiler has real float96 types but > I wonder about library support. This is really a build issue and it would be > good to have some feedback on what different platforms are doing so that we > know if we are doing things right. Is it possible that numpy is getting confused by being compiled with mingw on top of a visual studio python? Some further forensics seem to suggest that, despite the fact the math suggests float96 is float64, the storage format it in fact 80-bit extended precision: On OSX 32-bit where float128 is definitely 80 bit precision we see the sign bit being flipped to show us the beginning of the number: In [33]: bigbin(np.float128(2**53)-1) Out[33]: '00000000000000000000000000000000101111111111111101000000001100111111111111111111111111111111111111111111111111111111100000000000' In [34]: bigbin(-np.float128(2**53)+1) Out[34]: '00000000000000000000000000000000000000000000000011000000001100111111111111111111111111111111111111111111111111111111100000000000' I think that's 48 bits of padding followed by the number (bit 49 is being flipped with the sign). On windows (well, wine, but I think it's the same): bigbin(np.float96(2**53)-1) Out[14]: '000000000000000001000000001100111111111111111111111111111111111111111111111111111111100000000000' bigbin(np.float96(-2**53)+1) Out[15]: '000000000000000011000000001100111111111111111111111111111111111111111111111111111111100000000000' Thanks, Matthew import sys LE = sys.byteorder == 'little' import numpy as np def bigbin(val): val = np.asarray(val) nbytes = val.dtype.itemsize dt = [('f', np.uint8, nbytes)] out = [np.binary_repr(el, 8) for el in val.view(dt)['f']] if LE: out = out[::-1] return ''.join(out) From charlesr.harris at gmail.com Sun Nov 13 16:34:19 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Nov 2011 14:34:19 -0700 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: On Sun, Nov 13, 2011 at 2:25 PM, Matthew Brett wrote: > Hi, > > On Sun, Nov 13, 2011 at 8:21 AM, Charles R Harris > wrote: > > > > > > On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett > > > wrote: > >> > >> Hi, > >> > >> On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett < > matthew.brett at gmail.com> > >> wrote: > >> > Hi, > >> > > >> > Sorry for my continued confusion here. This is numpy 1.6.1 on windows > >> > XP 32 bit. > >> > > >> > In [2]: np.finfo(np.float96).nmant > >> > Out[2]: 52 > >> > > >> > In [3]: np.finfo(np.float96).nexp > >> > Out[3]: 15 > >> > > >> > In [4]: np.finfo(np.float64).nmant > >> > Out[4]: 52 > >> > > >> > In [5]: np.finfo(np.float64).nexp > >> > Out[5]: 11 > >> > > >> > If there are 52 bits of precision, 2**53+1 should not be > >> > representable, and sure enough: > >> > > >> > In [6]: np.float96(2**53)+1 > >> > Out[6]: 9007199254740992.0 > >> > > >> > In [7]: np.float64(2**53)+1 > >> > Out[7]: 9007199254740992.0 > >> > > >> > If the nexp is right, the max should be higher for the float96 type: > >> > > >> > In [9]: np.finfo(np.float64).max > >> > Out[9]: 1.7976931348623157e+308 > >> > > >> > In [10]: np.finfo(np.float96).max > >> > Out[10]: 1.#INF > >> > > >> > I see that long double in C is 12 bytes wide, and double is the usual > 8 > >> > bytes. > >> > >> Sorry - sizeof(long double) is 12 using mingw. I see that long double > >> is the same as double in MS Visual C++. > >> > >> http://en.wikipedia.org/wiki/Long_double > >> > >> but, as expected from the name: > >> > >> In [11]: np.dtype(np.float96).itemsize > >> Out[11]: 12 > >> > > > > Hmm, good point. There should not be a float96 on Windows using the MSVC > > compiler, and the longdouble types 'gG' should return float64 and > complex128 > > respectively. OTOH, I believe the mingw compiler has real float96 types > but > > I wonder about library support. This is really a build issue and it > would be > > good to have some feedback on what different platforms are doing so that > we > > know if we are doing things right. > > Is it possible that numpy is getting confused by being compiled with > mingw on top of a visual studio python? > > Some further forensics seem to suggest that, despite the fact the math > suggests float96 is float64, the storage format it in fact 80-bit > extended precision: > > Yes, extended precision is the type on Intel hardware with gcc, the 96/128 bits comes from alignment on 4 or 8 byte boundaries. With MSVC, double and long double are both ieee double, and on SPARC, long double is ieee quad precision. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sun Nov 13 17:56:50 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 13 Nov 2011 14:56:50 -0800 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: Hi, On Sun, Nov 13, 2011 at 1:34 PM, Charles R Harris wrote: > > > On Sun, Nov 13, 2011 at 2:25 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Sun, Nov 13, 2011 at 8:21 AM, Charles R Harris >> wrote: >> > >> > >> > On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett >> > >> > wrote: >> >> >> >> Hi, >> >> >> >> On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett >> >> >> >> wrote: >> >> > Hi, >> >> > >> >> > Sorry for my continued confusion here. ?This is numpy 1.6.1 on >> >> > windows >> >> > XP 32 bit. >> >> > >> >> > In [2]: np.finfo(np.float96).nmant >> >> > Out[2]: 52 >> >> > >> >> > In [3]: np.finfo(np.float96).nexp >> >> > Out[3]: 15 >> >> > >> >> > In [4]: np.finfo(np.float64).nmant >> >> > Out[4]: 52 >> >> > >> >> > In [5]: np.finfo(np.float64).nexp >> >> > Out[5]: 11 >> >> > >> >> > If there are 52 bits of precision, 2**53+1 should not be >> >> > representable, and sure enough: >> >> > >> >> > In [6]: np.float96(2**53)+1 >> >> > Out[6]: 9007199254740992.0 >> >> > >> >> > In [7]: np.float64(2**53)+1 >> >> > Out[7]: 9007199254740992.0 >> >> > >> >> > If the nexp is right, the max should be higher for the float96 type: >> >> > >> >> > In [9]: np.finfo(np.float64).max >> >> > Out[9]: 1.7976931348623157e+308 >> >> > >> >> > In [10]: np.finfo(np.float96).max >> >> > Out[10]: 1.#INF >> >> > >> >> > I see that long double in C is 12 bytes wide, and double is the usual >> >> > 8 >> >> > bytes. >> >> >> >> Sorry - sizeof(long double) is 12 using mingw. ?I see that long double >> >> is the same as double in MS Visual C++. >> >> >> >> http://en.wikipedia.org/wiki/Long_double >> >> >> >> but, as expected from the name: >> >> >> >> In [11]: np.dtype(np.float96).itemsize >> >> Out[11]: 12 >> >> >> > >> > Hmm, good point. There should not be a float96 on Windows using the MSVC >> > compiler, and the longdouble types 'gG' should return float64 and >> > complex128 >> > respectively. OTOH, I believe the mingw compiler has real float96 types >> > but >> > I wonder about library support. This is really a build issue and it >> > would be >> > good to have some feedback on what different platforms are doing so that >> > we >> > know if we are doing things right. >> >> Is it possible that numpy is getting confused by being compiled with >> mingw on top of a visual studio python? >> >> Some further forensics seem to suggest that, despite the fact the math >> suggests float96 is float64, the storage format it in fact 80-bit >> extended precision: >> > > Yes, extended precision is the type on Intel hardware with gcc, the 96/128 > bits comes from alignment on 4 or 8 byte boundaries. With MSVC, double and > long double are both ieee double, and on SPARC, long double is ieee quad > precision. Right - but I think my researches are showing that the longdouble numbers are being _stored_ as 80 bit, but the math on those numbers is 64 bit. Is there a reason than numpy can't do 80-bit math on these guys? If there is, is there any point in having a float96 on windows? See you, Matthew From charlesr.harris at gmail.com Sun Nov 13 20:03:23 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Nov 2011 18:03:23 -0700 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: On Sun, Nov 13, 2011 at 3:56 PM, Matthew Brett wrote: > Hi, > > On Sun, Nov 13, 2011 at 1:34 PM, Charles R Harris > wrote: > > > > > > On Sun, Nov 13, 2011 at 2:25 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Sun, Nov 13, 2011 at 8:21 AM, Charles R Harris > >> wrote: > >> > > >> > > >> > On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett > >> > > >> > wrote: > >> >> > >> >> Hi, > >> >> > >> >> On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett > >> >> > >> >> wrote: > >> >> > Hi, > >> >> > > >> >> > Sorry for my continued confusion here. This is numpy 1.6.1 on > >> >> > windows > >> >> > XP 32 bit. > >> >> > > >> >> > In [2]: np.finfo(np.float96).nmant > >> >> > Out[2]: 52 > >> >> > > >> >> > In [3]: np.finfo(np.float96).nexp > >> >> > Out[3]: 15 > >> >> > > >> >> > In [4]: np.finfo(np.float64).nmant > >> >> > Out[4]: 52 > >> >> > > >> >> > In [5]: np.finfo(np.float64).nexp > >> >> > Out[5]: 11 > >> >> > > >> >> > If there are 52 bits of precision, 2**53+1 should not be > >> >> > representable, and sure enough: > >> >> > > >> >> > In [6]: np.float96(2**53)+1 > >> >> > Out[6]: 9007199254740992.0 > >> >> > > >> >> > In [7]: np.float64(2**53)+1 > >> >> > Out[7]: 9007199254740992.0 > >> >> > > >> >> > If the nexp is right, the max should be higher for the float96 > type: > >> >> > > >> >> > In [9]: np.finfo(np.float64).max > >> >> > Out[9]: 1.7976931348623157e+308 > >> >> > > >> >> > In [10]: np.finfo(np.float96).max > >> >> > Out[10]: 1.#INF > >> >> > > >> >> > I see that long double in C is 12 bytes wide, and double is the > usual > >> >> > 8 > >> >> > bytes. > >> >> > >> >> Sorry - sizeof(long double) is 12 using mingw. I see that long > double > >> >> is the same as double in MS Visual C++. > >> >> > >> >> http://en.wikipedia.org/wiki/Long_double > >> >> > >> >> but, as expected from the name: > >> >> > >> >> In [11]: np.dtype(np.float96).itemsize > >> >> Out[11]: 12 > >> >> > >> > > >> > Hmm, good point. There should not be a float96 on Windows using the > MSVC > >> > compiler, and the longdouble types 'gG' should return float64 and > >> > complex128 > >> > respectively. OTOH, I believe the mingw compiler has real float96 > types > >> > but > >> > I wonder about library support. This is really a build issue and it > >> > would be > >> > good to have some feedback on what different platforms are doing so > that > >> > we > >> > know if we are doing things right. > >> > >> Is it possible that numpy is getting confused by being compiled with > >> mingw on top of a visual studio python? > >> > >> Some further forensics seem to suggest that, despite the fact the math > >> suggests float96 is float64, the storage format it in fact 80-bit > >> extended precision: > >> > > > > Yes, extended precision is the type on Intel hardware with gcc, the > 96/128 > > bits comes from alignment on 4 or 8 byte boundaries. With MSVC, double > and > > long double are both ieee double, and on SPARC, long double is ieee quad > > precision. > > Right - but I think my researches are showing that the longdouble > numbers are being _stored_ as 80 bit, but the math on those numbers is > 64 bit. > > Is there a reason than numpy can't do 80-bit math on these guys? If > there is, is there any point in having a float96 on windows? > It's a compiler/architecture thing and depends on how the compiler interprets the long double c type. The gcc compiler does do 80 bit math on Intel/AMD hardware. MSVC doesn't, and probably never will. MSVC shouldn't produce float96 numbers, if it does, it is a bug. Mingw uses the gcc compiler, so the numbers are there, but if it uses the MS library it will have to convert them to double to do computations like sin(x) since there are no microsoft routines for extended precision. I suspect that gcc/ms combo is what is producing the odd results you are seeing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From amueller at ais.uni-bonn.de Mon Nov 14 07:46:50 2011 From: amueller at ais.uni-bonn.de (=?ISO-8859-1?Q?Andreas_M=FCller?=) Date: Mon, 14 Nov 2011 13:46:50 +0100 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy Message-ID: <4EC10DBA.6080803@ais.uni-bonn.de> Hi everybody. When I did some normalization using numpy, I noticed that numpy.std uses more ram than I was expecting. A quick google search gave me this: http://luispedro.org/software/ncreduce The site claims that std and other reduce operations are implemented naively with many temporaries. Is that true? And if so, is there a particular reason for that? This issues seems quite easy to fix. In particular the link I gave above provides code. Cheers, Andy From marquett at iap.fr Mon Nov 14 09:04:56 2011 From: marquett at iap.fr (Jean-Baptiste Marquette) Date: Mon, 14 Nov 2011 15:04:56 +0100 Subject: [Numpy-discussion] ANN: SciPy 0.10.0 released In-Reply-To: References: Message-ID: Le 13 nov. 2011 ? 20:19, Ralf Gommers a ?crit : > I am pleased to announce the availability of SciPy 0.10.0. Hi all, Thanks for this great job. I've run nosetests on my Mac (64-bit 10.7.2 build on EPD) which fails on the following test: test_definition (test_basic.TestDoubleIFFT) ... FAIL test_definition_real (test_basic.TestDoubleIFFT) ... ok test_djbfft (test_basic.TestDoubleIFFT) ... python(60968) malloc: *** error for object 0x105435b58: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug Abort trap: 6 Thanks for any hint, Cheers Jean-Baptiste -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Nov 14 10:23:20 2011 From: cournape at gmail.com (David Cournapeau) Date: Mon, 14 Nov 2011 15:23:20 +0000 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: <4EC10DBA.6080803@ais.uni-bonn.de> References: <4EC10DBA.6080803@ais.uni-bonn.de> Message-ID: On Mon, Nov 14, 2011 at 12:46 PM, Andreas M?ller wrote: > Hi everybody. > When I did some normalization using numpy, I noticed that numpy.std uses > more ram than I was expecting. > A quick google search gave me this: > http://luispedro.org/software/ncreduce > The site claims that std and other reduce operations are implemented > naively with many temporaries. > Is that true? And if so, is there a particular reason for that? > This issues seems quite easy to fix. > In particular the link I gave above provides code. The code provided only implements a few special cases: being more efficient in those cases only is indeed easy. cheers, David From amueller at ais.uni-bonn.de Mon Nov 14 11:05:01 2011 From: amueller at ais.uni-bonn.de (=?UTF-8?B?QW5kcmVhcyBNw7xsbGVy?=) Date: Mon, 14 Nov 2011 17:05:01 +0100 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: References: <4EC10DBA.6080803@ais.uni-bonn.de> Message-ID: <4EC13C2D.2070504@ais.uni-bonn.de> On 11/14/2011 04:23 PM, David Cournapeau wrote: > On Mon, Nov 14, 2011 at 12:46 PM, Andreas M?ller > wrote: >> Hi everybody. >> When I did some normalization using numpy, I noticed that numpy.std uses >> more ram than I was expecting. >> A quick google search gave me this: >> http://luispedro.org/software/ncreduce >> The site claims that std and other reduce operations are implemented >> naively with many temporaries. >> Is that true? And if so, is there a particular reason for that? >> This issues seems quite easy to fix. >> In particular the link I gave above provides code. > The code provided only implements a few special cases: being more > efficient in those cases only is indeed easy. I am particularly interested in the std function. Is this implemented as a separate function or an instantiation of a general reduce operations? From sturla at molden.no Mon Nov 14 11:12:55 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 14 Nov 2011 17:12:55 +0100 Subject: [Numpy-discussion] Rebinning numpy array In-Reply-To: <4EBFEA77.6020102@gmx.de> References: <4EBFEA77.6020102@gmx.de> Message-ID: <4EC13E07.7010609@molden.no> Fit a poisson distribution (radioactive decay is a Poisson process), recompute lambda for whatever bin-size you need, and compute the new (estimated) bin counts by maximum likehood. It basically becomes a contrained optimization problem. Sturla Den 13.11.2011 17:04, skrev Johannes Bauer: > Hi group, > > I have a rather simple problem, or so it would seem. However I cannot > seem to find the right solution. Here's the problem: > > A Geiger counter measures counts in distinct time intervals. The time > intervals are not of constant length. Imaging for example that the > counter would always create a table entry when the counts reach 10. Then > we would have the following bins (made-up data for illustration): > > Seconds Counts Len CPS > 0 - 44 10 44 0.23 > 44 - 120 10 76 0.13 > 120 - 140 10 20 0.5 > 140 - 200 10 60 0.16 > > So we have n bins (in this example 4), but they're not equidistant. I > want to rebin samples to make them equidistant. For example, I would > like to rebin into 5 bins of 40 seconds time each. Then the rebinned > example (I calculate by hand so this might contain errors): > > 0-40 9.09 > 40-80 5.65 > 80-120 5.26 > 120-160 13.33 > 160-200 6.66 > > That means, if a destination bin completely overlaps a source bin, its > complete value is taken. If it overlaps partially, linear interpolation > of bin sizes should be used. > > It is very important that the overall count amount stays the same (in > this case 40, so my numbers seem to be correct, I checked that). In this > example I increased the bin size, but usually I will want to decrease > bin size (even dramatically). > > Now my pathetic attempts look something like this: > > interpolation_points = 4000 > xpts = [ time.mktime(x.timetuple()) for x in self.getx() ] > > interpolatedx = numpy.linspace(xpts[0], xpts[-1], interpolation_points) > interpolatedy = numpy.interp(interpolatedx, xpts, self.gety()) > > self._xreformatted = [ datetime.datetime.fromtimestamp(x) for x in > interpolatedx ] > self._yreformatted = interpolatedy > > This works somewhat, however I see artifacts depending on the > destination sample size: for example when I have a spike in the sample > input and reduce the number of interpolation points (i.e. increase > destination bin size) slowly, the spike will get smaller and smaller > (expected behaviour). After some amount of increasing, the spike however > will "magically" reappear. I believe this to be an interpolation artifact. > > Is there some standard way to get from a non-uniformally distributed bin > distribution to a unifomally distributed bin distribution of arbitrary > bin width? > > Best regards, > Joe > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From g.plantageneto at gmail.com Mon Nov 14 12:04:42 2011 From: g.plantageneto at gmail.com (Giovanni Plantageneto) Date: Mon, 14 Nov 2011 17:04:42 +0000 Subject: [Numpy-discussion] Copy netcdf attributes between different files Message-ID: Hi everybody, I am using netCDF4 library to read and write from netcdf files. I would like to copy all the attributes of one file to another one, in a way like this: --- from netCDF4 import Dataset as ncdf file1 = ncdf('file1.nc', mode='r', format='NETCDF4_CLASSIC') ... file2 = ncdf('file1.nc', mode='w', format='NETCDF4_CLASSIC') for att in file1.ncattrs(): file2.att = file1.getncatt(att) ... file1.close() file2.close() --- But this will not work as only one attribute named "att" in file2 will be created. How should I do this? Thanks. From jswhit at fastmail.fm Mon Nov 14 12:07:29 2011 From: jswhit at fastmail.fm (Jeff Whitaker) Date: Mon, 14 Nov 2011 10:07:29 -0700 Subject: [Numpy-discussion] Copy netcdf attributes between different files In-Reply-To: References: Message-ID: <4EC14AD1.20901@fastmail.fm> On 11/14/11 10:04 AM, Giovanni Plantageneto wrote: > Hi everybody, > I am using netCDF4 library to read and write from netcdf files. I > would like to copy all the attributes of one file to another one, in a > way like this: > > --- > > from netCDF4 import Dataset as ncdf > > file1 = ncdf('file1.nc', mode='r', format='NETCDF4_CLASSIC') > ... > file2 = ncdf('file1.nc', mode='w', format='NETCDF4_CLASSIC') > for att in file1.ncattrs(): > file2.att = file1.getncatt(att) > ... > file1.close() > file2.close() > > --- > > But this will not work as only one attribute named "att" in file2 will > be created. How should I do this? > Thanks. > Try this: for att in file1.ncattrs(): setattr(file2,att) = getattr(file1,att) -Jeff -- Jeffrey S. Whitaker Phone : (303)497-6313 Meteorologist FAX : (303)497-6449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker at noaa.gov 325 Broadway Office : Skaggs Research Cntr 1D-113 Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg From shish at keba.be Mon Nov 14 12:13:44 2011 From: shish at keba.be (Olivier Delalleau) Date: Mon, 14 Nov 2011 12:13:44 -0500 Subject: [Numpy-discussion] Copy netcdf attributes between different files In-Reply-To: References: Message-ID: In Python you use setattr to set an object's attribute whose name is stored into a variable: setattr(file2, att, file1.getncatt(att)) -=- Olivier 2011/11/14 Giovanni Plantageneto > Hi everybody, > I am using netCDF4 library to read and write from netcdf files. I > would like to copy all the attributes of one file to another one, in a > way like this: > > --- > > from netCDF4 import Dataset as ncdf > > file1 = ncdf('file1.nc', mode='r', format='NETCDF4_CLASSIC') > ... > file2 = ncdf('file1.nc', mode='w', format='NETCDF4_CLASSIC') > for att in file1.ncattrs(): > file2.att = file1.getncatt(att) > ... > file1.close() > file2.close() > > --- > > But this will not work as only one attribute named "att" in file2 will > be created. How should I do this? > Thanks. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Nov 14 12:17:11 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Mon, 14 Nov 2011 09:17:11 -0800 Subject: [Numpy-discussion] Rebinning numpy array In-Reply-To: References: <4EBFEA77.6020102@gmx.de> Message-ID: <4EC14D17.80009@noaa.gov> On 11/13/11 9:55 AM, Olivier Delalleau wrote: > idea, since it will throw out a lot of information if you decrease the > number of bins: I agree -- I'd think about looking at a smooth interpolation -- maybe kernel density estimation? On 11/14/11 8:12 AM, Sturla Molden wrote: > Fit a poisson distribution (radioactive decay is a Poisson process), even better -- if you have a physical process that fits a given function form -- us it! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Nov 14 12:24:40 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Mon, 14 Nov 2011 09:24:40 -0800 Subject: [Numpy-discussion] dedicated function for resize with averaging or rebin 2d arrays? In-Reply-To: <9DD9103D-D5CB-489A-A1CF-0273367651E1@me.com> References: <9DD9103D-D5CB-489A-A1CF-0273367651E1@me.com> Message-ID: <4EC14ED8.6080307@noaa.gov> On 11/11/11 8:28 PM, Craig Yoshioka wrote: > I once wrote a generic n-dimensional binning routine in C that I > could find if anyone is interested in integrating it into numpy... it > didn't do size increases though... and I think I implemented it so > that binning by a non-divisible factor trimmed the extras. It was > very-very fast though. And I've got one written in Cython. I'd been thinking of contributing it to ndimage, but never got around to it -- but I'd be glad to share. Let me know if you want it. >> I think the usefulness of the rebin function is to be simple and fast, >> and the best would be to implement it in the core numpy, as a simple >> method for smoothing and reshaping, Does it have common use=-cases outside image processing? If not, then no, it shouldn't be in numpy. > without the need of using scipy. I don't think you need all of scipy to use the ndimage toolkit -- at least I hope not. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Nov 14 12:26:04 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Mon, 14 Nov 2011 09:26:04 -0800 Subject: [Numpy-discussion] how to use the name of a ndarray as a string In-Reply-To: References: Message-ID: <4EC14F2C.3050608@noaa.gov> On 11/10/11 3:57 AM, Olivier Delalleau wrote: > In such a situation you should probably use a dictionary from the start, all good suggestions, but while we're at it: On 11/10/11 2:17 AM, Chao YUE wrote: > Does anyone know how I can quickly use the name of a ndarray as a string? This reflects a key misunderstanding of how Python works -- a ndarray ( or any python object) does not "have a name". when you write: index=np.arange(100) you are creating a ndarray object, and _binding_ the name "index" to it. the object exists apart from it's name: l =[] l.append(np.arange(100)) now there is a an andarray object in the list -- but it has no name. You can also bind multiple names to the same object: index=np.arange(100) fred = index john = index now "index", "fred", and "john" are all bound to the same object -- what would be its name now? Hence Olivier's suggestionss -- al various ways to store and refere to objects -- binding to a variable is only one way. This is a good read that should clarify it all: http://python.net/~mwh/hacks/objectthink.html -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From g.plantageneto at gmail.com Mon Nov 14 12:28:03 2011 From: g.plantageneto at gmail.com (Giovanni Plantageneto) Date: Mon, 14 Nov 2011 17:28:03 +0000 Subject: [Numpy-discussion] Copy netcdf attributes between different files In-Reply-To: <4EC14AD1.20901@fastmail.fm> References: <4EC14AD1.20901@fastmail.fm> Message-ID: Yes, thanks! It works, but the syntax is setattr(flle2,att,getattr(file1,att)) 2011/11/14 Jeff Whitaker : > On 11/14/11 10:04 AM, Giovanni Plantageneto wrote: >> >> Hi everybody, >> I am using netCDF4 library to read and write from netcdf files. I >> would like to copy all the attributes of one file to another one, in a >> way like this: >> >> --- >> >> from netCDF4 import Dataset as ncdf >> >> file1 = ncdf('file1.nc', mode='r', format='NETCDF4_CLASSIC') >> ... >> file2 = ncdf('file1.nc', mode='w', format='NETCDF4_CLASSIC') >> for att in file1.ncattrs(): >> ? ?file2.att = file1.getncatt(att) >> ... >> file1.close() >> file2.close() >> >> --- >> >> But this will not work as only one attribute named "att" in file2 will >> be created. How should I do this? >> Thanks. >> > Try this: > > for att in file1.ncattrs(): > ? setattr(file2,att) = getattr(file1,att) > > -Jeff > > > -- > Jeffrey S. Whitaker ? ? ? ? Phone ?: (303)497-6313 > Meteorologist ? ? ? ? ? ? ? FAX ? ?: (303)497-6449 > NOAA/OAR/PSD ?R/PSD1 ? ? ? ?Email ?: Jeffrey.S.Whitaker at noaa.gov > 325 Broadway ? ? ? ? ? ? ? ?Office : Skaggs Research Cntr 1D-113 > Boulder, CO, USA 80303-3328 Web ? ?: http://tinyurl.com/5telg > > From ralf.gommers at googlemail.com Mon Nov 14 14:54:20 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 14 Nov 2011 20:54:20 +0100 Subject: [Numpy-discussion] ANN: SciPy 0.10.0 released In-Reply-To: References: Message-ID: On Mon, Nov 14, 2011 at 3:04 PM, Jean-Baptiste Marquette wrote: > > Le 13 nov. 2011 ? 20:19, Ralf Gommers a ?crit : > > I am pleased to announce the availability of SciPy 0.10.0. > > > Hi all, > > Thanks for this great job. > I've run nosetests on my Mac (64-bit 10.7.2 build on EPD) which fails on > the following test: > > test_definition (test_basic.TestDoubleIFFT) ... FAIL > test_definition_real (test_basic.TestDoubleIFFT) ... ok > test_djbfft (test_basic.TestDoubleIFFT) ... python(60968) malloc: *** > error for object 0x105435b58: incorrect checksum for freed object - object > was probably modified after being freed. > *** set a breakpoint in malloc_error_break to debug > Abort trap: 6 > > The exact same issue was just reported on scipy-user, thread "test issues with 0.10". Can you please move the follow-up over there? What compilers did you use? And do you know if EPD was built with the same compilers? The error looks the same as the problem we still have with scipy.sparse under 64-bit python 2.7 (http://projects.scipy.org/scipy/ticket/1523), related to the Apple Accelerate Framework being broken. The problem doesn't exist on OS X 10.6 though, so it could be Apple broke it for 10.7 or it could be compiler related. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.MACKEITH at 3ds.com Mon Nov 14 15:18:36 2011 From: Andrew.MACKEITH at 3ds.com (MACKEITH Andrew) Date: Mon, 14 Nov 2011 20:18:36 +0000 Subject: [Numpy-discussion] numpy.int32 is not subclass of int, but numpy.int64 is Message-ID: <2922D4459F182B41ABD8C7AE501064200F6B95D7@AG-DCC-MBX03.dsone.3ds.com> Could someone explain this? An instance of numpy.int32 is not an instance of int or numpy.int. An instance of numpy.int64 is an instance of int and numpy.int. I don't know if it is a bug in my linux build. Andrew >python26 Python 2.6.2 (r262:71600, Jul 8 2010, 11:49:56) [GCC 4.1.2 20070115 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> >>> import numpy >>> a = numpy.array((1,2,3)) >>> type(a[0]) >>> isinstance(a[0], int) True >>> isinstance(a[0], numpy.int) True >>> seven = numpy.int64(7) >>> isinstance(seven, numpy.int) True >>> isinstance(seven, int) True >>> >>> five = numpy.int32(5) >>> isinstance(five, numpy.int) False >>> isinstance(five, int) False >>> five 5 >>> seven 7 >>> issubclass(numpy.int64, numpy.int) True >>> issubclass(numpy.int32, numpy.int) False >>> >>> import sys >>> sys.maxint 9223372036854775807 >>> import platform >>> print platform.platform() Linux-2.6.32.12-0.7-default-x86_64-with-SuSE-11-x86_64 >>> This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Nov 14 15:25:41 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Nov 2011 20:25:41 +0000 Subject: [Numpy-discussion] numpy.int32 is not subclass of int, but numpy.int64 is In-Reply-To: <2922D4459F182B41ABD8C7AE501064200F6B95D7@AG-DCC-MBX03.dsone.3ds.com> References: <2922D4459F182B41ABD8C7AE501064200F6B95D7@AG-DCC-MBX03.dsone.3ds.com> Message-ID: On Mon, Nov 14, 2011 at 20:18, MACKEITH Andrew wrote: > Could someone explain this? > > An instance of numpy.int32 is not an instance of int or numpy.int. > An instance of numpy.int64 is an instance of int and numpy.int. > > I don't know if it is a bug in my linux build. >>>> import sys >>>> sys.maxint > 9223372036854775807 >>>> import platform >>>> print? platform.platform() > Linux-2.6.32.12-0.7-default-x86_64-with-SuSE-11-x86_64 This is expected on a 64-bit platform. Note that numpy.int is just an alias for the builtin int type for backwards compatibility with an earlier version of numpy. We could probably remove it, since it seems to be causing more confusion than not. Anyways, we subclass the appropriately sized integer scalar type from Python's int type depending on the platform. So on a platform where Python's int type is 64-bits, numpy.int64 will include int in its inheritance tree. On platforms where the Python int type is 32-bit, numpy.int32 will include it instead. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From matthew.brett at gmail.com Mon Nov 14 16:01:15 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Nov 2011 13:01:15 -0800 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: Hi, On Sun, Nov 13, 2011 at 5:03 PM, Charles R Harris wrote: > > > On Sun, Nov 13, 2011 at 3:56 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Sun, Nov 13, 2011 at 1:34 PM, Charles R Harris >> wrote: >> > >> > >> > On Sun, Nov 13, 2011 at 2:25 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Sun, Nov 13, 2011 at 8:21 AM, Charles R Harris >> >> wrote: >> >> > >> >> > >> >> > On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett >> >> > >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett >> >> >> >> >> >> wrote: >> >> >> > Hi, >> >> >> > >> >> >> > Sorry for my continued confusion here. ?This is numpy 1.6.1 on >> >> >> > windows >> >> >> > XP 32 bit. >> >> >> > >> >> >> > In [2]: np.finfo(np.float96).nmant >> >> >> > Out[2]: 52 >> >> >> > >> >> >> > In [3]: np.finfo(np.float96).nexp >> >> >> > Out[3]: 15 >> >> >> > >> >> >> > In [4]: np.finfo(np.float64).nmant >> >> >> > Out[4]: 52 >> >> >> > >> >> >> > In [5]: np.finfo(np.float64).nexp >> >> >> > Out[5]: 11 >> >> >> > >> >> >> > If there are 52 bits of precision, 2**53+1 should not be >> >> >> > representable, and sure enough: >> >> >> > >> >> >> > In [6]: np.float96(2**53)+1 >> >> >> > Out[6]: 9007199254740992.0 >> >> >> > >> >> >> > In [7]: np.float64(2**53)+1 >> >> >> > Out[7]: 9007199254740992.0 >> >> >> > >> >> >> > If the nexp is right, the max should be higher for the float96 >> >> >> > type: >> >> >> > >> >> >> > In [9]: np.finfo(np.float64).max >> >> >> > Out[9]: 1.7976931348623157e+308 >> >> >> > >> >> >> > In [10]: np.finfo(np.float96).max >> >> >> > Out[10]: 1.#INF >> >> >> > >> >> >> > I see that long double in C is 12 bytes wide, and double is the >> >> >> > usual >> >> >> > 8 >> >> >> > bytes. >> >> >> >> >> >> Sorry - sizeof(long double) is 12 using mingw. ?I see that long >> >> >> double >> >> >> is the same as double in MS Visual C++. >> >> >> >> >> >> http://en.wikipedia.org/wiki/Long_double >> >> >> >> >> >> but, as expected from the name: >> >> >> >> >> >> In [11]: np.dtype(np.float96).itemsize >> >> >> Out[11]: 12 >> >> >> >> >> > >> >> > Hmm, good point. There should not be a float96 on Windows using the >> >> > MSVC >> >> > compiler, and the longdouble types 'gG' should return float64 and >> >> > complex128 >> >> > respectively. OTOH, I believe the mingw compiler has real float96 >> >> > types >> >> > but >> >> > I wonder about library support. This is really a build issue and it >> >> > would be >> >> > good to have some feedback on what different platforms are doing so >> >> > that >> >> > we >> >> > know if we are doing things right. >> >> >> >> Is it possible that numpy is getting confused by being compiled with >> >> mingw on top of a visual studio python? >> >> >> >> Some further forensics seem to suggest that, despite the fact the math >> >> suggests float96 is float64, the storage format it in fact 80-bit >> >> extended precision: >> >> >> > >> > Yes, extended precision is the type on Intel hardware with gcc, the >> > 96/128 >> > bits comes from alignment on 4 or 8 byte boundaries. With MSVC, double >> > and >> > long double are both ieee double, and on SPARC, long double is ieee quad >> > precision. >> >> Right - but I think my researches are showing that the longdouble >> numbers are being _stored_ as 80 bit, but the math on those numbers is >> 64 bit. >> >> Is there a reason than numpy can't do 80-bit math on these guys? ?If >> there is, is there any point in having a float96 on windows? > > It's a compiler/architecture thing and depends on how the compiler > interprets the long double c type. The gcc compiler does do 80 bit math on > Intel/AMD hardware. MSVC doesn't, and probably never will. MSVC shouldn't > produce float96 numbers, if it does, it is a bug. Mingw uses the gcc > compiler, so the numbers are there, but if it uses the MS library it will > have to convert them to double to do computations like sin(x) since there > are no microsoft routines for extended precision. I suspect that gcc/ms > combo is what is producing the odd results you are seeing. I think we might be talking past each other a bit. It seems to me that, if float96 must use float64 math, then it should be removed from the numpy namespace, because a) It implies higher precision than float64 but does not provide it b) It uses more memory to no obvious advantage On the other hand, it seems to me that raw gcc does use higher precision for basic math on long double, as expected. For example, this guy passes: #include #include int main(int argc, char* argv) { double d; long double ld; d = pow(2, 53); ld = d; assert(d == ld); d += 1; ld += 1; /* double rounds down because it doesn't have enough precision */ assert(d != ld); assert(d == ld - 1); } whereas numpy does not use the higher precision: In [10]: a = np.float96(2**53) In [11]: a Out[11]: 9007199254740992.0 In [12]: b = np.float64(2**53) In [13]: b Out[13]: 9007199254740992.0 In [14]: a == b Out[14]: True In [15]: (a + 1) == (b + 1) Out[15]: True So maybe there is a way of picking up the gcc math in numpy? Best, Matthew From zonca at deepspace.ucsb.edu Mon Nov 14 17:02:37 2011 From: zonca at deepspace.ucsb.edu (Andrea Zonca) Date: Mon, 14 Nov 2011 14:02:37 -0800 Subject: [Numpy-discussion] dedicated function for resize with averaging or rebin 2d arrays? In-Reply-To: <4EC14ED8.6080307@noaa.gov> References: <9DD9103D-D5CB-489A-A1CF-0273367651E1@me.com> <4EC14ED8.6080307@noaa.gov> Message-ID: thanks, On Mon, Nov 14, 2011 at 09:24, Chris.Barker wrote: > On 11/11/11 8:28 PM, Craig Yoshioka wrote: >> I once wrote a generic n-dimensional binning routine in C that I >> could find if anyone is interested in integrating it into numpy... it >> didn't do size increases though... and I think I implemented it so >> that binning by a non-divisible factor trimmed the extras. ?It was >> very-very fast though. > > And I've got one written in Cython. I'd been thinking of contributing it > to ndimage, but never got around to it -- but I'd be glad to share. Let > me know if you want it. Chris, I think this would be useful, can you make it available publicly? >>> I think the usefulness of the rebin function is to be simple and fast, >>> and the best would be to implement it in the core numpy, as a simple >>> method for smoothing and reshaping, > > Does it have common use=-cases outside image processing? If not, then > no, it shouldn't be in numpy. actually the most common use is outside of image processing, we typically use it when we have data that come from a detector scanning the sky in circles, so you have a row for each scan. It is nice to apply a quick and dirt decimation of the dataset by binning it. It is such a simple function that would make more sense, to me, to have in numpy. In image processing I imagine you typically want to apply window functions, so for that ndimage is more appropriate. cheers, andrea From nathan.faggian at gmail.com Tue Nov 15 00:08:45 2011 From: nathan.faggian at gmail.com (Nathan Faggian) Date: Tue, 15 Nov 2011 16:08:45 +1100 Subject: [Numpy-discussion] Using dtype=object Message-ID: Hi, I am interested in the use of numpy with native python objects, like so: In [91]: import collections In [92]: testContainer = collections.namedtuple('testContainer', 'att1 att2 att3') In [93]: test1 = testContainer(1, 2, 3) In [94]: test2 = testContainer(4, 5, 6) In [95]: test1 Out[95]: testContainer(att1=1, att2=2, att3=3) In [96]: test2 Out[96]: testContainer(att1=4, att2=5, att3=6) In [97]: x = np.empty((2,2), dtype=object) In [98]: x[0,0] = test1 In [99]: x[1,1] = test2 In [100]: x Out[100]: array([[testContainer(att1=1, att2=2, att3=3), None], [None, testContainer(att1=4, att2=5, att3=6)]], dtype=object) Does anyone know if it possible to form a mask using the attributes of the objects stored in the ndarray? After a few failed attempts I am left wondering if I should use a prepared dtype instead. Cheers, Nathan. From cournape at gmail.com Tue Nov 15 01:08:07 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 15 Nov 2011 06:08:07 +0000 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: On Mon, Nov 14, 2011 at 9:01 PM, Matthew Brett wrote: > Hi, > > On Sun, Nov 13, 2011 at 5:03 PM, Charles R Harris > wrote: >> >> >> On Sun, Nov 13, 2011 at 3:56 PM, Matthew Brett >> wrote: >>> >>> Hi, >>> >>> On Sun, Nov 13, 2011 at 1:34 PM, Charles R Harris >>> wrote: >>> > >>> > >>> > On Sun, Nov 13, 2011 at 2:25 PM, Matthew Brett >>> > wrote: >>> >> >>> >> Hi, >>> >> >>> >> On Sun, Nov 13, 2011 at 8:21 AM, Charles R Harris >>> >> wrote: >>> >> > >>> >> > >>> >> > On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett >>> >> > >>> >> > wrote: >>> >> >> >>> >> >> Hi, >>> >> >> >>> >> >> On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett >>> >> >> >>> >> >> wrote: >>> >> >> > Hi, >>> >> >> > >>> >> >> > Sorry for my continued confusion here. ?This is numpy 1.6.1 on >>> >> >> > windows >>> >> >> > XP 32 bit. >>> >> >> > >>> >> >> > In [2]: np.finfo(np.float96).nmant >>> >> >> > Out[2]: 52 >>> >> >> > >>> >> >> > In [3]: np.finfo(np.float96).nexp >>> >> >> > Out[3]: 15 >>> >> >> > >>> >> >> > In [4]: np.finfo(np.float64).nmant >>> >> >> > Out[4]: 52 >>> >> >> > >>> >> >> > In [5]: np.finfo(np.float64).nexp >>> >> >> > Out[5]: 11 >>> >> >> > >>> >> >> > If there are 52 bits of precision, 2**53+1 should not be >>> >> >> > representable, and sure enough: >>> >> >> > >>> >> >> > In [6]: np.float96(2**53)+1 >>> >> >> > Out[6]: 9007199254740992.0 >>> >> >> > >>> >> >> > In [7]: np.float64(2**53)+1 >>> >> >> > Out[7]: 9007199254740992.0 >>> >> >> > >>> >> >> > If the nexp is right, the max should be higher for the float96 >>> >> >> > type: >>> >> >> > >>> >> >> > In [9]: np.finfo(np.float64).max >>> >> >> > Out[9]: 1.7976931348623157e+308 >>> >> >> > >>> >> >> > In [10]: np.finfo(np.float96).max >>> >> >> > Out[10]: 1.#INF >>> >> >> > >>> >> >> > I see that long double in C is 12 bytes wide, and double is the >>> >> >> > usual >>> >> >> > 8 >>> >> >> > bytes. >>> >> >> >>> >> >> Sorry - sizeof(long double) is 12 using mingw. ?I see that long >>> >> >> double >>> >> >> is the same as double in MS Visual C++. >>> >> >> >>> >> >> http://en.wikipedia.org/wiki/Long_double >>> >> >> >>> >> >> but, as expected from the name: >>> >> >> >>> >> >> In [11]: np.dtype(np.float96).itemsize >>> >> >> Out[11]: 12 >>> >> >> >>> >> > >>> >> > Hmm, good point. There should not be a float96 on Windows using the >>> >> > MSVC >>> >> > compiler, and the longdouble types 'gG' should return float64 and >>> >> > complex128 >>> >> > respectively. OTOH, I believe the mingw compiler has real float96 >>> >> > types >>> >> > but >>> >> > I wonder about library support. This is really a build issue and it >>> >> > would be >>> >> > good to have some feedback on what different platforms are doing so >>> >> > that >>> >> > we >>> >> > know if we are doing things right. >>> >> >>> >> Is it possible that numpy is getting confused by being compiled with >>> >> mingw on top of a visual studio python? >>> >> >>> >> Some further forensics seem to suggest that, despite the fact the math >>> >> suggests float96 is float64, the storage format it in fact 80-bit >>> >> extended precision: >>> >> >>> > >>> > Yes, extended precision is the type on Intel hardware with gcc, the >>> > 96/128 >>> > bits comes from alignment on 4 or 8 byte boundaries. With MSVC, double >>> > and >>> > long double are both ieee double, and on SPARC, long double is ieee quad >>> > precision. >>> >>> Right - but I think my researches are showing that the longdouble >>> numbers are being _stored_ as 80 bit, but the math on those numbers is >>> 64 bit. >>> >>> Is there a reason than numpy can't do 80-bit math on these guys? ?If >>> there is, is there any point in having a float96 on windows? >> >> It's a compiler/architecture thing and depends on how the compiler >> interprets the long double c type. The gcc compiler does do 80 bit math on >> Intel/AMD hardware. MSVC doesn't, and probably never will. MSVC shouldn't >> produce float96 numbers, if it does, it is a bug. Mingw uses the gcc >> compiler, so the numbers are there, but if it uses the MS library it will >> have to convert them to double to do computations like sin(x) since there >> are no microsoft routines for extended precision. I suspect that gcc/ms >> combo is what is producing the odd results you are seeing. > > I think we might be talking past each other a bit. > > It seems to me that, if float96 must use float64 math, then it should > be removed from the numpy namespace, because If we were to do so, it would break too much code. > > a) It implies higher precision than float64 but does not provide it > b) It uses more memory to no obvious advantage There is an obvious advantage: to handle memory blocks which use long double, created outside numpy (or even python). Otherwise, while gcc indeed supports long double, the fact that the C runtime doesn't really mean it is hopeless to reach any kind of consistency. And I will reiterate what I said before about long double: if you care about your code behaving consistency across platforms, just forget about long double. cheers, David From matthew.brett at gmail.com Tue Nov 15 01:22:50 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Nov 2011 22:22:50 -0800 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: Hi, On Mon, Nov 14, 2011 at 10:08 PM, David Cournapeau wrote: > On Mon, Nov 14, 2011 at 9:01 PM, Matthew Brett wrote: >> Hi, >> >> On Sun, Nov 13, 2011 at 5:03 PM, Charles R Harris >> wrote: >>> >>> >>> On Sun, Nov 13, 2011 at 3:56 PM, Matthew Brett >>> wrote: >>>> >>>> Hi, >>>> >>>> On Sun, Nov 13, 2011 at 1:34 PM, Charles R Harris >>>> wrote: >>>> > >>>> > >>>> > On Sun, Nov 13, 2011 at 2:25 PM, Matthew Brett >>>> > wrote: >>>> >> >>>> >> Hi, >>>> >> >>>> >> On Sun, Nov 13, 2011 at 8:21 AM, Charles R Harris >>>> >> wrote: >>>> >> > >>>> >> > >>>> >> > On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett >>>> >> > >>>> >> > wrote: >>>> >> >> >>>> >> >> Hi, >>>> >> >> >>>> >> >> On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett >>>> >> >> >>>> >> >> wrote: >>>> >> >> > Hi, >>>> >> >> > >>>> >> >> > Sorry for my continued confusion here. ?This is numpy 1.6.1 on >>>> >> >> > windows >>>> >> >> > XP 32 bit. >>>> >> >> > >>>> >> >> > In [2]: np.finfo(np.float96).nmant >>>> >> >> > Out[2]: 52 >>>> >> >> > >>>> >> >> > In [3]: np.finfo(np.float96).nexp >>>> >> >> > Out[3]: 15 >>>> >> >> > >>>> >> >> > In [4]: np.finfo(np.float64).nmant >>>> >> >> > Out[4]: 52 >>>> >> >> > >>>> >> >> > In [5]: np.finfo(np.float64).nexp >>>> >> >> > Out[5]: 11 >>>> >> >> > >>>> >> >> > If there are 52 bits of precision, 2**53+1 should not be >>>> >> >> > representable, and sure enough: >>>> >> >> > >>>> >> >> > In [6]: np.float96(2**53)+1 >>>> >> >> > Out[6]: 9007199254740992.0 >>>> >> >> > >>>> >> >> > In [7]: np.float64(2**53)+1 >>>> >> >> > Out[7]: 9007199254740992.0 >>>> >> >> > >>>> >> >> > If the nexp is right, the max should be higher for the float96 >>>> >> >> > type: >>>> >> >> > >>>> >> >> > In [9]: np.finfo(np.float64).max >>>> >> >> > Out[9]: 1.7976931348623157e+308 >>>> >> >> > >>>> >> >> > In [10]: np.finfo(np.float96).max >>>> >> >> > Out[10]: 1.#INF >>>> >> >> > >>>> >> >> > I see that long double in C is 12 bytes wide, and double is the >>>> >> >> > usual >>>> >> >> > 8 >>>> >> >> > bytes. >>>> >> >> >>>> >> >> Sorry - sizeof(long double) is 12 using mingw. ?I see that long >>>> >> >> double >>>> >> >> is the same as double in MS Visual C++. >>>> >> >> >>>> >> >> http://en.wikipedia.org/wiki/Long_double >>>> >> >> >>>> >> >> but, as expected from the name: >>>> >> >> >>>> >> >> In [11]: np.dtype(np.float96).itemsize >>>> >> >> Out[11]: 12 >>>> >> >> >>>> >> > >>>> >> > Hmm, good point. There should not be a float96 on Windows using the >>>> >> > MSVC >>>> >> > compiler, and the longdouble types 'gG' should return float64 and >>>> >> > complex128 >>>> >> > respectively. OTOH, I believe the mingw compiler has real float96 >>>> >> > types >>>> >> > but >>>> >> > I wonder about library support. This is really a build issue and it >>>> >> > would be >>>> >> > good to have some feedback on what different platforms are doing so >>>> >> > that >>>> >> > we >>>> >> > know if we are doing things right. >>>> >> >>>> >> Is it possible that numpy is getting confused by being compiled with >>>> >> mingw on top of a visual studio python? >>>> >> >>>> >> Some further forensics seem to suggest that, despite the fact the math >>>> >> suggests float96 is float64, the storage format it in fact 80-bit >>>> >> extended precision: >>>> >> >>>> > >>>> > Yes, extended precision is the type on Intel hardware with gcc, the >>>> > 96/128 >>>> > bits comes from alignment on 4 or 8 byte boundaries. With MSVC, double >>>> > and >>>> > long double are both ieee double, and on SPARC, long double is ieee quad >>>> > precision. >>>> >>>> Right - but I think my researches are showing that the longdouble >>>> numbers are being _stored_ as 80 bit, but the math on those numbers is >>>> 64 bit. >>>> >>>> Is there a reason than numpy can't do 80-bit math on these guys? ?If >>>> there is, is there any point in having a float96 on windows? >>> >>> It's a compiler/architecture thing and depends on how the compiler >>> interprets the long double c type. The gcc compiler does do 80 bit math on >>> Intel/AMD hardware. MSVC doesn't, and probably never will. MSVC shouldn't >>> produce float96 numbers, if it does, it is a bug. Mingw uses the gcc >>> compiler, so the numbers are there, but if it uses the MS library it will >>> have to convert them to double to do computations like sin(x) since there >>> are no microsoft routines for extended precision. I suspect that gcc/ms >>> combo is what is producing the odd results you are seeing. >> >> I think we might be talking past each other a bit. >> >> It seems to me that, if float96 must use float64 math, then it should >> be removed from the numpy namespace, because > > If we were to do so, it would break too much code. David - please - obviously I'm not suggesting removing it without deprecating it. >> a) It implies higher precision than float64 but does not provide it >> b) It uses more memory to no obvious advantage > > There is an obvious advantage: to handle memory blocks which use long > double, created outside numpy (or even python). Right - but that's a bit arcane, and I would have thought np.longdouble would be a good enough name for that. Of course, the users may be surprised, as I was, that memory allocated for higher precision is using float64, and that may take them some time to work out. I'll say again that 'longdouble' says to me 'something specific to the compiler' and 'float96' says 'something standard in numpy', and that I - was surprised - when I found out what it was. > Otherwise, while gcc indeed supports long double, the fact that the C > runtime doesn't really mean it is hopeless to reach any kind of > consistency. I'm sorry for my ignorance, but which numerical algorithms come from the C runtime? I guess not addition from the snippet in my last mail. Is the numpy code deliberately using float64 for everything on the basis that it may not be able to use 80 bit precision for some things? > And I will reiterate what I said before about long > double: if you care about your code behaving consistency across > platforms, just forget about long double. Do you agree that the current state of float96 on Windows is hard to understand? If so, what do you think we can do to improve it? Best, Matthew From cournape at gmail.com Tue Nov 15 03:51:18 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 15 Nov 2011 08:51:18 +0000 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: On Tue, Nov 15, 2011 at 6:22 AM, Matthew Brett wrote: > Hi, > > On Mon, Nov 14, 2011 at 10:08 PM, David Cournapeau wrote: >> On Mon, Nov 14, 2011 at 9:01 PM, Matthew Brett wrote: >>> Hi, >>> >>> On Sun, Nov 13, 2011 at 5:03 PM, Charles R Harris >>> wrote: >>>> >>>> >>>> On Sun, Nov 13, 2011 at 3:56 PM, Matthew Brett >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> On Sun, Nov 13, 2011 at 1:34 PM, Charles R Harris >>>>> wrote: >>>>> > >>>>> > >>>>> > On Sun, Nov 13, 2011 at 2:25 PM, Matthew Brett >>>>> > wrote: >>>>> >> >>>>> >> Hi, >>>>> >> >>>>> >> On Sun, Nov 13, 2011 at 8:21 AM, Charles R Harris >>>>> >> wrote: >>>>> >> > >>>>> >> > >>>>> >> > On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett >>>>> >> > >>>>> >> > wrote: >>>>> >> >> >>>>> >> >> Hi, >>>>> >> >> >>>>> >> >> On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett >>>>> >> >> >>>>> >> >> wrote: >>>>> >> >> > Hi, >>>>> >> >> > >>>>> >> >> > Sorry for my continued confusion here. ?This is numpy 1.6.1 on >>>>> >> >> > windows >>>>> >> >> > XP 32 bit. >>>>> >> >> > >>>>> >> >> > In [2]: np.finfo(np.float96).nmant >>>>> >> >> > Out[2]: 52 >>>>> >> >> > >>>>> >> >> > In [3]: np.finfo(np.float96).nexp >>>>> >> >> > Out[3]: 15 >>>>> >> >> > >>>>> >> >> > In [4]: np.finfo(np.float64).nmant >>>>> >> >> > Out[4]: 52 >>>>> >> >> > >>>>> >> >> > In [5]: np.finfo(np.float64).nexp >>>>> >> >> > Out[5]: 11 >>>>> >> >> > >>>>> >> >> > If there are 52 bits of precision, 2**53+1 should not be >>>>> >> >> > representable, and sure enough: >>>>> >> >> > >>>>> >> >> > In [6]: np.float96(2**53)+1 >>>>> >> >> > Out[6]: 9007199254740992.0 >>>>> >> >> > >>>>> >> >> > In [7]: np.float64(2**53)+1 >>>>> >> >> > Out[7]: 9007199254740992.0 >>>>> >> >> > >>>>> >> >> > If the nexp is right, the max should be higher for the float96 >>>>> >> >> > type: >>>>> >> >> > >>>>> >> >> > In [9]: np.finfo(np.float64).max >>>>> >> >> > Out[9]: 1.7976931348623157e+308 >>>>> >> >> > >>>>> >> >> > In [10]: np.finfo(np.float96).max >>>>> >> >> > Out[10]: 1.#INF >>>>> >> >> > >>>>> >> >> > I see that long double in C is 12 bytes wide, and double is the >>>>> >> >> > usual >>>>> >> >> > 8 >>>>> >> >> > bytes. >>>>> >> >> >>>>> >> >> Sorry - sizeof(long double) is 12 using mingw. ?I see that long >>>>> >> >> double >>>>> >> >> is the same as double in MS Visual C++. >>>>> >> >> >>>>> >> >> http://en.wikipedia.org/wiki/Long_double >>>>> >> >> >>>>> >> >> but, as expected from the name: >>>>> >> >> >>>>> >> >> In [11]: np.dtype(np.float96).itemsize >>>>> >> >> Out[11]: 12 >>>>> >> >> >>>>> >> > >>>>> >> > Hmm, good point. There should not be a float96 on Windows using the >>>>> >> > MSVC >>>>> >> > compiler, and the longdouble types 'gG' should return float64 and >>>>> >> > complex128 >>>>> >> > respectively. OTOH, I believe the mingw compiler has real float96 >>>>> >> > types >>>>> >> > but >>>>> >> > I wonder about library support. This is really a build issue and it >>>>> >> > would be >>>>> >> > good to have some feedback on what different platforms are doing so >>>>> >> > that >>>>> >> > we >>>>> >> > know if we are doing things right. >>>>> >> >>>>> >> Is it possible that numpy is getting confused by being compiled with >>>>> >> mingw on top of a visual studio python? >>>>> >> >>>>> >> Some further forensics seem to suggest that, despite the fact the math >>>>> >> suggests float96 is float64, the storage format it in fact 80-bit >>>>> >> extended precision: >>>>> >> >>>>> > >>>>> > Yes, extended precision is the type on Intel hardware with gcc, the >>>>> > 96/128 >>>>> > bits comes from alignment on 4 or 8 byte boundaries. With MSVC, double >>>>> > and >>>>> > long double are both ieee double, and on SPARC, long double is ieee quad >>>>> > precision. >>>>> >>>>> Right - but I think my researches are showing that the longdouble >>>>> numbers are being _stored_ as 80 bit, but the math on those numbers is >>>>> 64 bit. >>>>> >>>>> Is there a reason than numpy can't do 80-bit math on these guys? ?If >>>>> there is, is there any point in having a float96 on windows? >>>> >>>> It's a compiler/architecture thing and depends on how the compiler >>>> interprets the long double c type. The gcc compiler does do 80 bit math on >>>> Intel/AMD hardware. MSVC doesn't, and probably never will. MSVC shouldn't >>>> produce float96 numbers, if it does, it is a bug. Mingw uses the gcc >>>> compiler, so the numbers are there, but if it uses the MS library it will >>>> have to convert them to double to do computations like sin(x) since there >>>> are no microsoft routines for extended precision. I suspect that gcc/ms >>>> combo is what is producing the odd results you are seeing. >>> >>> I think we might be talking past each other a bit. >>> >>> It seems to me that, if float96 must use float64 math, then it should >>> be removed from the numpy namespace, because >> >> If we were to do so, it would break too much code. > > David - please - obviously I'm not suggesting removing it without > deprecating it. Let's say I find it debatable that removing it (with all the deprecations) would be a good use of effort, especially given that there is no obviously better choice to be made. > >>> a) It implies higher precision than float64 but does not provide it >>> b) It uses more memory to no obvious advantage >> >> There is an obvious advantage: to handle memory blocks which use long >> double, created outside numpy (or even python). > > Right - but that's a bit arcane, and I would have thought > np.longdouble would be a good enough name for that. ? Of course, the > users may be surprised, as I was, that memory allocated for higher > precision is using float64, and that may take them some time to work > out. ?I'll say again that 'longdouble' says to me 'something specific > to the compiler' and 'float96' says 'something standard in numpy', and > that I - was surprised - when I found out what it was. I think the expectation is wrong, rather than the implementation :) If you use float96 on windows, each item will take 12 bytes (on 32 bits at least), so that part makes sense if you understand the number 96 as referring to its size in memory. Moreover, that's the *only* reasonable choice: if you want to create a numpy array from a C array of long double, each item needs to be separated by sizeof(long double) bytes (i.e. 12 bytes, i.e. 96 bits on x86). sizeof(long double) is the same with gcc and msvc. > >> Otherwise, while gcc indeed supports long double, the fact that the C >> runtime doesn't really mean it is hopeless to reach any kind of >> consistency. > > I'm sorry for my ignorance, but which numerical algorithms come from > the C runtime? Anything that looks like a function call. So while +, -, etc... are done by the compiler, pretty much everything else from isnan to exp is through the runtime (modulo our own replacements, which are significant on windows). > > Do you agree that the current state of float96 on Windows is hard to understand? Only if you expect float96 to behave like an actual extended precision (i.e. 80 bits). But what we actually do is to implement whatever the platform does for the C long double: on 32 bits windows, that means something that is exactly the same as double except for the fact that its sizeof is 12 instead of 8 bytes. As far as improvement, the only one I see is to have a standard-conformant quad precision implementation. cheers, David From matthew.brett at gmail.com Tue Nov 15 04:41:32 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Nov 2011 01:41:32 -0800 Subject: [Numpy-discussion] Odd-looking long double on windows 32 bit In-Reply-To: References: Message-ID: Hi, On Tue, Nov 15, 2011 at 12:51 AM, David Cournapeau wrote: > On Tue, Nov 15, 2011 at 6:22 AM, Matthew Brett wrote: >> Hi, >> >> On Mon, Nov 14, 2011 at 10:08 PM, David Cournapeau wrote: >>> On Mon, Nov 14, 2011 at 9:01 PM, Matthew Brett wrote: >>>> Hi, >>>> >>>> On Sun, Nov 13, 2011 at 5:03 PM, Charles R Harris >>>> wrote: >>>>> >>>>> >>>>> On Sun, Nov 13, 2011 at 3:56 PM, Matthew Brett >>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> On Sun, Nov 13, 2011 at 1:34 PM, Charles R Harris >>>>>> wrote: >>>>>> > >>>>>> > >>>>>> > On Sun, Nov 13, 2011 at 2:25 PM, Matthew Brett >>>>>> > wrote: >>>>>> >> >>>>>> >> Hi, >>>>>> >> >>>>>> >> On Sun, Nov 13, 2011 at 8:21 AM, Charles R Harris >>>>>> >> wrote: >>>>>> >> > >>>>>> >> > >>>>>> >> > On Sun, Nov 13, 2011 at 12:57 AM, Matthew Brett >>>>>> >> > >>>>>> >> > wrote: >>>>>> >> >> >>>>>> >> >> Hi, >>>>>> >> >> >>>>>> >> >> On Sat, Nov 12, 2011 at 11:35 PM, Matthew Brett >>>>>> >> >> >>>>>> >> >> wrote: >>>>>> >> >> > Hi, >>>>>> >> >> > >>>>>> >> >> > Sorry for my continued confusion here. ?This is numpy 1.6.1 on >>>>>> >> >> > windows >>>>>> >> >> > XP 32 bit. >>>>>> >> >> > >>>>>> >> >> > In [2]: np.finfo(np.float96).nmant >>>>>> >> >> > Out[2]: 52 >>>>>> >> >> > >>>>>> >> >> > In [3]: np.finfo(np.float96).nexp >>>>>> >> >> > Out[3]: 15 >>>>>> >> >> > >>>>>> >> >> > In [4]: np.finfo(np.float64).nmant >>>>>> >> >> > Out[4]: 52 >>>>>> >> >> > >>>>>> >> >> > In [5]: np.finfo(np.float64).nexp >>>>>> >> >> > Out[5]: 11 >>>>>> >> >> > >>>>>> >> >> > If there are 52 bits of precision, 2**53+1 should not be >>>>>> >> >> > representable, and sure enough: >>>>>> >> >> > >>>>>> >> >> > In [6]: np.float96(2**53)+1 >>>>>> >> >> > Out[6]: 9007199254740992.0 >>>>>> >> >> > >>>>>> >> >> > In [7]: np.float64(2**53)+1 >>>>>> >> >> > Out[7]: 9007199254740992.0 >>>>>> >> >> > >>>>>> >> >> > If the nexp is right, the max should be higher for the float96 >>>>>> >> >> > type: >>>>>> >> >> > >>>>>> >> >> > In [9]: np.finfo(np.float64).max >>>>>> >> >> > Out[9]: 1.7976931348623157e+308 >>>>>> >> >> > >>>>>> >> >> > In [10]: np.finfo(np.float96).max >>>>>> >> >> > Out[10]: 1.#INF >>>>>> >> >> > >>>>>> >> >> > I see that long double in C is 12 bytes wide, and double is the >>>>>> >> >> > usual >>>>>> >> >> > 8 >>>>>> >> >> > bytes. >>>>>> >> >> >>>>>> >> >> Sorry - sizeof(long double) is 12 using mingw. ?I see that long >>>>>> >> >> double >>>>>> >> >> is the same as double in MS Visual C++. >>>>>> >> >> >>>>>> >> >> http://en.wikipedia.org/wiki/Long_double >>>>>> >> >> >>>>>> >> >> but, as expected from the name: >>>>>> >> >> >>>>>> >> >> In [11]: np.dtype(np.float96).itemsize >>>>>> >> >> Out[11]: 12 >>>>>> >> >> >>>>>> >> > >>>>>> >> > Hmm, good point. There should not be a float96 on Windows using the >>>>>> >> > MSVC >>>>>> >> > compiler, and the longdouble types 'gG' should return float64 and >>>>>> >> > complex128 >>>>>> >> > respectively. OTOH, I believe the mingw compiler has real float96 >>>>>> >> > types >>>>>> >> > but >>>>>> >> > I wonder about library support. This is really a build issue and it >>>>>> >> > would be >>>>>> >> > good to have some feedback on what different platforms are doing so >>>>>> >> > that >>>>>> >> > we >>>>>> >> > know if we are doing things right. >>>>>> >> >>>>>> >> Is it possible that numpy is getting confused by being compiled with >>>>>> >> mingw on top of a visual studio python? >>>>>> >> >>>>>> >> Some further forensics seem to suggest that, despite the fact the math >>>>>> >> suggests float96 is float64, the storage format it in fact 80-bit >>>>>> >> extended precision: >>>>>> >> >>>>>> > >>>>>> > Yes, extended precision is the type on Intel hardware with gcc, the >>>>>> > 96/128 >>>>>> > bits comes from alignment on 4 or 8 byte boundaries. With MSVC, double >>>>>> > and >>>>>> > long double are both ieee double, and on SPARC, long double is ieee quad >>>>>> > precision. >>>>>> >>>>>> Right - but I think my researches are showing that the longdouble >>>>>> numbers are being _stored_ as 80 bit, but the math on those numbers is >>>>>> 64 bit. >>>>>> >>>>>> Is there a reason than numpy can't do 80-bit math on these guys? ?If >>>>>> there is, is there any point in having a float96 on windows? >>>>> >>>>> It's a compiler/architecture thing and depends on how the compiler >>>>> interprets the long double c type. The gcc compiler does do 80 bit math on >>>>> Intel/AMD hardware. MSVC doesn't, and probably never will. MSVC shouldn't >>>>> produce float96 numbers, if it does, it is a bug. Mingw uses the gcc >>>>> compiler, so the numbers are there, but if it uses the MS library it will >>>>> have to convert them to double to do computations like sin(x) since there >>>>> are no microsoft routines for extended precision. I suspect that gcc/ms >>>>> combo is what is producing the odd results you are seeing. >>>> >>>> I think we might be talking past each other a bit. >>>> >>>> It seems to me that, if float96 must use float64 math, then it should >>>> be removed from the numpy namespace, because >>> >>> If we were to do so, it would break too much code. >> >> David - please - obviously I'm not suggesting removing it without >> deprecating it. > > Let's say I find it debatable that removing it (with all the > deprecations) would be a good use of effort, especially given that > there is no obviously better choice to be made. > >> >>>> a) It implies higher precision than float64 but does not provide it >>>> b) It uses more memory to no obvious advantage >>> >>> There is an obvious advantage: to handle memory blocks which use long >>> double, created outside numpy (or even python). >> >> Right - but that's a bit arcane, and I would have thought >> np.longdouble would be a good enough name for that. ? Of course, the >> users may be surprised, as I was, that memory allocated for higher >> precision is using float64, and that may take them some time to work >> out. ?I'll say again that 'longdouble' says to me 'something specific >> to the compiler' and 'float96' says 'something standard in numpy', and >> that I - was surprised - when I found out what it was. > > I think the expectation is wrong, rather than the implementation :) If > you use float96 on windows, each item will take 12 bytes (on 32 bits > at least), so that part makes sense if you understand the number 96 as > referring to its size in memory. I recognize this from our previous float80 / float96 discussion. I think the user you have in mind does not think as I do. I wish there was a good way of knowing which user is more common, your user or my user. But anyway - nowhere much to go in the discussion - so I expect to learn to make friends with float96. See you, Matthew From shish at keba.be Tue Nov 15 07:02:30 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 15 Nov 2011 07:02:30 -0500 Subject: [Numpy-discussion] numpy.int32 is not subclass of int, but numpy.int64 is In-Reply-To: References: <2922D4459F182B41ABD8C7AE501064200F6B95D7@AG-DCC-MBX03.dsone.3ds.com> Message-ID: 2011/11/14 Robert Kern > On Mon, Nov 14, 2011 at 20:18, MACKEITH Andrew > wrote: > > Could someone explain this? > > > > An instance of numpy.int32 is not an instance of int or numpy.int. > > An instance of numpy.int64 is an instance of int and numpy.int. > > > > I don't know if it is a bug in my linux build. > > >>>> import sys > >>>> sys.maxint > > 9223372036854775807 > >>>> import platform > >>>> print platform.platform() > > Linux-2.6.32.12-0.7-default-x86_64-with-SuSE-11-x86_64 > > This is expected on a 64-bit platform. Note that numpy.int is just an > alias for the builtin int type for backwards compatibility with an > earlier version of numpy. We could probably remove it, since it seems > to be causing more confusion than not. > > Anyways, we subclass the appropriately sized integer scalar type from > Python's int type depending on the platform. So on a platform where > Python's int type is 64-bits, numpy.int64 will include int in its > inheritance tree. On platforms where the Python int type is 32-bit, > numpy.int32 will include it instead. I'll just add that there is a numpy.integer class that is parent of both numpy.int32 and numpy.int64 (see http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html). It's not a parent of numpy.int though, since as said above, numpy.int is an alias to the builtin int. -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.MACKEITH at 3ds.com Tue Nov 15 09:48:32 2011 From: Andrew.MACKEITH at 3ds.com (MACKEITH Andrew) Date: Tue, 15 Nov 2011 14:48:32 +0000 Subject: [Numpy-discussion] numpy.int32 is not subclass of int, but numpy.int64 is In-Reply-To: References: <2922D4459F182B41ABD8C7AE501064200F6B95D7@AG-DCC-MBX03.dsone.3ds.com> Message-ID: <2922D4459F182B41ABD8C7AE501064200F6B983C@AG-DCC-MBX03.dsone.3ds.com> From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Olivier Delalleau Sent: Tuesday, November 15, 2011 7:03 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy.int32 is not subclass of int, but numpy.int64 is 2011/11/14 Robert Kern > On Mon, Nov 14, 2011 at 20:18, MACKEITH Andrew > wrote: > Could someone explain this? > > An instance of numpy.int32 is not an instance of int or numpy.int. > An instance of numpy.int64 is an instance of int and numpy.int. > > I don't know if it is a bug in my linux build. >>>> import sys >>>> sys.maxint > 9223372036854775807 >>>> import platform >>>> print platform.platform() > Linux-2.6.32.12-0.7-default-x86_64-with-SuSE-11-x86_64 This is expected on a 64-bit platform. Note that numpy.int is just an alias for the builtin int type for backwards compatibility with an earlier version of numpy. We could probably remove it, since it seems to be causing more confusion than not. Anyways, we subclass the appropriately sized integer scalar type from Python's int type depending on the platform. So on a platform where Python's int type is 64-bits, numpy.int64 will include int in its inheritance tree. On platforms where the Python int type is 32-bit, numpy.int32 will include it instead. I'll just add that there is a numpy.integer class that is parent of both numpy.int32 and numpy.int64 (see http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html). It's not a parent of numpy.int though, since as said above, numpy.int is an alias to the builtin int. -=- Olivier Thanks you for the information. numpy.integer is what I was looking for. Is there an equivalent base class for float types? Do you know where these are documented? Andrew This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Tue Nov 15 10:01:33 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 15 Nov 2011 10:01:33 -0500 Subject: [Numpy-discussion] numpy.int32 is not subclass of int, but numpy.int64 is In-Reply-To: <2922D4459F182B41ABD8C7AE501064200F6B983C@AG-DCC-MBX03.dsone.3ds.com> References: <2922D4459F182B41ABD8C7AE501064200F6B95D7@AG-DCC-MBX03.dsone.3ds.com> <2922D4459F182B41ABD8C7AE501064200F6B983C@AG-DCC-MBX03.dsone.3ds.com> Message-ID: 2011/11/15 MACKEITH Andrew > *From:* numpy-discussion-bounces at scipy.org [mailto: > numpy-discussion-bounces at scipy.org] *On Behalf Of *Olivier Delalleau > *Sent:* Tuesday, November 15, 2011 7:03 AM > *To:* Discussion of Numerical Python > *Subject:* Re: [Numpy-discussion] numpy.int32 is not subclass of int, but > numpy.int64 is > > > > 2011/11/14 Robert Kern > > On Mon, Nov 14, 2011 at 20:18, MACKEITH Andrew > wrote: > > Could someone explain this? > > > > An instance of numpy.int32 is not an instance of int or numpy.int. > > An instance of numpy.int64 is an instance of int and numpy.int. > > > > I don't know if it is a bug in my linux build. > > >>>> import sys > >>>> sys.maxint > > 9223372036854775807 > >>>> import platform > >>>> print platform.platform() > > Linux-2.6.32.12-0.7-default-x86_64-with-SuSE-11-x86_64 > > This is expected on a 64-bit platform. Note that numpy.int is just an > alias for the builtin int type for backwards compatibility with an > earlier version of numpy. We could probably remove it, since it seems > to be causing more confusion than not. > > Anyways, we subclass the appropriately sized integer scalar type from > Python's int type depending on the platform. So on a platform where > Python's int type is 64-bits, numpy.int64 will include int in its > inheritance tree. On platforms where the Python int type is 32-bit, > numpy.int32 will include it instead. > > > I'll just add that there is a numpy.integer class that is parent of both > numpy.int32 and numpy.int64 (see > http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html). It's not > a parent of numpy.int though, since as said above, numpy.int is an alias > to the builtin int. > > -=- Olivier > > > > Thanks you for the information. numpy.integer is what I was looking for. > > > > Is there an equivalent base class for float types? > > Do you know where these are documented? > > > > Andrew > numpy.floating would be the one for non-complex float types (and numpy.inexact the parent for both complex and non-complex). The class hierarchy is shown in the link I provided in my previous mail. -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Tue Nov 15 10:28:47 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 15 Nov 2011 09:28:47 -0600 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: <4EC13C2D.2070504@ais.uni-bonn.de> References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> Message-ID: <4EC2852F.9050000@gmail.com> On 11/14/2011 10:05 AM, Andreas M?ller wrote: > On 11/14/2011 04:23 PM, David Cournapeau wrote: >> On Mon, Nov 14, 2011 at 12:46 PM, Andreas M?ller >> wrote: >>> Hi everybody. >>> When I did some normalization using numpy, I noticed that numpy.std uses >>> more ram than I was expecting. >>> A quick google search gave me this: >>> http://luispedro.org/software/ncreduce >>> The site claims that std and other reduce operations are implemented >>> naively with many temporaries. >>> Is that true? And if so, is there a particular reason for that? >>> This issues seems quite easy to fix. >>> In particular the link I gave above provides code. >> The code provided only implements a few special cases: being more >> efficient in those cases only is indeed easy. > I am particularly interested in the std function. > Is this implemented as a separate function or an instantiation > of a general reduce operations? > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion The'On-line algorithm' (http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm) could save you storage. I would presume if you know cython that you can probably make it quick as well (to address the loop over the data). Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.MACKEITH at 3ds.com Tue Nov 15 11:44:14 2011 From: Andrew.MACKEITH at 3ds.com (MACKEITH Andrew) Date: Tue, 15 Nov 2011 16:44:14 +0000 Subject: [Numpy-discussion] numpy.int32 is not subclass of int, but numpy.int64 is In-Reply-To: References: <2922D4459F182B41ABD8C7AE501064200F6B95D7@AG-DCC-MBX03.dsone.3ds.com> <2922D4459F182B41ABD8C7AE501064200F6B983C@AG-DCC-MBX03.dsone.3ds.com> Message-ID: <2922D4459F182B41ABD8C7AE501064200F6B9925@AG-DCC-MBX03.dsone.3ds.com> From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Olivier Delalleau Sent: Tuesday, November 15, 2011 10:02 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy.int32 is not subclass of int, but numpy.int64 is 2011/11/15 MACKEITH Andrew > From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Olivier Delalleau Sent: Tuesday, November 15, 2011 7:03 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy.int32 is not subclass of int, but numpy.int64 is 2011/11/14 Robert Kern > On Mon, Nov 14, 2011 at 20:18, MACKEITH Andrew > wrote: > Could someone explain this? > > An instance of numpy.int32 is not an instance of int or numpy.int. > An instance of numpy.int64 is an instance of int and numpy.int. > > I don't know if it is a bug in my linux build. >>>> import sys >>>> sys.maxint > 9223372036854775807 >>>> import platform >>>> print platform.platform() > Linux-2.6.32.12-0.7-default-x86_64-with-SuSE-11-x86_64 This is expected on a 64-bit platform. Note that numpy.int is just an alias for the builtin int type for backwards compatibility with an earlier version of numpy. We could probably remove it, since it seems to be causing more confusion than not. Anyways, we subclass the appropriately sized integer scalar type from Python's int type depending on the platform. So on a platform where Python's int type is 64-bits, numpy.int64 will include int in its inheritance tree. On platforms where the Python int type is 32-bit, numpy.int32 will include it instead. I'll just add that there is a numpy.integer class that is parent of both numpy.int32 and numpy.int64 (see http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html). It's not a parent of numpy.int though, since as said above, numpy.int is an alias to the builtin int. -=- Olivier Thanks you for the information. numpy.integer is what I was looking for. Is there an equivalent base class for float types? Do you know where these are documented? Andrew numpy.floating would be the one for non-complex float types (and numpy.inexact the parent for both complex and non-complex). The class hierarchy is shown in the link I provided in my previous mail. -=- Olivier Thanks. Andrew This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer -------------- next part -------------- An HTML attachment was scrubbed... URL: From amueller at ais.uni-bonn.de Tue Nov 15 11:46:52 2011 From: amueller at ais.uni-bonn.de (=?UTF-8?B?QW5kcmVhcyBNw7xsbGVy?=) Date: Tue, 15 Nov 2011 17:46:52 +0100 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: <4EC2852F.9050000@gmail.com> References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> <4EC2852F.9050000@gmail.com> Message-ID: <4EC2977C.1010009@ais.uni-bonn.de> On 11/15/2011 04:28 PM, Bruce Southey wrote: > On 11/14/2011 10:05 AM, Andreas M?ller wrote: >> On 11/14/2011 04:23 PM, David Cournapeau wrote: >>> On Mon, Nov 14, 2011 at 12:46 PM, Andreas M?ller >>> wrote: >>>> Hi everybody. >>>> When I did some normalization using numpy, I noticed that numpy.std uses >>>> more ram than I was expecting. >>>> A quick google search gave me this: >>>> http://luispedro.org/software/ncreduce >>>> The site claims that std and other reduce operations are implemented >>>> naively with many temporaries. >>>> Is that true? And if so, is there a particular reason for that? >>>> This issues seems quite easy to fix. >>>> In particular the link I gave above provides code. >>> The code provided only implements a few special cases: being more >>> efficient in those cases only is indeed easy. >> I am particularly interested in the std function. >> Is this implemented as a separate function or an instantiation >> of a general reduce operations? >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > The'On-line algorithm' > (http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm) > > could save you storage. I would presume if you know cython that you > can probably make it quick as well (to address the loop over the data). > My question was more along the lines of "why doesn't numpy do the online algorithm". -------------- next part -------------- An HTML attachment was scrubbed... URL: From amueller at ais.uni-bonn.de Tue Nov 15 11:48:44 2011 From: amueller at ais.uni-bonn.de (=?UTF-8?B?QW5kcmVhcyBNw7xsbGVy?=) Date: Tue, 15 Nov 2011 17:48:44 +0100 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: <4EC2977C.1010009@ais.uni-bonn.de> References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> <4EC2852F.9050000@gmail.com> <4EC2977C.1010009@ais.uni-bonn.de> Message-ID: <4EC297EC.6000205@ais.uni-bonn.de> On 11/15/2011 05:46 PM, Andreas M?ller wrote: > On 11/15/2011 04:28 PM, Bruce Southey wrote: >> On 11/14/2011 10:05 AM, Andreas M?ller wrote: >>> On 11/14/2011 04:23 PM, David Cournapeau wrote: >>>> On Mon, Nov 14, 2011 at 12:46 PM, Andreas M?ller >>>> wrote: >>>>> Hi everybody. >>>>> When I did some normalization using numpy, I noticed that numpy.std uses >>>>> more ram than I was expecting. >>>>> A quick google search gave me this: >>>>> http://luispedro.org/software/ncreduce >>>>> The site claims that std and other reduce operations are implemented >>>>> naively with many temporaries. >>>>> Is that true? And if so, is there a particular reason for that? >>>>> This issues seems quite easy to fix. >>>>> In particular the link I gave above provides code. >>>> The code provided only implements a few special cases: being more >>>> efficient in those cases only is indeed easy. >>> I am particularly interested in the std function. >>> Is this implemented as a separate function or an instantiation >>> of a general reduce operations? >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> The'On-line algorithm' >> (http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm) >> >> could save you storage. I would presume if you know cython that you >> can probably make it quick as well (to address the loop over the data). >> > > My question was more along the lines of "why doesn't numpy do the > online algorithm". > To be more precise, even not using the online version but computing E(X^2) and E(X)^2 would be good. It seems numpy centers the whole dataset. Otherwise I can't explain why the memory needed should depend on the number of examples. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Nov 15 11:55:35 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 15 Nov 2011 17:55:35 +0100 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: <4EC2977C.1010009@ais.uni-bonn.de> References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> <4EC2852F.9050000@gmail.com> <4EC2977C.1010009@ais.uni-bonn.de> Message-ID: <20111115165535.GA17201@phare.normalesup.org> On Tue, Nov 15, 2011 at 05:46:52PM +0100, Andreas M?ller wrote: > My question was more along the lines of "why doesn't numpy do the online > algorithm". It's probably a matter of nobody having had the time and the urge to code it, _and_ do all the extra steps necessary to integrate to the core numpy: working on n dimensions, suporting all the options of the current implementation, doing the tests, doing the docs. I don't think that anybody would frown upon a pull request that implements a better algorithm. It's just work, as you know given your contributions to other project. Gael From warren.weckesser at enthought.com Tue Nov 15 12:02:06 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 15 Nov 2011 11:02:06 -0600 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: <4EC297EC.6000205@ais.uni-bonn.de> References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> <4EC2852F.9050000@gmail.com> <4EC2977C.1010009@ais.uni-bonn.de> <4EC297EC.6000205@ais.uni-bonn.de> Message-ID: On Tue, Nov 15, 2011 at 10:48 AM, Andreas M?ller wrote: > ** > On 11/15/2011 05:46 PM, Andreas M?ller wrote: > > On 11/15/2011 04:28 PM, Bruce Southey wrote: > > On 11/14/2011 10:05 AM, Andreas M?ller wrote: > > On 11/14/2011 04:23 PM, David Cournapeau wrote: > > On Mon, Nov 14, 2011 at 12:46 PM, Andreas M?ller wrote: > > Hi everybody. > When I did some normalization using numpy, I noticed that numpy.std uses > more ram than I was expecting. > A quick google search gave me this:http://luispedro.org/software/ncreduce > The site claims that std and other reduce operations are implemented > naively with many temporaries. > Is that true? And if so, is there a particular reason for that? > This issues seems quite easy to fix. > In particular the link I gave above provides code. > > The code provided only implements a few special cases: being more > efficient in those cases only is indeed easy. > > I am particularly interested in the std function. > Is this implemented as a separate function or an instantiation > of a general reduce operations? > > _______________________________________________ > NumPy-Discussion mailing listNumPy-Discussion at scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion > > The 'On-line algorithm' ( > http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm)could save you storage. I would presume if you know cython that you can > probably make it quick as well (to address the loop over the data). > > > My question was more along the lines of "why doesn't numpy do the online > algorithm". > > To be more precise, even not using the online version but computing > E(X^2) and E(X)^2 would be good. > It seems numpy centers the whole dataset. Otherwise I can't explain why > the memory needed should depend > on the number of examples. > Yes, that is what it is doing. See line 63 in the function _var(), which is called by _std(): https://github.com/numpy/numpy/blob/master/numpy/core/_methods.py Warren > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amueller at ais.uni-bonn.de Tue Nov 15 12:07:54 2011 From: amueller at ais.uni-bonn.de (=?UTF-8?B?QW5kcmVhcyBNw7xsbGVy?=) Date: Tue, 15 Nov 2011 18:07:54 +0100 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> <4EC2852F.9050000@gmail.com> <4EC2977C.1010009@ais.uni-bonn.de> <4EC297EC.6000205@ais.uni-bonn.de> Message-ID: <4EC29C6A.8050802@ais.uni-bonn.de> On 11/15/2011 06:02 PM, Warren Weckesser wrote: > > > On Tue, Nov 15, 2011 at 10:48 AM, Andreas M?ller > > wrote: > > On 11/15/2011 05:46 PM, Andreas M?ller wrote: >> On 11/15/2011 04:28 PM, Bruce Southey wrote: >>> On 11/14/2011 10:05 AM, Andreas M?ller wrote: >>>> On 11/14/2011 04:23 PM, David Cournapeau wrote: >>>>> On Mon, Nov 14, 2011 at 12:46 PM, Andreas M?ller >>>>> wrote: >>>>>> Hi everybody. >>>>>> When I did some normalization using numpy, I noticed that numpy.std uses >>>>>> more ram than I was expecting. >>>>>> A quick google search gave me this: >>>>>> http://luispedro.org/software/ncreduce >>>>>> The site claims that std and other reduce operations are implemented >>>>>> naively with many temporaries. >>>>>> Is that true? And if so, is there a particular reason for that? >>>>>> This issues seems quite easy to fix. >>>>>> In particular the link I gave above provides code. >>>>> The code provided only implements a few special cases: being more >>>>> efficient in those cases only is indeed easy. >>>> I am particularly interested in the std function. >>>> Is this implemented as a separate function or an instantiation >>>> of a general reduce operations? >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> The'On-line algorithm' >>> (http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm) >>> >>> could save you storage. I would presume if you know cython that >>> you can probably make it quick as well (to address the loop over >>> the data). >>> >> >> My question was more along the lines of "why doesn't numpy do the >> online algorithm". >> > To be more precise, even not using the online version but > computing E(X^2) and E(X)^2 would be good. > It seems numpy centers the whole dataset. Otherwise I can't > explain why the memory needed should depend > on the number of examples. > > > > Yes, that is what it is doing. See line 63 in the function _var(), > which is called by _std(): > https://github.com/numpy/numpy/blob/master/numpy/core/_methods.py > Thanks for the clarification. I thought the function was somewhere in the C code - don't know why. I'll see if I can reformulate the function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Nov 15 12:14:47 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Tue, 15 Nov 2011 09:14:47 -0800 Subject: [Numpy-discussion] dedicated function for resize with averaging or rebin 2d arrays? In-Reply-To: References: <9DD9103D-D5CB-489A-A1CF-0273367651E1@me.com> <4EC14ED8.6080307@noaa.gov> Message-ID: <4EC29E07.5010707@noaa.gov> On 11/14/11 2:02 PM, Andrea Zonca wrote: > On Mon, Nov 14, 2011 at 09:24, Chris.Barker wrote: >> On 11/11/11 8:28 PM, Craig Yoshioka wrote: >>> I once wrote a generic n-dimensional binning routine in C that I >>> could find if anyone is interested in integrating it into numpy... it >>> didn't do size increases though... and I think I implemented it so >>> that binning by a non-divisible factor trimmed the extras. It was >>> very-very fast though. >> >> And I've got one written in Cython. I'd been thinking of contributing it >> to ndimage, but never got around to it -- but I'd be glad to share. Let >> me know if you want it. > > Chris, I think this would be useful, can you make it available publicly? hmm -- I thought I had a more generic version, but can't find it at the moment. So here is a cython file with a couple utilities in it -- only useful for one thing now, but it could be used as a starting point for a more general purpose function. Note that the trick to top performance with Cython is to specify the types of your inputs -- in this case np.uint8 for an RGB image. You could write it totally generic, but wouldn't get the big speed up you might want. Maybe write a generic version, and special case a couple types for the common case -- or do a templating thing like the bottleneck project. This code is in the public domain -- do with it whatever you want. -Chris >>>> I think the usefulness of the rebin function is to be simple and fast, >>>> and the best would be to implement it in the core numpy, as a simple >>>> method for smoothing and reshaping, >> >> Does it have common use=-cases outside image processing? If not, then >> no, it shouldn't be in numpy. > > actually the most common use is outside of image processing, we typically > use it when we have data that come from a detector scanning the sky in > circles, so you have a row for each scan. > It is nice to apply a quick and dirt decimation of the dataset by binning it. > It is such a simple function that would make more sense, to me, to > have in numpy. > In image processing I imagine you typically want to apply window functions, > so for that ndimage is more appropriate. > > cheers, > andrea > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Bitmap.pyx URL: From robert.kern at gmail.com Tue Nov 15 12:57:14 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 15 Nov 2011 17:57:14 +0000 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: <20111115165535.GA17201@phare.normalesup.org> References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> <4EC2852F.9050000@gmail.com> <4EC2977C.1010009@ais.uni-bonn.de> <20111115165535.GA17201@phare.normalesup.org> Message-ID: On Tue, Nov 15, 2011 at 16:55, Gael Varoquaux wrote: > On Tue, Nov 15, 2011 at 05:46:52PM +0100, Andreas M?ller wrote: >> ? ?My question was more along the lines of "why doesn't numpy do the online >> ? ?algorithm". > > It's probably a matter of nobody having had the time and the urge to code > it, _and_ do all the extra steps necessary to integrate to the core > numpy: working on n dimensions, suporting all the options of the current > implementation, doing the tests, doing the docs. > > I don't think that anybody would frown upon a pull request that > implements a better algorithm. It's just work, as you know given your > contributions to other project. Actually, last time I suggested it, it was brought up that the online algorithms can be worse numerically. I'll try to find the thread. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From gael.varoquaux at normalesup.org Tue Nov 15 13:03:48 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 15 Nov 2011 19:03:48 +0100 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> <4EC2852F.9050000@gmail.com> <4EC2977C.1010009@ais.uni-bonn.de> <20111115165535.GA17201@phare.normalesup.org> Message-ID: <20111115180348.GB9566@phare.normalesup.org> On Tue, Nov 15, 2011 at 05:57:14PM +0000, Robert Kern wrote: > Actually, last time I suggested it, it was brought up that the online > algorithms can be worse numerically. I'll try to find the thread. Indeed, especially for smallish datasets where the memory overhead is not an issue. I think that this is a common situation for which there is not _one_ good algorithm to solve the problem. IMHO the best way forward is to propose several options to the user, either with several function, or with a keyword argument. Gael From amueller at ais.uni-bonn.de Tue Nov 15 13:17:58 2011 From: amueller at ais.uni-bonn.de (=?ISO-8859-1?Q?Andreas_M=FCller?=) Date: Tue, 15 Nov 2011 19:17:58 +0100 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: <20111115180348.GB9566@phare.normalesup.org> References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> <4EC2852F.9050000@gmail.com> <4EC2977C.1010009@ais.uni-bonn.de> <20111115165535.GA17201@phare.normalesup.org> <20111115180348.GB9566@phare.normalesup.org> Message-ID: <4EC2ACD6.4000907@ais.uni-bonn.de> On 11/15/2011 07:03 PM, Gael Varoquaux wrote: > On Tue, Nov 15, 2011 at 05:57:14PM +0000, Robert Kern wrote: >> Actually, last time I suggested it, it was brought up that the online >> algorithms can be worse numerically. I'll try to find the thread. > Indeed, especially for smallish datasets where the memory overhead is not > an issue. I think that this is a common situation for which there is not > _one_ good algorithm to solve the problem. IMHO the best way forward is > to propose several options to the user, either with several function, or > with a keyword argument. > I thought it would be possible to do the "two pass algorithm" (as wikipedia calls it) without copying the dataset. I guess I didn't really think it through. That should be possible doing a custom reduce operation, right? I don't know in how far this would have numerical problems. From bsouthey at gmail.com Tue Nov 15 17:07:37 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 15 Nov 2011 16:07:37 -0600 Subject: [Numpy-discussion] Memory hungry reduce ops in Numpy In-Reply-To: References: <4EC10DBA.6080803@ais.uni-bonn.de> <4EC13C2D.2070504@ais.uni-bonn.de> <4EC2852F.9050000@gmail.com> <4EC2977C.1010009@ais.uni-bonn.de> <20111115165535.GA17201@phare.normalesup.org> Message-ID: On Tue, Nov 15, 2011 at 11:57 AM, Robert Kern wrote: > On Tue, Nov 15, 2011 at 16:55, Gael Varoquaux > wrote: >> On Tue, Nov 15, 2011 at 05:46:52PM +0100, Andreas M?ller wrote: >>> ? ?My question was more along the lines of "why doesn't numpy do the online >>> ? ?algorithm". >> >> It's probably a matter of nobody having had the time and the urge to code >> it, _and_ do all the extra steps necessary to integrate to the core >> numpy: working on n dimensions, suporting all the options of the current >> implementation, doing the tests, doing the docs. >> >> I don't think that anybody would frown upon a pull request that >> implements a better algorithm. It's just work, as you know given your >> contributions to other project. > > Actually, last time I suggested it, it was brought up that the online > algorithms can be worse numerically. I'll try to find the thread. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > There was this one: http://mail.scipy.org/pipermail/numpy-discussion/2010-November/054007.html I think part of the problem there was numerical precision as the difference is about right for double precision. Bruce From stefan at sun.ac.za Tue Nov 15 19:46:46 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 15 Nov 2011 16:46:46 -0800 Subject: [Numpy-discussion] Using dtype=object In-Reply-To: References: Message-ID: On Mon, Nov 14, 2011 at 9:08 PM, Nathan Faggian wrote: > I am interested in the use of numpy with native python objects, like so: > > In [91]: import collections > In [92]: testContainer = collections.namedtuple('testContainer', 'att1 > att2 att3') > In [93]: test1 = testContainer(1, 2, 3) > In [94]: test2 = testContainer(4, 5, 6) > In [95]: test1 > Out[95]: testContainer(att1=1, att2=2, att3=3) > In [96]: test2 > Out[96]: testContainer(att1=4, att2=5, att3=6) > In [97]: x = np.empty((2,2), dtype=object) > In [98]: x[0,0] = test1 > In [99]: x[1,1] = test2 > In [100]: x > Out[100]: > array([[testContainer(att1=1, att2=2, att3=3), None], > [None, testContainer(att1=4, att2=5, att3=6)]], dtype=object) > > Does anyone know if it possible to form a mask using the attributes of > the objects stored in the ndarray? > Maybe something like this: def attr_equal(attribute, value): def _func(x): if x is None: return False else: return getattr(x, attribute) == value return np.vectorize(_func) print attr_equal('att2', 5)(x) print attr_equal('att1', 1)(x) Regards St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.raybaut at gmail.com Wed Nov 16 04:43:00 2011 From: pierre.raybaut at gmail.com (Pierre Raybaut) Date: Wed, 16 Nov 2011 10:43:00 +0100 Subject: [Numpy-discussion] ANN: Spyder v2.1.2 Message-ID: Hi all, On the behalf of Spyder's development team (http://code.google.com/p/spyderlib/people/list), I'm pleased to announce that Spyder v2.1.2 has been released and is available for Windows XP/Vista/7, GNU/Linux and MacOS X: http://code.google.com/p/spyderlib/ As this is mostly a maintenance release, a lot of bugs were fixed and some minor features were added: http://code.google.com/p/spyderlib/wiki/ChangeLog Spyder is a free, open-source (MIT license) interactive development environment for the Python language with advanced editing, interactive testing, debugging and introspection features. Originally designed to provide MATLAB-like features (integrated help, interactive console, variable explorer with GUI-based editors for dictionaries, NumPy arrays, ...), it is strongly oriented towards scientific computing and software development. Thanks to the `spyderlib` library, Spyder also provides powerful ready-to-use widgets: embedded Python console (example: http://packages.python.org/guiqwt/_images/sift3.png), NumPy array editor (example: http://packages.python.org/guiqwt/_images/sift2.png), dictionary editor, source code editor, etc. Description of key features with tasty screenshots can be found at: http://code.google.com/p/spyderlib/wiki/Features On Windows platforms, Spyder is also available as a stand-alone executable (don't forget to disable UAC on Vista/7). This all-in-one portable version is still experimental (for example, it does not embed sphinx -- meaning no rich text mode for the object inspector) but it should provide a working version of Spyder for Windows platforms without having to install anything else (except Python 2.x itself, of course). Don't forget to follow Spyder updates/news: * on the project website: http://code.google.com/p/spyderlib/ * and on our official blog: http://spyder-ide.blogspot.com/ Last, but not least, we welcome any contribution that helps making Spyder an efficient scientific development/computing environment. Join us to help creating your favourite environment! (http://code.google.com/p/spyderlib/wiki/NoteForContributors) Enjoy! -Pierre From d.s.seljebotn at astro.uio.no Fri Nov 18 06:19:43 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 18 Nov 2011 12:19:43 +0100 Subject: [Numpy-discussion] BSD C port of FFTPACK incl. bluestein algorithm Message-ID: <4EC63F4F.7030405@astro.uio.no> I've been in touch with Martin Reinecke, author of the libpsht code for spherical harmonic transforms, about licensing issues. libpsht itself will remain under the GPL, but he is likely to release his C port of FFTPACK under BSD in the near future, as it is based on the public domain FFTPACK. I'm grateful for this change for my own purposes (allows releasing my own competing SHT library under the BSD) -- but it could perhaps be useful for NumPy or SciPy as well, depending on how complete the port is? E.g., perhaps make numpy.fft more complete (is the numpy.fft/scipy.fftpack split simply because of the Fortran dependency?). Not sure about whether all of FFTPACK or a subset is included, but it does include a Bluestein implementation in addition. http://libpsht.svn.sourceforge.net/viewvc/libpsht/libfftpack/ Dag Sverre From robert.kern at gmail.com Fri Nov 18 06:42:50 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 18 Nov 2011 11:42:50 +0000 Subject: [Numpy-discussion] BSD C port of FFTPACK incl. bluestein algorithm In-Reply-To: <4EC63F4F.7030405@astro.uio.no> References: <4EC63F4F.7030405@astro.uio.no> Message-ID: On Fri, Nov 18, 2011 at 11:19, Dag Sverre Seljebotn wrote: > I've been in touch with Martin Reinecke, author of the libpsht code for > spherical harmonic transforms, about licensing issues. > > libpsht itself will remain under the GPL, but he is likely to release > his C port of FFTPACK under BSD in the near future, as it is based on > the public domain FFTPACK. > > I'm grateful for this change for my own purposes (allows releasing my > own competing SHT library under the BSD) -- but it could perhaps be > useful for NumPy or SciPy as well, depending on how complete the port > is? E.g., perhaps make numpy.fft more complete (is the > numpy.fft/scipy.fftpack split simply because of the Fortran dependency?). It used to be the case that scipy.fftpack allowed one to build against multiple different, usually faster, FFT libraries like FFTW. I think we have backed away from that since the cost of maintaining the build configuration for all of those different backends was so high. It's worth noting that numpy.fft is already using a C translation of FFTPACK. I'm not sure what the differences are between this translation and Martin's. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From cournape at gmail.com Fri Nov 18 06:58:54 2011 From: cournape at gmail.com (David Cournapeau) Date: Fri, 18 Nov 2011 11:58:54 +0000 Subject: [Numpy-discussion] BSD C port of FFTPACK incl. bluestein algorithm In-Reply-To: References: <4EC63F4F.7030405@astro.uio.no> Message-ID: On Fri, Nov 18, 2011 at 11:42 AM, Robert Kern wrote: > On Fri, Nov 18, 2011 at 11:19, Dag Sverre Seljebotn > wrote: >> I've been in touch with Martin Reinecke, author of the libpsht code for >> spherical harmonic transforms, about licensing issues. >> >> libpsht itself will remain under the GPL, but he is likely to release >> his C port of FFTPACK under BSD in the near future, as it is based on >> the public domain FFTPACK. >> >> I'm grateful for this change for my own purposes (allows releasing my >> own competing SHT library under the BSD) -- but it could perhaps be >> useful for NumPy or SciPy as well, depending on how complete the port >> is? E.g., perhaps make numpy.fft more complete (is the >> numpy.fft/scipy.fftpack split simply because of the Fortran dependency?). > > It used to be the case that scipy.fftpack allowed one to build against > multiple different, usually faster, FFT libraries like FFTW. I think > we have backed away from that since the cost of maintaining the build > configuration for all of those different backends was so high. It's > worth noting that numpy.fft is already using a C translation of > FFTPACK. I'm not sure what the differences are between this > translation and Martin's. Having a Bluestein transformation alone would be worthwhile, as it would avoid the N^2 penalty for prime sizes. I am wondering about precision issues, though (when I tried implementing bluestein transforms on top of fftpack, it gave very bad results numerically-wise). A comparison with fftw would be good here. regards, David From d.s.seljebotn at astro.uio.no Fri Nov 18 07:18:10 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 18 Nov 2011 13:18:10 +0100 Subject: [Numpy-discussion] BSD C port of FFTPACK incl. bluestein algorithm In-Reply-To: References: <4EC63F4F.7030405@astro.uio.no> Message-ID: <4EC64D02.5000007@astro.uio.no> On 11/18/2011 12:58 PM, David Cournapeau wrote: > On Fri, Nov 18, 2011 at 11:42 AM, Robert Kern wrote: >> On Fri, Nov 18, 2011 at 11:19, Dag Sverre Seljebotn >> wrote: >>> I've been in touch with Martin Reinecke, author of the libpsht code for >>> spherical harmonic transforms, about licensing issues. >>> >>> libpsht itself will remain under the GPL, but he is likely to release >>> his C port of FFTPACK under BSD in the near future, as it is based on >>> the public domain FFTPACK. >>> >>> I'm grateful for this change for my own purposes (allows releasing my >>> own competing SHT library under the BSD) -- but it could perhaps be >>> useful for NumPy or SciPy as well, depending on how complete the port >>> is? E.g., perhaps make numpy.fft more complete (is the >>> numpy.fft/scipy.fftpack split simply because of the Fortran dependency?). >> >> It used to be the case that scipy.fftpack allowed one to build against >> multiple different, usually faster, FFT libraries like FFTW. I think >> we have backed away from that since the cost of maintaining the build >> configuration for all of those different backends was so high. It's >> worth noting that numpy.fft is already using a C translation of >> FFTPACK. I'm not sure what the differences are between this >> translation and Martin's. Here's some more info forwarded from Martin: """ - only FFTs are supported (no DCTs/DSTs) - only double precision is supported (extension to single precision might not be much work, though) - both complex and real FFTs are supported - real FFTs allow various storage schemes for the (half)complex frequency domain data (classic FFTPACK scheme, FFTW or halfcomplex scheme, uncompressed complex storage) - precision of transforms involving large prime factors should be slightly better than with original FFTPACK - Bluestein's algorithm is automatically selected if considered profitable - small accuracy self-testing code is provided. Fairly complete interface documentation is available in Doxygen format. I'll prepare a source package later in the afternoon and send it around. Best regards, Martin """ > > Having a Bluestein transformation alone would be worthwhile, as it > would avoid the N^2 penalty for prime sizes. > > I am wondering about precision issues, though (when I tried > implementing bluestein transforms on top of fftpack, it gave very bad > results numerically-wise). A comparison with fftw would be good here. Well, there's an indirect comparison: My SHT code currently uses FFTW3, and it manages to agree with Reinecke's SHT code to a precision of better than ~1e-12 for full SHTs. That includes several other sources of errors though. (That's an average over several different-sized FFTs, of which half has n=8192 and the other half all have different size, starting from 4 and increasing up to 8192 in steps of 4 -- meaning prime factors on the order of 1000). I agree, a more direct comparison with FFTW would be good. In more detail from the README: I replaced the iterative sine and cosine calculations in radfg() and radbg() 17 by an exact calculation, which slightly improves the transform accuracy for 18 real FFTs with lengths containing large prime factors. Dag Sverre From ndbecker2 at gmail.com Fri Nov 18 08:12:33 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 18 Nov 2011 08:12:33 -0500 Subject: [Numpy-discussion] magma? Message-ID: Is there any possibility of incorporating this work into numpy? http://icl.cs.utk.edu/magma/software/index.html From robert.kern at gmail.com Fri Nov 18 08:38:48 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 18 Nov 2011 13:38:48 +0000 Subject: [Numpy-discussion] magma? In-Reply-To: References: Message-ID: On Fri, Nov 18, 2011 at 13:12, Neal Becker wrote: > Is there any possibility of incorporating this work into numpy? > > http://icl.cs.utk.edu/magma/software/index.html Into numpy itself? Very low probability, I think. The dependency on nVidia hardware and proprietary compilers makes supporting it as a build option in numpy itself too much effort for too little gain, not to mention the heuristics necessary to decide when to switch from the CPU implementations to the GPU implementations. The best value/cost ratio can probably be achieved by a separate package that wraps these routines with functions signature-compatible with those in numpy.linalg. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From charlesr.harris at gmail.com Fri Nov 18 10:08:37 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 18 Nov 2011 08:08:37 -0700 Subject: [Numpy-discussion] BSD C port of FFTPACK incl. bluestein algorithm In-Reply-To: <4EC64D02.5000007@astro.uio.no> References: <4EC63F4F.7030405@astro.uio.no> <4EC64D02.5000007@astro.uio.no> Message-ID: On Fri, Nov 18, 2011 at 5:18 AM, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > On 11/18/2011 12:58 PM, David Cournapeau wrote: > > On Fri, Nov 18, 2011 at 11:42 AM, Robert Kern > wrote: > >> On Fri, Nov 18, 2011 at 11:19, Dag Sverre Seljebotn > >> wrote: > >>> I've been in touch with Martin Reinecke, author of the libpsht code for > >>> spherical harmonic transforms, about licensing issues. > >>> > >>> libpsht itself will remain under the GPL, but he is likely to release > >>> his C port of FFTPACK under BSD in the near future, as it is based on > >>> the public domain FFTPACK. > >>> > >>> I'm grateful for this change for my own purposes (allows releasing my > >>> own competing SHT library under the BSD) -- but it could perhaps be > >>> useful for NumPy or SciPy as well, depending on how complete the port > >>> is? E.g., perhaps make numpy.fft more complete (is the > >>> numpy.fft/scipy.fftpack split simply because of the Fortran > dependency?). > >> > >> It used to be the case that scipy.fftpack allowed one to build against > >> multiple different, usually faster, FFT libraries like FFTW. I think > >> we have backed away from that since the cost of maintaining the build > >> configuration for all of those different backends was so high. It's > >> worth noting that numpy.fft is already using a C translation of > >> FFTPACK. I'm not sure what the differences are between this > >> translation and Martin's. > > Here's some more info forwarded from Martin: > > """ > - only FFTs are supported (no DCTs/DSTs) > - only double precision is supported (extension to single precision might > not be much work, though) > - both complex and real FFTs are supported > - real FFTs allow various storage schemes for the (half)complex frequency > domain data (classic FFTPACK scheme, FFTW or halfcomplex scheme, > uncompressed complex storage) > - precision of transforms involving large prime factors should be slightly > better than with original FFTPACK > - Bluestein's algorithm is automatically selected if considered profitable > - small accuracy self-testing code is provided. > > Fairly complete interface documentation is available in Doxygen format. > I'll prepare a source package later in the afternoon and send it around. > > Best regards, > Martin > """ > > > > > Having a Bluestein transformation alone would be worthwhile, as it > > would avoid the N^2 penalty for prime sizes. > > > > I am wondering about precision issues, though (when I tried > > implementing bluestein transforms on top of fftpack, it gave very bad > > results numerically-wise). A comparison with fftw would be good here. > > Well, there's an indirect comparison: My SHT code currently uses FFTW3, > and it manages to agree with Reinecke's SHT code to a precision of > better than ~1e-12 for full SHTs. That includes several other sources of > errors though. > > (That's an average over several different-sized FFTs, of which half has > n=8192 and the other half all have different size, starting from 4 and > increasing up to 8192 in steps of 4 -- meaning prime factors on the > order of 1000). > > I agree, a more direct comparison with FFTW would be good. > > In more detail from the README: > > I replaced the iterative sine and cosine calculations in radfg() and > radbg() > 17 by an exact calculation, which slightly improves the transform > accuracy for > 18 real FFTs with lengths containing large prime factors. > > The current numpy FFT code could use a review in any case, IIRC, there were some possible integer related issues in the arguments. Having clean documented C code would probably be an improvement and also help with maintenance. I suspect there were iterative computations of sin/cos that were spoiling the precision of the single precision versions (scipy), so fixing those might also be useful. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Fri Nov 18 11:22:02 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 18 Nov 2011 17:22:02 +0100 Subject: [Numpy-discussion] BSD C port of FFTPACK incl. bluestein algorithm In-Reply-To: <4EC64D02.5000007@astro.uio.no> References: <4EC63F4F.7030405@astro.uio.no> <4EC64D02.5000007@astro.uio.no> Message-ID: <4EC6862A.8090909@astro.uio.no> On 11/18/2011 01:18 PM, Dag Sverre Seljebotn wrote: > On 11/18/2011 12:58 PM, David Cournapeau wrote: >> On Fri, Nov 18, 2011 at 11:42 AM, Robert Kern wrote: >>> On Fri, Nov 18, 2011 at 11:19, Dag Sverre Seljebotn >>> wrote: >>>> I've been in touch with Martin Reinecke, author of the libpsht code for >>>> spherical harmonic transforms, about licensing issues. >>>> >>>> libpsht itself will remain under the GPL, but he is likely to release >>>> his C port of FFTPACK under BSD in the near future, as it is based on >>>> the public domain FFTPACK. >>>> >>>> I'm grateful for this change for my own purposes (allows releasing my >>>> own competing SHT library under the BSD) -- but it could perhaps be >>>> useful for NumPy or SciPy as well, depending on how complete the port >>>> is? E.g., perhaps make numpy.fft more complete (is the >>>> numpy.fft/scipy.fftpack split simply because of the Fortran dependency?). >>> >>> It used to be the case that scipy.fftpack allowed one to build against >>> multiple different, usually faster, FFT libraries like FFTW. I think >>> we have backed away from that since the cost of maintaining the build >>> configuration for all of those different backends was so high. It's >>> worth noting that numpy.fft is already using a C translation of >>> FFTPACK. I'm not sure what the differences are between this >>> translation and Martin's. > > Here's some more info forwarded from Martin: > > """ > - only FFTs are supported (no DCTs/DSTs) > - only double precision is supported (extension to single precision might > not be much work, though) > - both complex and real FFTs are supported > - real FFTs allow various storage schemes for the (half)complex frequency > domain data (classic FFTPACK scheme, FFTW or halfcomplex scheme, > uncompressed complex storage) > - precision of transforms involving large prime factors should be slightly > better than with original FFTPACK > - Bluestein's algorithm is automatically selected if considered profitable > - small accuracy self-testing code is provided. > > Fairly complete interface documentation is available in Doxygen format. > I'll prepare a source package later in the afternoon and send it around. OK, Martin sent me a BSD-ed version of his libfftpack, and I stuck it on github: https://github.com/dagss/libfftpack Note that ls_fft.h contains Martin's API for it (plan construction and execution etc.) Dag Sverre From silva at lma.cnrs-mrs.fr Fri Nov 18 11:34:14 2011 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Fri, 18 Nov 2011 17:34:14 +0100 Subject: [Numpy-discussion] Subclassing ndarray Message-ID: <1321634054.5925.37.camel@lma-98.cnrs-mrs.fr> Hi, I wish I could subclass ndarray so that I can use the ufuncs, but that I could ensure the array remains Fortran contiguous, and that the number of dimension is always 3 (even if the dtype can be more complex than a simple numerical type). My intention is to deal with fields that can be scalar, vectorial or even tensorial, but it must stay compatible with a library we are using that assumes 3D problems with column-major order, and some more attributes concerning spatial steps, strides and comments... I have been looking at __array_{wrap, prepare, finalize}__ methods. I would believe __array_prepare__ would be the good starting point to impose the 'F'-ordered array and atleast_3d, but the point is that I don't know at that time if the ufunc will transform the datatype (from vector to scalar, from int to float, etc...) Some thought? Regards -- Fabrice Silva From chaoyuejoy at gmail.com Fri Nov 18 11:40:05 2011 From: chaoyuejoy at gmail.com (Chao YUE) Date: Fri, 18 Nov 2011 17:40:05 +0100 Subject: [Numpy-discussion] how to use the name of a ndarray as a string In-Reply-To: <26db5d71-23d5-4c8e-a86b-f3d24ab37fd9@x7g2000yqb.googlegroups.com> References: <26db5d71-23d5-4c8e-a86b-f3d24ab37fd9@x7g2000yqb.googlegroups.com> Message-ID: Thanks to all people for this very nice discussions. the solutions are more that what I want!! and help me to clarify some concepts, and really begin to use class as a beginner :) I would say either Olivier or denis's solution can solve my problem completely. cheers, Chao 2011/11/18 denis > as Olivier and Chris say, use a dict from the start. > To make this a bit easier / more readable, > use this tiny class Dotdict: > > class Dotdict( dict ): > """ d = dotdict(), d.key same as d["key"] """ > # aka Bag, Bunch > # cPickle.dumps( d, -1 ): PicklingError in 2.6.4 > def __init__(self, **kwargs): > dict.__init__(self, kwargs) > self.__dict__ = self > > d = Dotdict() > d.a = np.arange(10) > d.b = np.ones((3,4)) > d.b[1,2] += 1 > print d.b > > > One can also save all the d.* arrays like this: > > def savedotdict( d, prefix="d.", save=np.savetxt, **kwargs ): > """ save all d.* to files """ > for name, val in d.items(): > out = prefix + name > if isinstance( val, basestring ): > with open( out, "w" ) as f: > f.write( val.rstrip() ) > f.write( "\n" ) > else: > print "saving %s to %s" % (val.shape, out) > save( out, val, **kwargs ) > > d.info = "2011-11-18 nov kilroy" # From() > savedotdict( d, "d.", fmt="%.3g" ) > > cheers > -- denis > > (If you use this, could you post it to the numpy-discussion group > please ? > It rejects me, grr.) > > > On Nov 10, 11:17 am, Chao YUE wrote: > > Hi, > > > > Does anyone know how I can quickly use the name of a ndarray as a string? > -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************ -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Fri Nov 18 11:48:28 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Fri, 18 Nov 2011 10:48:28 -0600 Subject: [Numpy-discussion] how to use the name of a ndarray as a string In-Reply-To: References: <26db5d71-23d5-4c8e-a86b-f3d24ab37fd9@x7g2000yqb.googlegroups.com> Message-ID: On Fri, Nov 18, 2011 at 10:40 AM, Chao YUE wrote: > Thanks to all people for this very nice discussions. > > the solutions are more that what I want!! and help me to clarify some > concepts, and really begin to use class as a beginner :) > FYI: Just a day or so ago, I stumbled across the same question in the Python FAQ: http://docs.python.org/faq/programming.html#how-can-my-code-discover-the-name-of-an-object Warren > > I would say either Olivier or denis's solution can solve my problem > completely. > > cheers, > > Chao > > 2011/11/18 denis > >> as Olivier and Chris say, use a dict from the start. >> To make this a bit easier / more readable, >> use this tiny class Dotdict: >> >> class Dotdict( dict ): >> """ d = dotdict(), d.key same as d["key"] """ >> # aka Bag, Bunch >> # cPickle.dumps( d, -1 ): PicklingError in 2.6.4 >> def __init__(self, **kwargs): >> dict.__init__(self, kwargs) >> self.__dict__ = self >> >> d = Dotdict() >> d.a = np.arange(10) >> d.b = np.ones((3,4)) >> d.b[1,2] += 1 >> print d.b >> >> >> One can also save all the d.* arrays like this: >> >> def savedotdict( d, prefix="d.", save=np.savetxt, **kwargs ): >> """ save all d.* to files """ >> for name, val in d.items(): >> out = prefix + name >> if isinstance( val, basestring ): >> with open( out, "w" ) as f: >> f.write( val.rstrip() ) >> f.write( "\n" ) >> else: >> print "saving %s to %s" % (val.shape, out) >> save( out, val, **kwargs ) >> >> d.info = "2011-11-18 nov kilroy" # From() >> savedotdict( d, "d.", fmt="%.3g" ) >> >> cheers >> -- denis >> >> (If you use this, could you post it to the numpy-discussion group >> please ? >> It rejects me, grr.) >> >> >> >> On Nov 10, 11:17 am, Chao YUE wrote: >> > Hi, >> > >> > Does anyone know how I can quickly use the name of a ndarray as a >> string? >> > > > > -- > > *********************************************************************************** > Chao YUE > Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) > UMR 1572 CEA-CNRS-UVSQ > Batiment 712 - Pe 119 > 91191 GIF Sur YVETTE Cedex > Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 > > ************************************************************************************ > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Nov 18 12:31:48 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 18 Nov 2011 10:31:48 -0700 Subject: [Numpy-discussion] BSD C port of FFTPACK incl. bluestein algorithm In-Reply-To: <4EC6862A.8090909@astro.uio.no> References: <4EC63F4F.7030405@astro.uio.no> <4EC64D02.5000007@astro.uio.no> <4EC6862A.8090909@astro.uio.no> Message-ID: On Fri, Nov 18, 2011 at 9:22 AM, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > On 11/18/2011 01:18 PM, Dag Sverre Seljebotn wrote: > > On 11/18/2011 12:58 PM, David Cournapeau wrote: > >> On Fri, Nov 18, 2011 at 11:42 AM, Robert Kern > wrote: > >>> On Fri, Nov 18, 2011 at 11:19, Dag Sverre Seljebotn > >>> wrote: > >>>> I've been in touch with Martin Reinecke, author of the libpsht code > for > >>>> spherical harmonic transforms, about licensing issues. > >>>> > >>>> libpsht itself will remain under the GPL, but he is likely to release > >>>> his C port of FFTPACK under BSD in the near future, as it is based on > >>>> the public domain FFTPACK. > >>>> > >>>> I'm grateful for this change for my own purposes (allows releasing my > >>>> own competing SHT library under the BSD) -- but it could perhaps be > >>>> useful for NumPy or SciPy as well, depending on how complete the port > >>>> is? E.g., perhaps make numpy.fft more complete (is the > >>>> numpy.fft/scipy.fftpack split simply because of the Fortran > dependency?). > >>> > >>> It used to be the case that scipy.fftpack allowed one to build against > >>> multiple different, usually faster, FFT libraries like FFTW. I think > >>> we have backed away from that since the cost of maintaining the build > >>> configuration for all of those different backends was so high. It's > >>> worth noting that numpy.fft is already using a C translation of > >>> FFTPACK. I'm not sure what the differences are between this > >>> translation and Martin's. > > > > Here's some more info forwarded from Martin: > > > > """ > > - only FFTs are supported (no DCTs/DSTs) > > - only double precision is supported (extension to single precision might > > not be much work, though) > > - both complex and real FFTs are supported > > - real FFTs allow various storage schemes for the (half)complex frequency > > domain data (classic FFTPACK scheme, FFTW or halfcomplex scheme, > > uncompressed complex storage) > > - precision of transforms involving large prime factors should be > slightly > > better than with original FFTPACK > > - Bluestein's algorithm is automatically selected if considered > profitable > > - small accuracy self-testing code is provided. > > > > Fairly complete interface documentation is available in Doxygen format. > > I'll prepare a source package later in the afternoon and send it around. > > OK, Martin sent me a BSD-ed version of his libfftpack, and I stuck it on > github: > > https://github.com/dagss/libfftpack > > Note that ls_fft.h contains Martin's API for it (plan construction and > execution etc.) > > #define CC(a,b,c) cc[(a)+ido*((b)+cdim*(c))] Hmm... ;) All in all, the code looks pretty readable, although the coding style needs fixups and some of the macros using complex types are never used and should be removed. The size_t type also needs to be replaced by npy_intp for numpy. I think it is worth taking a closer look at integrating the code. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lev at columbia.edu Fri Nov 18 14:21:30 2011 From: lev at columbia.edu (Lev Givon) Date: Fri, 18 Nov 2011 14:21:30 -0500 Subject: [Numpy-discussion] magma? In-Reply-To: References: Message-ID: <20111118192130.GI24018@avicenna.ee.columbia.edu> Received from Robert Kern on Fri, Nov 18, 2011 at 08:38:48AM EST: > On Fri, Nov 18, 2011 at 13:12, Neal Becker wrote: > > Is there any possibility of incorporating this work into numpy? > > > > http://icl.cs.utk.edu/magma/software/index.html > > Into numpy itself? Very low probability, I think. The dependency on > nVidia hardware and proprietary compilers makes supporting it as a > build option in numpy itself too much effort for too little gain, not > to mention the heuristics necessary to decide when to switch from the > CPU implementations to the GPU implementations. The best value/cost > ratio can probably be achieved by a separate package that wraps these > routines with functions signature-compatible with those in > numpy.linalg. Apropos of the above, I've written a BSD-licensed scikit [1] that contains a variety of numpy/scipy-like functions that can take advantage of NVIDIA GPUs via PyCUDA and ctypes interfaces to several libraries provided by NVIDIA and others. I would like to add interfaces to the MAGMA library at some point, but I'm afraid I don't have time to do so in the immediate future (I'm currently adding support for some library interface changes introduced in CUDA 4.0). If anyone is interested in contributing MAGMA wrappers to the scikit, I would be happy to consider incorporating them. L.G. [1] http://github.com/lebedov/scikits.cuda From nicolasbock at gmail.com Fri Nov 18 17:27:24 2011 From: nicolasbock at gmail.com (Nicolas Bock) Date: Fri, 18 Nov 2011 15:27:24 -0700 Subject: [Numpy-discussion] matrix product in parallel? Message-ID: Hello list, is it possible to get numpy to do a matrix product in parallel? I presume that the multiply in the back-end is really done by dgemm() or equivalent, so is it possible to use a parallel dgemm() there? Thanks, nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolasbock at gmail.com Fri Nov 18 17:55:40 2011 From: nicolasbock at gmail.com (Nicolas Bock) Date: Fri, 18 Nov 2011 15:55:40 -0700 Subject: [Numpy-discussion] matrix product in parallel? In-Reply-To: References: Message-ID: Hello list, please ignore my previous email. I just found http://www.scipy.org/ParallelProgramming Thanks, nick On Fri, Nov 18, 2011 at 15:27, Nicolas Bock wrote: > Hello list, > > is it possible to get numpy to do a matrix product in parallel? I presume > that the multiply in the back-end is really done by dgemm() or equivalent, > so is it possible to use a parallel dgemm() there? > > Thanks, > > nick > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From giannicristian at msn.com Sat Nov 19 07:23:39 2011 From: giannicristian at msn.com (Solimyr) Date: Sat, 19 Nov 2011 04:23:39 -0800 (PST) Subject: [Numpy-discussion] Vectorizing!!!! Message-ID: <32868162.post@talk.nabble.com> Hi all!!!I'm pretty new here!! I'm just use my free time to learn something more about python....I just discovered a new word named 'Vectorizing'.... I'm going to explain my question...I have a matrix (no care about the size) and I want made some mathematical operations like mean,standard deviation,variance,ecc... I post my result so can be usefull for all the newbie to understand the vectorizing mean: Mean using a 3x3 window and 2 FOR cicle (a=matrix of interest, b same size then a but filled with zeros at the start): for i in range(1,row-1): for j in range(1,col-1): b[i][j]= numpy.mean([a[i-1][j-1],a[i][j-1],a[i+1][j-1],a[i-1][j],a[i][j],a[i+1][j],a[i-1][j+1],a[i][j+1],a[i+1][j+1]]) Very disappointing in term of time..This is with the vectorizing(a=matrix of interest, c same size then a but filled with zeros at the start): c[1:row-1,1:col-1]=(a[0:row-2,0:col-2]+a[0:row-2,1:col-1]+a[2:row,2:col]+a[1:row-1,1:col-1]+a[1:row-1,0:col-2]+a[0:row-2,2:col]+a[1:row-1,2:col]+a[2:row,0:col-2]+a[2:row,1:col-1])/9 I have seen that I get a big advantage!!!But my question is: If I want to calculate the variance in a 3x3 window, can I use the vectorizing method and 'numpy.var' or I must explain the variance formula? I don't know if the question is understandable! I have thought something like: c[1:row-1,1:col-1]=numpy.var(a[0:row-2,0:col-2]+a[0:row-2,1:col-1]+a[2:row,2:col]+a[1:row-1,1:col-1]+a[1:row-1,0:col-2]+a[0:row-2,2:col]+a[1:row-1,2:col]+a[2:row,0:col-2]+a[2:row,1:col-1]) But doesn't work because I think that numpy.var work over all the matrix and not only in the 3x3 window!!Is it correct??? Thanks in advance for any answers!!! Solimyr -- View this message in context: http://old.nabble.com/Vectorizing%21%21%21%21-tp32868162p32868162.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From ecarlson at eng.ua.edu Sat Nov 19 10:55:52 2011 From: ecarlson at eng.ua.edu (Eric Carlson) Date: Sat, 19 Nov 2011 09:55:52 -0600 Subject: [Numpy-discussion] Vectorizing!!!! In-Reply-To: <32868162.post@talk.nabble.com> References: <32868162.post@talk.nabble.com> Message-ID: I'm not sure if I am addressing your question on vectorizing directly, but consider the following code, which does (maybe?) what your asking. import scipy from numpy import reshape,ones, zeros, arange, array A=reshape(arange(100),[10,10]) nr,nc=A.shape B=zeros(A.shape) #initialize array #calculate averages filt=ones([3,3])/9.0 #the weighting matrix B[1:-1,1:-1]=(scipy.signal.convolve2d(A,filt, mode='same'))[1:-1,1:-1] #initialize variance matrix for interior elements B_var =zeros(array(A.shape)-2) for i in arange(-1,2): for j in arange(-1,2): B_var+=(A[(1+i):(nr-1+i),(1+j):(nc-1+j)]-B[1:-1,1:-1])**2 B_var/=9.0 #variance of 8x8 interior elements This may or may not be the best or fastest way to get your answer. In particular, it may be just as fast to use the same loop structure for B as for B_var since convolve2D is not known as a speed demon. On the whole, though, I usually like to use scipy/numpy functions when available. If you need to do this calculation a lot, it would be pretty straightforward to parallelize with c or cuda. You'd just need to take care with minimizing memory accesses (by doing things like calculating average and variance after doing one fetch of the 9 elements). From madsipsen at gmail.com Sun Nov 20 12:04:07 2011 From: madsipsen at gmail.com (Mads Ipsen) Date: Sun, 20 Nov 2011 18:04:07 +0100 Subject: [Numpy-discussion] Build warnings Message-ID: <4EC93307.2050800@gmail.com> Hi, I am currently building numpy 1.6.1 on Ubuntu 11.04 using gcc 4.4.5. Some compiler warnings are printed during the compilation stage. Is it of any interest to post these somewhere in order for some of the core developers to fix these? Best regards, Mads -- +-----------------------------------------------------+ | Mads Ipsen | +----------------------+------------------------------+ | G?seb?ksvej 7, 4. tv | | | DK-2500 Valby | phone: +45-29716388 | | Denmark | email: mads.ipsen at gmail.com | +----------------------+------------------------------+ From numpy-discussion at maubp.freeserve.co.uk Mon Nov 21 06:26:49 2011 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Mon, 21 Nov 2011 11:26:49 +0000 Subject: [Numpy-discussion] PyPy guys deserve some help on micronumpy from you, numpy gurus! In-Reply-To: References: Message-ID: On Fri, Sep 16, 2011 at 10:49 AM, Peter wrote: > On Thu, Sep 15, 2011 at 11:34 AM, Peter > wrote: >> This may not be the best place to ask, but how should a >> python script (e.g. setup.py) distinguish between real NumPy >> and micronumpy? Or should I instead be looking to distinguish >> PyPy versus another Python implementation? > > For anyone interested, over on the pypy-dev list Alex recommended: > import platform; platform.python_implementation == 'PyPy' > http://mail.python.org/pipermail/pypy-dev/2011-September/008315.html Good news, in PyPy 1.7 they have fixed the namespace clash. http://morepypy.blogspot.com/2011/11/pypy-17-widening-sweet-spot.html In PyPy 1.6, "import numpy as np" would give you "fake" numpy, the micronumpy written by the PyPy team - which was a frustratingly limited subset of the full NumPy API. As of PyPy 1.7, you must explicitly use "import numpypy as np" to get their micronumpy. This makes life simpler for 3rd party libraries since "import numpy" will just fail, and they can use and test for the PyPy mini-numpy explicitly if they want to. Cheers, Peter From questions.anon at gmail.com Mon Nov 21 19:12:22 2011 From: questions.anon at gmail.com (questions anon) Date: Tue, 22 Nov 2011 11:12:22 +1100 Subject: [Numpy-discussion] mask one array using another array Message-ID: I am trying to mask one array using another array. I have created a masked array using mask=MA.masked_equal(myarray, 0), that looks something like: [1 - - 1, 1 1 - 1, 1 1 1 1, - 1 - 1] I have an array of values that I want to mask whereever my mask has a a '-'. how do I do this? I have looked at http://www.cawcr.gov.au/bmrc/climdyn/staff/lih/pubs/docs/masks.pdf but the command: d = array(a, mask=c.mask() results in this error: TypeError: 'numpy.ndarray' object is not callable I basically want to do exactly what that article does in that equation. Any feedback will be greatly appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Mon Nov 21 19:21:24 2011 From: shish at keba.be (Olivier Delalleau) Date: Mon, 21 Nov 2011 19:21:24 -0500 Subject: [Numpy-discussion] mask one array using another array In-Reply-To: References: Message-ID: If your new array is x, you can use: numpy.ma.masked_array(x, mask=mask.mask) -=- Olivier 2011/11/21 questions anon > I am trying to mask one array using another array. > > I have created a masked array using > mask=MA.masked_equal(myarray, > 0), > that looks something like: > [1 - - 1, > 1 1 - 1, > 1 1 1 1, > - 1 - 1] > > I have an array of values that I want to mask whereever my mask has a a > '-'. > how do I do this? > I have looked at > http://www.cawcr.gov.au/bmrc/climdyn/staff/lih/pubs/docs/masks.pdf but > the command: > > d = array(a, mask=c.mask() > > results in this error: > TypeError: 'numpy.ndarray' object is not callable > > I basically want to do exactly what that article does in that equation. > > Any feedback will be greatly appreciated. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Nov 21 20:18:36 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Mon, 21 Nov 2011 17:18:36 -0800 Subject: [Numpy-discussion] subclassing ndarray Message-ID: <4ECAF86C.7050504@noaa.gov> Hi folks, I'm working on a "ragged array" class -- an array that can store and work with what can be considered tabular data, with the rows of different lengths: """ ragged_array A "ragged" array class -- build on numpy The idea is to be able to store data that is essentially 2-d, but each row is an arbitrary length, like: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 ... At the moment, my implementation (see enclosed) stores the data in a 1-d numpy array as an attribute, and also an index array that stores the indexes into the rows. This is working fine. However, I'd like to have it support any of the usual numpy operations that make sense for a ragged array: arr.sum() arr *= a_scalar arr * a_scalar etc, etc, etc. So I thought maybe I'd do a subclass, instead of having the data array an attribute of the class. But I can't figure out how to solve the indexing problem: I want to re-map indexing, so that: arr[i] returns the ith "row": In [2]: ra = ragged_array([(1,2), (3,4,5), (6,7)]) In [4]: print ra ragged array: [1 2] [3 4 5] [6 7] In [5]: ra[1] Out[5]: array([3, 4, 5]) I'm currently doing (error checking removed): def __getitem__(self,index): """ returns a numpy array of one row. """ row = (self._data_array[self._index_array[index]:self._index_array[index+1]] ) return row But if I subclass ndarray, then self._data_array becomes jsut plain "self", and I've overloaded indexing (and slicing), so I don't know how I could index into the "flat" array to get the subset of the array I need. any ideas? Other comments about the class would be great, too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: ragged_array.py Type: text/x-python-script Size: 3727 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_ragged_array.py Type: text/x-python-script Size: 4208 bytes Desc: not available URL: From questions.anon at gmail.com Mon Nov 21 20:36:43 2011 From: questions.anon at gmail.com (questions anon) Date: Tue, 22 Nov 2011 12:36:43 +1100 Subject: [Numpy-discussion] mask one array using another array In-Reply-To: References: Message-ID: Excellent, thank you. I just realised this does not work with my data because of the extra dimension. I have a mask that matches my 2-dimensional array but my data is for every hour over a month so the arrays do not match. Is there a way to make them match or mask each time? thanks again This is some of my code: for ncfile in files: if ncfile[-3:]=='.nc': print "dealing with ncfiles:", path+ncfile ncfile=os.path.join(path,ncfile) ncfile=Dataset(ncfile, 'r+', 'NETCDF4') TSFC=ncfile.variables['T_SFC'][:] TIME=ncfile.variables['time'][:] fillvalue=ncfile.variables['T_SFC']._FillValue TSFC=MA.masked_values(TSFC, fillvalue) ncfile.close() TSFC=MA.masked_array(TSFC, mask=newmask.mask) On Tue, Nov 22, 2011 at 11:21 AM, Olivier Delalleau wrote: > If your new array is x, you can use: > > numpy.ma.masked_array(x, mask=mask.mask) > > -=- Olivier > > 2011/11/21 questions anon > >> I am trying to mask one array using another array. >> >> I have created a masked array using >> mask=MA.masked_equal(myarray, >> 0), >> that looks something like: >> [1 - - 1, >> 1 1 - 1, >> 1 1 1 1, >> - 1 - 1] >> >> I have an array of values that I want to mask whereever my mask has a a >> '-'. >> how do I do this? >> I have looked at >> http://www.cawcr.gov.au/bmrc/climdyn/staff/lih/pubs/docs/masks.pdf but >> the command: >> >> d = array(a, mask=c.mask() >> >> results in this error: >> TypeError: 'numpy.ndarray' object is not callable >> >> I basically want to do exactly what that article does in that equation. >> >> Any feedback will be greatly appreciated. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Mon Nov 21 21:13:39 2011 From: shish at keba.be (Olivier Delalleau) Date: Mon, 21 Nov 2011 21:13:39 -0500 Subject: [Numpy-discussion] mask one array using another array In-Reply-To: References: Message-ID: I can't really figure out if that's the case in your code, but if you need to repeat the mask along a new dimension (for instance, the first one), you can do: numpy.tile(mask.mask, [number_of_repeats] + [1] * len(mask.mask.shape)) (not sure that's the most elegant way to do it, but it should work) -=- Olivier 2011/11/21 questions anon > Excellent, thank you. > I just realised this does not work with my data because of the extra > dimension. > I have a mask that matches my 2-dimensional array but my data is for every > hour over a month so the arrays do not match. Is there a way to make them > match or mask each time? > thanks again > > This is some of my code: > > > for ncfile in files: > if ncfile[-3:]=='.nc': > print "dealing with ncfiles:", > path+ncfile > ncfile=os.path.join(path,ncfile) > ncfile=Dataset(ncfile, 'r+', 'NETCDF4') > TSFC=ncfile.variables['T_SFC'][:] > TIME=ncfile.variables['time'][:] > > fillvalue=ncfile.variables['T_SFC']._FillValue > TSFC=MA.masked_values(TSFC, fillvalue) > ncfile.close() > > TSFC=MA.masked_array(TSFC, > mask=newmask.mask) > > > > > > > On Tue, Nov 22, 2011 at 11:21 AM, Olivier Delalleau wrote: > >> If your new array is x, you can use: >> >> numpy.ma.masked_array(x, mask=mask.mask) >> >> -=- Olivier >> >> 2011/11/21 questions anon >> >>> I am trying to mask one array using another array. >>> >>> I have created a masked array using >>> mask=MA.masked_equal(myarray, >>> 0), >>> that looks something like: >>> [1 - - 1, >>> 1 1 - 1, >>> 1 1 1 1, >>> - 1 - 1] >>> >>> I have an array of values that I want to mask whereever my mask has a a >>> '-'. >>> how do I do this? >>> I have looked at >>> http://www.cawcr.gov.au/bmrc/climdyn/staff/lih/pubs/docs/masks.pdf but >>> the command: >>> >>> d = array(a, mask=c.mask() >>> >>> results in this error: >>> TypeError: 'numpy.ndarray' object is not callable >>> >>> I basically want to do exactly what that article does in that equation. >>> >>> Any feedback will be greatly appreciated. >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From questions.anon at gmail.com Mon Nov 21 21:35:03 2011 From: questions.anon at gmail.com (questions anon) Date: Tue, 22 Nov 2011 13:35:03 +1100 Subject: [Numpy-discussion] mask one array using another array In-Reply-To: References: Message-ID: numpy.tile is what I was after. Thank you! On Tue, Nov 22, 2011 at 1:13 PM, Olivier Delalleau wrote: > I can't really figure out if that's the case in your code, but if you need > to repeat the mask along a new dimension (for instance, the first one), you > can do: > > numpy.tile(mask.mask, [number_of_repeats] + [1] * len(mask.mask.shape)) > > (not sure that's the most elegant way to do it, but it should work) > > > -=- Olivier > > 2011/11/21 questions anon > >> Excellent, thank you. >> I just realised this does not work with my data because of the extra >> dimension. >> I have a mask that matches my 2-dimensional array but my data is for >> every hour over a month so the arrays do not match. Is there a way to make >> them match or mask each time? >> thanks again >> >> This is some of my code: >> >> >> for ncfile in files: >> if ncfile[-3:]=='.nc': >> print "dealing with ncfiles:", >> path+ncfile >> ncfile=os.path.join(path,ncfile) >> ncfile=Dataset(ncfile, 'r+', >> 'NETCDF4') >> TSFC=ncfile.variables['T_SFC'][:] >> TIME=ncfile.variables['time'][:] >> >> fillvalue=ncfile.variables['T_SFC']._FillValue >> TSFC=MA.masked_values(TSFC, fillvalue) >> ncfile.close() >> >> TSFC=MA.masked_array(TSFC, >> mask=newmask.mask) >> >> >> >> >> >> >> On Tue, Nov 22, 2011 at 11:21 AM, Olivier Delalleau wrote: >> >>> If your new array is x, you can use: >>> >>> numpy.ma.masked_array(x, mask=mask.mask) >>> >>> -=- Olivier >>> >>> 2011/11/21 questions anon >>> >>>> I am trying to mask one array using another array. >>>> >>>> I have created a masked array using >>>> mask=MA.masked_equal(myarray, >>>> 0), >>>> that looks something like: >>>> [1 - - 1, >>>> 1 1 - 1, >>>> 1 1 1 1, >>>> - 1 - 1] >>>> >>>> I have an array of values that I want to mask whereever my mask has a a >>>> '-'. >>>> how do I do this? >>>> I have looked at >>>> http://www.cawcr.gov.au/bmrc/climdyn/staff/lih/pubs/docs/masks.pdf but >>>> the command: >>>> >>>> d = array(a, mask=c.mask() >>>> >>>> results in this error: >>>> TypeError: 'numpy.ndarray' object is not callable >>>> >>>> I basically want to do exactly what that article does in that equation. >>>> >>>> Any feedback will be greatly appreciated. >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xabart at gmail.com Tue Nov 22 18:42:11 2011 From: xabart at gmail.com (Xavier Barthelemy) Date: Wed, 23 Nov 2011 10:42:11 +1100 Subject: [Numpy-discussion] compiling and linking MKL with numpy/scipy Message-ID: dear everyone, I am hard trying to compile numpy / and scipy with mkl. unfortunately it does not work. i have tried a lot of solution, and the closest for me to work is: *I have * Linux manning 2.6.32-27-server #49-Ubuntu SMP Thu Dec 2 02:05:21 UTC 2010 x86_64 GNU/Linux python -c 'import os,sys;print os.name,sys.platform' -> posix linux python 2.6 numpy 1.6.1 scipy 0.10 Intel? Fortran Compiler XE Version 12.1 Update 1, Intel? Debugger Version 12.1 Update 1, Intel? Math Kernel Library (Intel? MKL) Version 10.3 Update 7 *I used this to help me*: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ *the relevant part of site.cfg* in numpy: [mkl] library_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/ include_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/include/ lapack_libs = mmkl_libs = mkl_intel_ilp64 ,mkl_core ,mkl_intel_thread,mkl_mc *i**n distutils/intecompiler.py* cc_exe = 'icc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC -xHost -fp-model strict -DMKL_ILP64 ' ccxx_exe = 'icpc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC -xHost -fp-model strict -DMKL_ILP64 ' * in distutils/fcompiler/intel.py* def get_flags_opt(self): return ['-O3 -openmp -i8 -fpe0 -L$(MKLROOT)/lib/intel64 -lpthread,'] *I run in .bashrc* source /opt/intel/composer_xe_2011_sp1.7.256/bin/compilervars.sh intel64 source /opt/intel/composer_xe_2011_sp1.7.256/mkl/bin/mklvars.sh intel64 ilp64 *and my complation script is:* export LD_LIBRARY_PATH=/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:$LD_LIBRARY_PATH python setup.py config --compiler=intelem build_clib --compiler=intelem build_ext --compiler=intelem install *I had to use* "-fp-model strict" to pass the simple test of numpy numpy.test() *but when I run numpy.test('full')* test_assumed_shape.TestAssumedShapeSumExample.test_all ... Segmentation fault *and then with the same config for scipy * scipy.test('full') test_definition (test_basic.TestDoubleFFT) ... Segmentation fault *which obviously come from numpy* So what do I miss to make it work? Thanks Xavier -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Nov 22 19:45:37 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 22 Nov 2011 19:45:37 -0500 Subject: [Numpy-discussion] bug in reshape ? Message-ID: might be an old story >>> np.__version__ -> '1.5.1' It thought for once it's easier to use reshape to add a new axis instead of ...,None but my results got weird (normal(0,1) sample of 2.13795875e-314) >>> x = 1 >>> y = np.arange(3) >>> z = np.arange(2)[:,None] >>> np.broadcast(x,y,z) >>> np.broadcast_arrays(x,y,z) [array([[1, 1, 1], [1, 1, 1]]), array([[0, 1, 2], [0, 1, 2]]), array([[0, 0, 0], [1, 1, 1]])] >>> x1, y1, z1 = np.broadcast_arrays(x,y,z) >>> map(np.shape, (x1, y1, z1)) [(2, 3), (2, 3), (2, 3)] shape looks fine, let's add an extra axis with reshape >>> x1.reshape(2,3,1) array([[[ 1], [ 1], [ 1099464714]], [[-2147481592], [ 184], [ 1]]]) what's that ? >>> (0+x1).reshape(2,3,1) array([[[1], [1], [1]], [[1], [1], [1]]]) >>> (y1*1.).reshape(2,3,1) array([[[ 0.], [ 1.], [ 2.]], [[ 0.], [ 1.], [ 2.]]]) >>> (y1).reshape(2,3,1) array([[[ 0], [ 1], [ 2]], [[ 0], [ 1099447643], [-2147483648]]]) >>> x1, y1, z1 = np.broadcast_arrays(x,y,z) >>> x1[...,None] array([[[1], [1], [1]], [[1], [1], [1]]]) >>> x1.shape (2, 3) >>> x1.reshape(2,3,1) array([[[ 1], [ 1], [ 1099464730]], [[-2147479536], [ -445054780], [ 1063686842]]]) the background story: playing broadcasting tricks for http://projects.scipy.org/scipy/ticket/1544 Josef From friedrichromstedt at gmail.com Tue Nov 22 20:57:28 2011 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Wed, 23 Nov 2011 02:57:28 +0100 Subject: [Numpy-discussion] bug in reshape ? In-Reply-To: References: Message-ID: 2011/11/23 : > might be an old story >>> np.__version__ ?-> '1.5.1' I have only a numpy 1.4.1 for testing atm, but I believe this corner case never occured and hence never got fixed since then. Confirming your issue on that. > It thought for once it's easier to use reshape to add a new axis > instead of ...,None > but my results got weird (normal(0,1) sample of 2.13795875e-314) What do you mean by "sample of 2.13....e-314"? >>>> x = 1 >>>> y = np.arange(3) >>>> z = np.arange(2)[:,None] >>>> np.broadcast(x,y,z) > >>>> np.broadcast_arrays(x,y,z) > [array([[1, 1, 1], > ? ? ? [1, 1, 1]]), array([[0, 1, 2], > ? ? ? [0, 1, 2]]), array([[0, 0, 0], > ? ? ? [1, 1, 1]])] >>>> x1, y1, z1 = np.broadcast_arrays(x,y,z) >>>> map(np.shape, (x1, y1, z1)) > [(2, 3), (2, 3), (2, 3)] > > shape looks fine, let's add an extra axis with reshape > >>>> x1.reshape(2,3,1) > array([[[ ? ? ? ? ?1], > ? ? ? ?[ ? ? ? ? ?1], > ? ? ? ?[ 1099464714]], > > ? ? ? [[-2147481592], > ? ? ? ?[ ? ? ? ?184], > ? ? ? ?[ ? ? ? ? ?1]]]) > > what's that ? I think it has to do with this: >>> x1.strides (0, 0) Let's have a look at http://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html#numpy.reshape. From there we have this nice examples: " >>> a = np.array([[1,2,3], [4,5,6]]) >>> np.reshape(a, 6) array([1, 2, 3, 4, 5, 6]) >>> np.reshape(a, 6, order='F') array([1, 4, 2, 5, 3, 6]) " So my guess would be that numpy.reshape either assumes C or F order, I don't know what the "preserve" 'A' flag has to be interpreted as? At least this would explain the behaviour. >>>> (0+x1).reshape(2,3,1) > array([[[1], > ? ? ? ?[1], > ? ? ? ?[1]], > > ? ? ? [[1], > ? ? ? ?[1], > ? ? ? ?[1]]]) > >>>> (y1*1.).reshape(2,3,1) > array([[[ 0.], > ? ? ? ?[ 1.], > ? ? ? ?[ 2.]], > > ? ? ? [[ 0.], > ? ? ? ?[ 1.], > ? ? ? ?[ 2.]]]) > >>>> (y1).reshape(2,3,1) > array([[[ ? ? ? ? ?0], > ? ? ? ?[ ? ? ? ? ?1], > ? ? ? ?[ ? ? ? ? ?2]], > > ? ? ? [[ ? ? ? ? ?0], > ? ? ? ?[ 1099447643], > ? ? ? ?[-2147483648]]]) Guess the zero in the second block of (y1) is just coincidentally. >>>> x1, y1, z1 = np.broadcast_arrays(x,y,z) >>>> x1[...,None] > array([[[1], > ? ? ? ?[1], > ? ? ? ?[1]], > > ? ? ? [[1], > ? ? ? ?[1], > ? ? ? ?[1]]]) > >>>> x1.shape > (2, 3) >>>> x1.reshape(2,3,1) > array([[[ ? ? ? ? ?1], > ? ? ? ?[ ? ? ? ? ?1], > ? ? ? ?[ 1099464730]], > > ? ? ? [[-2147479536], > ? ? ? ?[ -445054780], > ? ? ? ?[ 1063686842]]]) > > > the background story: playing broadcasting tricks for > http://projects.scipy.org/scipy/ticket/1544 Have not looked at that. > Josef Some more: >>> x1.reshape((2, 3, 1), order='A') array([[[ 1], [2506242], [6755216]], [[ 262144], [7272256], [7278624]]]) >>> x1.reshape((2, 3, 1), order='C') array([[[ 1], [2506242], [6755216]], [[ 262144], [7272256], [7278624]]]) >>> x1.reshape((2, 3, 1), order='F') array([[[ 1], [6755216], [7272256]], [[2506242], [ 262144], [7278624]]]) So I guess it's not a bug, it's a feature ;-) Reshape is low-level plumbing apparently. It should be, since the strides involved not necessarily are compatible with each other, you can reshape (11, 7) to (7, 11). Maybe a warning if the strides contain zeros might be appropriate to avoid reading in arbitrary memory, though. Friedrich From xabart at gmail.com Wed Nov 23 02:35:46 2011 From: xabart at gmail.com (Xavier Barthelemy) Date: Wed, 23 Nov 2011 18:35:46 +1100 Subject: [Numpy-discussion] compiling and linking MKL with numpy/scipy Message-ID: Dear everyone I have been a bit better patching the files I finally get something in .basrc source /opt/intel/composer_xe_2011_sp1.7.256/bin/compilervars.sh intel64 source /opt/intel/composer_xe_2011_sp1.7.256/mkl/bin/mklvars.sh intel64 lp64 export OMP_NUM_THREADS=4 export KMP_AFFINITY=verbose,compact env variables: export LD_LIBRARY_PATH=/opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/:$LD_LIBRARY_PATH site.cfg [mkl] library_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/ include_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/ lapack_libs = mkl_libs = mkl_def, mkl_intel_lp64, mkl_core , mkl_intel_thread, mkl_mc distutils/intelcompiler.py cc_exe = 'icc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC -xHost -fp-model strict -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -lm' ccxx_exe = 'icpc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC -xHost -fp-model strict -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -lm' distutils/fcompiler/intel.py return ['-O3 -openmp -fpe0 -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -lpthread -xHost -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include -lm'] in numpy.test("full") i get test_double (test_linalg.TestCond2) ... python: symbol lookup error: /opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/libmkl_sequential.so: undefined symbol: mkl_serv_lock I am trying to look for mkl_serv_lock, but impossible to find. some ideas, someone? Xavier -------------- next part -------------- An HTML attachment was scrubbed... URL: From silva at lma.cnrs-mrs.fr Wed Nov 23 09:01:02 2011 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Wed, 23 Nov 2011 15:01:02 +0100 Subject: [Numpy-discussion] dtype and pep3118 Message-ID: <1322056862.2398.4.camel@lma-98.cnrs-mrs.fr> Hi folks, how should I specify a PEP3118 buffer format that could be understand by numpy as the following dtype: dtype = [('t11', '|f8'), ('t22', '|f8'), ('t33', '|f8'), ('t23', '|f8'), ('t13', '|f8'), ('t12', '|f8')] so that I can manipulate a structured array and its fields ? I tried strings like "T{d:t11: d:t22: d:t33: d:t23: d:t13: d:t12:}:Tensor2d:" "d:t11: d:t22: d:t33: d:t23: d:t13: d:t12:" without success -- Fabrice Silva From pav at iki.fi Wed Nov 23 09:52:38 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 23 Nov 2011 15:52:38 +0100 Subject: [Numpy-discussion] dtype and pep3118 In-Reply-To: <1322056862.2398.4.camel@lma-98.cnrs-mrs.fr> References: <1322056862.2398.4.camel@lma-98.cnrs-mrs.fr> Message-ID: 23.11.2011 15:01, Fabrice Silva kirjoitti: > Hi folks, > how should I specify a PEP3118 buffer format that could be understand by > numpy as the following dtype: > > dtype = [('t11', '|f8'), ('t22', '|f8'), ('t33', '|f8'), > ('t23', '|f8'), ('t13', '|f8'), ('t12', '|f8')] > > so that I can manipulate a structured array and its fields ? > > I tried strings like > "T{d:t11: d:t22: d:t33: d:t23: d:t13: d:t12:}:Tensor2d:" > "d:t11: d:t22: d:t33: d:t23: d:t13: d:t12:" > > without success >>> dtype = [('t11', '|f8'), ('t22', '|f8'), ('t33', '|f8'), ... ('t23', '|f8'), ('t13', '|f8'), ('t12', '|f8')] >>> x = np.zeros([1], dtype=dtype) >>> memoryview(x).format 'T{d:t11:d:t22:d:t33:d:t23:d:t13:d:t12:}' >>> np.array(memoryview(x)) array([(0.0, 0.0, 0.0, 0.0, 0.0, 0.0)], dtype=[('t11', ' References: <1322056862.2398.4.camel@lma-98.cnrs-mrs.fr> Message-ID: <1322061123.2398.10.camel@lma-98.cnrs-mrs.fr> Le mercredi 23 novembre 2011 ? 15:52 +0100, Pauli Virtanen a ?crit : > > >>> dtype = [('t11', '|f8'), ('t22', '|f8'), ('t33', '|f8'), > ... ('t23', '|f8'), ('t13', '|f8'), ('t12', '|f8')] > >>> x = np.zeros([1], dtype=dtype) > >>> memoryview(x).format > 'T{d:t11:d:t22:d:t33:d:t23:d:t13:d:t12:}' > >>> np.array(memoryview(x)) > array([(0.0, 0.0, 0.0, 0.0, 0.0, 0.0)], > dtype=[('t11', ' ' References: Message-ID: I attached a site.cfg file for numpy 1.3 compiled with MKL on some Linux 64 bit architecture, in case it might help. I always had trouble getting programs (other than numpy though) to link and execute properly with MKL. You might also try to play with LD_PRELOAD. Good luck, -=- Olivier 2011/11/23 Xavier Barthelemy > Dear everyone > > I have been a bit better > > patching the files I finally get something > > in .basrc > > source /opt/intel/composer_xe_2011_sp1.7.256/bin/compilervars.sh intel64 > source /opt/intel/composer_xe_2011_sp1.7.256/mkl/bin/mklvars.sh intel64 > lp64 > export OMP_NUM_THREADS=4 > export KMP_AFFINITY=verbose,compact > > env variables: > export > LD_LIBRARY_PATH=/opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/:$LD_LIBRARY_PATH > > site.cfg > > [mkl] > library_dirs = > /opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/ > include_dirs = > /opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/ > lapack_libs = > mkl_libs = mkl_def, mkl_intel_lp64, mkl_core , mkl_intel_thread, mkl_mc > > distutils/intelcompiler.py > > cc_exe = 'icc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC -xHost > -fp-model strict -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 > -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -lm' > ccxx_exe = 'icpc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC > -xHost -fp-model strict > -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 > -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -lm' > > > distutils/fcompiler/intel.py > > return ['-O3 -openmp -fpe0 > -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -lpthread -xHost > -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 > -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include -lm'] > > in numpy.test("full") > > i get test_double (test_linalg.TestCond2) ... python: symbol lookup error: > /opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/libmkl_sequential.so: > undefined symbol: mkl_serv_lock > > > > I am trying to look for mkl_serv_lock, but impossible to find. > > some ideas, someone? > > Xavier > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: site.cfg Type: application/octet-stream Size: 5962 bytes Desc: not available URL: From xabart at gmail.com Wed Nov 23 22:38:15 2011 From: xabart at gmail.com (Xavier Barthelemy) Date: Thu, 24 Nov 2011 14:38:15 +1100 Subject: [Numpy-discussion] compiling and linking MKL with numpy/scipy In-Reply-To: References: Message-ID: thanks it was in the order when linking mkl libs: everything above is ok, with mkl_libs = mkl_def, mkl_intel_lp64 , mkl_intel_thread, mkl_core, mkl_mc thanks xavier 2011/11/24 Olivier Delalleau > I attached a site.cfg file for numpy 1.3 compiled with MKL on some Linux > 64 bit architecture, in case it might help. > I always had trouble getting programs (other than numpy though) to link > and execute properly with MKL. > You might also try to play with LD_PRELOAD. > Good luck, > > -=- Olivier > > 2011/11/23 Xavier Barthelemy > >> Dear everyone >> >> I have been a bit better >> >> patching the files I finally get something >> >> in .basrc >> >> source /opt/intel/composer_xe_2011_sp1.7.256/bin/compilervars.sh intel64 >> source /opt/intel/composer_xe_2011_sp1.7.256/mkl/bin/mklvars.sh intel64 >> lp64 >> export OMP_NUM_THREADS=4 >> export KMP_AFFINITY=verbose,compact >> >> env variables: >> export >> LD_LIBRARY_PATH=/opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/:$LD_LIBRARY_PATH >> >> site.cfg >> >> [mkl] >> library_dirs = >> /opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/ >> include_dirs = >> /opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/ >> lapack_libs = >> mkl_libs = mkl_def, mkl_intel_lp64, mkl_core , mkl_intel_thread, mkl_mc >> >> distutils/intelcompiler.py >> >> cc_exe = 'icc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC >> -xHost -fp-model strict >> -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 >> -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -lm' >> ccxx_exe = 'icpc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC >> -xHost -fp-model strict >> -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 >> -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -lm' >> >> >> distutils/fcompiler/intel.py >> >> return ['-O3 -openmp -fpe0 >> -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -lpthread -xHost >> -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 >> -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include -lm'] >> >> in numpy.test("full") >> >> i get test_double (test_linalg.TestCond2) ... python: symbol lookup >> error: >> /opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/libmkl_sequential.so: >> undefined symbol: mkl_serv_lock >> >> >> >> I am trying to look for mkl_serv_lock, but impossible to find. >> >> some ideas, someone? >> >> Xavier >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- ? Quand le gouvernement viole les droits du peuple, l'insurrection est, pour le peuple et pour chaque portion du peuple, le plus sacr? des droits et le plus indispensable des devoirs ? D?claration des droits de l'homme et du citoyen, article 35, 1793 -------------- next part -------------- An HTML attachment was scrubbed... URL: From morph at debian.org Thu Nov 24 09:22:51 2011 From: morph at debian.org (Sandro Tosi) Date: Thu, 24 Nov 2011 15:22:51 +0100 Subject: [Numpy-discussion] Numpy 1.5.1/1.6.1 doesn't build doc with sphinx 1.1.2 Message-ID: Hello, with the new sphinx, v1.1.2, numpy documentation of 1.5.1 or 1.6.1 doesn't build. The full log of the debian package build is at: http://people.debian.org/~jwilk/tmp/python-numpy_1.6.1-3_i386.build attached is the file left in /tmp by sphinx for the error. Could you please give it a look? Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi -------------- next part -------------- # Sphinx version: 1.1.2 # Python version: 2.7.2+ # Docutils version: 0.8.1 release # Jinja2 version: 2.6 Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/sphinx/cmdline.py", line 189, in main app.build(force_all, filenames) File "/usr/lib/pymodules/python2.7/sphinx/application.py", line 204, in build self.builder.build_update() File "/usr/lib/pymodules/python2.7/sphinx/builders/__init__.py", line 196, in build_update 'out of date' % len(to_build)) File "/usr/lib/pymodules/python2.7/sphinx/builders/__init__.py", line 216, in build purple, length): File "/usr/lib/pymodules/python2.7/sphinx/builders/__init__.py", line 120, in status_iterator for item in iterable: File "/usr/lib/pymodules/python2.7/sphinx/environment.py", line 613, in update_generator self.read_doc(docname, app=app) File "/usr/lib/pymodules/python2.7/sphinx/environment.py", line 761, in read_doc pub.publish() File "/usr/lib/pymodules/python2.7/docutils/core.py", line 211, in publish self.settings) File "/usr/lib/pymodules/python2.7/docutils/readers/__init__.py", line 69, in read self.parse() File "/usr/lib/pymodules/python2.7/docutils/readers/__init__.py", line 75, in parse self.parser.parse(self.input, document) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/__init__.py", line 157, in parse self.statemachine.run(inputlines, document, inliner=self.inliner) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 170, in run input_source=document['source']) File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 237, in run context, state, transitions) File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 458, in check_line return method(match, context, next_state) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2943, in text self.section(title.lstrip(), source, style, lineno + 1, messages) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 329, in section self.new_subsection(title, lineno, messages) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 398, in new_subsection node=section_node, match_titles=1) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 284, in nested_parse node=node, match_titles=match_titles) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 195, in run results = StateMachineWS.run(self, input_lines, input_offset) File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 237, in run context, state, transitions) File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 458, in check_line return method(match, context, next_state) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2285, in explicit_markup self.explicit_list(blank_finish) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2316, in explicit_list match_titles=self.state_machine.match_titles) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 321, in nested_list_parse node=node, match_titles=match_titles) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 195, in run results = StateMachineWS.run(self, input_lines, input_offset) File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 237, in run context, state, transitions) File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 458, in check_line return method(match, context, next_state) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2589, in explicit_markup nodelist, blank_finish = self.explicit_construct(match) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2295, in explicit_construct return method(self, expmatch) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2036, in directive directive_class, match, type_name, option_presets) File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2087, in run_directive result = directive_instance.run() File "/usr/lib/pymodules/python2.7/sphinx/domains/__init__.py", line 190, in run return BaseDirective.run(self) File "/usr/lib/pymodules/python2.7/sphinx/domains/std.py", line 318, in run self.state.nested_parse(definition, definition.items[0][1], defnode) IndexError: list index out of range From pav at iki.fi Thu Nov 24 09:31:19 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 24 Nov 2011 15:31:19 +0100 Subject: [Numpy-discussion] Numpy 1.5.1/1.6.1 doesn't build doc with sphinx 1.1.2 In-Reply-To: References: Message-ID: 24.11.2011 15:22, Sandro Tosi kirjoitti: [clip] > The full log of the debian package build is at: > http://people.debian.org/~jwilk/tmp/python-numpy_1.6.1-3_i386.build > > attached is the file left in /tmp by sphinx for the error. > > Could you please give it a look? Seems like some sort of a Sphinx bug (rather than having to do with the Sphinx extensions Numpy uses). It appears it doesn't like having a `glossary::` specified in a docstring used in `automodule::`. Or, alternatively, there's something wrong with the formatting in the file `numpy/doc/glossary.py`. Workarounds: copy-paste the glossary list from `glossary.py` to `doc/source/glossary.txt` in place of `automodule:: numpy.doc.glossary` If that doesn't help, then something in the formatting of the glossary list that makes Sphinx to choke (and it's certainly then a Sphinx bug). -- Pauli Virtanen From pearu at cens.ioc.ee Fri Nov 25 09:48:03 2011 From: pearu at cens.ioc.ee (Dave) Date: Fri, 25 Nov 2011 20:48:03 +0600 Subject: [Numpy-discussion] 21 Success Secrets of Self-Made Millionaires Message-ID: 498143029.1854759596@zr.com http://jm-a.nm.ru - 21 Scucess Secrets of Self-Made Millionaires From seb.haase at gmail.com Fri Nov 25 13:20:49 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Fri, 25 Nov 2011 19:20:49 +0100 Subject: [Numpy-discussion] scipy.org still says source in some subversion repo -- should be git !? Message-ID: google search for: numpy browse source points here: http://new.scipy.org/download.html which talks about: svn co http://svn.scipy.org/svn/numpy/trunk numpy the link should probably go here: https://github.com/numpy/numpy as already stated here: http://www.scipy.org/Developer_Zone HTH, Sebastian Haase From faltet at pytables.org Sun Nov 27 08:00:48 2011 From: faltet at pytables.org (Francesc Alted) Date: Sun, 27 Nov 2011 14:00:48 +0100 Subject: [Numpy-discussion] ANN: Numexpr 2.0 released Message-ID: <201111271400.48560.faltet@pytables.org> ======================== Announcing Numexpr 2.0 ======================== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's VML library, which allows for squeezing the last drop of performance out of your multi-core processors. What's new ========== This version comes with support for the new iterator in NumPy (introduced in NumPy 1.6), allowing for improved performance in practically all the scenarios (the exception being very small arrays), and most specially for operations implying broadcasting, fortran-ordered or non-native byte orderings. The carefully crafted mix of the new NumPy iterator and direct access to data buffers turned out to be so powerful and flexible, that the internal virtual machine has been completely revamped around this combination. The drawback is that you will need NumPy >= 1.6 to run numexpr 2.0. However, NumPy 1.6 has been released more than 6 months ago now, so we think this is a good time for taking advantage of it. Many thanks to Mark Wiebe for such an important contribution! For some benchmarks on the new virtual machine, see: http://code.google.com/p/numexpr/wiki/NewVM Also, Ga?tan de Menten contributed important bug fixes, code cleanup as well as speed enhancements. Francesc Alted contributed some fixes, and added compatibility code with existing applications (PyTables) too. In case you want to know more in detail what has changed in this version, see: http://code.google.com/p/numexpr/wiki/ReleaseNotes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at Google code in: http://code.google.com/p/numexpr/ You can get the packages from PyPI as well: http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy! -- Francesc Alted From paul.anton.letnes at gmail.com Sun Nov 27 08:44:22 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sun, 27 Nov 2011 14:44:22 +0100 Subject: [Numpy-discussion] numpy.savetxt for complex arrays Message-ID: Hello numpy fans, a patch and pull request for ticket 1573 has been posted, giving numpy.savetxt the possibility to save complex arrays to text files. Formatting of the output is supported roughly along the same lines as for real numbers. https://github.com/numpy/numpy/pull/172 Share, test, discuss, and enjoy! Cheers Paul. From christoph.gohle at mpq.mpg.de Sun Nov 27 11:47:15 2011 From: christoph.gohle at mpq.mpg.de (Christoph Gohle) Date: Sun, 27 Nov 2011 17:47:15 +0100 Subject: [Numpy-discussion] recarray field access: cannot call setfield on an object array Message-ID: <8634746D-56FD-4C03-8150-EAD21C9CBDAC@mpq.mpg.de> Hi, I have a problem with recarray field access using attributes: Microsoft Windows [Version 6.1.7601] Copyright (c) 2009 Microsoft Corporation. All rights reserved. C:\Users\labuser>python Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import * >>> a = recarray(5,formats = ['i4','f8','O'],names = ['a','b','c']) >>> a.a = 5 Traceback (most recent call last): File "", line 1, in File "C:\Python26\lib\site-packages\numpy\core\records.py", line 454, in __set attr__ return self.setfield(val, *res) RuntimeError: cannot call setfield on an object array This is on numpy version 1.6.0 from the pythonxy distribution on windows 7. Access using a['a'] works as expected. What am I doing wrong? Cheers, Christoph From madsipsen at gmail.com Mon Nov 28 03:10:25 2011 From: madsipsen at gmail.com (Mads Ipsen) Date: Mon, 28 Nov 2011 09:10:25 +0100 Subject: [Numpy-discussion] dtype Message-ID: <4ED341F1.20705@gmail.com> Hi, Where can I find a complete and exhaustive description of the dtype syntax and arguments. For example, something like a = arange(4, dtype='=H') seems hard to extract from the documentation that I can find on the web and the numpy docstrings. Best regards Mads -- +-----------------------------------------------------+ | Mads Ipsen | +----------------------+------------------------------+ | G?seb?ksvej 7, 4. tv | | | DK-2500 Valby | phone: +45-29716388 | | Denmark | email: mads.ipsen at gmail.com | +----------------------+------------------------------+ -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Mon Nov 28 03:28:15 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Mon, 28 Nov 2011 09:28:15 +0100 Subject: [Numpy-discussion] dtype In-Reply-To: <4ED341F1.20705@gmail.com> References: <4ED341F1.20705@gmail.com> Message-ID: <29213E58-D72A-47B9-80B8-140E84107452@gmail.com> Hi, I don't know about documentation, but I always use syntax like zeros(10, dtype=numpy.float64) where you have dtypes like numpy.int8 numpy.uint32 numpy.complex128 # == two numpy.float64, one for real, one for imag. etc. This is usually less confusing to my mind. One caveat: floats above 64 bits are not always supported and rarely what you think - see another discussion on this list a few weeks back. Cheers Paul On 28. nov. 2011, at 09:10, Mads Ipsen wrote: > Hi, > > Where can I find a complete and exhaustive description of the dtype syntax and arguments. For example, something like > > a = arange(4, dtype='=H') > > seems hard to extract from the documentation that I can find on the web and the numpy docstrings. > > Best regards > > Mads > -- > +-----------------------------------------------------+ > | Mads Ipsen | > +----------------------+------------------------------+ > | G?seb?ksvej 7, 4. tv | | > | DK-2500 Valby | phone: +45-29716388 | > | Denmark | email: > mads.ipsen at gmail.com > | > +----------------------+------------------------------+ > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ralf.gommers at googlemail.com Mon Nov 28 16:01:05 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 28 Nov 2011 22:01:05 +0100 Subject: [Numpy-discussion] scipy.org still says source in some subversion repo -- should be git !? In-Reply-To: References: Message-ID: On Fri, Nov 25, 2011 at 7:20 PM, Sebastian Haase wrote: > google search for: numpy browse source > > points here: http://new.scipy.org/download.html > > which talks about: > svn co http://svn.scipy.org/svn/numpy/trunk numpy > The problem is that "new".scipy.org duplicates content from scipy.org, and is not so new anymore. I suspect that there's more out of date info (like installation instructions). Is anyone still working on this, or planning to do so in the near future? If not, it may be better to disable this site until someone volunteers to spend time on it again. Ralf > the link should probably go here: > > https://github.com/numpy/numpy > > as already stated here: > http://www.scipy.org/Developer_Zone > > > HTH, > Sebastian Haase > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Nov 28 16:19:49 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 28 Nov 2011 13:19:49 -0800 Subject: [Numpy-discussion] scipy.org still says source in some subversion repo -- should be git !? In-Reply-To: References: Message-ID: Hi, On Mon, Nov 28, 2011 at 1:01 PM, Ralf Gommers wrote: > > > On Fri, Nov 25, 2011 at 7:20 PM, Sebastian Haase > wrote: >> >> google search for: ?numpy browse source >> >> points ?here: ?http://new.scipy.org/download.html >> >> which talks about: >> svn co http://svn.scipy.org/svn/numpy/trunk numpy > > The problem is that "new".scipy.org duplicates content from scipy.org, and > is not so new anymore. I suspect that there's more out of date info (like > installation instructions). Is anyone still working on this, or planning to > do so in the near future? If not, it may be better to disable this site > until someone volunteers to spend time on it again. > Who controls the new.scipy.org site? Maybe the content could be put in http://github.com/scipy/scipy.github.com so we can make pull requests there? And redirect new.scipy.org to http://scipy.github.com ? Best, Matthew From ater1980 at gmail.com Tue Nov 29 15:13:55 2011 From: ater1980 at gmail.com (Alex Ter-Sarkissov) Date: Wed, 30 Nov 2011 09:13:55 +1300 Subject: [Numpy-discussion] binary to ascii Message-ID: hi eveyone, is there a simple command in numpy similar to matlab char(bin2dec('//some binary value//')) to convert binary to characters and back? thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Tue Nov 29 15:19:56 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 29 Nov 2011 15:19:56 -0500 Subject: [Numpy-discussion] binary to ascii In-Reply-To: References: Message-ID: Would numpy.fromstring and ndarray.tostring fit your needs? -=- Olivier 2011/11/29 Alex Ter-Sarkissov > hi eveyone, > > is there a simple command in numpy similar to matlab char(bin2dec('//some > binary value//')) to convert binary to characters and back? > > thanks > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy-discussion at maubp.freeserve.co.uk Tue Nov 29 15:33:04 2011 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Tue, 29 Nov 2011 20:33:04 +0000 Subject: [Numpy-discussion] binary to ascii In-Reply-To: References: Message-ID: On Tue, Nov 29, 2011 at 8:13 PM, Alex Ter-Sarkissov wrote: > hi eveyone, > > is there a simple command in numpy similar to matlab char(bin2dec('//some > binary value//')) to convert binary to characters and back? > > thanks Would the Python struct library do? You'd get a tuple back from struct.unpack(...) which you can turn into a numpy array if you need to. Peter From glenn.caltech at gmail.com Tue Nov 29 16:14:23 2011 From: glenn.caltech at gmail.com (G Jones) Date: Tue, 29 Nov 2011 13:14:23 -0800 Subject: [Numpy-discussion] binary to ascii In-Reply-To: References: Message-ID: With pure python you can do: chr(int('01000001', base=2)) 'A' On Tue, Nov 29, 2011 at 12:13 PM, Alex Ter-Sarkissov wrote: > hi eveyone, > > is there a simple command in numpy similar to matlab char(bin2dec('//some > binary value//')) to convert binary to characters and back? > > thanks > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.plantageneto at gmail.com Wed Nov 30 06:09:56 2011 From: g.plantageneto at gmail.com (Giovanni Plantageneto) Date: Wed, 30 Nov 2011 12:09:56 +0100 Subject: [Numpy-discussion] Reading automatically all the parameters from a file Message-ID: Dear all, I have a simple question. I would like to have all the parameters of a model written in a configuration file (text), and I would like to have all the parameters in the file automatically defined inside a program. I find ConfigParser a bit low level, is there any function that automatically reads everything from a file? Thanks. From robert.kern at gmail.com Wed Nov 30 06:21:14 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 30 Nov 2011 11:21:14 +0000 Subject: [Numpy-discussion] Reading automatically all the parameters from a file In-Reply-To: References: Message-ID: On Wed, Nov 30, 2011 at 11:09, Giovanni Plantageneto wrote: > Dear all, > I have a simple question. I would like to have all the parameters of a > model written in a configuration file (text), and I would like to have > all the parameters in the file automatically defined inside a program. > I find ConfigParser a bit low level, is there any function that > automatically reads everything from a file? You may want to give something like configobj a try. http://pypi.python.org/pypi/configobj It builds on ConfigParser to read all of the parameters in and creates a hierarchical object will all of the parameters as attributes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From alan.isaac at gmail.com Wed Nov 30 08:00:08 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Wed, 30 Nov 2011 08:00:08 -0500 Subject: [Numpy-discussion] Reading automatically all the parameters from a file In-Reply-To: References: Message-ID: <4ED628D8.7010706@gmail.com> On 11/30/2011 6:09 AM, Giovanni Plantageneto wrote: > I find ConfigParser a bit low level, is there any function that > automatically reads everything from a file? You could just use a dictionary for your params, and import it from your "configuration file". If you insist on an ini format, ConfigParser/configparser looks pretty good. fwiw, Alan Isaac From paul.anton.letnes at gmail.com Wed Nov 30 09:43:11 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 30 Nov 2011 15:43:11 +0100 Subject: [Numpy-discussion] Reading automatically all the parameters from a file In-Reply-To: References: Message-ID: <478BEB06-E04C-429B-958C-AACFFF54E856@gmail.com> On 30. nov. 2011, at 12:09, Giovanni Plantageneto wrote: > Dear all, > I have a simple question. I would like to have all the parameters of a > model written in a configuration file (text), and I would like to have > all the parameters in the file automatically defined inside a program. > I find ConfigParser a bit low level, is there any function that > automatically reads everything from a file? > Thanks. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion I like having my input files simply be python files, on the form bar = 'foo' ham = ['spam', 'eggs'] Then I import them as import imp parameters = imp.load_source("parameters", "myinpufile.py") Now the object 'parameters' is a python module so I can say print parameters.bar and 'foo' will be printed. Paul From ben.root at ou.edu Wed Nov 30 10:11:05 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 30 Nov 2011 09:11:05 -0600 Subject: [Numpy-discussion] Reading automatically all the parameters from a file In-Reply-To: References: Message-ID: On Wednesday, November 30, 2011, Robert Kern wrote: > On Wed, Nov 30, 2011 at 11:09, Giovanni Plantageneto > wrote: >> Dear all, >> I have a simple question. I would like to have all the parameters of a >> model written in a configuration file (text), and I would like to have >> all the parameters in the file automatically defined inside a program. >> I find ConfigParser a bit low level, is there any function that >> automatically reads everything from a file? > > You may want to give something like configobj a try. > > http://pypi.python.org/pypi/configobj > > It builds on ConfigParser to read all of the parameters in and creates > a hierarchical object will all of the parameters as attributes. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > +1 on configobj. I use this module extensively for my simulation configuration. It can even do some validation of parameters and allows for saving of comments. Furthermore, it utilizes the dictionary idiom, which makes it very easy to work with, especially for passing kwargs to functions. Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaoyuejoy at gmail.com Wed Nov 30 13:16:44 2011 From: chaoyuejoy at gmail.com (Chao YUE) Date: Wed, 30 Nov 2011 19:16:44 +0100 Subject: [Numpy-discussion] what statistical module to use for python? Message-ID: Hi all, I just want to broadly ask what statistical package are you guys using? I mean routine statistical function like linear regression, GLM, ANOVA... etc. I know there is SciKits packages like statsmodels, but are there more general and complete ones? thanks to all, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Wed Nov 30 13:49:34 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 30 Nov 2011 13:49:34 -0500 Subject: [Numpy-discussion] Reading automatically all the parameters from a file References: Message-ID: My suggestion is: don't. It's easier to script runs if you read parameters from the command line. I recommend argparse. Giovanni Plantageneto wrote: > Dear all, > I have a simple question. I would like to have all the parameters of a > model written in a configuration file (text), and I would like to have > all the parameters in the file automatically defined inside a program. > I find ConfigParser a bit low level, is there any function that > automatically reads everything from a file? > Thanks. From tsyu80 at gmail.com Wed Nov 30 14:14:39 2011 From: tsyu80 at gmail.com (Tony Yu) Date: Wed, 30 Nov 2011 14:14:39 -0500 Subject: [Numpy-discussion] Reading automatically all the parameters from a file In-Reply-To: References: Message-ID: On Wed, Nov 30, 2011 at 1:49 PM, Neal Becker wrote: > My suggestion is: don't. > > It's easier to script runs if you read parameters from the command line. > I recommend argparse. > > I think setting parameters in a config file and setting them on the command line both have their merits. I like to combine ConfigObj with argparse; something like: #~~~ parser = argparse.ArgumentParser() cfg = configobj.ConfigObj("params_file.cfg") parser.set_defaults(**cfg.config) #~~~ Then call parser.parse_args, which will override parameters in the config file by specifying values on the command line. Cheers, -Tony -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Nov 30 15:30:47 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 30 Nov 2011 15:30:47 -0500 Subject: [Numpy-discussion] who owns the data? Message-ID: just a basic question (since I haven't looked at this in some time) I'm creating a structured array in a function. However, I want to return the array with just a simple dtype uni = uni.view(dt).reshape(-1, ncols) return uni the returned uni has owndata=False. Who owns the data, since the underlying, original array went out of scope? alternatives 1) uni = np.asarray(uni, dt).reshape(-1, ncols) return uni looks obvious but raises exception 2) uni.dtype = dt uni.reshape(-1, ncols) return uni this works and uni owns the data. I'm only worried whether assigning to dtype directly is not a dangerous thing to do. >>> u array([0, 0, 0, 1, 1, 0, 1, 1]) >>> u.dtype = np.dtype("float") >>> u array([ 0.00000000e+000, 2.12199579e-314, 4.94065646e-324, 2.12199579e-314]) adding a safety check: for t in uni.dtype.fields.values(): assert (t[0] == dt) maybe I shouldn't care if nobody owns the data. Thanks, Josef From robert.kern at gmail.com Wed Nov 30 16:00:57 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 30 Nov 2011 21:00:57 +0000 Subject: [Numpy-discussion] who owns the data? In-Reply-To: References: Message-ID: On Wed, Nov 30, 2011 at 20:30, wrote: > just a basic question (since I haven't looked at this in some time) > > I'm creating a structured array in a function. However, I want to > return the array with just a simple dtype > > uni = uni.view(dt).reshape(-1, ncols) > return uni > > the returned uni has owndata=False. Who owns the data, since the > underlying, original array went out of scope? Every time you make a view through .view(), slicing, .T, certain restricted .reshape() calls , etc. a reference to the original object is stored on the view. Consequently, the original object does not get garbage collected until all of the views go away too. Making view of a view just adds another link in the chain. In your example, the original object that was assigned to `uni` before that last assignment statement was executed maintains ownership of the memory. The new ndarray object that gets assigned to `uni` for the return statement refers to the temporary ndarray returned by .view() which in turn refers to the original `uni` array which owns the actual memory. > 2) > uni.dtype = dt > uni.reshape(-1, ncols) > return uni > > this works and uni owns the data. uni.reshape() doesn't reshape `uni` inplace, though. It is possible that your `uni` array wasn't contiguous to begin with. In all of the cases that your first example would have owndata=False, this one should too. > I'm only worried whether assigning > to dtype directly is not a dangerous thing to do. It's no worse than .view(dt). The same kind of checking goes on in both places. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Wed Nov 30 16:21:00 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 30 Nov 2011 16:21:00 -0500 Subject: [Numpy-discussion] who owns the data? In-Reply-To: References: Message-ID: On Wed, Nov 30, 2011 at 4:00 PM, Robert Kern wrote: > On Wed, Nov 30, 2011 at 20:30, ? wrote: >> just a basic question (since I haven't looked at this in some time) >> >> I'm creating a structured array in a function. However, I want to >> return the array with just a simple dtype >> >> uni = uni.view(dt).reshape(-1, ncols) >> return uni >> >> the returned uni has owndata=False. Who owns the data, since the >> underlying, original array went out of scope? > > Every time you make a view through .view(), slicing, .T, certain > restricted .reshape() calls , etc. a reference to the original object > is stored on the view. Consequently, the original object does not get > garbage collected until all of the views go away too. Making view of a > view just adds another link in the chain. In your example, the > original object that was assigned to `uni` before that last assignment > statement was executed maintains ownership of the memory. The new > ndarray object that gets assigned to `uni` for the return statement > refers to the temporary ndarray returned by .view() which in turn > refers to the original `uni` array which owns the actual memory. Thanks for the explanation. There where cases on the mailing list where views created problem, so I just thought of trying to own the data, but I don't think it's really relevant. > >> 2) >> uni.dtype = dt >> uni.reshape(-1, ncols) >> return uni >> >> this works and uni owns the data. > > uni.reshape() doesn't reshape `uni` inplace, though. It is possible > that your `uni` array wasn't contiguous to begin with. In all of the > cases that your first example would have owndata=False, this one > should too. this bug happened to me a few times now. I found it but only checked the flags before fixing it. Since reshape again creates a view, the next step is to assign to shape uni.shape = (uni.size//ncols, ncols) but that starts to look like too much inplace modifications just to avoid a view Thanks, Josef > >> I'm only worried whether assigning >> to dtype directly is not a dangerous thing to do. > > It's no worse than .view(dt). The same kind of checking goes on in both places. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From questions.anon at gmail.com Wed Nov 30 20:16:34 2011 From: questions.anon at gmail.com (questions anon) Date: Thu, 1 Dec 2011 12:16:34 +1100 Subject: [Numpy-discussion] ignore NAN in numpy.true_divide() Message-ID: I am trying to calculate the mean across many netcdf files. I cannot use numpy.mean because there are too many files to concatenate and I end up with a memory error. I have enabled the below code to do what I need but I have a few nan values in some of my arrays. Is there a way to ignore these somewhere in my code. I seem to face this problem often so I would love a command that ignores blanks in my array before I continue on to the next processing step. Any feedback is greatly appreciated. netCDF_list=[] for dir in glob.glob(MainFolder + '*/01/')+ glob.glob(MainFolder + '*/02/')+ glob.glob(MainFolder + '*/12/'): for ncfile in glob.glob(dir + '*.nc'): netCDF_list.append(ncfile) slice_counter=0 print netCDF_list for filename in netCDF_list: ncfile=netCDF4.Dataset(filename) TSFC=ncfile.variables['T_SFC'][:] fillvalue=ncfile.variables['T_SFC']._FillValue TSFC=MA.masked_values(TSFC, fillvalue) for i in xrange(0,len(TSFC)-1,1): slice_counter +=1 #print slice_counter try: running_sum=N.add(running_sum, TSFC[i]) except NameError: print "Initiating the running total of my variable..." running_sum=N.array(TSFC[i]) TSFC_avg=N.true_divide(running_sum, slice_counter) N.set_printoptions(threshold='nan') print "the TSFC_avg is:", TSFC_avg -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Nov 30 20:43:37 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 30 Nov 2011 20:43:37 -0500 Subject: [Numpy-discussion] what statistical module to use for python? In-Reply-To: References: Message-ID: On Wed, Nov 30, 2011 at 1:16 PM, Chao YUE wrote: > Hi all, > > I just want to broadly ask what statistical package are you guys using? I > mean routine statistical function like linear regression, GLM, ANOVA... etc. > > I know there is SciKits packages like statsmodels, but are there more > general and complete ones? > > thanks to all, I forwarded it to the scipy-user mailing list since that is more suitable. Josef > > Chao > -- > *********************************************************************************** > Chao YUE > Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) > UMR 1572 CEA-CNRS-UVSQ > Batiment 712 - Pe 119 > 91191 GIF Sur YVETTE Cedex > Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 > ************************************************************************************ > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From magnetotellurics at gmail.com Wed Nov 30 20:44:41 2011 From: magnetotellurics at gmail.com (Karl Kappler) Date: Wed, 30 Nov 2011 17:44:41 -0800 Subject: [Numpy-discussion] Apparently non-deterministic behaviour of complex array multiplication Message-ID: Hello, I am somewhat new to scipy/numpy so please point me in the right direction if I am posting to an incorrect forum. The experience which has prompted my post is the following: I have a numpy array Y where the elements of Y are type(Y[0,0]) Out[709]: The absolute values of the real and complex values do not far exceed say 1e-10. The shape of Y is (24, 49218). When I perform the operation: C = dot(Y,Y.conj().transpose), i.e. I form the covariance matrix by multiplying T by its conjugate transpose, I sometimes get NaN in the array C. I can imagine some reasons why this may happen, but what is truly puzzling to me is that I will be working in ipython and will execute for example: find(isnan(C)) and will be returned an list of elements of C which are NaN, fine, but then I recalculate C, and repeat the find(isnan(C)) command and I get a different answer. I type: find(isnan(dot(Y,Y.conj().transpose))) and an empty array is returned. Repeated calls of the same command however result in a non-empty array. In fact, the sequence of arrays returned from several consecutive calls varies. Sometimes there are tens of NaNs, sometimes none. I have been performing a collection of experiments for some hours and cannot get to the bottom of this; Some things I have tried: 1. Cast the array Y as a matrix X and calculate X*X.H --- in this case i get the same result in that sometimes I have NaN and sometimes I do not. 2. set A=X.H and calculate X*A --- same results* 3. set b=A.copy() and calc X*b --- same results*. 4. find(isnan(real(X*X.H))) --- same results* 5. find(isnan(real(X)*real(X.H))) - no NaN appear *N.B. "Same results" does not mean that the same indices were going NaN, simply that I was getting back a different result if I ran the command say a dozen times. So it would seem that it has something to do with the complex multiplication. I am wondering if there is too much dynamic range being used in the calculation? It absolutely amazes me that I can perform the same complex-arithmetic operation sitting at the command line and obtain different results each time. In one case I ran a for loop where I performed the multiplication 1000 times and found that 694 trials had no NaN and 306 trials had NaN. Saving X to file and then reloading it in a new ipython interpreter typically resulted in no NaN. For a fixed interpretter and instance of X or Y, the indices which go NaN (when they do) sometimes repeat many times and sometimes they vary apparently at random. Also note that I have had a similar problem with much smaller arrays, say 24 x 3076 I have also tried 'upping' the numpy array to complex256, I have like 12GB of RAM... This happens both in ipython and when I call my function from the command line. Does this sound familiar to anyone? Is my machine possessed? -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Wed Nov 30 20:57:11 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 30 Nov 2011 20:57:11 -0500 Subject: [Numpy-discussion] Apparently non-deterministic behaviour of complex array multiplication In-Reply-To: References: Message-ID: I guess it's just a typo on your part, but just to make sure, you are using .transpose(), not .transpose, correct? -=- Olivier 2011/11/30 Karl Kappler > Hello, > I am somewhat new to scipy/numpy so please point me in the right direction > if I am posting to an incorrect forum. > > The experience which has prompted my post is the following: > I have a numpy array Y where the elements of Y are > type(Y[0,0]) > Out[709]: > > The absolute values of the real and complex values do not far exceed say > 1e-10. The shape of Y is (24, 49218). > When I perform the operation: C = dot(Y,Y.conj().transpose), i.e. I form > the covariance matrix by multiplying T by its conjugate transpose, I > sometimes get NaN in the array C. > > I can imagine some reasons why this may happen, but what is truly puzzling > to me is that I will be working in ipython and will execute for example: > find(isnan(C)) and will be returned an list of elements of C which are > NaN, > fine, but then I recalculate C, and repeat the find(isnan(C)) command and > I get a different answer. > > I type: > find(isnan(dot(Y,Y.conj().transpose))) > and an empty array is returned. Repeated calls of the same command > however result in a non-empty array. In fact, the sequence of arrays > returned from several consecutive calls varies. Sometimes there are tens of > NaNs, sometimes none. > > I have been performing a collection of experiments for some hours and > cannot get to the bottom of this; > Some things I have tried: > 1. Cast the array Y as a matrix X and calculate X*X.H --- in this case i > get the same result in that sometimes I have NaN and sometimes I do not. > 2. set A=X.H and calculate X*A --- same results* > 3. set b=A.copy() and calc X*b --- same results*. > 4. find(isnan(real(X*X.H))) --- same results* > 5. find(isnan(real(X)*real(X.H))) - no NaN appear > > *N.B. "Same results" does not mean that the same indices were going NaN, > simply that I was getting back a different result if I ran the command say > a dozen times. > > So it would seem that it has something to do with the complex > multiplication. I am wondering if there is too much dynamic range being > used in the calculation? It absolutely amazes me that I can perform the > same complex-arithmetic operation sitting at the command line and obtain > different results each time. In one case I ran a for loop where I > performed the multiplication 1000 times and found that 694 trials had no > NaN and 306 trials had NaN. > > Saving X to file and then reloading it in a new ipython interpreter > typically resulted in no NaN. > > For a fixed interpretter and instance of X or Y, the indices which go NaN > (when they do) sometimes repeat many times and sometimes they vary > apparently at random. > > Also note that I have had a similar problem with much smaller arrays, say > 24 x 3076 > > I have also tried 'upping' the numpy array to complex256, I have like 12GB > of RAM... > > This happens both in ipython and when I call my function from the command > line. > > Does this sound familiar to anyone? Is my machine possessed? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From questions.anon at gmail.com Wed Nov 30 21:03:11 2011 From: questions.anon at gmail.com (questions anon) Date: Thu, 1 Dec 2011 13:03:11 +1100 Subject: [Numpy-discussion] loop through arrays and find numpy maximum Message-ID: I would like to calculate the max and min of many netcdf files. I know how to create one big array and then concatenate and find the numpy.max but when I run this on 1000's of arrays I have a memory error. What I would prefer is to loop through the arrays and produce the maximum without having the make a big array. My idea goes something like: netCDF_list=[] maxarray=[] for dir in glob.glob(MainFolder + '*/01/')+ glob.glob(MainFolder + '*/02/')+ glob.glob(MainFolder + '*/12/'): for ncfile in glob.glob(dir + '*.nc'): netCDF_list.append(ncfile) for filename in netCDF_list: ncfile=netCDF4.Dataset(filename) TSFC=ncfile.variables['T_SFC'][:] fillvalue=ncfile.variables['T_SFC']._FillValue TSFC=MA.masked_values(TSFC, fillvalue) for i in TSFC: if i == N.max(TSFC, axis=0): maxarray.append(i) else: pass print maxarray -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Nov 30 21:31:57 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 30 Nov 2011 21:31:57 -0500 Subject: [Numpy-discussion] np.dot and array order Message-ID: np.__version__ '1.5.1' official win32 installer (playing with ipython for once) I thought np.dot is Lapack based and favors fortran order, but if the second array is fortran ordered, then dot takes twice as long. The order of the first array seems irrelevant (or maybe just with my shapes, in case it matters: the first array is float64, the second is bool, and I'm low in left over memory) In [93]: %timeit np.dot(x.T, indi) 1 loops, best of 3: 1.33 s per loop In [94]: %timeit np.dot(xf.T, indi) 1 loops, best of 3: 1.27 s per loop In [95]: %timeit np.dot(xf.T, indif) 1 loops, best of 3: 3 s per loop In [100]: %timeit np.dot(x.T, indif) 1 loops, best of 3: 3.05 s per loop In [96]: x.flags.c_contiguous Out[96]: True In [97]: xf.flags.c_contiguous Out[97]: False In [98]: indi.flags.c_contiguous Out[98]: True In [99]: indif.flags.c_contiguous Out[99]: False In [101]: x.shape Out[101]: (200000, 20) In [102]: indi.shape Out[102]: (200000, 500) It's just the way it is, or does it depend on ....? Josef