From marquett at iap.fr Thu Sep 1 06:15:30 2011 From: marquett at iap.fr (Jean-Baptiste Marquette) Date: Thu, 1 Sep 2011 12:15:30 +0200 Subject: [Numpy-discussion] A question about dtype syntax In-Reply-To: References: <6AB4E3BA-C9B9-4D99-A470-259CD81589A7@gmail.com> <8701E943-4F16-435B-9EA9-B39639E2754C@iap.fr> <64CA5B4C-E000-4372-85FD-35767715DE3B@gmail.com> <44057DAD-05DF-4513-A7D1-A80238702C21@gmail.com> <725FB731-F708-4B56-B084-4C8E1F34CDF1@iap.fr> Message-ID: <6C8447AA-22C6-49CE-9DDC-A3D63C0B6EC7@iap.fr> Hi Warren & Pierre, > Notice that your array is actually a 2D structured array with shape (n, 1). Try reshaping it to (n,) or apply np.squeeze before calling savetxt. Thanks to both of you, I made huge progress to understand how numpy arrays work. I use PyDev under Eclipse, and the debugger showed me that the vstack function generated a 2D array. So I used append instead, and my script now works! Cheers JB -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpmuller at gmail.com Thu Sep 1 14:20:46 2011 From: rpmuller at gmail.com (Rick Muller) Date: Thu, 1 Sep 2011 12:20:46 -0600 Subject: [Numpy-discussion] Eigenvalues did not converge In-Reply-To: References: <9992db82607f9fe061235882402b64f7@telecom-paristech.fr> Message-ID: Yes, as I pointed out, the problem does run on the Macintosh systems. But I'd like to be able to run these on our linux supercomputers. Surely this is possible, right? On Mon, Aug 29, 2011 at 9:31 AM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > I recently got into trouble with these calculations (although I used > scipy). I actually got segfaults and "bus errors". The solution for me was > to not link against ATLAS, but rather link against Apple's blas/lapack > libraries. That got everything working again. I would suggest trying to > install against something other than ATLAS and see if that helps (or, more > generally, determining which blas/lapack you are linking against, and try > something else). > > Paul > > > On 29. aug. 2011, at 16.21, Charanpal Dhanjal wrote: > > > I posted a similar question about the non-convergence of > > numpy.linalg.svd a few weeks ago. I'm not sure I can help but I wonder > > if you compiled numpy with ATLAS/MKL support (try numpy.show_config()) > > and whether it made a difference? Also what is the condition number and > > Frobenius norm of the matrix in question? > > > > Charanpal > > > > On Mon, 29 Aug 2011 08:56:31 -0600, Rick Muller wrote: > >> Im bumping into the old "Eigenvalues did not converge" error using > >> numpy.linalg.eigh() on several different linux builds of numpy > >> (1.4.1). The matrix is 166x166. I can compute the eigenvalues on a > >> Macintosh build of numpy, and I can confirm that there arent > >> degenerate eigenvalues, and that the matrix appears to be negative > >> definite. > >> > >> Ive seen this before (though not for several years), and what I > >> normally do is to build lapack with -O0. This trick did not work in > >> the current instance. Does anyone have any tricks to getting eigh to > >> work? > >> > >> Other weird things that Ive noticed about this case: I can compute > >> the eigenvalues using eigvals and eigvalsh, and can compute the > >> eigenvals/vecs using eig(). The matrix is real symmetric, and Ive > >> tested that its symmetric enough by forcibly symmetrizing it. > >> > >> Thanks in advance for any help you can offer. > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Rick Muller rpmuller at gmail.com 505-750-7557 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dhanjal at telecom-paristech.fr Thu Sep 1 15:10:17 2011 From: dhanjal at telecom-paristech.fr (Charanpal Dhanjal) Date: Thu, 01 Sep 2011 21:10:17 +0200 Subject: [Numpy-discussion] Eigenvalues did not converge In-Reply-To: References: <9992db82607f9fe061235882402b64f7@telecom-paristech.fr> Message-ID: <410f5df41ace99703c7e5fffe8d99ce2@telecom-paristech.fr> As Paul suggested I'd try compiling numpy with something other than the BLAS/LAPACK libraries currently in use. Here is a good place to start: http://www.scipy.org/Installing_SciPy/Linux. Charanpal On Thu, 1 Sep 2011 12:20:46 -0600, Rick Muller wrote: > Yes, as I pointed out, the problem does run on the Macintosh systems. > But Id like to be able to run these on our linux supercomputers. > Surely this is possible, right? > > On Mon, Aug 29, 2011 at 9:31 AM, Paul Anton Letnes wrote: > >> I recently got into trouble with these calculations (although I used >> scipy). I actually got segfaults and "bus errors". The solution for >> me was to not link against ATLAS, but rather link against Apples >> blas/lapack libraries. That got everything working again. I would >> suggest trying to install against something other than ATLAS and see >> if that helps (or, more generally, determining which blas/lapack you >> are linking against, and try something else). >> >> Paul >> >> On 29. aug. 2011, at 16.21, Charanpal Dhanjal wrote: >> >> > I posted a similar question about the non-convergence of >> > numpy.linalg.svd a few weeks ago. Im not sure I can help but I >> wonder >> > if you compiled numpy with ATLAS/MKL support (try >> numpy.show_config()) >> > and whether it made a difference? Also what is the condition >> number and >> > Frobenius norm of the matrix in question? >> > >> > Charanpal >> > >> > On Mon, 29 Aug 2011 08:56:31 -0600, Rick Muller wrote: >> >> Im bumping into the old "Eigenvalues did not converge" error >> using >> >> numpy.linalg.eigh() on several different linux builds of numpy >> >> (1.4.1). The matrix is 166x166. I can compute the eigenvalues on >> a >> >> Macintosh build of numpy, and I can confirm that there arent >> >> degenerate eigenvalues, and that the matrix appears to be >> negative >> >> definite. >> >> >> >> Ive seen this before (though not for several years), and what I >> >> normally do is to build lapack with -O0. This trick did not work >> in >> >> the current instance. Does anyone have any tricks to getting >> eigh to >> >> work? >> >> >> >> Other weird things that Ive noticed about this case: I can >> compute >> >> the eigenvalues using eigvals and eigvalsh, and can compute the >> >> eigenvals/vecs using eig(). The matrix is real symmetric, and >> Ive >> >> tested that its symmetric enough by forcibly symmetrizing it. >> >> >> >> Thanks in advance for any help you can offer. >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org [1] >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion [2] >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org [3] >> http://mail.scipy.org/mailman/listinfo/numpy-discussion [4] > > -- > Rick Muller > rpmuller at gmail.com [6] > 505-750-7557 > > > > Links: > ------ > [1] mailto:NumPy-Discussion at scipy.org > [2] http://mail.scipy.org/mailman/listinfo/numpy-discussion > [3] mailto:NumPy-Discussion at scipy.org > [4] http://mail.scipy.org/mailman/listinfo/numpy-discussion > [5] mailto:paul.anton.letnes at gmail.com > [6] mailto:rpmuller at gmail.com From jonas.wallin81 at gmail.com Thu Sep 1 17:33:20 2011 From: jonas.wallin81 at gmail.com (Jonas Wallin) Date: Thu, 1 Sep 2011 23:33:20 +0200 Subject: [Numpy-discussion] question about subtraction and shape Message-ID: Hello, I implemented the following line of code: Gami[~index0].shape ----> (100,) sigma.shape ----> (1,1) Gami[~index0] = Gam[~index0] - sigma**2 I get the error message: *** ValueError: array is not broadcastable to correct shape apparently *temp* = Gam[~index0] - sigma**2 *temp*.shape --> (1,100) Which seems strange to me is there any reason why *temp* gets this shape ? mvh Jonas Wallin -------------- next part -------------- An HTML attachment was scrubbed... URL: From tsyu80 at gmail.com Thu Sep 1 17:44:49 2011 From: tsyu80 at gmail.com (Tony Yu) Date: Thu, 1 Sep 2011 17:44:49 -0400 Subject: [Numpy-discussion] question about subtraction and shape In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 5:33 PM, Jonas Wallin wrote: > Hello, > > I implemented the following line of code: > > Gami[~index0].shape ----> (100,) > sigma.shape ----> (1,1) > Gami[~index0] = Gam[~index0] - sigma**2 > > I get the error message: > *** ValueError: array is not broadcastable to correct shape > > apparently > *temp* = Gam[~index0] - sigma**2 > *temp*.shape --> (1,100) > > Which seems strange to me is there any reason why *temp* gets this shape ? > Jonas, Numpy changes the shape of the output array based on certain broadcasting rules (See http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html). Broadcasting is really useful when you need it, but it can be confusing when you don't. Best, -Tony -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjordan1 at uw.edu Thu Sep 1 18:02:28 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Thu, 1 Sep 2011 17:02:28 -0500 Subject: [Numpy-discussion] Pull Request Review: R-like sample function Message-ID: Hi--I've just submitted a numpy 2.0 pull request for a function sample in np.random. It's essentially an implementation of R's sample function. It allows possibly non-uniform, possibly without-replacement sampling from a given 1-D array-like. This is very useful for quickly and cleanly creating samples from, for example, a list of strings or a list of non-contiguous, non-evenly spaced integers. Both occur in data analysis with categorical data. It is, essentially, a convenience function that wraps a number of existing ways to take a random sample. I think it belongs in numpy.random rather than scipy.stats because it's just a random sampler, rather than a probability distribution. It isn't possible to define a scipy.stats discrete random variable on strings--it would have to instead be done on the indices of the list containing the possible samples. And (as far as I can tell) the scipy.stats distributions can't be used for sampling without replacement. https://github.com/numpy/numpy/pull/151 -Chris JS From rowen at uw.edu Thu Sep 1 19:47:36 2011 From: rowen at uw.edu (Russell E. Owen) Date: Thu, 01 Sep 2011 16:47:36 -0700 Subject: [Numpy-discussion] Efficient way to load a 1Gb file? References: <781AF0C6-B761-4ABB-9798-9385582536E5@astro.physik.uni-goettingen.de> Message-ID: In article <781AF0C6-B761-4ABB-9798-9385582536E5 at astro.physik.uni-goettingen.de>, Derek Homeier wrote: > On 11.08.2011, at 8:50PM, Russell E. Owen wrote: > > > It seems a shame that loadtxt has no argument for predicted length, > > which would allow preallocation and less appending/copying data. > > > > And yes...reading the whole file first to figure out how many elements > > it has seems sensible to me -- at least as a switchable behavior, and > > preferably the default. 1Gb isn't that large in modern systems, but > > loadtxt is filing up all 6Gb of RAM reading it! > > 1 GB is indeed not much in terms of disk space these days, but using text > files for such data amounts is nonetheless very much non-state-of-the-art ;-) > That said, of course there is no justification to use excessive amounts of > memory where it could be avoided! > Implementing the above scheme for npyio is not quite as straightforward > as in the example I gave before, mainly for the following reasons: > > loadtxt also has to deal with more complex data like structured arrays, > plus comments, empty lines etc., meaning it has to find and count the > actual valid data lines. > > Ideally, genfromtxt, which offers yet more functionality to deal with missing > data, should offer the same options, but they would be certainly more > difficult to implement there. > > More than 6 GB is still remarkable - from what info I found in the web, lists > seem to consume ~24 Bytes/element, i.e. 3 times more than a final float64 > array. The text representation would typically take 10-20 char's for one > float (though with <12 digits, they could usually be read as float32 without > loss of precision). Thus a factor >6 seems quite extreme, unless the file > is full of (relatively) short integers... > But this also means copying of the final array would still have a relatively > low memory footprint compared to the buffer list, thus using some kind of > mutable array type for reading should be a reasonable solution as well. > Unfortunately fromiter is not of that much use here since it only reads > 1D-arrays. I haven't tried to use Chris' accumulator class yet, so for now > I did go the 2x read approach with loadtxt, it turned out to add only ~10% > to the read-in time. For compressed files this goes up to 30-50%, but > once physical memory is exhausted it should probably actually become > faster. > > I've made a pull request > https://github.com/numpy/numpy/pull/144 > implementing that option as a switch 'prescan'; could you review it in > particular regarding the following: > > Is the option reasonably named and documented? > > In the case the allocated array does not match the input data (which > really should never happen), right now just a warning is issued, > filling any excess buffer with zeros or discarding remaining input data - > should this rather raise an IndexError? > > No prediction if/when I might be able to provide this for genfromtxt, sorry! > > Cheers, > Derek This looks like a great improvement to me! I think the name is well chosen and the help is very clear. A few comments: - Might you rename the variable "l"? It is easily confused with the digit 1. - I don't understand the l < n_valid test, so this may be off base, but I'm surprised that you first massage the data and then raise an exception. Is the massaged data any use after the exception is raised? Naively I would expect you to issue a warning instead of raising an exception if you are going to handle the error by massaging the data. (It is a pity that your patch duplicates so much parsing code, but I don't see a better way to do it. Putting conditionals in the parsing loop to decide how to handle each line based on prescan would presumably slow things down too much.) Regards, -- Russell From josef.pktd at gmail.com Thu Sep 1 22:01:36 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 1 Sep 2011 22:01:36 -0400 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 6:02 PM, Christopher Jordan-Squire wrote: > Hi--I've just submitted a numpy 2.0 pull request for a function sample > in np.random. It's essentially an implementation of R's sample > function. It allows possibly non-uniform, possibly without-replacement > sampling from a given 1-D array-like. This is very useful for quickly > and cleanly creating samples from, for example, a list of strings or a > list of non-contiguous, non-evenly spaced integers. Both occur in data > analysis with categorical data. > > It is, essentially, a convenience function that wraps a number of > existing ways to take a random sample. I think it belongs in > numpy.random rather than scipy.stats because it's just a random > sampler, rather than a probability distribution. It isn't possible to > define a scipy.stats discrete random variable on strings--it would > have to instead be done on the indices of the list containing the > possible samples. And (as far as I can tell) the scipy.stats > distributions can't be used for sampling without replacement. > > https://github.com/numpy/numpy/pull/151 I don't think you can kill numpy.random.random and similar mixed in with an adding a new function commit. First these functions would need to be deprecated. "it does not break the API as the previous function was not in the docs" This is a doc bug, I assume. I don't think it means users/developers don't rely on it. searching for np.random.random shows 120 threads in my gmail reader, python uses random.random() dir(np.random) shows it I copied it from mailing list examples. It's used quite a bit in scipy, as I saw because of your work. I also find the historical multiplicity of aliases confusing, but which names should be deprecated would at least require a discussion and a separate commit. Josef > > -Chris JS > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cjordan1 at uw.edu Thu Sep 1 22:39:48 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Thu, 1 Sep 2011 22:39:48 -0400 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 10:01 PM, wrote: > On Thu, Sep 1, 2011 at 6:02 PM, Christopher Jordan-Squire > wrote: >> Hi--I've just submitted a numpy 2.0 pull request for a function sample >> in np.random. It's essentially an implementation of R's sample >> function. It allows possibly non-uniform, possibly without-replacement >> sampling from a given 1-D array-like. This is very useful for quickly >> and cleanly creating samples from, for example, a list of strings or a >> list of non-contiguous, non-evenly spaced integers. Both occur in data >> analysis with categorical data. >> >> It is, essentially, a convenience function that wraps a number of >> existing ways to take a random sample. I think it belongs in >> numpy.random rather than scipy.stats because it's just a random >> sampler, rather than a probability distribution. It isn't possible to >> define a scipy.stats discrete random variable on strings--it would >> have to instead be done on the indices of the list containing the >> possible samples. And (as far as I can tell) the scipy.stats >> distributions can't be used for sampling without replacement. >> >> https://github.com/numpy/numpy/pull/151 > > I don't think you can kill numpy.random.random and similar mixed in > with an adding a new function commit. > Killjoy. > First these functions would need to be deprecated. > I discussed this with a few other people, and they suggested that it could be alright since it's for numpy 2.0 rather than numpy 1.x. For the 2.0 version it would be perfectly reasonable to have a break with the API. (Though, as I said, it's not a break with the API.) > "it does not break the API as the previous function was not in the docs" > > This is a doc bug, I assume. I don't think it means users/developers > don't rely on it. > You apparently don't subscribe to the view that the API is infallible revelation revealed. (That's a joke, if it's not obvious.) > searching for np.random.random shows 120 threads in my gmail reader, > python uses random.random() > dir(np.random) shows it > I copied it from mailing list examples. It's used quite a bit in > scipy, as I saw because of your work. > > I also find the historical multiplicity of aliases confusing, but > which names should be deprecated would at least require a discussion > and a separate commit. > > Josef > I hadn't thought about the random.random connection. I'm fine with leaving random.random as an alias for random.random_sample. I just wanted to claim random.sample for my own function. I can't think of many other instances of aliased functions like that in numpy, though--but perhaps I'm not thinking hard enough. It certainly seemed strange to have 4 names for the same function. Now that you mention the standard library connection, my use of random.sample seems more in line with the standard library random package than the alias to random.random_sample. Though that would suggest using the default replace=True, which I'd prefer not to do. -Chris JS > >> >> -Chris JS >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Sep 1 22:48:26 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 1 Sep 2011 21:48:26 -0500 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 21:39, Christopher Jordan-Squire wrote: > On Thu, Sep 1, 2011 at 10:01 PM, ? wrote: >> First these functions would need to be deprecated. > > I discussed this with a few other people, and they suggested that it > could be alright since it's for numpy 2.0 rather than numpy 1.x. For > the 2.0 version it would be perfectly reasonable to have a break with > the API. (Though, as I said, it's not a break with the API.) Yes it is. A very long-standing API. The fact that you had to go remove a number of actual uses of the aliases should have told you this. The documentation is not the API. You cannot remove these aliases without a deprecation period lasting one full minor release. 2.0 is not license to make backwards-incompatible changes solely for aesthetic reasons. There is no reason not to follow the standard deprecation schedule here. > I can't think of many other instances of aliased functions like that > in numpy, though--but perhaps I'm not thinking hard enough. It > certainly seemed strange to have 4 names for the same function. numpy.random was actually replacing multiple libraries at once. The aliases kind of accreted. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From cjordan1 at uw.edu Thu Sep 1 23:07:47 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Thu, 1 Sep 2011 23:07:47 -0400 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 10:48 PM, Robert Kern wrote: > On Thu, Sep 1, 2011 at 21:39, Christopher Jordan-Squire wrote: >> On Thu, Sep 1, 2011 at 10:01 PM, ? wrote: > >>> First these functions would need to be deprecated. >> >> I discussed this with a few other people, and they suggested that it >> could be alright since it's for numpy 2.0 rather than numpy 1.x. For >> the 2.0 version it would be perfectly reasonable to have a break with >> the API. (Though, as I said, it's not a break with the API.) > > Yes it is. A very long-standing API. The fact that you had to go > remove a number of actual uses of the aliases should have told you > this. The documentation is not the API. You cannot remove these > aliases without a deprecation period lasting one full minor release. > 2.0 is not license to make backwards-incompatible changes solely for > aesthetic reasons. There is no reason not to follow the standard > deprecation schedule here. > Then I was misinformed, and I hadn't realized it was a long-standing use. I suspected it might be problematic for np.random.random. But np.random.sample was only used once, and np.random.ranf not at all. Those two didn't appear in the statsmodels code base either, I don't think. (And random.random only appeared once in a docstring.) So in the mean time, are there any suggestions for what this R sample function should be called, since random.sample is apparently taken? -Chris >> I can't think of many other instances of aliased functions like that >> in numpy, though--but perhaps I'm not thinking hard enough. It >> certainly seemed strange to have 4 names for the same function. > > numpy.random was actually replacing multiple libraries at once. The > aliases kind of accreted. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Sep 1 23:14:40 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 1 Sep 2011 22:14:40 -0500 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 22:07, Christopher Jordan-Squire wrote: > So in the mean time, are there any suggestions for what this R sample > function should be called, since random.sample is apparently taken? If you default to size=1 (which you probably should anyways), then np.random.choice() makes sense, in analogy to random.choice() from the standard library. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From cjordan1 at uw.edu Thu Sep 1 23:31:25 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Thu, 1 Sep 2011 23:31:25 -0400 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 11:14 PM, Robert Kern wrote: > On Thu, Sep 1, 2011 at 22:07, Christopher Jordan-Squire wrote: > >> So in the mean time, are there any suggestions for what this R sample >> function should be called, since random.sample is apparently taken? > > If you default to size=1 (which you probably should anyways), then > np.random.choice() makes sense, in analogy to random.choice() from the > standard library. > Alright. I can make that change tomorrow. I'd prefer np.sample in the long-run, for compatibility with R. (False friends are loathsome things.) How does one petition to get function names deprecated? -Chris JS > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Sep 1 23:38:25 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 1 Sep 2011 22:38:25 -0500 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 22:31, Christopher Jordan-Squire wrote: > On Thu, Sep 1, 2011 at 11:14 PM, Robert Kern wrote: >> On Thu, Sep 1, 2011 at 22:07, Christopher Jordan-Squire wrote: >> >>> So in the mean time, are there any suggestions for what this R sample >>> function should be called, since random.sample is apparently taken? >> >> If you default to size=1 (which you probably should anyways), then >> np.random.choice() makes sense, in analogy to random.choice() from the >> standard library. >> > Alright. I can make that change tomorrow. I'd prefer np.sample in the > long-run, for compatibility with R. (False friends are loathsome > things.) We have a number of software packages we might want to be compatible with and a limited English vocabulary. There is always going to be some overlap with someone that is misleading in some contexts. It's a concern, but not an overriding one. >?How does one petition to get function names deprecated? Ask here. Obtain general agreement. Write a patch. Not necessarily in that order. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From njs at pobox.com Thu Sep 1 23:55:15 2011 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 1 Sep 2011 20:55:15 -0700 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 8:31 PM, Christopher Jordan-Squire wrote: > On Thu, Sep 1, 2011 at 11:14 PM, Robert Kern wrote: >> On Thu, Sep 1, 2011 at 22:07, Christopher Jordan-Squire wrote: >> >>> So in the mean time, are there any suggestions for what this R sample >>> function should be called, since random.sample is apparently taken? >> >> If you default to size=1 (which you probably should anyways), then >> np.random.choice() makes sense, in analogy to random.choice() from the >> standard library. >> > Alright. I can make that change tomorrow. I'd prefer np.sample in the > long-run, for compatibility with R. (False friends are loathsome > things.) ?How does one petition to get function names deprecated? I was about to argue that "random.choice" was a better name anyway, but then I remembered that the standard library "random.sample" exists and does something similar. So instead I'd like to argue that making this compatible with Python is more important than making it compatible with R :-). Specifically, 'np.random.sample(array_like, k)', with no further arguments, should perform random sampling *without* replacement. Ideally it should also efficiently handle the case where array_like is an xrange object. Docs are here: http://docs.python.org/library/random.html -- Nathaniel From stefan-usenet at bytereef.org Fri Sep 2 02:49:45 2011 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Fri, 2 Sep 2011 08:49:45 +0200 Subject: [Numpy-discussion] PyBUF_SIMPLE/PyBUF_FORMAT: casts to unsigned bytes In-Reply-To: <9d39119a-723a-4018-8ba2-149416f59658@email.android.com> References: <20110823121048.GA14594@sleipnir.bytereef.org> <9d39119a-723a-4018-8ba2-149416f59658@email.android.com> Message-ID: <20110902064945.GA5538@sleipnir.bytereef.org> Dag Sverre Seljebotn wrote: > Under 2), would it make sense to also export the contents of a > Fortran-contiguous buffer as a raw byte stream? I was just the other week > writing code to serialize an array in Fortran order to a binary stream. Probably, since it works now and people might have gotten used to it. It breaks the current hierarchy of requests though: PyBUF_INDIRECT -> suboffsets + strides (PIL-style) PyBUF_STRIDES -> strides (NumPy-style) PyBUF_ND -> C-contiguous PyBUF_SIMPLE -> cast from C or Fortran contiguous to unsigned bytes The last one would be a step up in complexity again. If Fortran contiguous weren't allowed, one could assume that all buffers below (and including) PyBUF_ND are C-contiguous. It is still not clear to me what value itemsize should have if if PyBUF_FORMAT is not given. As I see it, the rules are [1]: Request without PyBUF_FORMAT -> view.format must be NULL -> 'B'. Then itemsize = 'number of bytes implied by the format' = 1. This would work for viewing contiguous buffers as byte streams, but what about non-contiguous buffers? PyBUF_STRIDED: (PyBUF_STRIDES | PyBUF_WRITABLE) How should the buffer be used if itemsize is set to 1? For example, it seems impossible to implement tobytes() if the real itemsize is missing. Travis, if you have time, it would be very nice to have your input on this one, too. Stefan Krah [1] format: "a NULL-terminated format-string (following the struct-style syntax including extensions) indicating what is in each element of memory. The number of elements is len / itemsize, where itemsize is the number of bytes implied by the format. This can be NULL which implies standard unsigned bytes ("B")." From stefan-usenet at bytereef.org Fri Sep 2 10:03:56 2011 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Fri, 2 Sep 2011 16:03:56 +0200 Subject: [Numpy-discussion] PyBUF_SIMPLE/PyBUF_FORMAT: casts to unsigned bytes In-Reply-To: <20110902064945.GA5538@sleipnir.bytereef.org> References: <20110823121048.GA14594@sleipnir.bytereef.org> <9d39119a-723a-4018-8ba2-149416f59658@email.android.com> <20110902064945.GA5538@sleipnir.bytereef.org> Message-ID: <20110902140356.GA7595@sleipnir.bytereef.org> Stefan Krah wrote: > Dag Sverre Seljebotn wrote: > > Under 2), would it make sense to also export the contents of a > > Fortran-contiguous buffer as a raw byte stream? I was just the other week > > writing code to serialize an array in Fortran order to a binary stream. > > Probably, since it works now and people might have gotten used to it. There are two other considerations why a "simple" view should be C-contiguous: 1) memoryview.tobytes() converts a Fortran-contiguous buffer to the C layout. I would find it odd if simple_view.tobytes() is different from the raw memory. 2) There is an initiative (with very broad support) to add a cast method to memoryview. The following casts are allowed: - any ND (C-contiguous) -> 1D bytes - any 1D bytes -> ND (C-contiguous) This is how it looks (in my private repo): >>> from _testbuffer import * >>> nd = ndarray(list(range(12)), shape=[3,4], format="Q") >>> m1 = memoryview(nd) >>> m2 = m1.cast('B') >>> m3 = m2.cast('Q', shape=[3,4]) >>> m4 = m2.cast('Q', shape=[2,2,3]) To summarize, I think it will be more consistent if an implicit cast via PyBUF_SIMPLE also disallows Fortran arrays. Unless there are waves of protest here... ;) Stefan Krah From derek at astro.physik.uni-goettingen.de Fri Sep 2 11:22:19 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Fri, 2 Sep 2011 17:22:19 +0200 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E5D0DEC.2070507@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> Message-ID: <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> On 30.08.2011, at 6:21PM, Chris.Barker wrote: >> I've submitted a pull request for a new method for loading data from >> text files into a record array/masked record array. > >> Click on the link for more info, but the general idea is to create a >> regular expression for what entries should look like and loop over the >> file, updating the regular expression if it's wrong. Once the types >> are determined the file is loaded line by line into a pre-allocated >> numpy array. > > nice stuff. > > Have you looked at my "accumulator" class, rather than pre-allocating? > Less the class itself than that ideas behind it. It's easy enough to do, > and would keep you from having to run through the file twice. The cost > of memory re-allocation as the array grows is very small. > > I've posted the code recently, but let me know if you want it again. I agree it would make a very nice addition, and could complement my pre-allocation option for loadtxt - however there I've also been made aware that this approach breaks streamed input etc., so the buffer.resize(?) methods in accumulator would be the better way to go. For load table this is not quite as straightforward, though, because the type auto-detection, strictly done, requires to scan the entire input, because a column full of int could still produce a float in the last row? I'd say one just has to accept that this kind of auto-detection is incompatible with input streams, and with the necessity to scan the entire data first anyway, pre-allocating the array makes sense as well. For better consistency with what people have likely got used to from npyio, I'd recommend some minor changes: make spaces the default delimiter enable automatic decompression (given the modularity, could you simply use np.lib._datasource.open() like genfromtxt?) Cheers, Derek -- ---------------------------------------------------------------- Derek Homeier Centre de Recherche Astrophysique de Lyon ENS Lyon 46, All?e d'Italie 69364 Lyon Cedex 07, France +33 1133 47272-8894 ---------------------------------------------------------------- From Chris.Barker at noaa.gov Fri Sep 2 11:50:54 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Fri, 02 Sep 2011 08:50:54 -0700 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> Message-ID: <4E60FB5E.6010606@noaa.gov> On 9/2/11 8:22 AM, Derek Homeier wrote: > I agree it would make a very nice addition, and could complement my > pre-allocation option for loadtxt - however there I've also been made > aware that this approach breaks streamed input etc., so the buffer.resize(?) > methods in accumulator would be the better way to go. Good point, that would be nice. > For load table this is not quite as straightforward, though, because the type > auto-detection, strictly done, requires to scan the entire input, because a > column full of int could still produce a float in the last row? hmmm -- it seems you could jsut as well be building the array as you go, and if you hit a change in the imput, re-set and start again. In my tests, I'm pretty sure that the time spent file io and string parsing swamp the time it takes to allocate memory and set the values. So there is little cost, and for the common use case, it would be faster and cleaner. There is a chance, of course, that you might have to re-wind and start over more than once, but I suspect that that is the rare case. > For better consistency with what people have likely got used to from npyio, > I'd recommend some minor changes: > > make spaces the default delimiter +1 > enable automatic decompression (given the modularity, could you simply > use np.lib._datasource.open() like genfromtxt?) I _think_this would benefit from a one-pass solution as well -- so you don't need to de-compress twice. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cjordan1 at uw.edu Fri Sep 2 12:16:08 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Fri, 2 Sep 2011 11:16:08 -0500 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E60FB5E.6010606@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> Message-ID: Sorry I'm only now getting around to thinking more about this. Been side-tracked by stats stuff. On Fri, Sep 2, 2011 at 10:50 AM, Chris.Barker wrote: > On 9/2/11 8:22 AM, Derek Homeier wrote: >> I agree it would make a very nice addition, and could complement my >> pre-allocation option for loadtxt - however there I've also been made >> aware that this approach breaks streamed input etc., so the buffer.resize(?) >> methods in accumulator would be the better way to go. > I'll read more about this soon. I haven't thought about it, and I didn't realize it was breaking anything. > Good point, that would be nice. > >> For load table this is not quite as straightforward, though, because the type >> auto-detection, strictly done, requires to scan the entire input, because a >> column full of int could still produce a float in the last row? > > hmmm -- it seems you could jsut as well be building the array as you go, > and if you hit a change in the imput, re-set and start again. > I hadn't thought of that. Interesting idea. I'm surprised that completely resetting the array could be faster. > In my tests, I'm pretty sure that the time spent file io and string > parsing swamp the time it takes to allocate memory and set the values. In my tests, at least for a medium sized csv file (about 3000 rows by 30 columns), about 10% of the time was determine the types in the first read through and 90% of the time was sticking the data in the array. However, that particular test took more time for reading in because the data was quoted (so converting '"3,25"' to a float took between 1.5x and 2x as long as '3.25') and the datetime conversion is costly. Regardless, that suggests making the data loading faster is more important than avoiding reading through the file twice. I guess that intuition probably breaks if the data doesn't fit until memory, though. But I haven't worked with extremely large data files before, so I'd appreciate refutation/confirmation of my priors. > > So there is little cost, and for the common use case, it would be faster > and cleaner. > > There is a chance, of course, that you might have to re-wind and start > over more than once, but I suspect that that is the rare case. > Perhaps. I know that in the 'really annoying dataset that loading quickly and easily should be your use case' example I was given, about half-way through the data one of the columns got its first observation. (It was time series data where one of the columns didn't start being observed until 1/2 through the observation period.) So I'm not sure it'd be as rare we'd like. >> For better consistency with what people have likely got used to from npyio, >> I'd recommend some minor changes: >> >> make spaces the default delimiter > > +1 > Sure. >> enable automatic decompression (given the modularity, could you simply >> use np.lib._datasource.open() like genfromtxt?) > > I _think_this would benefit from a one-pass solution as well -- so you > don't need to de-compress twice. > > -Chris > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From derek at astro.physik.uni-goettingen.de Fri Sep 2 12:17:41 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Fri, 2 Sep 2011 18:17:41 +0200 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E60FB5E.6010606@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> Message-ID: <4296732A-1E54-4447-8F49-C56FCD4B4ED0@astro.physik.uni-goettingen.de> On 02.09.2011, at 5:50PM, Chris.Barker wrote: > hmmm -- it seems you could jsut as well be building the array as you go, > and if you hit a change in the imput, re-set and start again. > > In my tests, I'm pretty sure that the time spent file io and string > parsing swamp the time it takes to allocate memory and set the values. > > So there is little cost, and for the common use case, it would be faster > and cleaner. > > There is a chance, of course, that you might have to re-wind and start > over more than once, but I suspect that that is the rare case. > I still haven't studied your class in detail, but one could probably actually just create a copy of the array read in so far, e.g. changing it from a dtype=[('f0', ' >> enable automatic decompression (given the modularity, could you simply >> use np.lib._datasource.open() like genfromtxt?) > > I _think_this would benefit from a one-pass solution as well -- so you > don't need to de-compress twice. Absolutely; on compressed data the time for the extra pass jumps up to +30-50%. Cheers, Derek -- ---------------------------------------------------------------- Derek Homeier Centre de Recherche Astrophysique de Lyon ENS Lyon 46, All?e d'Italie 69364 Lyon Cedex 07, France +33 1133 47272-8894 ---------------------------------------------------------------- From derek at astro.physik.uni-goettingen.de Fri Sep 2 12:42:07 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Fri, 2 Sep 2011 18:42:07 +0200 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> Message-ID: <625E35AD-5CDB-46F3-87B5-83EE62D57D21@astro.physik.uni-goettingen.de> On 02.09.2011, at 6:16PM, Christopher Jordan-Squire wrote: > I hadn't thought of that. Interesting idea. I'm surprised that > completely resetting the array could be faster. > I had experimented a bit with the fromiter function, which also increases the output array as needed, and this creates negligible overhead compared to parsing the text input (it is implemented in C, though, I don't know how the .resize() calls would compare to that; and unfortunately it's for 1D-arrays only). >> In my tests, I'm pretty sure that the time spent file io and string >> parsing swamp the time it takes to allocate memory and set the values. > > In my tests, at least for a medium sized csv file (about 3000 rows by > 30 columns), about 10% of the time was determine the types in the > first read through and 90% of the time was sticking the data in the > array. > This would be consistent with my experience (basically testing for comment characters and the length of line.split(delimiter) in the first pass). > However, that particular test took more time for reading in because > the data was quoted (so converting '"3,25"' to a float took between > 1.5x and 2x as long as '3.25') and the datetime conversion is costly. > > Regardless, that suggests making the data loading faster is more > important than avoiding reading through the file twice. I guess that > intuition probably breaks if the data doesn't fit until memory, > though. But I haven't worked with extremely large data files before, > so I'd appreciate refutation/confirmation of my priors. > The lion's share in the data loading time, by my experience, is still the string operations (like the comma conversion you quote above), so I'd always expect any subsequent manipulations of the numpy array data to be very fast compared to that. Maybe this changes slightly with more complex data types like string records or datetime instances, but as you indicate, even for those the conversion seems to dominate the cost. Cheers, Derek -- ---------------------------------------------------------------- Derek Homeier Centre de Recherche Astrophysique de Lyon ENS Lyon 46, All?e d'Italie 69364 Lyon Cedex 07, France +33 1133 47272-8894 ---------------------------------------------------------------- From cjordan1 at uw.edu Fri Sep 2 13:14:00 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Fri, 2 Sep 2011 12:14:00 -0500 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: I made the changes discussed here and pushed them to pull request. https://github.com/numpy/numpy/pull/143#issuecomment-1980897 I changed the new function's name from sample to choice and added the size=1 default as Robert suggested. I also reverted all the changes for sample, random, and ranf and added them to the reference docs. -Chris JS On Thu, Sep 1, 2011 at 10:55 PM, Nathaniel Smith wrote: > On Thu, Sep 1, 2011 at 8:31 PM, Christopher Jordan-Squire > wrote: >> On Thu, Sep 1, 2011 at 11:14 PM, Robert Kern wrote: >>> On Thu, Sep 1, 2011 at 22:07, Christopher Jordan-Squire wrote: >>> >>>> So in the mean time, are there any suggestions for what this R sample >>>> function should be called, since random.sample is apparently taken? >>> >>> If you default to size=1 (which you probably should anyways), then >>> np.random.choice() makes sense, in analogy to random.choice() from the >>> standard library. >>> >> Alright. I can make that change tomorrow. I'd prefer np.sample in the >> long-run, for compatibility with R. (False friends are loathsome >> things.) ?How does one petition to get function names deprecated? > > I was about to argue that "random.choice" was a better name anyway, > but then I remembered that the standard library "random.sample" exists > and does something similar. So instead I'd like to argue that making > this compatible with Python is more important than making it > compatible with R :-). > > Specifically, 'np.random.sample(array_like, k)', with no further > arguments, should perform random sampling *without* replacement. > Ideally it should also efficiently handle the case where array_like is > an xrange object. Docs are here: > http://docs.python.org/library/random.html > > -- Nathaniel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cjordan1 at uw.edu Fri Sep 2 13:17:18 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Fri, 2 Sep 2011 12:17:18 -0500 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Thu, Sep 1, 2011 at 10:55 PM, Nathaniel Smith wrote: > On Thu, Sep 1, 2011 at 8:31 PM, Christopher Jordan-Squire > wrote: >> On Thu, Sep 1, 2011 at 11:14 PM, Robert Kern wrote: >>> On Thu, Sep 1, 2011 at 22:07, Christopher Jordan-Squire wrote: >>> >>>> So in the mean time, are there any suggestions for what this R sample >>>> function should be called, since random.sample is apparently taken? >>> >>> If you default to size=1 (which you probably should anyways), then >>> np.random.choice() makes sense, in analogy to random.choice() from the >>> standard library. >>> >> Alright. I can make that change tomorrow. I'd prefer np.sample in the >> long-run, for compatibility with R. (False friends are loathsome >> things.) ?How does one petition to get function names deprecated? > > I was about to argue that "random.choice" was a better name anyway, > but then I remembered that the standard library "random.sample" exists > and does something similar. So instead I'd like to argue that making > this compatible with Python is more important than making it > compatible with R :-). > > Specifically, 'np.random.sample(array_like, k)', with no further > arguments, should perform random sampling *without* replacement. > Ideally it should also efficiently handle the case where array_like is > an xrange object. Docs are here: > http://docs.python.org/library/random.html > I mentioned this at the end of my initial response to Josef. I agree that my use of sample is more consistent with the python random library's usage, except that I don't want to have the same name but a different default. I think that also counts as a 'false friend'. So I think np.random.choice is a better name for right now. Simpler to have the same function with the same defaults but a different name than the same function with the same name and different defaults. -Chris > -- Nathaniel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From njs at pobox.com Fri Sep 2 13:58:08 2011 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Sep 2011 10:58:08 -0700 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Fri, Sep 2, 2011 at 10:14 AM, Christopher Jordan-Squire wrote: > I made the changes discussed here and pushed them to pull request. > > https://github.com/numpy/numpy/pull/143#issuecomment-1980897 I think you mean https://github.com/numpy/numpy/pull/151 > I changed the new function's name from sample to choice and added the > size=1 default as Robert suggested. I also reverted all the changes > for sample, random, and ranf and added them to the reference docs. I still think the default should be sampling without replacement, so as to reduce confusion later when we make this available as np.random.sample. You should also deprecate the current 'sample' alias (I believe this is basically a matter of documenting that it is deprecated, and making sure that it fires a deprecation warning when used). But perhaps you want to do that in a separate pull request, I dunno. -- Nathaniel From cjordan1 at uw.edu Fri Sep 2 14:45:05 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Fri, 2 Sep 2011 13:45:05 -0500 Subject: [Numpy-discussion] Pull Request Review: R-like sample function In-Reply-To: References: Message-ID: On Fri, Sep 2, 2011 at 12:58 PM, Nathaniel Smith wrote: > On Fri, Sep 2, 2011 at 10:14 AM, Christopher Jordan-Squire > wrote: >> I made the changes discussed here and pushed them to pull request. >> >> https://github.com/numpy/numpy/pull/143#issuecomment-1980897 > > I think you mean https://github.com/numpy/numpy/pull/151 > Yes. Sorry for the mix-up. >> I changed the new function's name from sample to choice and added the >> size=1 default as Robert suggested. I also reverted all the changes >> for sample, random, and ranf and added them to the reference docs. > > I still think the default should be sampling without replacement, so > as to reduce confusion later when we make this available as > np.random.sample. > np.random.sample could be just a call to np.random.choice(blah, blah, replace=False, blah). I'm not opposed to python compatibility, I just would like a function with the defaults I want. > You should also deprecate the current 'sample' alias (I believe this > is basically a matter of documenting that it is deprecated, and making > sure that it fires a deprecation warning when used). But perhaps you > want to do that in a separate pull request, I dunno. > Yes, I think I need a separate pull request after getting some agreement that it should be changed. -Chris > -- Nathaniel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Fri Sep 2 16:54:56 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Fri, 02 Sep 2011 13:54:56 -0700 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> Message-ID: <4E6142A0.4040507@noaa.gov> On 9/2/11 9:16 AM, Christopher Jordan-Squire wrote: >>> I agree it would make a very nice addition, and could complement my >>> pre-allocation option for loadtxt - however there I've also been made >>> aware that this approach breaks streamed input etc., so the buffer.resize(?) >>> methods in accumulator would be the better way to go. >> > I'll read more about this soon. I haven't thought about it, and I > didn't realize it was breaking anything. you could call it a missing feature, rather than breaking... >> hmmm -- it seems you could jsut as well be building the array as you go, >> and if you hit a change in the imput, re-set and start again. >> > > I hadn't thought of that. Interesting idea. I'm surprised that > completely resetting the array could be faster. releasing memory an re-allocating doesn't take long at all. >> In my tests, I'm pretty sure that the time spent file io and string >> parsing swamp the time it takes to allocate memory and set the values. > > In my tests, at least for a medium sized csv file (about 3000 rows by > 30 columns), about 10% of the time was determine the types in the > first read through and 90% of the time was sticking the data in the > array. I don't know how that can even be possible: Don't you have to load and parse the entire file to determine the data types? Once you've allocated, then all you are doing is setting a value in the array -- that has got to be fast. Also, the second time around, you may be taking advantage of disk cache, so that should be faster for that reason. Even so -- you may be able to save much of that 10%. > However, that particular test took more time for reading in because > the data was quoted (so converting '"3,25"' to a float took between > 1.5x and 2x as long as '3.25') and the datetime conversion is costly. Didn't you have to do all that on the first pass as well? Or are you only checking for gross info -- length of rows, etc? > Regardless, that suggests making the data loading faster is more > important than avoiding reading through the file twice. I guess that > intuition probably breaks if the data doesn't fit until memory, > though. if the data don't fit into memory, then you need to go to memmapped arrays or something -- a whole new ball of wax. >> There is a chance, of course, that you might have to re-wind and start >> over more than once, but I suspect that that is the rare case. >> > Perhaps. I know that in the 'really annoying dataset that loading > quickly and easily should be your use case' example I was given, about > half-way through the data one of the columns got its first > observation. OK -- but isn't that just one re-wind? On 9/2/11 9:17 AM, Derek Homeier wrote: >> There is a chance, of course, that you might have to re-wind and start >> over more than once, but I suspect that that is the rare case. >> > I still haven't studied your class in detail, but one could probably actually > just create a copy of the array read in so far, e.g. changing it from a > dtype=[('f0', ' or even first implement it as a list or dict of arrays, that could be individually > changed and only create a record array from that at the end. I think that's a method that the OP is specifically trying to avoid -- a list of arrays uses substantially more storage than an array. Though less than a list of lists If each row is long, infact, the list overhead would be small. > The required copying and extra memory use would definitely pale compared > to the text parsing or the current memory usage for the input list. That's what I expected -- the OP's timing seems to indicate otherwise, but I'm still skeptical as to what has been timed. > In my loadtxt version [https://github.com/numpy/numpy/pull/144] just parsing > the text for comment lines adds ca. 10% time, while any of the array allocation > and copying operations should at most be at the 1% level. much more what I'd expect. > I had experimented a bit with the fromiter function, which also increases > the output array as needed, and this creates negligible overhead compared > to parsing the text input (it is implemented in C, though, I don't know how > the .resize() calls would compare to that; it's probably using pretty much the same code as .resize() internally anyway. >and unfortunately it's for 1D-arrays only). That's not bad for this use -- make a row a struct dtype, and you've got a 1-d array anyway -- you can optionally convert to a 2-d array after the fact. I don't know why I didn't think of using fromiter() when I build accumulator. Though what I did is a bit more flexible -- you can add stuff later on, too, you don't need to do it allat once. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cjordan1 at uw.edu Fri Sep 2 17:45:17 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Fri, 2 Sep 2011 16:45:17 -0500 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E6142A0.4040507@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> Message-ID: On Fri, Sep 2, 2011 at 3:54 PM, Chris.Barker wrote: > On 9/2/11 9:16 AM, Christopher Jordan-Squire wrote: >>>> I agree it would make a very nice addition, and could complement my >>>> pre-allocation option for loadtxt - however there I've also been made >>>> aware that this approach breaks streamed input etc., so the buffer.resize(?) >>>> methods in accumulator would be the better way to go. >>> >> I'll read more about this soon. I haven't thought about it, and I >> didn't realize it was breaking anything. > > you could call it a missing feature, rather than breaking... > >>> hmmm -- it seems you could jsut as well be building the array as you go, >>> and if you hit a change in the imput, re-set and start again. >>> >> >> I hadn't thought of that. Interesting idea. I'm surprised that >> completely resetting the array could be faster. > > releasing memory an re-allocating doesn't take long at all. > >>> In my tests, I'm pretty sure that the time spent file io and string >>> parsing swamp the time it takes to allocate memory and set the values. >> >> In my tests, at least for a medium sized csv file (about 3000 rows by >> 30 columns), about 10% of the time was determine the types in the >> first read through and 90% of the time was sticking the data in the >> array. > > I don't know how that can even be possible: > > Don't you have to load and parse the entire file to determine the data > types? > > Once you've allocated, then all you are doing is setting a value in the > array -- that has got to be fast. > It doesn't have to parse the entire file to determine the dtypes. It builds up a regular expression for what it expects to see, in terms of dtypes. Then it just loops over the lines, only parsing if the regular expression doesn't match. It seems that a regex match is fast, but a regex fail is expensive. But the regex fails should be fairly rare, and are generally simple to catch. It was more expensive to keep track of the sizes for each line, as the doc string for loadtable describes. I couldn't find a good solution to cover all cases, so there's a combination of options to allow the user to pick the best case for them. Setting array elements is not as fast for the masked record arrays. You must set entire rows at a time, so I have to build up each row as a list, and convert to a tuple, and then stuff it in the array. And it's even slower for the record arrays with missing data because I must branch between adding missing data versus adding real data. Might that be the reason for the slower performance than you'd expect? > Also, the second time around, you may be taking advantage of disk cache, > so that should be faster for that reason. > > Even so -- you may be able to save much of that 10%. I don't understand your meaning. > >> However, that particular test took more time for reading in because >> the data was quoted (so converting '"3,25"' to a float took between >> 1.5x and 2x as long as '3.25') and the datetime conversion is costly. > > Didn't you have to do all that on the first pass as well? Or are you > only checking for gross info -- length of rows, etc? > >> Regardless, that suggests making the data loading faster is more >> important than avoiding reading through the file twice. I guess that >> intuition probably breaks if the data doesn't fit until memory, >> though. > > if the data don't fit into memory, then you need to go to memmapped > arrays or something -- a whole new ball of wax. > >>> There is a chance, of course, that you might have to re-wind and start >>> over more than once, but I suspect that that is the rare case. >>> >> Perhaps. I know that in the 'really annoying dataset that loading >> quickly and easily should be your use case' example I was given, about >> half-way through the data one of the columns got its first >> observation. > > OK -- but isn't that just one re-wind? > Sure. If it only happens for one column. But suppose your data is a bunch of time series, one per column, where they each start at different dates. You'd have a restart for each column. But I guess that point is pedantic since regardless the number of columns should be many less than the number of rows. I wonder if there are any really important cases where you'd actually lose something by simply recasting an entry to another dtype, as Derek suggested. That would avoid having to go back to the start simply by doing an in-place conversion of the data. > On 9/2/11 9:17 AM, Derek Homeier wrote: > >>> There is a chance, of course, that you might have to re-wind and start >>> over more than once, but I suspect that that is the rare case. >>> >> I still haven't studied your class in detail, but one could probably actually >> just create a copy of the array read in so far, e.g. changing it from a >> dtype=[('f0', ' > good point -- that would be a better way to do it, and only a tiny bit > harder. > >> or even first implement it as a list or dict of arrays, that could be individually >> changed and only create a record array from that at the end. > > I think that's a method that the OP is specifically trying to avoid -- a > list of arrays uses substantially more storage than an array. Though > less than a list of lists If each row is long, infact, the list overhead > would be small. > >> The required copying and extra memory use would definitely pale compared >> to the text parsing or the current memory usage for the input list. > > That's what I expected -- the OP's timing seems to indicate otherwise, > but I'm still skeptical as to what has been timed. > So here's some of the important output from prun (in ipython) on the following call: np.loadtable('biggun.csv', quoted=True, comma_decimals=True, NA_re=r'#N/A N/A|', date_re='\d{1,2}/\d{1,2}/\d{2}', date_strp='%m/%d/%y', header=True) where biggun.csv was the 2857 by 25 csv file with datetimes and quoted data I'd mentioned earlier. (It's proprietary data, so I can't share the csv file itself.) ---------------------------------------------------------------------------------------------------------- 377584 function calls (377532 primitive calls) in 2.242 CPU seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.540 0.540 1.866 1.866 loadtable.py:859(get_data_missing) 91615 0.460 0.000 0.460 0.000 {built-in method match} 58835 0.286 0.000 0.405 0.000 loadtable.py:301(comma_float_conv) 126267 0.270 0.000 0.270 0.000 {method 'replace' of 'str' objects} 2857 0.125 0.000 0.151 0.000 core.py:2975(__setitem__) 2857 0.113 0.000 0.295 0.000 _strptime.py:295(_strptime) 2862 0.071 0.000 0.071 0.000 {numpy.core.multiarray.array} 2857 0.053 0.000 0.200 0.000 loadtable.py:1304(update_sizes) 2857 0.039 0.000 0.066 0.000 locale.py:316(normalize) 1 0.028 0.028 0.373 0.373 loadtable.py:1165(get_nrows_sizes_coltypes) 2857 0.024 0.000 0.319 0.000 {built-in method strptime} 5796 0.021 0.000 0.021 0.000 {built-in method groups} 2857 0.018 0.000 0.342 0.000 loadtable.py:784() 2857 0.017 0.000 0.102 0.000 locale.py:481(getlocale) 8637 0.016 0.000 0.016 0.000 {method 'get' of 'dict' objects} 2857 0.016 0.000 0.016 0.000 {map} 8631 0.015 0.000 0.015 0.000 {len} --------------------------------------------------------------------------------------------------------------------- It goes on, but those seem to be the important calls. So I wasn't quite right on the 90-10 split, but 99.9% of the time is in two methods: getting the data (get_data_missing) and determining the sizes, dtypes (get_nrows_sizes_coltypes). Between those two the split is 17-83. >> In my loadtxt version [https://github.com/numpy/numpy/pull/144] just parsing >> the text for comment lines adds ca. 10% time, while any of the array allocation >> and copying operations should at most be at the 1% level. > > much more what I'd expect. > >> I had experimented a bit with the fromiter function, which also increases >> the output array as needed, and this creates negligible overhead compared >> to parsing the text input (it is implemented in C, though, I don't know how >> the .resize() calls would compare to that; > > it's probably using pretty much the same code as .resize() internally > anyway. > >>and unfortunately it's for 1D-arrays only). > > That's not bad for this use -- make a row a struct dtype, and you've got > a 1-d array anyway -- you can optionally convert to a 2-d array after > the fact. > > I don't know why I didn't think of using fromiter() when I build > accumulator. ?Though what I did is a bit more flexible -- you can add > stuff later on, too, you don't need to do it allat once. > I'm unsure how to use fromiter for missing data. It sounds like a potential solution when no data is missing, though. -Chris > -Chris > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From gael.varoquaux at normalesup.org Fri Sep 2 18:15:04 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 3 Sep 2011 00:15:04 +0200 Subject: [Numpy-discussion] Hiring a junior developer on the scikit-learn Message-ID: <20110902221504.GE27778@phare.normalesup.org> Hi list, Once again, we are looking for a junior developer to work on the scikit-learn. Below is the official job posting. Unofficially I would like to stress that this is a unique opportunity to be payed for two years to work on learning and improving the scientific Python toolstack. Gael ________________________________________________________________________________ **Job Description** INRIA is looking to hire a young graduate on a 2-year position to help with the community-driven development of the open source machine learning in Python library, scikit-learn. The scikit-learn is one of the major machine-learning libraries in Python. It aims to be state-of-the-art on mid-size to large datasets by harnessing the power of the scientific Python toolstack. Speaking French is not a requirement, as it is an international team. **Requirements** * Programming skills in Python and C/C++ * Understanding of quality assurance in software development: test-driven programming, version control, technical documentation. * Some knowledge of Linux/Unix * Software design skills * Knowledge of open-source development and community-driven environments * Good technical English level * An experience in statistical learning or a mathematical-oriented mindset is a plus * We can only hire a young-graduate that has received a masters or equivalent degree at most a year ago. **About the company** INRIA is the French computer science research institute. It recognized word-wide as one of the leading research institutions and has a strong expertise in machine learning. You will be working in the `Parietal team `_ that makes a heavy use of Python for brain imaging analysis. Parietal is a small research team (around 10 people) with an excellent technical knowledge of scientific and numerical computing in Python as well as a fine understanding of algorithmic issues in machine learning and statistics. Parietal is committed to investing in scikit-learn. Working at Parietal is a unique opportunity to improve your skills in machine learning and numerical computing in Python. In addition, working full time on the scikit-learn, a very active open-source project, will give you premium experience of open source community management and collaborative project development. **Contact Info:** * **Technical Contact**: Bertand Thirion * **E-mail contact**: bertrand.thirion at inria.fr * **HR Contact**: Marie Domingues * **E-mail Contact**: marie.domingues at inria.fr * **No telecommuting** From bryce.ready at gmail.com Sat Sep 3 22:22:25 2011 From: bryce.ready at gmail.com (Bryce Ready) Date: Sat, 3 Sep 2011 20:22:25 -0600 Subject: [Numpy-discussion] converting standard array to record array In-Reply-To: References: Message-ID: Ah, I see. So basically I need to prepend dimensions myself to make this work - makes sense. Might be nice if view could do that automatically, but this is workable. Thanks Josef! On Tue, Aug 30, 2011 at 9:49 PM, wrote: > On Tue, Aug 30, 2011 at 4:34 PM, Bryce Ready > wrote: > > Hello all, > > > > So i'm using numpy 1.6.0, and trying to convert a (4,4) numpy array of > dtype > > 'f8' into a record array of this dtype: > > > >> dt = np.dtype([('mat','(4,4)f8')]) > > > > Here is the code snippet: > > > >> In [21]: a = np.random.randn(4,4) > >> > >> In [22]: a.view(dt) > > > > and the resulting error: > > > >> ValueError: new type not compatible with array. > > > > Can anyone shed some light for me on why this conversion is not possible? > > It is certainly technically possible, since the memory layout of the two > > arrays should be the same. > > > > Can anyone recommend a better way to do this conversion? > > I guess it can only convert rows, each row needs the memory size of the dt > > >>> np.random.randn(4,4).ravel().view(dt).shape > (1,) > >>> np.random.randn(2,4,4).reshape(-1,16).view(dt) > array([[ ([[1.7107996212005496, 0.64334162481360346, > -2.1589367225479004, 1.2302260107072134], [0.90703092017458831, > -1.0297890301610224, -0.095086304368665275, 0.35407366904038495], > [-1.1083969421298907, 0.83307347286837752, 0.39886399402076494, > 0.26313136034262563], [0.81860729029038914, -1.1443047382313905, > 0.73326737255810859, 0.34482475392499168]],)], > [ ([[0.69027418489768777, 0.25867753263599164, > 1.0320990807184023, 0.21836691513066409], [0.45913017094388614, > -0.89570247025515981, 0.76452726059163534, -2.2953009964941642], > [0.60248580944596275, 1.0863090037733505, -0.10849220482850662, > -0.19176089514256078], [-1.0700600508627109, -1.4743316703511105, > 0.79193567523155062, 0.82243321942810521]],)]], > dtype=[('mat', ' > Josef > > > > > Thanks in advance! > > > > -Bryce Ready > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Sun Sep 4 10:17:07 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sun, 4 Sep 2011 16:17:07 +0200 Subject: [Numpy-discussion] Eigenvalues did not converge In-Reply-To: References: <9992db82607f9fe061235882402b64f7@telecom-paristech.fr> Message-ID: I'm not sure if I got my point across. My point was that the ATLAS installation that numpy linked against was broken on Mac OS X but not on Linux (afaik). Hence, your code may run better on your supercomputer. So, try linking against a different BLAS/LAPACK implementation, and, with some luck, your problem could potentially disappear. ymmv, Paul. On Thu, Sep 1, 2011 at 8:20 PM, Rick Muller wrote: > Yes, as I pointed out, the problem does run on the Macintosh systems. But > I'd like to be able to run these on our linux supercomputers. Surely this is > possible, right? > > > On Mon, Aug 29, 2011 at 9:31 AM, Paul Anton Letnes < > paul.anton.letnes at gmail.com> wrote: > >> I recently got into trouble with these calculations (although I used >> scipy). I actually got segfaults and "bus errors". The solution for me was >> to not link against ATLAS, but rather link against Apple's blas/lapack >> libraries. That got everything working again. I would suggest trying to >> install against something other than ATLAS and see if that helps (or, more >> generally, determining which blas/lapack you are linking against, and try >> something else). >> >> Paul >> >> >> On 29. aug. 2011, at 16.21, Charanpal Dhanjal wrote: >> >> > I posted a similar question about the non-convergence of >> > numpy.linalg.svd a few weeks ago. I'm not sure I can help but I wonder >> > if you compiled numpy with ATLAS/MKL support (try numpy.show_config()) >> > and whether it made a difference? Also what is the condition number and >> > Frobenius norm of the matrix in question? >> > >> > Charanpal >> > >> > On Mon, 29 Aug 2011 08:56:31 -0600, Rick Muller wrote: >> >> Im bumping into the old "Eigenvalues did not converge" error using >> >> numpy.linalg.eigh() on several different linux builds of numpy >> >> (1.4.1). The matrix is 166x166. I can compute the eigenvalues on a >> >> Macintosh build of numpy, and I can confirm that there arent >> >> degenerate eigenvalues, and that the matrix appears to be negative >> >> definite. >> >> >> >> Ive seen this before (though not for several years), and what I >> >> normally do is to build lapack with -O0. This trick did not work in >> >> the current instance. Does anyone have any tricks to getting eigh to >> >> work? >> >> >> >> Other weird things that Ive noticed about this case: I can compute >> >> the eigenvalues using eigvals and eigvalsh, and can compute the >> >> eigenvals/vecs using eig(). The matrix is real symmetric, and Ive >> >> tested that its symmetric enough by forcibly symmetrizing it. >> >> >> >> Thanks in advance for any help you can offer. >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > Rick Muller > rpmuller at gmail.com > 505-750-7557 > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fredmfp at gmail.com Mon Sep 5 06:43:34 2011 From: fredmfp at gmail.com (Fred) Date: Mon, 05 Sep 2011 12:43:34 +0200 Subject: [Numpy-discussion] gradient with memmap array bug (?) In-Reply-To: References: <4D4FE3BB.90405@gmail.com> <4D67782A.3000205@gmail.com> <20110225094239.GF24960@phare.normalesup.org> Message-ID: <4E64A7D6.9060901@gmail.com> Hi all, I get some issue using gradient on an array created from memmap: PC-Fred[pts/10]:~/{11}/> ipython -p numpy Python 2.6.7 (r267:88850, Jul 10 2011, 08:11:54) Type "copyright", "credits" or "license" for more information. IPython 0.10.2 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. IPython profile: numpy PC-Fred[12:39:18]:~/{1}/> a=array([[1.,2.], [3., 4.]], dtype='f') PC-Fred[12:40:24]:~/{2}/> a.tofile('a.sep') PC-Fred[12:40:32]:~/{3}/> del a PC-Fred[12:40:45]:~/{4}/> a = memmap('a.sep', mode='r', dtype='f', shape=(2,2)) PC-Fred[12:40:49]:~/{5}/> x, y = gradient(a, 1, 1) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /mnt/common/home/fred/ in () /usr/lib/pymodules/python2.6/numpy/lib/function_base.py in gradient(f, *varargs) 842 for axis in range(N): 843 # select out appropriate parts for this dimension --> 844 out = np.zeros_like(f).astype(otype) 845 slice1[axis] = slice(1, -1) 846 slice2[axis] = slice(2, None) /usr/lib/pymodules/python2.6/numpy/core/memmap.py in __array_finalize__(self, obj) 255 if hasattr(obj, '_mmap'): 256 self._mmap = obj._mmap --> 257 self.filename = obj.filename 258 self.offset = obj.offset 259 self.mode = obj.mode AttributeError: 'memmap' object has no attribute 'filename' Sounds like a bug or not? Any clue? TIA. Cheers, PS : NumPy 1.5.1 on wheezy debian box -- Fred From ralf.gommers at googlemail.com Mon Sep 5 12:41:14 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 5 Sep 2011 18:41:14 +0200 Subject: [Numpy-discussion] gradient with memmap array bug (?) In-Reply-To: <4E64A7D6.9060901@gmail.com> References: <4D4FE3BB.90405@gmail.com> <4D67782A.3000205@gmail.com> <20110225094239.GF24960@phare.normalesup.org> <4E64A7D6.9060901@gmail.com> Message-ID: On Mon, Sep 5, 2011 at 12:43 PM, Fred wrote: > Hi all, > > I get some issue using gradient on an array created from memmap: > > PC-Fred[pts/10]:~/{11}/> ipython -p numpy > Python 2.6.7 (r267:88850, Jul 10 2011, 08:11:54) > Type "copyright", "credits" or "license" for more information. > > IPython 0.10.2 -- An enhanced Interactive Python. > ? -> Introduction and overview of IPython's features. > %quickref -> Quick reference. > help -> Python's own help system. > object? -> Details about 'object'. ?object also works, ?? prints more. > > IPython profile: numpy > > PC-Fred[12:39:18]:~/{1}/> a=array([[1.,2.], [3., 4.]], dtype='f') > > PC-Fred[12:40:24]:~/{2}/> a.tofile('a.sep') > > PC-Fred[12:40:32]:~/{3}/> del a > > PC-Fred[12:40:45]:~/{4}/> a = memmap('a.sep', mode='r', dtype='f', > shape=(2,2)) > > PC-Fred[12:40:49]:~/{5}/> x, y = gradient(a, 1, 1) > --------------------------------------------------------------------------- > AttributeError Traceback (most recent call last) > > /mnt/common/home/fred/ in () > > /usr/lib/pymodules/python2.6/numpy/lib/function_base.py in gradient(f, > *varargs) > 842 for axis in range(N): > 843 # select out appropriate parts for this dimension > > --> 844 out = np.zeros_like(f).astype(otype) > 845 slice1[axis] = slice(1, -1) > 846 slice2[axis] = slice(2, None) > > /usr/lib/pymodules/python2.6/numpy/core/memmap.py in > __array_finalize__(self, obj) > 255 if hasattr(obj, '_mmap'): > 256 self._mmap = obj._mmap > --> 257 self.filename = obj.filename > 258 self.offset = obj.offset > 259 self.mode = obj.mode > > AttributeError: 'memmap' object has no attribute 'filename' > > > Sounds like a bug or not? > > Any clue? > > That rings a bell. I can reproduce this with 1.5.1, and it's fixed in master. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Tue Sep 6 10:32:49 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 6 Sep 2011 16:32:49 +0200 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> Message-ID: <158E151F-3C9A-4CF3-8841-C347ECC7ABE3@astro.physik.uni-goettingen.de> On 02.09.2011, at 11:45PM, Christopher Jordan-Squire wrote: >>> >>> and unfortunately it's for 1D-arrays only). >> >> That's not bad for this use -- make a row a struct dtype, and you've got >> a 1-d array anyway -- you can optionally convert to a 2-d array after >> the fact. >> >> I don't know why I didn't think of using fromiter() when I build >> accumulator. Though what I did is a bit more flexible -- you can add >> stuff later on, too, you don't need to do it allat once. >> > > I'm unsure how to use fromiter for missing data. It sounds like a > potential solution when no data is missing, though. Strange I haven't thought about it before either; I guess for record arrays it comes more natural to view them as a collection of 1D arrays. However, you'd need to construct a list or something of ncolumn iterators from the input - should not be too hard; but then how do you feed the ncolumn fromiter() instances synchronously from that?? As far as I can see there is no way to make them read one item at a time, row by row. Then there are additional complications with multi-D dtypes, and in your case, especially datetime instances, but the problem that all columns have to be read in in parallel really seems to be the showstopper here. Of course for "flat" 2D arrays of data (all the same dtype) this would work with simply reshaping the array - that's probably even the most common use case for loadtxt, but that method lacks way too much generality for my taste. Back to accumulator, I suppose. Cheers, Derek -- ---------------------------------------------------------------- Derek Homeier Centre de Recherche Astrophysique de Lyon ENS Lyon 46, All?e d'Italie 69364 Lyon Cedex 07, France +33 1133 47272-8894 ---------------------------------------------------------------- From cjordan1 at uw.edu Tue Sep 6 10:36:40 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Tue, 6 Sep 2011 09:36:40 -0500 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <158E151F-3C9A-4CF3-8841-C347ECC7ABE3@astro.physik.uni-goettingen.de> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <158E151F-3C9A-4CF3-8841-C347ECC7ABE3@astro.physik.uni-goettingen.de> Message-ID: On Tue, Sep 6, 2011 at 9:32 AM, Derek Homeier wrote: > On 02.09.2011, at 11:45PM, Christopher Jordan-Squire wrote: >>>> >>>> and unfortunately it's for 1D-arrays only). >>> >>> That's not bad for this use -- make a row a struct dtype, and you've got >>> a 1-d array anyway -- you can optionally convert to a 2-d array after >>> the fact. >>> >>> I don't know why I didn't think of using fromiter() when I build >>> accumulator. ?Though what I did is a bit more flexible -- you can add >>> stuff later on, too, you don't need to do it allat once. >>> >> >> I'm unsure how to use fromiter for missing data. It sounds like a >> potential solution when no data is missing, though. > > Strange I haven't thought about it before either; I guess for record arrays it > comes more natural to view them as a collection of 1D arrays. > However, you'd need to construct a list or something of ncolumn iterators from > the input - should not be too hard; but then > how do you feed the ncolumn fromiter() instances synchronously from that?? > As far as I can see there is no way to make them read one item at a time, > row by row. Then there are additional complications with multi-D dtypes, > and in your case, especially datetime instances, but the problem that all columns > have to be read in in parallel really seems to be the showstopper here. > Of course for "flat" 2D arrays of data (all the same dtype) this would work with > simply reshaping the array - that's probably even the most common use case for > loadtxt, but that method lacks way too much generality for my taste. > Back to accumulator, I suppose. > Yes, I believe the thinking was that if your data is all one dtype that's simple enough to figure out, and there are other method for reading in such an array to produce a 2-d array. This is strictly for structured arrays currently, though I suppose that could change. -Chris > Cheers, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Derek > -- > ---------------------------------------------------------------- > Derek Homeier ? ? ? ? ?Centre de Recherche Astrophysique de Lyon > ENS Lyon ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?46, All?e d'Italie > 69364 Lyon Cedex 07, France ? ? ? ? ? ? ? ? ?+33 1133 47272-8894 > ---------------------------------------------------------------- > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From derek at astro.physik.uni-goettingen.de Tue Sep 6 11:04:36 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 6 Sep 2011 17:04:36 +0200 Subject: [Numpy-discussion] Efficient way to load a 1Gb file? In-Reply-To: References: <781AF0C6-B761-4ABB-9798-9385582536E5@astro.physik.uni-goettingen.de> Message-ID: On 02.09.2011, at 1:47AM, Russell E. Owen wrote: >> I've made a pull request >> https://github.com/numpy/numpy/pull/144 >> implementing that option as a switch 'prescan'; could you review it in >> particular regarding the following: >> >> Is the option reasonably named and documented? >> >> In the case the allocated array does not match the input data (which >> really should never happen), right now just a warning is issued, >> filling any excess buffer with zeros or discarding remaining input data - >> should this rather raise an IndexError? >> >> No prediction if/when I might be able to provide this for genfromtxt, sorry! >> >> Cheers, >> Derek > > This looks like a great improvement to me! I think the name is well > chosen and the help is very clear. > Thanks for your feedback, just a few quick comments: > A few comments: > - Might you rename the variable "l"? It is easily confused with the > digit 1. > - I don't understand the l < n_valid test, so this may be off base, but > I'm surprised that you first massage the data and then raise an > exception. Is the massaged data any use after the exception is raised? > Naively I would expect you to issue a warning instead of raising an > exception if you are going to handle the error by massaging the data. > The exception is currently caught right after the loop, which might seem a bit illogical, but the idea was to handle both cases in one place (if l > n_valid, trying to assign to X[l] will also raise an IndexError, meaning there are data left in the input that could not be stored) - so the present version indeed just issues a warning for both cases, but that could easily be changed... > (It is a pity that your patch duplicates so much parsing code, but I > don't see a better way to do it. Putting conditionals in the parsing > loop to decide how to handle each line based on prescan would presumably > slow things down too much.) That was my idea behind it - otherwise I would also have considered moving it out into its own functions, but as long as the entire code more or less fits into one editor window, this did not appear an obstacle to me. The main update on the issue is however, that all this is currently on hold because some concerns have been raised about not using dynamic resizing instead (the extra reading pass would break streamed input), and we have been discussing the best way to do this in another thread related to pull request https://github.com/numpy/numpy/pull/143 (which implements similar functionality, plus a lot more, for a genfromtxt-like function). So don't be surprised if the loadtxt patch comes back later, in a completely revised form? Cheers, Derek -- ---------------------------------------------------------------- Derek Homeier Centre de Recherche Astrophysique de Lyon ENS Lyon 46, All?e d'Italie 69364 Lyon Cedex 07, France +33 1133 47272-8894 ---------------------------------------------------------------- From totonixsame at gmail.com Tue Sep 6 14:15:46 2011 From: totonixsame at gmail.com (Thiago Franco Moraes) Date: Tue, 6 Sep 2011 15:15:46 -0300 Subject: [Numpy-discussion] Smoothing binary volumes Message-ID: Hi all, I've been implementing the algorithm from this paper "Reducing Aliasing Artifacts in Iso-Surfaces of Binary Volumes" from Ross T. Whitaker. Because I develop a opensource software which works with segmentation of CT and MRI medical images, and the results of segmentation is a binary volume image. The problem is that the surface generate from binary images are rough and have those staircase artifacts. Bellow some examples after and before using the algorithm: Binary ball - http://i.imgur.com/DlIwP.png Smoothed ball - http://i.imgur.com/G04zN.png Binary CT - http://i.imgur.com/Ah1LB.png Smoothed CT - http://i.imgur.com/ps1Nz.png The algorithm works in the volumetric image. After the algorithm I use the Marching Cubes (from VTK) to generate the surface. I could use a gaussian filter, but some fine features could be lost. The source-code is here [1]. The first attempt I use python, but it used a lot of memory. Now I'm using Cython, it's not completed. Maybe I will try to code directly in C. I'm sending this email to this mail-list because someone could have some interest in this, and maybe it could be inserted in some python library, like mahotas. Thanks! [1] - https://github.com/tfmoraes/binvol From david.cottrell at gmail.com Tue Sep 6 14:38:35 2011 From: david.cottrell at gmail.com (David Cottrell) Date: Tue, 6 Sep 2011 14:38:35 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: Message-ID: I posted on stackoverflow but then noticed this message board: http://stackoverflow.com/questions/7311869/python-numpy-on-solaris-blas-slow-or-not-linked I'm reposting the full post below: Matrix-Matrix multiplies are very slow on my Solaris install (running on a sparc server) compared to my OSX install (on a laptop!). The laptop runs 100 times faster (for matrix-matrix multiplies of 3000x3000 dense random matrices of doubles). It must be because the Solaris install is not using blas, but the numpy scripts are reporting that the libs are 'found'. ??? $python3 -c "import numpy.distutils.system_info as f; d = f.get_info('blas',0); print(d); d = f.get_info('lapack',0); print(d)" ???? {'libraries': ['sunperf'], 'library_dirs': ['/home/$myname/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib'], 'language': 'f77'} ???? {'libraries': ['sunmath'], 'library_dirs': ['/home/$myname/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib'],'language': 'f77'} The following import FAILS on the Solaris install but succeeds on OSX: import numpy.core._dotblas There is no ATLAS available for the Solaris install. I wouldn't think this would make such a huge different in computational efficiency. Any suggestions for other tests? Does the distutils.system_info not necessarily mean anything about the install? I tried the numpy.alterdot() command but that seems to have no effect. Even more notes: And I basically followed the setup on http://www.scipy.org/Installing_SciPy/Solaris, except that my site.cfg looked like: ??? [DEFAULT] ??? library_dirs = $PATH_TO_MY_SOLARIS_DIR/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib ??? [blas] ??? blas_libs = sunperf ??? [lapack] ??? lapack_libs = sunmath From cournape at gmail.com Tue Sep 6 14:59:52 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Sep 2011 14:59:52 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: Message-ID: On Tue, Sep 6, 2011 at 2:38 PM, David Cottrell wrote: > I posted on stackoverflow but then noticed this message board: > > http://stackoverflow.com/questions/7311869/python-numpy-on-solaris-blas-slow-or-not-linked > > I'm reposting the full post below: > > Matrix-Matrix multiplies are very slow on my Solaris install (running > on a sparc server) compared to my OSX install (on a laptop!). The > laptop runs 100 times faster (for matrix-matrix multiplies of > 3000x3000 dense random matrices of doubles). > > It must be because the Solaris install is not using blas, but the > numpy scripts are reporting that the libs are 'found'. > > ??? $python3 -c "import numpy.distutils.system_info as f; d = > f.get_info('blas',0); print(d); d = f.get_info('lapack',0); print(d)" > ???? {'libraries': ['sunperf'], 'library_dirs': > ['/home/$myname/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib'], > 'language': 'f77'} > ???? {'libraries': ['sunmath'], 'library_dirs': > ['/home/$myname/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib'],'language': > 'f77'} > > The following import FAILS on the Solaris install but succeeds on OSX: > > import numpy.core._dotblas > > There is no ATLAS available for the Solaris install. I wouldn't think > this would make such a huge different in computational efficiency. Actually, it will make a different for this exact case, because dotblas (which implements numpy.dot) depends on ATLAS, not on BLAS itself (this is a bug in numpy that I meant to fix for ages). Mac OS X includes ATLAS in its accelerate framework. If BLAS/LAPACK from Solaris was correctly linked, you should be able to see fast operations for pretty much everything else. For example, how fast is SVD or eig for a 1000x1000 matrix ? cheers, David From david.cottrell at gmail.com Tue Sep 6 16:13:10 2011 From: david.cottrell at gmail.com (David Cottrell) Date: Tue, 6 Sep 2011 16:13:10 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: Message-ID: Thanks, I didn't realize dot was not just calling dgemm or some variant which I assume would be reasonably fast. I see dgemm appears in the numpy code in various places such as the lapack_lite module. I ran the svd test on the solaris setup and will check the OSX run when back at my laptop. 8.4 seconds is slightly slower than matlab but still seems reasonable. $ ./test_03.py No ATLAS: (1000, 1000) (1000,) (1000, 1000) 8.17235898972 $ cat test_03.py #!/usr/bin/env python3 import numpy import time try: import numpy.core._dotblas print('Using ATLAS:') except ImportError: print('No ATLAS:') import numpy.linalg N = 1000 x = numpy.random.random((N,N)) t = time.time() (U, s, V) = numpy.linalg.svd(x) print(U.shape, s.shape, V.shape) # S = numpy.matrix(numpy.diag(s)) # y = U * S * V #print(y.shape) print(time.time()-t) On Tue, Sep 6, 2011 at 2:59 PM, David Cournapeau wrote: > On Tue, Sep 6, 2011 at 2:38 PM, David Cottrell wrote: >> I posted on stackoverflow but then noticed this message board: >> >> http://stackoverflow.com/questions/7311869/python-numpy-on-solaris-blas-slow-or-not-linked >> >> I'm reposting the full post below: >> >> Matrix-Matrix multiplies are very slow on my Solaris install (running >> on a sparc server) compared to my OSX install (on a laptop!). The >> laptop runs 100 times faster (for matrix-matrix multiplies of >> 3000x3000 dense random matrices of doubles). >> >> It must be because the Solaris install is not using blas, but the >> numpy scripts are reporting that the libs are 'found'. >> >> ??? $python3 -c "import numpy.distutils.system_info as f; d = >> f.get_info('blas',0); print(d); d = f.get_info('lapack',0); print(d)" >> ???? {'libraries': ['sunperf'], 'library_dirs': >> ['/home/$myname/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib'], >> 'language': 'f77'} >> ???? {'libraries': ['sunmath'], 'library_dirs': >> ['/home/$myname/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib'],'language': >> 'f77'} >> >> The following import FAILS on the Solaris install but succeeds on OSX: >> >> import numpy.core._dotblas >> >> There is no ATLAS available for the Solaris install. I wouldn't think >> this would make such a huge different in computational efficiency. > > Actually, it will make a different for this exact case, because > dotblas (which implements numpy.dot) depends on ATLAS, not on BLAS > itself (this is a bug in numpy that I meant to fix for ages). Mac OS X > includes ATLAS in its accelerate framework. If BLAS/LAPACK from > Solaris was correctly linked, you should be able to see fast > operations for pretty much everything else. For example, how fast is > SVD or eig for a 1000x1000 matrix ? > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- David Cottrell +1 416 995 9860 http://ca.linkedin.com/in/dcottrell From fredmfp at gmail.com Tue Sep 6 16:16:50 2011 From: fredmfp at gmail.com (Fred) Date: Tue, 06 Sep 2011 22:16:50 +0200 Subject: [Numpy-discussion] gradient with memmap array bug (?) In-Reply-To: References: <4D4FE3BB.90405@gmail.com> <4D67782A.3000205@gmail.com> <20110225094239.GF24960@phare.normalesup.org> <4E64A7D6.9060901@gmail.com> Message-ID: <4E667FB2.9000005@gmail.com> Le 05/09/2011 18:41, Ralf Gommers a ?crit : > That rings a bell. I can reproduce this with 1.5.1, and it's fixed in > master. Thanks! Cheers, -- Fred From david.cottrell at gmail.com Tue Sep 6 17:12:46 2011 From: david.cottrell at gmail.com (David Cottrell) Date: Tue, 6 Sep 2011 17:12:46 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: Message-ID: Actually this link: http://www.scipy.org/PerformanceTips seems to indicate that numpy.dot does use blas ... Is there some way of running ldd on the install to see what libraries are being pulled in? On Tue, Sep 6, 2011 at 4:13 PM, David Cottrell wrote: > Thanks, I didn't realize dot was not just calling dgemm or some > variant which I assume would be reasonably fast. I see dgemm appears > in the numpy code in various places such as the lapack_lite module. > > I ran the svd test on the solaris setup and will check the OSX run > when back at my laptop. 8.4 seconds is slightly slower than matlab but > still seems reasonable. > > $ ./test_03.py > No ATLAS: > (1000, 1000) (1000,) (1000, 1000) > 8.17235898972 > > ?$ cat test_03.py > #!/usr/bin/env python3 > import numpy > import time > > try: > ? import numpy.core._dotblas > ? print('Using ATLAS:') > except ImportError: > ? print('No ATLAS:') > > import numpy.linalg > > N = 1000 > x = numpy.random.random((N,N)) > t = time.time() > (U, s, V) = numpy.linalg.svd(x) > print(U.shape, s.shape, V.shape) > # S = numpy.matrix(numpy.diag(s)) > # y = U * S * V > #print(y.shape) > > print(time.time()-t) > > > > > On Tue, Sep 6, 2011 at 2:59 PM, David Cournapeau wrote: >> On Tue, Sep 6, 2011 at 2:38 PM, David Cottrell wrote: >>> I posted on stackoverflow but then noticed this message board: >>> >>> http://stackoverflow.com/questions/7311869/python-numpy-on-solaris-blas-slow-or-not-linked >>> >>> I'm reposting the full post below: >>> >>> Matrix-Matrix multiplies are very slow on my Solaris install (running >>> on a sparc server) compared to my OSX install (on a laptop!). The >>> laptop runs 100 times faster (for matrix-matrix multiplies of >>> 3000x3000 dense random matrices of doubles). >>> >>> It must be because the Solaris install is not using blas, but the >>> numpy scripts are reporting that the libs are 'found'. >>> >>> ??? $python3 -c "import numpy.distutils.system_info as f; d = >>> f.get_info('blas',0); print(d); d = f.get_info('lapack',0); print(d)" >>> ???? {'libraries': ['sunperf'], 'library_dirs': >>> ['/home/$myname/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib'], >>> 'language': 'f77'} >>> ???? {'libraries': ['sunmath'], 'library_dirs': >>> ['/home/$myname/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib'],'language': >>> 'f77'} >>> >>> The following import FAILS on the Solaris install but succeeds on OSX: >>> >>> import numpy.core._dotblas >>> >>> There is no ATLAS available for the Solaris install. I wouldn't think >>> this would make such a huge different in computational efficiency. >> >> Actually, it will make a different for this exact case, because >> dotblas (which implements numpy.dot) depends on ATLAS, not on BLAS >> itself (this is a bug in numpy that I meant to fix for ages). Mac OS X >> includes ATLAS in its accelerate framework. If BLAS/LAPACK from >> Solaris was correctly linked, you should be able to see fast >> operations for pretty much everything else. For example, how fast is >> SVD or eig for a 1000x1000 matrix ? >> >> cheers, >> >> David >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > David Cottrell > +1 416 995 9860 > http://ca.linkedin.com/in/dcottrell > -- David Cottrell +1 416 995 9860 http://ca.linkedin.com/in/dcottrell From cournape at gmail.com Tue Sep 6 17:35:04 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Sep 2011 17:35:04 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: Message-ID: On Tue, Sep 6, 2011 at 5:12 PM, David Cottrell wrote: > Actually this link: http://www.scipy.org/PerformanceTips seems to > indicate that numpy.dot does use blas ... This not true (you can check by looking into numpy/core/setup.py, which explicitly checks for ATLAS for _dotblas). The idea is that numpy can be built without a fortran compiler, but linking against blas requires a fortran compiler. By depending on ATLAS, we can still get a fast numpy.dot in the cases where a fortran compiler is not available. But strictly speaking, the code itself only depends on CBLAS availability. The issue being that some CBLAS libraries still require a fortran compiler to be linked against... > > Is there some way of running ldd on the install to see what libraries > are being pulled in? Yes, just use ldd on the .so inside a numpy installation. Generally, python extensions are simply binaries that be be dlopened, cheers, David From scipy at samueljohn.de Wed Sep 7 09:08:27 2011 From: scipy at samueljohn.de (Samuel John) Date: Wed, 7 Sep 2011 15:08:27 +0200 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: Message-ID: <919343ED-3F54-4B20-82BB-3267630579E3@samueljohn.de> On 06.09.2011, at 22:13, David Cottrell wrote: > Thanks, I didn't realize dot was not just calling dgemm or some > variant which I assume would be reasonably fast. I see dgemm appears > in the numpy code in various places such as the lapack_lite module. > > I ran the svd test on the solaris setup and will check the OSX run > when back at my laptop. 8.4 seconds is slightly slower than matlab but > still seems reasonable. > > $ ./test_03.py > No ATLAS: > (1000, 1000) (1000,) (1000, 1000) > 8.17235898972 I just ran your benchmark code on OSX 10.7.1 on an 2011 MacBook Pro (core-i7) with numpy.version.version '2.0.0.dev-900d82e': Using ATLAS: ((1000, 1000), (1000,), (1000, 1000)) 0.908223152161 cheers, Samuel From maillist at schwertberger.de Wed Sep 7 14:27:15 2011 From: maillist at schwertberger.de (Dietmar Schwertberger) Date: Wed, 07 Sep 2011 20:27:15 +0200 Subject: [Numpy-discussion] Numeric 24.2 for Python 2.4 on Windows 7? Message-ID: <4E67B783.90604@schwertberger.de> Hi! Does anybody have Numeric 24.2 binaries for Python 2.4 which are compatible with Windows 7 SP1? Regards, Dietmar From cgohlke at uci.edu Wed Sep 7 15:07:02 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Wed, 07 Sep 2011 12:07:02 -0700 Subject: [Numpy-discussion] Numeric 24.2 for Python 2.4 on Windows 7? In-Reply-To: <4E67B783.90604@schwertberger.de> References: <4E67B783.90604@schwertberger.de> Message-ID: <4E67C0D6.1070803@uci.edu> On 9/7/2011 11:27 AM, Dietmar Schwertberger wrote: > Hi! > > Does anybody have Numeric 24.2 binaries for Python 2.4 which are > compatible with Windows 7 SP1? > > > Regards, > > Dietmar > simply recompiling from source seems to work: Christoph From maillist at schwertberger.de Wed Sep 7 15:39:28 2011 From: maillist at schwertberger.de (Dietmar Schwertberger) Date: Wed, 07 Sep 2011 21:39:28 +0200 Subject: [Numpy-discussion] Numeric 24.2 for Python 2.4 on Windows 7? In-Reply-To: <4E67C0D6.1070803@uci.edu> References: <4E67B783.90604@schwertberger.de> <4E67C0D6.1070803@uci.edu> Message-ID: <4E67C870.2030700@schwertberger.de> Am 07.09.2011 21:07, schrieb Christoph Gohlke: > On 9/7/2011 11:27 AM, Dietmar Schwertberger wrote: >> Does anybody have Numeric 24.2 binaries for Python 2.4 which are >> compatible with Windows 7 SP1? > simply recompiling from source seems to work: > > A cool page, but not too obvious to find. Thanks a lot. Usually I'm using Python 2.7 and will build my the modules myself, but for this old project on 2.4 I did not want to install the old compilers again... Regards, Dietmar From Chris.Barker at noaa.gov Wed Sep 7 15:52:44 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Wed, 07 Sep 2011 12:52:44 -0700 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> Message-ID: <4E67CB8C.3000902@noaa.gov> On 9/2/11 2:45 PM, Christopher Jordan-Squire wrote: > It doesn't have to parse the entire file to determine the dtypes. It > builds up a regular expression for what it expects to see, in terms of > dtypes. Then it just loops over the lines, only parsing if the regular > expression doesn't match. It seems that a regex match is fast, but a > regex fail is expensive. interesting -- I wouldn't have expected a regex to be faster that simple parsing, but that's why you profile! > Setting array elements is not as fast for the masked record arrays. > You must set entire rows at a time, so I have to build up each row as > a list, and convert to a tuple, and then stuff it in the array. hmmm -- that is a lot -- I was thinking of simple "set a value in an array". I"ve also done a bunch of this in C, where's it's really fast. However, rather than: build a row as a list build a row as a tuple stuff into array could you create an empty array scalar, and fill that, then put that in your array: In [4]: dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)]) In [5]: dt Out[5]: dtype([('x', ' it's even slower for the record arrays with missing data because I > must branch between adding missing data versus adding real data. Might > that be the reason for the slower performance than you'd expect? could be -- I haven't thought about the missing data part much. > I wonder if there are any really important cases where you'd actually > lose something by simply recasting an entry to another dtype, as Derek > suggested. In general, it won't be a simple re-cast -- it will be a copy to a subset -- which may be hard to write the code, but would save having to re-parse the data. Anyway, you know the issues, this is good stuff either way. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cekees at gmail.com Wed Sep 7 16:52:55 2011 From: cekees at gmail.com (Chris Kees) Date: Wed, 7 Sep 2011 15:52:55 -0500 Subject: [Numpy-discussion] what to use in buildbot config for numpy testing Message-ID: Hi, Is there a recommended way to run the numpy test suite as a buildbot test? Just run ad python -c "import numpy; numpy.test" as ShellCommand object? Thanks, Chris From derek at astro.physik.uni-goettingen.de Wed Sep 7 17:22:39 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Wed, 7 Sep 2011 23:22:39 +0200 Subject: [Numpy-discussion] what to use in buildbot config for numpy testing In-Reply-To: References: Message-ID: On 07.09.2011, at 10:52PM, Chris Kees wrote: > Is there a recommended way to run the numpy test suite as a buildbot > test? Just run ad python -c "import numpy; numpy.test" as ShellCommand > object? It would be numpy.test() [or numpy.test('full')]; then it depends on what you need as the return value of your test. I am using for package verification python -c 'import numpy, sys; ret=numpy.test("full"); sys.exit(2*len(ret.errors+ret.failures))' so python will return with a value != 0 in case of an unsuccessful test (which it otherwise would not do). But this is just within a simple shell script. Cheers, Derek -- ---------------------------------------------------------------- Derek Homeier Centre de Recherche Astrophysique de Lyon ENS Lyon 46, All?e d'Italie 69364 Lyon Cedex 07, France +33 1133 47272-8894 ---------------------------------------------------------------- From cekees at gmail.com Wed Sep 7 17:44:11 2011 From: cekees at gmail.com (Chris Kees) Date: Wed, 7 Sep 2011 16:44:11 -0500 Subject: [Numpy-discussion] what to use in buildbot config for numpy testing In-Reply-To: References: Message-ID: Hi Derek, Thanks! I forgot that python would exit with 0 even if numpy.test() fails. That could have taken a while to realize. Chris On Wed, Sep 7, 2011 at 4:22 PM, Derek Homeier wrote: > On 07.09.2011, at 10:52PM, Chris Kees wrote: > >> Is there a recommended way to run the numpy test suite as a buildbot >> test? Just run ad python -c "import numpy; numpy.test" as ShellCommand >> object? > > It would be numpy.test() [or numpy.test('full')]; then it depends on what you need > as the return value of your test. I am using for package verification > > python -c 'import numpy, sys; ret=numpy.test("full"); sys.exit(2*len(ret.errors+ret.failures))' > > so python will return with a value != 0 in case of an unsuccessful test (which it > otherwise would not do). But this is just within a simple shell script. > > Cheers, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Derek > -- > ---------------------------------------------------------------- > Derek Homeier ? ? ? ? ?Centre de Recherche Astrophysique de Lyon > ENS Lyon ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?46, All?e d'Italie > 69364 Lyon Cedex 07, France ? ? ? ? ? ? ? ? ?+33 1133 47272-8894 > ---------------------------------------------------------------- > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pav at iki.fi Thu Sep 8 03:43:48 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 8 Sep 2011 07:43:48 +0000 (UTC) Subject: [Numpy-discussion] load from text files Pull Request Review References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> Message-ID: Wed, 07 Sep 2011 12:52:44 -0700, Chris.Barker wrote: [clip] > In [9]: temp['x'] = 3 > > In [10]: temp['y'] = 4 > > In [11]: temp['z'] = 5 [clip] > maybe it wouldn't be any faster, but with re-using temp, and one less > list-tuple conversion, and fewer python type to numpy type conversions, > maybe it would. Structured array assignments have plenty of overhead in Numpy, so it could be slower, too: x = np.array((1,2), dtype=[('a', int), ('b', float)]) x2 = [1,2,3] %timeit x['a'] = 9 100000 loops, best of 3: 2.83 us per loop %timeit x2[0] = 9 1000000 loops, best of 3: 368 ns per loop From marquett at iap.fr Thu Sep 8 03:27:01 2011 From: marquett at iap.fr (Jean-Baptiste Marquette) Date: Thu, 8 Sep 2011 09:27:01 +0200 Subject: [Numpy-discussion] what to use in buildbot config for numpy testing In-Reply-To: References: Message-ID: <23E78CA0-B740-4C32-AAD6-485B99EB2DB5@iap.fr> There is much more simple: the command nosetests numpy Of course, nosetests needs to be installed... Cheers JB Le 7 sept. 2011 ? 23:44, Chris Kees a ?crit : > Hi Derek, > > Thanks! I forgot that python would exit with 0 even if numpy.test() > fails. That could have taken a while to realize. > > Chris > > On Wed, Sep 7, 2011 at 4:22 PM, Derek Homeier > wrote: >> On 07.09.2011, at 10:52PM, Chris Kees wrote: >> >>> Is there a recommended way to run the numpy test suite as a buildbot >>> test? Just run ad python -c "import numpy; numpy.test" as ShellCommand >>> object? >> >> It would be numpy.test() [or numpy.test('full')]; then it depends on what you need >> as the return value of your test. I am using for package verification >> >> python -c 'import numpy, sys; ret=numpy.test("full"); sys.exit(2*len(ret.errors+ret.failures))' >> >> so python will return with a value != 0 in case of an unsuccessful test (which it >> otherwise would not do). But this is just within a simple shell script. >> >> Cheers, >> Derek >> -- >> ---------------------------------------------------------------- >> Derek Homeier Centre de Recherche Astrophysique de Lyon >> ENS Lyon 46, All?e d'Italie >> 69364 Lyon Cedex 07, France +33 1133 47272-8894 >> ---------------------------------------------------------------- >> >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From Jim.Vickroy at noaa.gov Thu Sep 8 07:56:43 2011 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Thu, 08 Sep 2011 05:56:43 -0600 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 Message-ID: <4E68AD7B.2080705@noaa.gov> Hello All, I'm attempting to create a python wrapper, for a Fortran subroutine, using f2py. My system details are: >>> sys.version '2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)]' >>> sys.getwindowsversion() (5, 1, 2600, 2, 'Service Pack 3') >>> scipy.__version__ '0.7.1' >>> numpy.__version__ '1.4.0' C:\>gfortran -dumpversion 4.7.0 C:\Python26\Lib\site-packages\numpy\f2py>f2py.py -c --help-fcompiler Traceback (most recent call last): File "C:\Python26\Scripts\f2py.py", line 24, in main() File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557, in main run_compile() File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543, in run_compile setup(ext_modules = [ext]) File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line 186, in setup return old_setup(**new_attr) File "C:\Python26\lib\distutils\core.py", line 138, in setup ok = dist.parse_command_line() File "C:\Python26\lib\distutils\dist.py", line 460, in parse_command_line args = self._parse_command_opts(parser, args) File "C:\Python26\lib\distutils\dist.py", line 574, in _parse_command_opts func() File "C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py", line 13, in show_fortran_compilers show_fcompilers(dist) File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", line 855, in show_fcompilers c.customize(dist) File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", line 525, in customize self.set_libraries(self.get_libraries()) File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\gnu.py", line 306, in get_libraries raise NotImplementedError("Only MS compiler supported with gfortran on win64") NotImplementedError: Only MS compiler supported with gfortran on win64 Could someone help me to resolve this? Thanks, -- jv From sole at esrf.fr Thu Sep 8 08:09:52 2011 From: sole at esrf.fr (=?ISO-8859-1?Q?=22V=2E_Armando_Sol=E9=22?=) Date: Thu, 08 Sep 2011 14:09:52 +0200 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E68AD7B.2080705@noaa.gov> References: <4E68AD7B.2080705@noaa.gov> Message-ID: <4E68B090.7030001@esrf.fr> Have you tried to install Visual Studio 2008 Express edition (plus the windows SDK to be able to compile 64 bit code)? Armando On 08/09/2011 13:56, Jim Vickroy wrote: > Hello All, I'm attempting to create a python wrapper, for a Fortran > subroutine, using f2py. > > My system details are: > > >>> sys.version '2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 > 32 bit (Intel)]' > >>> sys.getwindowsversion() (5, 1, 2600, 2, 'Service Pack 3') > >>> scipy.__version__ '0.7.1' > >>> numpy.__version__ '1.4.0' > C:\>gfortran -dumpversion > 4.7.0 > > > C:\Python26\Lib\site-packages\numpy\f2py>f2py.py -c --help-fcompiler > Traceback (most recent call last): > File "C:\Python26\Scripts\f2py.py", line 24, in > main() > File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557, > in main > run_compile() > File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543, > in run_compile > setup(ext_modules = [ext]) > File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line > 186, in setup > return old_setup(**new_attr) > File "C:\Python26\lib\distutils\core.py", line 138, in setup > ok = dist.parse_command_line() > File "C:\Python26\lib\distutils\dist.py", line 460, in parse_command_line > args = self._parse_command_opts(parser, args) > File "C:\Python26\lib\distutils\dist.py", line 574, in > _parse_command_opts > func() > File > "C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py", > line 13, in show_fortran_compilers > show_fcompilers(dist) > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", > line 855, in show_fcompilers > c.customize(dist) > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", > line 525, in customize > self.set_libraries(self.get_libraries()) > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\gnu.py", line > 306, in get_libraries > raise NotImplementedError("Only MS compiler supported with gfortran > on win64") > NotImplementedError: Only MS compiler supported with gfortran on win64 > > > Could someone help me to resolve this? > > Thanks, -- jv > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Jim.Vickroy at noaa.gov Thu Sep 8 10:16:20 2011 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Thu, 08 Sep 2011 08:16:20 -0600 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E68B090.7030001@esrf.fr> References: <4E68AD7B.2080705@noaa.gov> <4E68B090.7030001@esrf.fr> Message-ID: <4E68CE34.2050406@noaa.gov> On 9/8/2011 6:09 AM, "V. Armando Sol?" wrote: > Have you tried to install Visual Studio 2008 Express edition (plus the > windows SDK to be able to compile 64 bit code)? > > Armando Armando, "Visual Studio 2008 Professional" is installed on the computer as well as "Intel Visual Fortran Composer XE 2011". f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I tried gfortran. The "Win64" reference, in the Exception, is puzzling to me since this is a 32-bit computer. > > On 08/09/2011 13:56, Jim Vickroy wrote: >> Hello All, I'm attempting to create a python wrapper, for a Fortran >> subroutine, using f2py. >> >> My system details are: >> >> >>> sys.version '2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 >> 32 bit (Intel)]' >> >>> sys.getwindowsversion() (5, 1, 2600, 2, 'Service Pack 3') >> >>> scipy.__version__ '0.7.1' >> >>> numpy.__version__ '1.4.0' >> C:\>gfortran -dumpversion >> 4.7.0 >> >> >> C:\Python26\Lib\site-packages\numpy\f2py>f2py.py -c --help-fcompiler >> Traceback (most recent call last): >> File "C:\Python26\Scripts\f2py.py", line 24, in >> main() >> File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557, >> in main >> run_compile() >> File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543, >> in run_compile >> setup(ext_modules = [ext]) >> File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line >> 186, in setup >> return old_setup(**new_attr) >> File "C:\Python26\lib\distutils\core.py", line 138, in setup >> ok = dist.parse_command_line() >> File "C:\Python26\lib\distutils\dist.py", line 460, in parse_command_line >> args = self._parse_command_opts(parser, args) >> File "C:\Python26\lib\distutils\dist.py", line 574, in >> _parse_command_opts >> func() >> File >> "C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py", >> line 13, in show_fortran_compilers >> show_fcompilers(dist) >> File >> "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", >> line 855, in show_fcompilers >> c.customize(dist) >> File >> "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", >> line 525, in customize >> self.set_libraries(self.get_libraries()) >> File >> "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\gnu.py", line >> 306, in get_libraries >> raise NotImplementedError("Only MS compiler supported with gfortran >> on win64") >> NotImplementedError: Only MS compiler supported with gfortran on win64 >> >> >> Could someone help me to resolve this? >> >> Thanks, -- jv >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From sole at esrf.fr Thu Sep 8 12:44:29 2011 From: sole at esrf.fr (=?ISO-8859-1?Q?=22V=2E_Armando_Sol=E9=22?=) Date: Thu, 08 Sep 2011 18:44:29 +0200 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E68CE34.2050406@noaa.gov> References: <4E68AD7B.2080705@noaa.gov> <4E68B090.7030001@esrf.fr> <4E68CE34.2050406@noaa.gov> Message-ID: <4E68F0ED.2090702@esrf.fr> On 08/09/2011 16:16, Jim Vickroy wrote: > On 9/8/2011 6:09 AM, "V. Armando Sol?" wrote: >> Have you tried to install Visual Studio 2008 Express edition (plus the >> windows SDK to be able to compile 64 bit code)? >> >> Armando > Armando, "Visual Studio 2008 Professional" is installed on the computer > as well as "Intel Visual Fortran Composer XE 2011". > > f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I > tried gfortran. > > The "Win64" reference, in the Exception, is puzzling to me since this is > a 32-bit computer. > Oh! I totally misunderstood the situation. I thought the problem was the missing compiler. All what I do with python and the intel fortran compiler is to compile numpy. Just in case it helps you, I set my environment from the console by running a bat file with the following content (I am on 64 bit but you could easily tailor it to your needs): "C:\Program Files\Microsoft SDKs\Windows\v7.0\Setup\WindowsSdkVer.exe" -version:v7.0 call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat" call "C:\Program Files (x86)\Intel\ComposerXE-2011\bin\ipsxe-comp-vars.bat" intel64 vs2008shell rem call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64" rem call "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\setenv.cmd" /x64 /Release set PATH=C:\Python27;C:\Python27\Scripts;%PATH% set PATH="C:\Program Files (x86)\Intel\ComposerXE-2011\redist\intel64\mkl";"C:\Program Files (x86)\Intel\ComposerXE-2011\mkl\lib\intel64";%PATH% Perhaps that helps you to set a working environment. All what I can tell you is that with that environment, if I run "python f2py.py -c --help-fcompiler" it finds the intel compiler. Good luck, Armando From ralf.gommers at googlemail.com Thu Sep 8 15:59:50 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 8 Sep 2011 21:59:50 +0200 Subject: [Numpy-discussion] strange test failure in Cython code with numpy master Message-ID: Hi, There's a test failure in scipy/io/matlab/mio_utils that shows up with numpy master but not 1.5.1, see http://projects.scipy.org/scipy/ticket/1512 I have a fix here: https://github.com/rgommers/scipy/commit/4ade7829649b9e2c251f5f7f370781b16fc13612, but I don't really understand why the code is failing in the first place. This is the relevant part: - arr = np.squeeze(arr) - if not arr.shape and arr.dtype.isbuiltin: # 0d coverted to scalar - return arr.item() - return arr + arr2 = np.squeeze(arr) + if (not arr2.shape) and arr2.dtype.isbuiltin: # 0d coverted to scalar + return arr2.item() + return arr2 All it does is rename arr to arr2. It fails with both Cython 0.13 and 0.15. Any idea? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Thu Sep 8 16:24:37 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Thu, 8 Sep 2011 13:24:37 -0700 Subject: [Numpy-discussion] strange test failure in Cython code with numpy master In-Reply-To: References: Message-ID: Probably has something to do with 'arr' having a statically defined type of cnp.ndarray. -Mark On Thu, Sep 8, 2011 at 12:59 PM, Ralf Gommers wrote: > Hi, > > There's a test failure in scipy/io/matlab/mio_utils that shows up with > numpy master but not 1.5.1, see > http://projects.scipy.org/scipy/ticket/1512 > I have a fix here: > https://github.com/rgommers/scipy/commit/4ade7829649b9e2c251f5f7f370781b16fc13612, but I don't really understand why the code is failing in the first place. > This is the relevant part: > > - arr = np.squeeze(arr) > - if not arr.shape and arr.dtype.isbuiltin: # 0d coverted to scalar > - return arr.item() > - return arr > + arr2 = np.squeeze(arr) > + if (not arr2.shape) and arr2.dtype.isbuiltin: # 0d coverted to scalar > + return arr2.item() > + return arr2 > > All it does is rename arr to arr2. It fails with both Cython 0.13 and 0.15. > Any idea? > > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjordan1 at uw.edu Thu Sep 8 16:43:43 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Thu, 8 Sep 2011 15:43:43 -0500 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E67CB8C.3000902@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> Message-ID: On Wed, Sep 7, 2011 at 2:52 PM, Chris.Barker wrote: > On 9/2/11 2:45 PM, Christopher Jordan-Squire wrote: >> It doesn't have to parse the entire file to determine the dtypes. It >> builds up a regular expression for what it expects to see, in terms of >> dtypes. Then it just loops over the lines, only parsing if the regular >> expression doesn't match. It seems that a regex match is fast, but a >> regex fail is expensive. > > interesting -- I wouldn't have expected a regex to be faster that simple > parsing, but that's why you profile! > >> Setting array elements is not as fast for the masked record arrays. >> You must set entire rows at a time, so I have to build up each row as >> a list, and convert to a tuple, and then stuff it in the array. > > hmmm -- that is a lot -- I was thinking of simple "set a value in an > array". I"ve also done a bunch of this in C, where's it's really fast. > > However, rather than: > > ? build a row as a list > ? build a row as a tuple > ? stuff into array > > could you create an empty array scalar, and fill that, then put that in > your array: > > In [4]: dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)]) > > In [5]: dt > Out[5]: dtype([('x', ' > In [6]: temp = np.empty((), dtype=dt) > > In [9]: temp['x'] = 3 > > In [10]: temp['y'] = 4 > > In [11]: temp['z'] = 5 > > In [13]: a = np.zeros((4,), dtype = dt) > > In [14]: a[0] = temp > > In [15]: a > Out[15]: > array([(3.0, 4, 5.0), (0.0, 0, 0.0), (0.0, 0, 0.0), (0.0, 0, 0.0)], > ? ? ? dtype=[('x', ' > > (and you could pass the array scalar into accumulator as well) > > maybe it wouldn't be any faster, but with re-using temp, and one less > list-tuple conversion, and fewer python type to numpy type conversions, > maybe it would. > I just ran a quick test on my machine of this idea. With dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)]) temp = np.empty((), dtype=dt) temp2 = np.zeros(1,dtype=dt) In [96]: def f(): ...: l=[0]*3 ...: l[0] = 2.54 ...: l[1] = 4 ...: l[2] = 2.3645 ...: j = tuple(l) ...: temp2[0] = j vs In [97]: def g(): ...: temp['x'] = 2.54 ...: temp['y'] = 4 ...: temp['z'] = 2.3645 ...: temp2[0] = temp ...: The timing results were 2.73 us for f and 3.43 us for g. So good idea, but it doesn't appear to be faster. (Though the difference wasn't nearly as dramatic as I thought it would be, based on Pauli's comment.) -Chris JS >> it's even slower for the record arrays with missing data because I >> must branch between adding missing data versus adding real data. Might >> that be the reason for the slower performance than you'd expect? > > could be -- I haven't thought about the missing data part much. > >> I wonder if there are any really important cases where you'd actually >> lose something by simply recasting an entry to another dtype, as Derek >> suggested. > > In general, it won't be a simple re-cast -- it will be a copy to a > subset -- which may be hard to write the code, but would save having to > re-parse the data. > > > Anyway, you know the issues, this is good stuff either way. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Thu Sep 8 16:57:59 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Thu, 08 Sep 2011 13:57:59 -0700 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> Message-ID: <4E692C57.2040501@noaa.gov> On 9/8/11 1:43 PM, Christopher Jordan-Squire wrote: > I just ran a quick test on my machine of this idea. With > > dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)]) > temp = np.empty((), dtype=dt) > temp2 = np.zeros(1,dtype=dt) > > In [96]: def f(): > ...: l=[0]*3 > ...: l[0] = 2.54 > ...: l[1] = 4 > ...: l[2] = 2.3645 > ...: j = tuple(l) > ...: temp2[0] = j > > vs > > > In [97]: def g(): > ...: temp['x'] = 2.54 > ...: temp['y'] = 4 > ...: temp['z'] = 2.3645 > ...: temp2[0] = temp > ...: > > The timing results were 2.73 us for f and 3.43 us for g. So good idea, > but it doesn't appear to be faster. (Though the difference wasn't > nearly as dramatic as I thought it would be, based on Pauli's > comment.) my guess is that the lines like: temp['x'] = 2.54 are slower (it requires a dict lookup, and a conversion from a python type to a "raw" type) and temp2[0] = temp is faster, as that doesn't require any conversion. Which means that if you has a larger struct dtype, it would be even slower, so clearly not the way to go for performance. It would be nice to have a higher performing struct dtype scalar -- as it is ordered, it might be nice to be able to index it with either the name or an numeric index. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Jim.Vickroy at noaa.gov Fri Sep 9 10:22:33 2011 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Fri, 09 Sep 2011 08:22:33 -0600 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E68F0ED.2090702@esrf.fr> References: <4E68AD7B.2080705@noaa.gov> <4E68B090.7030001@esrf.fr> <4E68CE34.2050406@noaa.gov> <4E68F0ED.2090702@esrf.fr> Message-ID: <4E6A2129.9050602@noaa.gov> On 9/8/2011 10:44 AM, "V. Armando Sol?" wrote: > On 08/09/2011 16:16, Jim Vickroy wrote: >> On 9/8/2011 6:09 AM, "V. Armando Sol?" wrote: >>> Have you tried to install Visual Studio 2008 Express edition (plus the >>> windows SDK to be able to compile 64 bit code)? >>> >>> Armando >> Armando, "Visual Studio 2008 Professional" is installed on the computer >> as well as "Intel Visual Fortran Composer XE 2011". >> >> f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I >> tried gfortran. >> >> The "Win64" reference, in the Exception, is puzzling to me since this is >> a 32-bit computer. >> > Oh! I totally misunderstood the situation. I thought the problem was the > missing compiler. > > All what I do with python and the intel fortran compiler is to compile > numpy. Just in case it helps you, I set my environment from the console > by running a bat file with the following content (I am on 64 bit but you > could easily tailor it to your needs): > > "C:\Program Files\Microsoft SDKs\Windows\v7.0\Setup\WindowsSdkVer.exe" > -version:v7.0 > call "C:\Program Files (x86)\Microsoft Visual Studio > 9.0\VC\bin\vcvars64.bat" > call "C:\Program Files > (x86)\Intel\ComposerXE-2011\bin\ipsxe-comp-vars.bat" intel64 vs2008shell > rem call "C:\Program Files (x86)\Microsoft Visual Studio > 9.0\VC\bin\vcvars64" > rem call "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\setenv.cmd" > /x64 /Release > set PATH=C:\Python27;C:\Python27\Scripts;%PATH% > set PATH="C:\Program Files > (x86)\Intel\ComposerXE-2011\redist\intel64\mkl";"C:\Program Files > (x86)\Intel\ComposerXE-2011\mkl\lib\intel64";%PATH% > > Perhaps that helps you to set a working environment. All what I can tell > you is that with that environment, if I run "python f2py.py -c > --help-fcompiler" it finds the intel compiler. > > Good luck, > > Armando Thanks for the suggestion. So far I have not been able to modify the above for use on my 32-bit machine such that ifort is found by f2py. From the f2py documentation, I assumed this was going to be rather straightforward, but it is not. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From Jim.Vickroy at noaa.gov Fri Sep 9 11:14:28 2011 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Fri, 09 Sep 2011 09:14:28 -0600 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E68AD7B.2080705@noaa.gov> References: <4E68AD7B.2080705@noaa.gov> Message-ID: <4E6A2D54.6050406@noaa.gov> On 9/8/2011 5:56 AM, Jim Vickroy wrote: > Hello All, I'm attempting to create a python wrapper, for a Fortran > subroutine, using f2py. > > My system details are: > > >>> sys.version '2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 > 32 bit (Intel)]' > >>> sys.getwindowsversion() (5, 1, 2600, 2, 'Service Pack 3') > >>> scipy.__version__ '0.7.1' > >>> numpy.__version__ '1.4.0' > C:\>gfortran -dumpversion > 4.7.0 > > > C:\Python26\Lib\site-packages\numpy\f2py>f2py.py -c --help-fcompiler > Traceback (most recent call last): > File "C:\Python26\Scripts\f2py.py", line 24, in > main() > File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557, > in main > run_compile() > File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543, > in run_compile > setup(ext_modules = [ext]) > File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line > 186, in setup > return old_setup(**new_attr) > File "C:\Python26\lib\distutils\core.py", line 138, in setup > ok = dist.parse_command_line() > File "C:\Python26\lib\distutils\dist.py", line 460, in parse_command_line > args = self._parse_command_opts(parser, args) > File "C:\Python26\lib\distutils\dist.py", line 574, in > _parse_command_opts > func() > File > "C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py", > line 13, in show_fortran_compilers > show_fcompilers(dist) > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", > line 855, in show_fcompilers > c.customize(dist) > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", > line 525, in customize > self.set_libraries(self.get_libraries()) > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\gnu.py", line > 306, in get_libraries > raise NotImplementedError("Only MS compiler supported with gfortran > on win64") > NotImplementedError: Only MS compiler supported with gfortran on win64 > > > Could someone help me to resolve this? > > Thanks, -- jv I eliminated the above NotImplementedError by replacing this raise exception line with a pass statement. Now, gfortran is being found and used by f2py but subsequently fails as follows: gfortran.exe: error: unrecognized command line option '-mno-cygwin' Why is f2py invoking gfortran with a cygwin option on a Windows XP machine? I supposedly installed the stand-alone version of gfortran and my take on the f2py documentation was that cygwin was not required. Note, gfortran can be used from a Windows commandline to successfully compile the code. Is anyone successfully using f2py and gfortran on a Windows machine without relying on cygwin? I'm not committed to gfortran; I also have the Intel Visual Fortran compiler (ifort) installed, but so far, I have been unable to get f2py to acknowledge that it exists. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From cgohlke at uci.edu Fri Sep 9 13:46:41 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Fri, 09 Sep 2011 10:46:41 -0700 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E6A2129.9050602@noaa.gov> References: <4E68AD7B.2080705@noaa.gov> <4E68B090.7030001@esrf.fr> <4E68CE34.2050406@noaa.gov> <4E68F0ED.2090702@esrf.fr> <4E6A2129.9050602@noaa.gov> Message-ID: <4E6A5101.90002@uci.edu> On 9/9/2011 7:22 AM, Jim Vickroy wrote: > On 9/8/2011 10:44 AM, "V. Armando Sol?" wrote: >> On 08/09/2011 16:16, Jim Vickroy wrote: >>> On 9/8/2011 6:09 AM, "V. Armando Sol?" wrote: >>>> Have you tried to install Visual Studio 2008 Express edition (plus the >>>> windows SDK to be able to compile 64 bit code)? >>>> >>>> Armando >>> Armando, "Visual Studio 2008 Professional" is installed on the computer >>> as well as "Intel Visual Fortran Composer XE 2011". >>> >>> f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I >>> tried gfortran. >>> >>> The "Win64" reference, in the Exception, is puzzling to me since this is >>> a 32-bit computer. >>> >> Oh! I totally misunderstood the situation. I thought the problem was the >> missing compiler. >> >> All what I do with python and the intel fortran compiler is to compile >> numpy. Just in case it helps you, I set my environment from the console >> by running a bat file with the following content (I am on 64 bit but you >> could easily tailor it to your needs): >> >> "C:\Program Files\Microsoft SDKs\Windows\v7.0\Setup\WindowsSdkVer.exe" >> -version:v7.0 >> call "C:\Program Files (x86)\Microsoft Visual Studio >> 9.0\VC\bin\vcvars64.bat" >> call "C:\Program Files >> (x86)\Intel\ComposerXE-2011\bin\ipsxe-comp-vars.bat" intel64 vs2008shell >> rem call "C:\Program Files (x86)\Microsoft Visual Studio >> 9.0\VC\bin\vcvars64" >> rem call "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\setenv.cmd" >> /x64 /Release >> set PATH=C:\Python27;C:\Python27\Scripts;%PATH% >> set PATH="C:\Program Files >> (x86)\Intel\ComposerXE-2011\redist\intel64\mkl";"C:\Program Files >> (x86)\Intel\ComposerXE-2011\mkl\lib\intel64";%PATH% >> >> Perhaps that helps you to set a working environment. All what I can tell >> you is that with that environment, if I run "python f2py.py -c >> --help-fcompiler" it finds the intel compiler. >> >> Good luck, >> >> Armando > > Thanks for the suggestion. So far I have not been able to modify the > above for use on my 32-bit machine such that ifort is found by f2py. > > From the f2py documentation, I assumed this was going to be rather > straightforward, but it is not. > > There should be a file named "ifortvars.bat" in the Intel Compiler bin directory. Call it with the right arguments before using f2py. On my system it is: "C:\Program Files (x86)\Intel\Compiler\11.1\070\bin\ifortvars.bat" intel64 vs2008 Christoph From Jim.Vickroy at noaa.gov Fri Sep 9 14:43:34 2011 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Fri, 09 Sep 2011 12:43:34 -0600 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E6A5101.90002@uci.edu> References: <4E68AD7B.2080705@noaa.gov> <4E68B090.7030001@esrf.fr> <4E68CE34.2050406@noaa.gov> <4E68F0ED.2090702@esrf.fr> <4E6A2129.9050602@noaa.gov> <4E6A5101.90002@uci.edu> Message-ID: <4E6A5E56.6040606@noaa.gov> On 9/9/2011 11:46 AM, Christoph Gohlke wrote: > > On 9/9/2011 7:22 AM, Jim Vickroy wrote: >> On 9/8/2011 10:44 AM, "V. Armando Sol?" wrote: >>> On 08/09/2011 16:16, Jim Vickroy wrote: >>>> On 9/8/2011 6:09 AM, "V. Armando Sol?" wrote: >>>>> Have you tried to install Visual Studio 2008 Express edition (plus the >>>>> windows SDK to be able to compile 64 bit code)? >>>>> >>>>> Armando >>>> Armando, "Visual Studio 2008 Professional" is installed on the computer >>>> as well as "Intel Visual Fortran Composer XE 2011". >>>> >>>> f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I >>>> tried gfortran. >>>> >>>> The "Win64" reference, in the Exception, is puzzling to me since this is >>>> a 32-bit computer. >>>> >>> Oh! I totally misunderstood the situation. I thought the problem was the >>> missing compiler. >>> >>> All what I do with python and the intel fortran compiler is to compile >>> numpy. Just in case it helps you, I set my environment from the console >>> by running a bat file with the following content (I am on 64 bit but you >>> could easily tailor it to your needs): >>> >>> "C:\Program Files\Microsoft SDKs\Windows\v7.0\Setup\WindowsSdkVer.exe" >>> -version:v7.0 >>> call "C:\Program Files (x86)\Microsoft Visual Studio >>> 9.0\VC\bin\vcvars64.bat" >>> call "C:\Program Files >>> (x86)\Intel\ComposerXE-2011\bin\ipsxe-comp-vars.bat" intel64 vs2008shell >>> rem call "C:\Program Files (x86)\Microsoft Visual Studio >>> 9.0\VC\bin\vcvars64" >>> rem call "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\setenv.cmd" >>> /x64 /Release >>> set PATH=C:\Python27;C:\Python27\Scripts;%PATH% >>> set PATH="C:\Program Files >>> (x86)\Intel\ComposerXE-2011\redist\intel64\mkl";"C:\Program Files >>> (x86)\Intel\ComposerXE-2011\mkl\lib\intel64";%PATH% >>> >>> Perhaps that helps you to set a working environment. All what I can tell >>> you is that with that environment, if I run "python f2py.py -c >>> --help-fcompiler" it finds the intel compiler. >>> >>> Good luck, >>> >>> Armando >> Thanks for the suggestion. So far I have not been able to modify the >> above for use on my 32-bit machine such that ifort is found by f2py. >> >> From the f2py documentation, I assumed this was going to be rather >> straightforward, but it is not. >> >> > There should be a file named "ifortvars.bat" in the Intel Compiler bin > directory. Call it with the right arguments before using f2py. On my > system it is: > > "C:\Program Files (x86)\Intel\Compiler\11.1\070\bin\ifortvars.bat" > intel64 vs2008 > > Christoph Thanks Christoph. Unfortunately, this does not seem to enable f2py to find ifort. Here is the result of running ifortvars.bat immediately followed by f2py: >"C:\Program Files\Intel\ComposerXE-2011\bin\ifortvars.bat" ia32 Intel(R) Parallel Studio XE 2011 Update 1 Copyright (C) 1985-2011 Intel Corporation. All rights reserved. Intel(R) Composer XE 2011 Update 3 (package 175) Setting environment for using Microsoft Visual Studio 2008 x86 tools. >f2py.py -c --help-fcompiler Gnu95FCompiler instance properties: archiver = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- cr'] compile_switch = '-c' compiler_f77 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-ffixed-form', '-fno-second-underscore', '-mno- cygwin', '-O3', '-funroll-loops'] compiler_f90 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-fno-second-underscore', '-mno-cygwin', '-O3', '- funroll-loops'] compiler_fix = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-ffixed-form', '-fno-second-underscore', '-mno- cygwin', '-Wall', '-fno-second-underscore', '-mno-cygwin', '-O3', '-funroll-loops'] libraries = ['gfortran'] library_dirs = ["gfortran.exe: error: unrecognized command line option '-mno-cygwin'\nc:\\program files\\gfortran\\lib\\gcc\\i586 -pc-mingw32\\4.7.0"] linker_exe = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-mno-cygwin', '-Wall', '-mno-cygwin'] linker_so = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-mno-cygwin', '-Wall', '-mno-cygwin', '-shared'] object_switch = '-o ' ranlib = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe'] version = LooseVersion ('4.7.0') version_cmd = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '-- version', '-mno-cygwin'] Fortran compilers found: --fcompiler=gnu95 GNU Fortran 95 compiler (4.7.0) Compilers available for this platform, but not found: --fcompiler=absoft Absoft Corp Fortran Compiler --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler --fcompiler=g95 G95 Fortran Compiler --fcompiler=gnu GNU Fortran 77 compiler --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps Compilers not available on this platform: --fcompiler=compaq Compaq Fortran Compiler --fcompiler=hpux HP Fortran 90 Compiler --fcompiler=ibm IBM XL Fortran Compiler --fcompiler=intel Intel Fortran Compiler for 32-bit apps --fcompiler=intele Intel Fortran Compiler for Itanium apps --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler --fcompiler=mips MIPSpro Fortran Compiler --fcompiler=nag NAGWare Fortran 95 Compiler --fcompiler=none Fake Fortran compiler --fcompiler=pg Portland Group Fortran Compiler --fcompiler=sun Sun or Forte Fortran 95 Compiler --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler For compiler details, run 'config_fc --verbose' setup command. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From cgohlke at uci.edu Fri Sep 9 15:14:51 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Fri, 09 Sep 2011 12:14:51 -0700 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E6A5E56.6040606@noaa.gov> References: <4E68AD7B.2080705@noaa.gov> <4E68B090.7030001@esrf.fr> <4E68CE34.2050406@noaa.gov> <4E68F0ED.2090702@esrf.fr> <4E6A2129.9050602@noaa.gov> <4E6A5101.90002@uci.edu> <4E6A5E56.6040606@noaa.gov> Message-ID: <4E6A65AB.8070607@uci.edu> On 9/9/2011 11:43 AM, Jim Vickroy wrote: > On 9/9/2011 11:46 AM, Christoph Gohlke wrote: >> >> On 9/9/2011 7:22 AM, Jim Vickroy wrote: >>> On 9/8/2011 10:44 AM, "V. Armando Sol?" wrote: >>>> On 08/09/2011 16:16, Jim Vickroy wrote: >>>>> On 9/8/2011 6:09 AM, "V. Armando Sol?" wrote: >>>>>> Have you tried to install Visual Studio 2008 Express edition (plus the >>>>>> windows SDK to be able to compile 64 bit code)? >>>>>> >>>>>> Armando >>>>> Armando, "Visual Studio 2008 Professional" is installed on the computer >>>>> as well as "Intel Visual Fortran Composer XE 2011". >>>>> >>>>> f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I >>>>> tried gfortran. >>>>> >>>>> The "Win64" reference, in the Exception, is puzzling to me since this is >>>>> a 32-bit computer. >>>>> >>>> Oh! I totally misunderstood the situation. I thought the problem was the >>>> missing compiler. >>>> >>>> All what I do with python and the intel fortran compiler is to compile >>>> numpy. Just in case it helps you, I set my environment from the console >>>> by running a bat file with the following content (I am on 64 bit but you >>>> could easily tailor it to your needs): >>>> >>>> "C:\Program Files\Microsoft SDKs\Windows\v7.0\Setup\WindowsSdkVer.exe" >>>> -version:v7.0 >>>> call "C:\Program Files (x86)\Microsoft Visual Studio >>>> 9.0\VC\bin\vcvars64.bat" >>>> call "C:\Program Files >>>> (x86)\Intel\ComposerXE-2011\bin\ipsxe-comp-vars.bat" intel64 vs2008shell >>>> rem call "C:\Program Files (x86)\Microsoft Visual Studio >>>> 9.0\VC\bin\vcvars64" >>>> rem call "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\setenv.cmd" >>>> /x64 /Release >>>> set PATH=C:\Python27;C:\Python27\Scripts;%PATH% >>>> set PATH="C:\Program Files >>>> (x86)\Intel\ComposerXE-2011\redist\intel64\mkl";"C:\Program Files >>>> (x86)\Intel\ComposerXE-2011\mkl\lib\intel64";%PATH% >>>> >>>> Perhaps that helps you to set a working environment. All what I can tell >>>> you is that with that environment, if I run "python f2py.py -c >>>> --help-fcompiler" it finds the intel compiler. >>>> >>>> Good luck, >>>> >>>> Armando >>> Thanks for the suggestion. So far I have not been able to modify the >>> above for use on my 32-bit machine such that ifort is found by f2py. >>> >>> From the f2py documentation, I assumed this was going to be rather >>> straightforward, but it is not. >>> >>> >> There should be a file named "ifortvars.bat" in the Intel Compiler bin >> directory. Call it with the right arguments before using f2py. On my >> system it is: >> >> "C:\Program Files (x86)\Intel\Compiler\11.1\070\bin\ifortvars.bat" >> intel64 vs2008 >> >> Christoph > > Thanks Christoph. > > Unfortunately, this does not seem to enable f2py to find ifort. > > Here is the result of running ifortvars.bat immediately followed by f2py: > > > >"C:\Program Files\Intel\ComposerXE-2011\bin\ifortvars.bat" ia32 > Intel(R) Parallel Studio XE 2011 Update 1 > Copyright (C) 1985-2011 Intel Corporation. All rights reserved. > Intel(R) Composer XE 2011 Update 3 (package 175) > Setting environment for using Microsoft Visual Studio 2008 x86 tools. > > > >f2py.py -c --help-fcompiler > Gnu95FCompiler instance properties: > archiver = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- > cr'] > compile_switch = '-c' > compiler_f77 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- > Wall', '-ffixed-form', '-fno-second-underscore', '-mno- > cygwin', '-O3', '-funroll-loops'] > compiler_f90 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- > Wall', '-fno-second-underscore', '-mno-cygwin', > '-O3', '- > funroll-loops'] > compiler_fix = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- > Wall', '-ffixed-form', '-fno-second-underscore', '-mno- > cygwin', '-Wall', '-fno-second-underscore', > '-mno-cygwin', > '-O3', '-funroll-loops'] > libraries = ['gfortran'] > library_dirs = ["gfortran.exe: error: unrecognized command line option > '-mno-cygwin'\nc:\\program > files\\gfortran\\lib\\gcc\\i586 > -pc-mingw32\\4.7.0"] > linker_exe = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- > Wall', '-mno-cygwin', '-Wall', '-mno-cygwin'] > linker_so = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- > Wall', '-mno-cygwin', '-Wall', '-mno-cygwin', > '-shared'] > object_switch = '-o ' > ranlib = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe'] > version = LooseVersion ('4.7.0') > version_cmd = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '-- > version', '-mno-cygwin'] > Fortran compilers found: > --fcompiler=gnu95 GNU Fortran 95 compiler (4.7.0) > Compilers available for this platform, but not found: > --fcompiler=absoft Absoft Corp Fortran Compiler > --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler > --fcompiler=g95 G95 Fortran Compiler > --fcompiler=gnu GNU Fortran 77 compiler > --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps > --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps > Compilers not available on this platform: > --fcompiler=compaq Compaq Fortran Compiler > --fcompiler=hpux HP Fortran 90 Compiler > --fcompiler=ibm IBM XL Fortran Compiler > --fcompiler=intel Intel Fortran Compiler for 32-bit apps > --fcompiler=intele Intel Fortran Compiler for Itanium apps > --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps > --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler > --fcompiler=mips MIPSpro Fortran Compiler > --fcompiler=nag NAGWare Fortran 95 Compiler > --fcompiler=none Fake Fortran compiler > --fcompiler=pg Portland Group Fortran Compiler > --fcompiler=sun Sun or Forte Fortran 95 Compiler > --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler > For compiler details, run 'config_fc --verbose' setup command. > > What is your numpy version and the output of ifort.exe? Maybe numpy.distutils needs to be updated for ComposerXE-2011. Christoph From Jim.Vickroy at noaa.gov Fri Sep 9 15:22:01 2011 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Fri, 09 Sep 2011 13:22:01 -0600 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E6A65AB.8070607@uci.edu> References: <4E68AD7B.2080705@noaa.gov> <4E68B090.7030001@esrf.fr> <4E68CE34.2050406@noaa.gov> <4E68F0ED.2090702@esrf.fr> <4E6A2129.9050602@noaa.gov> <4E6A5101.90002@uci.edu> <4E6A5E56.6040606@noaa.gov> <4E6A65AB.8070607@uci.edu> Message-ID: <4E6A6759.2080601@noaa.gov> On 9/9/2011 1:14 PM, Christoph Gohlke wrote: > > On 9/9/2011 11:43 AM, Jim Vickroy wrote: >> On 9/9/2011 11:46 AM, Christoph Gohlke wrote: >>> On 9/9/2011 7:22 AM, Jim Vickroy wrote: >>>> On 9/8/2011 10:44 AM, "V. Armando Sol?" wrote: >>>>> On 08/09/2011 16:16, Jim Vickroy wrote: >>>>>> On 9/8/2011 6:09 AM, "V. Armando Sol?" wrote: >>>>>>> Have you tried to install Visual Studio 2008 Express edition (plus the >>>>>>> windows SDK to be able to compile 64 bit code)? >>>>>>> >>>>>>> Armando >>>>>> Armando, "Visual Studio 2008 Professional" is installed on the computer >>>>>> as well as "Intel Visual Fortran Composer XE 2011". >>>>>> >>>>>> f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I >>>>>> tried gfortran. >>>>>> >>>>>> The "Win64" reference, in the Exception, is puzzling to me since this is >>>>>> a 32-bit computer. >>>>>> >>>>> Oh! I totally misunderstood the situation. I thought the problem was the >>>>> missing compiler. >>>>> >>>>> All what I do with python and the intel fortran compiler is to compile >>>>> numpy. Just in case it helps you, I set my environment from the console >>>>> by running a bat file with the following content (I am on 64 bit but you >>>>> could easily tailor it to your needs): >>>>> >>>>> "C:\Program Files\Microsoft SDKs\Windows\v7.0\Setup\WindowsSdkVer.exe" >>>>> -version:v7.0 >>>>> call "C:\Program Files (x86)\Microsoft Visual Studio >>>>> 9.0\VC\bin\vcvars64.bat" >>>>> call "C:\Program Files >>>>> (x86)\Intel\ComposerXE-2011\bin\ipsxe-comp-vars.bat" intel64 vs2008shell >>>>> rem call "C:\Program Files (x86)\Microsoft Visual Studio >>>>> 9.0\VC\bin\vcvars64" >>>>> rem call "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\setenv.cmd" >>>>> /x64 /Release >>>>> set PATH=C:\Python27;C:\Python27\Scripts;%PATH% >>>>> set PATH="C:\Program Files >>>>> (x86)\Intel\ComposerXE-2011\redist\intel64\mkl";"C:\Program Files >>>>> (x86)\Intel\ComposerXE-2011\mkl\lib\intel64";%PATH% >>>>> >>>>> Perhaps that helps you to set a working environment. All what I can tell >>>>> you is that with that environment, if I run "python f2py.py -c >>>>> --help-fcompiler" it finds the intel compiler. >>>>> >>>>> Good luck, >>>>> >>>>> Armando >>>> Thanks for the suggestion. So far I have not been able to modify the >>>> above for use on my 32-bit machine such that ifort is found by f2py. >>>> >>>> From the f2py documentation, I assumed this was going to be rather >>>> straightforward, but it is not. >>>> >>>> >>> There should be a file named "ifortvars.bat" in the Intel Compiler bin >>> directory. Call it with the right arguments before using f2py. On my >>> system it is: >>> >>> "C:\Program Files (x86)\Intel\Compiler\11.1\070\bin\ifortvars.bat" >>> intel64 vs2008 >>> >>> Christoph >> Thanks Christoph. >> >> Unfortunately, this does not seem to enable f2py to find ifort. >> >> Here is the result of running ifortvars.bat immediately followed by f2py: >> >> >> >"C:\Program Files\Intel\ComposerXE-2011\bin\ifortvars.bat" ia32 >> Intel(R) Parallel Studio XE 2011 Update 1 >> Copyright (C) 1985-2011 Intel Corporation. All rights reserved. >> Intel(R) Composer XE 2011 Update 3 (package 175) >> Setting environment for using Microsoft Visual Studio 2008 x86 tools. >> >> >> >f2py.py -c --help-fcompiler >> Gnu95FCompiler instance properties: >> archiver = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >> cr'] >> compile_switch = '-c' >> compiler_f77 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >> Wall', '-ffixed-form', '-fno-second-underscore', '-mno- >> cygwin', '-O3', '-funroll-loops'] >> compiler_f90 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >> Wall', '-fno-second-underscore', '-mno-cygwin', >> '-O3', '- >> funroll-loops'] >> compiler_fix = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >> Wall', '-ffixed-form', '-fno-second-underscore', '-mno- >> cygwin', '-Wall', '-fno-second-underscore', >> '-mno-cygwin', >> '-O3', '-funroll-loops'] >> libraries = ['gfortran'] >> library_dirs = ["gfortran.exe: error: unrecognized command line option >> '-mno-cygwin'\nc:\\program >> files\\gfortran\\lib\\gcc\\i586 >> -pc-mingw32\\4.7.0"] >> linker_exe = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >> Wall', '-mno-cygwin', '-Wall', '-mno-cygwin'] >> linker_so = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >> Wall', '-mno-cygwin', '-Wall', '-mno-cygwin', >> '-shared'] >> object_switch = '-o ' >> ranlib = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe'] >> version = LooseVersion ('4.7.0') >> version_cmd = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '-- >> version', '-mno-cygwin'] >> Fortran compilers found: >> --fcompiler=gnu95 GNU Fortran 95 compiler (4.7.0) >> Compilers available for this platform, but not found: >> --fcompiler=absoft Absoft Corp Fortran Compiler >> --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler >> --fcompiler=g95 G95 Fortran Compiler >> --fcompiler=gnu GNU Fortran 77 compiler >> --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps >> --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps >> Compilers not available on this platform: >> --fcompiler=compaq Compaq Fortran Compiler >> --fcompiler=hpux HP Fortran 90 Compiler >> --fcompiler=ibm IBM XL Fortran Compiler >> --fcompiler=intel Intel Fortran Compiler for 32-bit apps >> --fcompiler=intele Intel Fortran Compiler for Itanium apps >> --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps >> --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler >> --fcompiler=mips MIPSpro Fortran Compiler >> --fcompiler=nag NAGWare Fortran 95 Compiler >> --fcompiler=none Fake Fortran compiler >> --fcompiler=pg Portland Group Fortran Compiler >> --fcompiler=sun Sun or Forte Fortran 95 Compiler >> --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler >> For compiler details, run 'config_fc --verbose' setup command. >> >> > What is your numpy version and the output of ifort.exe? Maybe > numpy.distutils needs to be updated for ComposerXE-2011. The (edited) output from f2py -v is: Version: 2 numpy Version: 1.4.0 Requires: Python 2.3 or higher. License: NumPy license (see LICENSE.txt in the NumPy source code) Copyright 1999 - 2005 Pearu Peterson all rights reserved. http://cens.ioc.ee/projects/f2py2e/ and for ifort: >ifort Intel(R) Visual Fortran Compiler XE for applications running on IA-32, Version 12.0.3.175 Build 20110309 Copyright (C) 1985-2011 Intel Corporation. All rights reserved. Thanks for your assistance. > > Christoph > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From cgohlke at uci.edu Fri Sep 9 15:59:08 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Fri, 09 Sep 2011 12:59:08 -0700 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E6A6759.2080601@noaa.gov> References: <4E68AD7B.2080705@noaa.gov> <4E68B090.7030001@esrf.fr> <4E68CE34.2050406@noaa.gov> <4E68F0ED.2090702@esrf.fr> <4E6A2129.9050602@noaa.gov> <4E6A5101.90002@uci.edu> <4E6A5E56.6040606@noaa.gov> <4E6A65AB.8070607@uci.edu> <4E6A6759.2080601@noaa.gov> Message-ID: <4E6A700C.3050809@uci.edu> On 9/9/2011 12:22 PM, Jim Vickroy wrote: > On 9/9/2011 1:14 PM, Christoph Gohlke wrote: >> >> On 9/9/2011 11:43 AM, Jim Vickroy wrote: >>> On 9/9/2011 11:46 AM, Christoph Gohlke wrote: >>>> On 9/9/2011 7:22 AM, Jim Vickroy wrote: >>>>> On 9/8/2011 10:44 AM, "V. Armando Sol?" wrote: >>>>>> On 08/09/2011 16:16, Jim Vickroy wrote: >>>>>>> On 9/8/2011 6:09 AM, "V. Armando Sol?" wrote: >>>>>>>> Have you tried to install Visual Studio 2008 Express edition (plus the >>>>>>>> windows SDK to be able to compile 64 bit code)? >>>>>>>> >>>>>>>> Armando >>>>>>> Armando, "Visual Studio 2008 Professional" is installed on the computer >>>>>>> as well as "Intel Visual Fortran Composer XE 2011". >>>>>>> >>>>>>> f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I >>>>>>> tried gfortran. >>>>>>> >>>>>>> The "Win64" reference, in the Exception, is puzzling to me since this is >>>>>>> a 32-bit computer. >>>>>>> >>>>>> Oh! I totally misunderstood the situation. I thought the problem was the >>>>>> missing compiler. >>>>>> >>>>>> All what I do with python and the intel fortran compiler is to compile >>>>>> numpy. Just in case it helps you, I set my environment from the console >>>>>> by running a bat file with the following content (I am on 64 bit but you >>>>>> could easily tailor it to your needs): >>>>>> >>>>>> "C:\Program Files\Microsoft SDKs\Windows\v7.0\Setup\WindowsSdkVer.exe" >>>>>> -version:v7.0 >>>>>> call "C:\Program Files (x86)\Microsoft Visual Studio >>>>>> 9.0\VC\bin\vcvars64.bat" >>>>>> call "C:\Program Files >>>>>> (x86)\Intel\ComposerXE-2011\bin\ipsxe-comp-vars.bat" intel64 vs2008shell >>>>>> rem call "C:\Program Files (x86)\Microsoft Visual Studio >>>>>> 9.0\VC\bin\vcvars64" >>>>>> rem call "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\setenv.cmd" >>>>>> /x64 /Release >>>>>> set PATH=C:\Python27;C:\Python27\Scripts;%PATH% >>>>>> set PATH="C:\Program Files >>>>>> (x86)\Intel\ComposerXE-2011\redist\intel64\mkl";"C:\Program Files >>>>>> (x86)\Intel\ComposerXE-2011\mkl\lib\intel64";%PATH% >>>>>> >>>>>> Perhaps that helps you to set a working environment. All what I can tell >>>>>> you is that with that environment, if I run "python f2py.py -c >>>>>> --help-fcompiler" it finds the intel compiler. >>>>>> >>>>>> Good luck, >>>>>> >>>>>> Armando >>>>> Thanks for the suggestion. So far I have not been able to modify the >>>>> above for use on my 32-bit machine such that ifort is found by f2py. >>>>> >>>>> From the f2py documentation, I assumed this was going to be rather >>>>> straightforward, but it is not. >>>>> >>>>> >>>> There should be a file named "ifortvars.bat" in the Intel Compiler bin >>>> directory. Call it with the right arguments before using f2py. On my >>>> system it is: >>>> >>>> "C:\Program Files (x86)\Intel\Compiler\11.1\070\bin\ifortvars.bat" >>>> intel64 vs2008 >>>> >>>> Christoph >>> Thanks Christoph. >>> >>> Unfortunately, this does not seem to enable f2py to find ifort. >>> >>> Here is the result of running ifortvars.bat immediately followed by f2py: >>> >>> >>> >"C:\Program Files\Intel\ComposerXE-2011\bin\ifortvars.bat" ia32 >>> Intel(R) Parallel Studio XE 2011 Update 1 >>> Copyright (C) 1985-2011 Intel Corporation. All rights reserved. >>> Intel(R) Composer XE 2011 Update 3 (package 175) >>> Setting environment for using Microsoft Visual Studio 2008 x86 tools. >>> >>> >>> >f2py.py -c --help-fcompiler >>> Gnu95FCompiler instance properties: >>> archiver = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>> cr'] >>> compile_switch = '-c' >>> compiler_f77 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>> Wall', '-ffixed-form', '-fno-second-underscore', '-mno- >>> cygwin', '-O3', '-funroll-loops'] >>> compiler_f90 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>> Wall', '-fno-second-underscore', '-mno-cygwin', >>> '-O3', '- >>> funroll-loops'] >>> compiler_fix = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>> Wall', '-ffixed-form', '-fno-second-underscore', '-mno- >>> cygwin', '-Wall', '-fno-second-underscore', >>> '-mno-cygwin', >>> '-O3', '-funroll-loops'] >>> libraries = ['gfortran'] >>> library_dirs = ["gfortran.exe: error: unrecognized command line option >>> '-mno-cygwin'\nc:\\program >>> files\\gfortran\\lib\\gcc\\i586 >>> -pc-mingw32\\4.7.0"] >>> linker_exe = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>> Wall', '-mno-cygwin', '-Wall', '-mno-cygwin'] >>> linker_so = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>> Wall', '-mno-cygwin', '-Wall', '-mno-cygwin', >>> '-shared'] >>> object_switch = '-o ' >>> ranlib = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe'] >>> version = LooseVersion ('4.7.0') >>> version_cmd = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '-- >>> version', '-mno-cygwin'] >>> Fortran compilers found: >>> --fcompiler=gnu95 GNU Fortran 95 compiler (4.7.0) >>> Compilers available for this platform, but not found: >>> --fcompiler=absoft Absoft Corp Fortran Compiler >>> --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler >>> --fcompiler=g95 G95 Fortran Compiler >>> --fcompiler=gnu GNU Fortran 77 compiler >>> --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps >>> --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps >>> Compilers not available on this platform: >>> --fcompiler=compaq Compaq Fortran Compiler >>> --fcompiler=hpux HP Fortran 90 Compiler >>> --fcompiler=ibm IBM XL Fortran Compiler >>> --fcompiler=intel Intel Fortran Compiler for 32-bit apps >>> --fcompiler=intele Intel Fortran Compiler for Itanium apps >>> --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps >>> --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler >>> --fcompiler=mips MIPSpro Fortran Compiler >>> --fcompiler=nag NAGWare Fortran 95 Compiler >>> --fcompiler=none Fake Fortran compiler >>> --fcompiler=pg Portland Group Fortran Compiler >>> --fcompiler=sun Sun or Forte Fortran 95 Compiler >>> --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler >>> For compiler details, run 'config_fc --verbose' setup command. >>> >>> >> What is your numpy version and the output of ifort.exe? Maybe >> numpy.distutils needs to be updated for ComposerXE-2011. > > The (edited) output from f2py -v is: > > Version: 2 > numpy Version: 1.4.0 > Requires: Python 2.3 or higher. > License: NumPy license (see LICENSE.txt in the NumPy source code) > Copyright 1999 - 2005 Pearu Peterson all rights reserved. > http://cens.ioc.ee/projects/f2py2e/ > > and for ifort: > > >ifort > Intel(R) Visual Fortran Compiler XE for applications running on IA-32, > Version 12.0.3.175 Build 20110309 > Copyright (C) 1985-2011 Intel Corporation. All rights reserved. > > > Thanks for your assistance. > Numpy 1.4.0 doesn't work for me either. Try upgrading to numpy 1.6.1. Christoph From Jim.Vickroy at noaa.gov Fri Sep 9 16:21:45 2011 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Fri, 09 Sep 2011 14:21:45 -0600 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E6A700C.3050809@uci.edu> References: <4E68AD7B.2080705@noaa.gov> <4E68B090.7030001@esrf.fr> <4E68CE34.2050406@noaa.gov> <4E68F0ED.2090702@esrf.fr> <4E6A2129.9050602@noaa.gov> <4E6A5101.90002@uci.edu> <4E6A5E56.6040606@noaa.gov> <4E6A65AB.8070607@uci.edu> <4E6A6759.2080601@noaa.gov> <4E6A700C.3050809@uci.edu> Message-ID: <4E6A7559.9070405@noaa.gov> On 9/9/2011 1:59 PM, Christoph Gohlke wrote: > > On 9/9/2011 12:22 PM, Jim Vickroy wrote: >> On 9/9/2011 1:14 PM, Christoph Gohlke wrote: >>> On 9/9/2011 11:43 AM, Jim Vickroy wrote: >>>> On 9/9/2011 11:46 AM, Christoph Gohlke wrote: >>>>> On 9/9/2011 7:22 AM, Jim Vickroy wrote: >>>>>> On 9/8/2011 10:44 AM, "V. Armando Sol?" wrote: >>>>>>> On 08/09/2011 16:16, Jim Vickroy wrote: >>>>>>>> On 9/8/2011 6:09 AM, "V. Armando Sol?" wrote: >>>>>>>>> Have you tried to install Visual Studio 2008 Express edition (plus the >>>>>>>>> windows SDK to be able to compile 64 bit code)? >>>>>>>>> >>>>>>>>> Armando >>>>>>>> Armando, "Visual Studio 2008 Professional" is installed on the computer >>>>>>>> as well as "Intel Visual Fortran Composer XE 2011". >>>>>>>> >>>>>>>> f2py was not finding the Intel compiler (f2py -c --help-fcompiler) so I >>>>>>>> tried gfortran. >>>>>>>> >>>>>>>> The "Win64" reference, in the Exception, is puzzling to me since this is >>>>>>>> a 32-bit computer. >>>>>>>> >>>>>>> Oh! I totally misunderstood the situation. I thought the problem was the >>>>>>> missing compiler. >>>>>>> >>>>>>> All what I do with python and the intel fortran compiler is to compile >>>>>>> numpy. Just in case it helps you, I set my environment from the console >>>>>>> by running a bat file with the following content (I am on 64 bit but you >>>>>>> could easily tailor it to your needs): >>>>>>> >>>>>>> "C:\Program Files\Microsoft SDKs\Windows\v7.0\Setup\WindowsSdkVer.exe" >>>>>>> -version:v7.0 >>>>>>> call "C:\Program Files (x86)\Microsoft Visual Studio >>>>>>> 9.0\VC\bin\vcvars64.bat" >>>>>>> call "C:\Program Files >>>>>>> (x86)\Intel\ComposerXE-2011\bin\ipsxe-comp-vars.bat" intel64 vs2008shell >>>>>>> rem call "C:\Program Files (x86)\Microsoft Visual Studio >>>>>>> 9.0\VC\bin\vcvars64" >>>>>>> rem call "C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\setenv.cmd" >>>>>>> /x64 /Release >>>>>>> set PATH=C:\Python27;C:\Python27\Scripts;%PATH% >>>>>>> set PATH="C:\Program Files >>>>>>> (x86)\Intel\ComposerXE-2011\redist\intel64\mkl";"C:\Program Files >>>>>>> (x86)\Intel\ComposerXE-2011\mkl\lib\intel64";%PATH% >>>>>>> >>>>>>> Perhaps that helps you to set a working environment. All what I can tell >>>>>>> you is that with that environment, if I run "python f2py.py -c >>>>>>> --help-fcompiler" it finds the intel compiler. >>>>>>> >>>>>>> Good luck, >>>>>>> >>>>>>> Armando >>>>>> Thanks for the suggestion. So far I have not been able to modify the >>>>>> above for use on my 32-bit machine such that ifort is found by f2py. >>>>>> >>>>>> From the f2py documentation, I assumed this was going to be rather >>>>>> straightforward, but it is not. >>>>>> >>>>>> >>>>> There should be a file named "ifortvars.bat" in the Intel Compiler bin >>>>> directory. Call it with the right arguments before using f2py. On my >>>>> system it is: >>>>> >>>>> "C:\Program Files (x86)\Intel\Compiler\11.1\070\bin\ifortvars.bat" >>>>> intel64 vs2008 >>>>> >>>>> Christoph >>>> Thanks Christoph. >>>> >>>> Unfortunately, this does not seem to enable f2py to find ifort. >>>> >>>> Here is the result of running ifortvars.bat immediately followed by f2py: >>>> >>>> >>>> >"C:\Program Files\Intel\ComposerXE-2011\bin\ifortvars.bat" ia32 >>>> Intel(R) Parallel Studio XE 2011 Update 1 >>>> Copyright (C) 1985-2011 Intel Corporation. All rights reserved. >>>> Intel(R) Composer XE 2011 Update 3 (package 175) >>>> Setting environment for using Microsoft Visual Studio 2008 x86 tools. >>>> >>>> >>>> >f2py.py -c --help-fcompiler >>>> Gnu95FCompiler instance properties: >>>> archiver = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>>> cr'] >>>> compile_switch = '-c' >>>> compiler_f77 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>>> Wall', '-ffixed-form', '-fno-second-underscore', '-mno- >>>> cygwin', '-O3', '-funroll-loops'] >>>> compiler_f90 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>>> Wall', '-fno-second-underscore', '-mno-cygwin', >>>> '-O3', '- >>>> funroll-loops'] >>>> compiler_fix = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>>> Wall', '-ffixed-form', '-fno-second-underscore', '-mno- >>>> cygwin', '-Wall', '-fno-second-underscore', >>>> '-mno-cygwin', >>>> '-O3', '-funroll-loops'] >>>> libraries = ['gfortran'] >>>> library_dirs = ["gfortran.exe: error: unrecognized command line option >>>> '-mno-cygwin'\nc:\\program >>>> files\\gfortran\\lib\\gcc\\i586 >>>> -pc-mingw32\\4.7.0"] >>>> linker_exe = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>>> Wall', '-mno-cygwin', '-Wall', '-mno-cygwin'] >>>> linker_so = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- >>>> Wall', '-mno-cygwin', '-Wall', '-mno-cygwin', >>>> '-shared'] >>>> object_switch = '-o ' >>>> ranlib = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe'] >>>> version = LooseVersion ('4.7.0') >>>> version_cmd = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '-- >>>> version', '-mno-cygwin'] >>>> Fortran compilers found: >>>> --fcompiler=gnu95 GNU Fortran 95 compiler (4.7.0) >>>> Compilers available for this platform, but not found: >>>> --fcompiler=absoft Absoft Corp Fortran Compiler >>>> --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler >>>> --fcompiler=g95 G95 Fortran Compiler >>>> --fcompiler=gnu GNU Fortran 77 compiler >>>> --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps >>>> --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps >>>> Compilers not available on this platform: >>>> --fcompiler=compaq Compaq Fortran Compiler >>>> --fcompiler=hpux HP Fortran 90 Compiler >>>> --fcompiler=ibm IBM XL Fortran Compiler >>>> --fcompiler=intel Intel Fortran Compiler for 32-bit apps >>>> --fcompiler=intele Intel Fortran Compiler for Itanium apps >>>> --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps >>>> --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler >>>> --fcompiler=mips MIPSpro Fortran Compiler >>>> --fcompiler=nag NAGWare Fortran 95 Compiler >>>> --fcompiler=none Fake Fortran compiler >>>> --fcompiler=pg Portland Group Fortran Compiler >>>> --fcompiler=sun Sun or Forte Fortran 95 Compiler >>>> --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler >>>> For compiler details, run 'config_fc --verbose' setup command. >>>> >>>> >>> What is your numpy version and the output of ifort.exe? Maybe >>> numpy.distutils needs to be updated for ComposerXE-2011. >> The (edited) output from f2py -v is: >> >> Version: 2 >> numpy Version: 1.4.0 >> Requires: Python 2.3 or higher. >> License: NumPy license (see LICENSE.txt in the NumPy source code) >> Copyright 1999 - 2005 Pearu Peterson all rights reserved. >> http://cens.ioc.ee/projects/f2py2e/ >> >> and for ifort: >> >> >ifort >> Intel(R) Visual Fortran Compiler XE for applications running on IA-32, >> Version 12.0.3.175 Build 20110309 >> Copyright (C) 1985-2011 Intel Corporation. All rights reserved. >> >> >> Thanks for your assistance. >> > Numpy 1.4.0 doesn't work for me either. Try upgrading to numpy 1.6.1. > > Christoph That worked! Thanks so much for your help! P.S. ... been away from Python for a year and have fallen behind in updates so I also upgraded scipy and matplotlib. >f2py.py -c --help-fcompiler IntelVisualFCompiler instance properties: archiver = ['C:\\Program Files\\Intel\\ComposerXE- 2011\\bin\\ia32\\ifort.exe', '/verbose', '/OUT:'] compile_switch = '/c ' compiler_f77 = ['C:\\Program Files\\Intel\\ComposerXE- 2011\\bin\\ia32\\ifort.exe', '-FI', '-w90', '-w95', '/nologo', '/MD', '/nbs', '/Qlowercase', '/us', '/O1', '/QaxM'] compiler_f90 = ['C:\\Program Files\\Intel\\ComposerXE- 2011\\bin\\ia32\\ifort.exe', '-FR', '/nologo', '/MD', '/nbs', '/Qlowercase', '/us', '/O1', '/QaxM'] compiler_fix = ['C:\\Program Files\\Intel\\ComposerXE- 2011\\bin\\ia32\\ifort.exe', '-FI', '-4L72', '-w', '/nologo', '/MD', '/nbs', '/Qlowercase', '/us', '/O1', '/QaxM'] libraries = [] library_dirs = [] linker_exe = None linker_so = ['C:\\Program Files\\Intel\\ComposerXE- 2011\\bin\\ia32\\ifort.exe', '-shared', '-shared'] object_switch = '/Fo' ranlib = None version = LooseVersion ('12.0.3.175') version_cmd = ['C:\\Program Files\\Intel\\ComposerXE- 2011\\bin\\ia32\\ifort.exe', '/FI', '/c', 'c:\\docume~1\\jim~1.vic\\locals~1\\temp\\tmp4ypyfm\\3zikj 7.f', '/o', 'c:\\docume~1\\jim~1.vic\\locals~1\\temp\\tmp4ypyfm\\3zikj 7.o'] Gnu95FCompiler instance properties: archiver = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- cr'] compile_switch = '-c' compiler_f77 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-ffixed-form', '-fno-second-underscore', '-O3', '- funroll-loops'] compiler_f90 = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-fno-second-underscore', '-O3', '-funroll-loops'] compiler_fix = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-ffixed-form', '-fno-second-underscore', '-Wall', '-fno-second-underscore', '-O3', '-funroll-loops'] libraries = ['gfortran'] library_dirs = ['c:\\program files\\gfortran\\lib', 'c:\\program files\\gfortran\\lib\\gcc\\i586-pc-mingw32\\4.7.0'] linker_exe = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-Wall'] linker_so = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '- Wall', '-Wall', '-shared'] object_switch = '-o ' ranlib = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe'] version = LooseVersion ('4.7.0') version_cmd = ['C:\\Program Files\\gfortran\\bin\\gfortran.exe', '-- version'] Fortran compilers found: --fcompiler=gnu95 GNU Fortran 95 compiler (4.7.0) --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps (12.0.3.175) > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From jsilter at gmail.com Fri Sep 9 17:57:19 2011 From: jsilter at gmail.com (Jacob Silterra) Date: Fri, 9 Sep 2011 17:57:19 -0400 Subject: [Numpy-discussion] Functions for finding the relative extrema of numeric data Message-ID: Hello all, I'd like to see functions for calculating the relative extrema in a set of data included in numpy. I use that functionality frequently, and always seem to be writing my own version. It seems like this functionality would be useful to the community at large, as it's a fairly common operation. For numeric data (which is presumably noisy), the definition of a relative extrema isn't completely obvious. The implementation I am proposing finds a point in an ndarray along an axis which is larger (or smaller) than it's `order` nearest neighbors (`order` being an optional parameter, default 1). This is likely to find more points than may be desired, which I believe is preferable to the alternative. The output is formatted the same as numpy.where. Code available here: https://github.com/numpy/numpy/pull/154 I'm not sure whether this belongs in numpy or scipy, that question is somewhat debatable. More sophisticated peak-finding functions (in N dimensions, as opposed to 1) may also be useful to the community, and those would definitely belong in scipy. An alternative implementation would be to require that function be continuously descending (or ascending) for `order` points, which would enforce a minimum width on a peak. -Jacob Silterra -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at vaught.net Sun Sep 11 01:30:23 2011 From: travis at vaught.net (Travis Vaught) Date: Sun, 11 Sep 2011 00:30:23 -0500 Subject: [Numpy-discussion] recarrays and lists of lists Message-ID: <5871873B-0588-4485-8425-32798294BC26@vaught.net> Greetings, Is there a particular reason why a list of lists can't be passed in to create a recarray given a particular dtype? A list of tuples works fine. I keep getting bitten by this and was thinking it should be an easy check/convert for an allowance for a row to be a list _or_ a tuple? Here's a session: ~~~~~~~~~~ In [2]: import numpy as np In [3]: dt = np.dtype({'names':['a', 'b', 'c'], 'formats':[float]*3}) In [4]: dt Out[4]: dtype([('a', ' in () ----> 1 ary = np.array(rows, dtype=dt) TypeError: expected a readable buffer object In [7]: rows2 = [tuple(row) for row in rows] In [8]: ary = np.array(rows2, dtype=dt) In [9]: ary Out[9]: array([(1.0, 2.0, 3.0), (2.0, 3.0, 4.0), (3.0, 4.0, 5.0)], dtype=[('a', ' References: <5871873B-0588-4485-8425-32798294BC26@vaught.net> Message-ID: On Sun, Sep 11, 2011 at 00:30, Travis Vaught wrote: > Greetings, > > Is there a particular reason why a list of lists can't be passed in to create a recarray given a particular dtype? > > A list of tuples works fine. ?I keep getting bitten by this and was thinking it should be an easy check/convert for an allowance for a row to be a list _or_ a tuple? As a rule, tuples are considered "scalar" records and lists are recursed upon. This rule helps numpy.array() figure out which sequences are records and which are other sequences to be recursed upon; i.e. which sequences create another dimension and which are the atomic elements. Otherwise, it has to make some guesses and do some intelligent backtracking. It's not that intelligent. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pengkui.luo at gmail.com Sun Sep 11 04:03:26 2011 From: pengkui.luo at gmail.com (Pengkui Luo) Date: Sun, 11 Sep 2011 03:03:26 -0500 Subject: [Numpy-discussion] How to apply Numpy ufunc to Scipy sparse matrices? Message-ID: It seems that numpy universal functions only apply to ndarray (or dense matrix). Is there a way to apply them to scipy sparse matrices also? For example, suppose S is an large sparse matrix (lil_matrix format, dtype=np.float). I want to get another sparse matrix B (of the same shape) that represents the signs of corresponding entries of S. I wrote down "B=np.sign(S)" first, but it did not give me the desired output. I realized that np.sign() might not work for a sparse matrix, so I have to convert S to a dense matrix, i.e. B = np.sign( S.todense() ) However, converting a large sparse matrix to dense would easily eat up the memory. Is there a way for np.sign (as well as other ufunc) to take a sparse matrix as parameter, and return a sparse matrix? (If I recall correctly, in Matlab all functions work on both dense matrices and sparse matrices.) -- Pengkui -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sun Sep 11 11:12:27 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 11 Sep 2011 17:12:27 +0200 Subject: [Numpy-discussion] [ANN] ESCO2012 and PyHPC2011: Python in science in major computational science conference Message-ID: <20110911151227.GA23609@phare.normalesup.org> ESCO 2012 - European Seminar on Coupled Problems ================================================= ESCO2012 http://esco2012.femhub.com/ is the 3rd event in a series of interdisciplineary meetings dedicated to computational science challenges in multi-physics and PDEs. I was invited as ESCO last year. It was an aboslute pleasure, because it is a small conference that is very focused on discussions. I learned a lot and could sit down with people who code top notch PDE libraries such as FEniCS and have technical discussions. Besides, it is hosted in the historical brewery where the Pilsner was invented. Plenty of great beer. Application areas ------------------ Theoretical results as well as applications are welcome. Application areas include, but are not limited to: Computational electromagnetics, Civil engineering, Nuclear engineering, Mechanical engineering, Computational fluid dynamics, Computational geophysics, Geomechanics and rock mechanics, Computational hydrology, Subsurface modeling, Biomechanics, Computational chemistry, Climate and weather modeling, Wave propagation, Acoustics, Stochastic differential equations, and Uncertainty quantification. Minisymposia * Multiphysics and Multiscale Problems in Civil Engineering * Modern Numerical Methods for ODE * Porous Media Hydrodynamics * Nuclear Fuel Recycling Simulations * Adaptive Methods for Eigenproblems * Discontinuous Galerkin Methods for Electromagnetics * Undergraduate Projects in Technical Computing Software afternoon ------------------- Important part of each ESCO conference is a software afternoon featuring software projects by participants. Presented can be any computational software that has reached certain level of maturity, i.e., it is used outside of the author's institution, and it has a web page and a user documentation. Proceedings ----------- For each ESCO we strive to reserve a special issue of an international journal with impact factor. Proceedings of ESCO 2008 appeared in Math. Comput. Simul., proceedings of ESCO 2010 in CiCP and Appl. Math. Comput. Proceedings of ESCO 2012 will appear in Computing. Important Dates * December 15, 2011: Abstract submission deadline. * December 15, 2011: Minisymposia proposals. * January 15, 2012: Notification of acceptance. PyHPC: Python for High performance computing -------------------------------------------- If you are doing super computing, SC11, ( http://sc11.supercomputing.org/) the Super Computing conference is the reference conference. This year there will a workshop on high performance computing with Python: PyHPC (http://www.dlr.de/sc/desktopdefault.aspx/tabid-1183/1638_read-31733/). At the scipy conference, I was having a discussion with some of the attendees on how people often still do process management and I/O with Fortran in the big computing environment. This is counter productive. However, has success stories of supercomputing folks using high-level languages are not advertized, this is bound to stay. Come and tell us how you use Python for high performance computing! Topics * Python-based scientific applications and libraries * High performance computing * Parallel Python-based programming languages * Scientific visualization * Scientific computing education * Python performance and language issues * Problem solving environments with Python * Performance analysis tools for Python application Papers We invite you to submit a paper of up to 10 pages via the submission site. Authors are encouraged to use IEEE two column format. Important Dates * Full paper submission: September 19, 2011 * Notification of acceptance: October 7, 2011 * Camera-ready papers: October 31, 2011 From pav at iki.fi Sun Sep 11 14:23:45 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 11 Sep 2011 18:23:45 +0000 (UTC) Subject: [Numpy-discussion] How to apply Numpy ufunc to Scipy sparse matrices? References: Message-ID: Sun, 11 Sep 2011 03:03:26 -0500, Pengkui Luo wrote: [clip] > However, converting a large sparse matrix to dense would easily eat up > the memory. Is there a way for np.sign (as well as other ufunc) to take > a sparse matrix as parameter, and return a sparse matrix? For CSR, CSC, and DIA you can do x.data = some_ufunc(x.data) to operate on the nonzero entries. If some_ufunc(0) == 0, that's all you need. Otherwise, the matrix becomes dense. Pauli From pengkui.luo at gmail.com Sun Sep 11 15:27:54 2011 From: pengkui.luo at gmail.com (Pengkui Luo) Date: Sun, 11 Sep 2011 14:27:54 -0500 Subject: [Numpy-discussion] How to apply Numpy ufunc to Scipy sparse matrices? In-Reply-To: References: Message-ID: On Sun, Sep 11, 2011 at 13:23, Pauli Virtanen wrote: > Sun, 11 Sep 2011 03:03:26 -0500, Pengkui Luo wrote: > [clip] > > However, converting a large sparse matrix to dense would easily eat up > > the memory. Is there a way for np.sign (as well as other ufunc) to take > > a sparse matrix as parameter, and return a sparse matrix? > > For CSR, CSC, and DIA you can do > > x.data = some_ufunc(x.data) > > to operate on the nonzero entries. If some_ufunc(0) == 0, that's all you > need. Otherwise, the matrix becomes dense. > > Pauli > Hi Pauli, Thanks for your ".data" tip. It works! def _test_ufunc(): import numpy as np from scipy.sparse import csr_matrix A = np.mat( '2 3 0 0; 0 0 0 6; 1 0 0 -7; 0 -2 -1 0' ) print 'A=\n', A AS = csr_matrix(A) AS.data = np.sign( AS.data ) print '\ntype(AS.data)=%s' % type( AS.data ) print '\nnp.sign(AS)=\n%s' % AS.todense() print '\ntype(AS)=%s' % type(AS) if __name__=="__main__": _test_ufunc() >>> A= [[ 2 3 0 0] [ 0 0 0 6] [ 1 0 0 -7] [ 0 -2 -1 0]] type(AS.data)= np.sign(AS)= [[ 1 1 0 0] [ 0 0 0 1] [ 1 0 0 -1] [ 0 -1 -1 0]] type(AS)= --Pengkui -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at vaught.net Sun Sep 11 15:44:29 2011 From: travis at vaught.net (Travis Vaught) Date: Sun, 11 Sep 2011 14:44:29 -0500 Subject: [Numpy-discussion] recarrays and lists of lists In-Reply-To: References: <5871873B-0588-4485-8425-32798294BC26@vaught.net> Message-ID: On Sep 11, 2011, at 2:58 AM, Robert Kern wrote: > On Sun, Sep 11, 2011 at 00:30, Travis Vaught wrote: >> Greetings, >> >> Is there a particular reason why a list of lists can't be passed in to create a recarray given a particular dtype? >> >> A list of tuples works fine. I keep getting bitten by this and was thinking it should be an easy check/convert for an allowance for a row to be a list _or_ a tuple? > > As a rule, tuples are considered "scalar" records and lists are > recursed upon. This rule helps numpy.array() figure out which > sequences are records and which are other sequences to be recursed > upon; i.e. which sequences create another dimension and which are the > atomic elements. Otherwise, it has to make some guesses and do some > intelligent backtracking. It's not that intelligent. > > -- > Robert Kern Thanks for the explanation. It makes a bit of sense -- at least to answer the question of what is being traded off here. Looks like simplicity and performance, perhaps, is being chosen over my convenience. Best, Travis From robert.kern at gmail.com Sun Sep 11 15:56:52 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 11 Sep 2011 14:56:52 -0500 Subject: [Numpy-discussion] recarrays and lists of lists In-Reply-To: References: <5871873B-0588-4485-8425-32798294BC26@vaught.net> Message-ID: On Sun, Sep 11, 2011 at 14:44, Travis Vaught wrote: > > On Sep 11, 2011, at 2:58 AM, Robert Kern wrote: > >> On Sun, Sep 11, 2011 at 00:30, Travis Vaught wrote: >>> Greetings, >>> >>> Is there a particular reason why a list of lists can't be passed in to create a recarray given a particular dtype? >>> >>> A list of tuples works fine. ?I keep getting bitten by this and was thinking it should be an easy check/convert for an allowance for a row to be a list _or_ a tuple? >> >> As a rule, tuples are considered "scalar" records and lists are >> recursed upon. This rule helps numpy.array() figure out which >> sequences are records and which are other sequences to be recursed >> upon; i.e. which sequences create another dimension and which are the >> atomic elements. Otherwise, it has to make some guesses and do some >> intelligent backtracking. It's not that intelligent. >> >> -- >> Robert Kern > > Thanks for the explanation. ?It makes a bit of sense -- at least to answer the question of what is being traded off here. ?Looks like simplicity and performance, perhaps, is being chosen over my convenience. Mostly unambiguity. numpy.array() has to figure out several interrelated things simultaneously, so there is an identifiability problem. Too many equations, not enough data points; the tuple/list distinction provides an additional data point that helps numpy.array() do what it needs to do. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From david.froger at gmail.com Mon Sep 12 02:52:02 2011 From: david.froger at gmail.com (David Froger) Date: Mon, 12 Sep 2011 08:52:02 +0200 Subject: [Numpy-discussion] iterate over multiple arrays Message-ID: <1315810245-sup-1371@david-desktop> Hy everybody, I'm wondering what is the (best) way to apply the same function to multiple arrays. For example, in the following code: from numpy import * def f(arr): return arr*2 a = array( [1,1,1] ) b = array( [2,2,2] ) c = array( [3,3,3] ) d = array( [4,4,4] ) a = f(a) b = f(b) c = f(c) d = f(d) I would like to replace : a = f(a) b = f(b) c = f(c) d = f(d) with something like that, but which really modify a,b,c and d: for x in [a,b,c,d]: x = f(x) So having something like a pointer on the arrays. Thanks for your help ! David -- From shish at keba.be Mon Sep 12 07:00:00 2011 From: shish at keba.be (Olivier Delalleau) Date: Mon, 12 Sep 2011 07:00:00 -0400 Subject: [Numpy-discussion] iterate over multiple arrays In-Reply-To: <1315810245-sup-1371@david-desktop> References: <1315810245-sup-1371@david-desktop> Message-ID: If you can make f work in-place then you can just call map(f, [a, b, c, d]): def f(arr): arr *= 2 Otherwise, you can: - Work with a list instead (a_b_c_d = map(f, a_b_c_d), with a_b_c_d = [a, b, c, d], but this won't update the local definitions of a, b, c, d). - Use locals(): for x in ('a', 'b', 'c', 'd'): locals()[x] = f(eval(x)) -=- Olivier 2011/9/12 David Froger > Hy everybody, > > I'm wondering what is the (best) way to apply the same function to multiple > arrays. > > For example, in the following code: > > from numpy import * > > def f(arr): > return arr*2 > > a = array( [1,1,1] ) > b = array( [2,2,2] ) > c = array( [3,3,3] ) > d = array( [4,4,4] ) > > a = f(a) > b = f(b) > c = f(c) > d = f(d) > > I would like to replace : > > a = f(a) > b = f(b) > c = f(c) > d = f(d) > > with something like that, but which really modify a,b,c and d: > > for x in [a,b,c,d]: > x = f(x) > > So having something like a pointer on the arrays. > > Thanks for your help ! > > David > -- > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonas.wallin81 at gmail.com Mon Sep 12 07:18:35 2011 From: jonas.wallin81 at gmail.com (Jonas Wallin) Date: Mon, 12 Sep 2011 13:18:35 +0200 Subject: [Numpy-discussion] Matrix addition, += Message-ID: Hello, I am sure this question has been answered before but I can't find the right search word to find it. Why does MuY += MuY.transpose() and MuY = MuY + MuY.transpose() give different answers? thanks /Jonas Wallin -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Mon Sep 12 07:58:31 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Mon, 12 Sep 2011 07:58:31 -0400 Subject: [Numpy-discussion] Matrix addition, += In-Reply-To: References: Message-ID: <4E6DF3E7.7000909@gmail.com> On 9/12/2011 7:18 AM, Jonas Wallin wrote: > Why does > > MuY += MuY.transpose() > > and > > MuY = MuY + MuY.transpose() > > give different answers? Because the first one is done in-place, so you are changing MuY (and thus MuY.transpose) as the operation proceeds. MuY.transpose() is generally a *view* of MuY. http://docs.scipy.org/doc/numpy/reference/generated/numpy.transpose.html Use the second one. Alan Isaac From jniehof at lanl.gov Mon Sep 12 13:17:16 2011 From: jniehof at lanl.gov (Jonathan T. Niehof) Date: Mon, 12 Sep 2011 11:17:16 -0600 (MDT) Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <4E6A2D54.6050406@noaa.gov> References: <4E68AD7B.2080705@noaa.gov> <4E6A2D54.6050406@noaa.gov> Message-ID: <52204.128.165.0.81.1315847836.squirrel@webmail.lanl.gov> > Is anyone successfully using f2py and gfortran on a Windows machine > without relying on cygwin? SpacePy uses mingw32 for both gcc and gfortran; I didn't have any trouble with f2py. I haven't tried a build with 64-bit Python or with EPD; I just build the installer against python.org's python and a bog-standard mingw32 and numpy install. Feel free to take a look at our setup.py (http://spacepy.lanl.gov/). It runs f2py with: --fcompiler=gnu95 --compiler=mingw32 (Note that gfortran normally refers to the new f90/95 chain, called gnu95 by f2py, and g77 is the old, f2c-derived chain, called gnu by f2py.) -- Jonathan Niehof ISR-3 Space Data Systems Los Alamos National Laboratory MS-D466 Los Alamos, NM 87545 Phone: 505-667-9595 email: jniehof at lanl.gov Correspondence / Technical data or Software Publicly Available From Jim.Vickroy at noaa.gov Mon Sep 12 13:54:04 2011 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Mon, 12 Sep 2011 11:54:04 -0600 Subject: [Numpy-discussion] f2py : NotImplementedError: Only MS compiler supported with gfortran on win64 In-Reply-To: <52204.128.165.0.81.1315847836.squirrel@webmail.lanl.gov> References: <4E68AD7B.2080705@noaa.gov> <4E6A2D54.6050406@noaa.gov> <52204.128.165.0.81.1315847836.squirrel@webmail.lanl.gov> Message-ID: <4E6E473C.7080805@noaa.gov> On 9/12/2011 11:17 AM, Jonathan T. Niehof wrote: >> Is anyone successfully using f2py and gfortran on a Windows machine >> without relying on cygwin? > SpacePy uses mingw32 for both gcc and gfortran; I didn't have any trouble > with f2py. I haven't tried a build with 64-bit Python or with EPD; I just > build the installer against python.org's python and a bog-standard mingw32 > and numpy install. Feel free to take a look at our setup.py > (http://spacepy.lanl.gov/). It runs f2py with: > --fcompiler=gnu95 --compiler=mingw32 > > (Note that gfortran normally refers to the new f90/95 chain, called gnu95 > by f2py, and g77 is the old, f2c-derived chain, called gnu by f2py.) > Thanks for this information; that site certainly looks interesting. I will look at it more carefully as I'm developing software at the NOAA Space Weather Prediction Center . Christoph Gohlke determined that if I upgraded from numpy 1.3 to 1.4, f2py would successfully find and use the Intel Fortran compiler which was my preferred choice all along. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 12 14:41:38 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Sep 2011 13:41:38 -0500 Subject: [Numpy-discussion] iterate over multiple arrays In-Reply-To: <1315810245-sup-1371@david-desktop> References: <1315810245-sup-1371@david-desktop> Message-ID: On Mon, Sep 12, 2011 at 01:52, David Froger wrote: > Hy everybody, > > I'm wondering what is the (best) way to apply the same function to multiple > arrays. > > For example, in the following code: > > from numpy import * > > def f(arr): > ? ? return arr*2 > > a = array( [1,1,1] ) > b = array( [2,2,2] ) > c = array( [3,3,3] ) > d = array( [4,4,4] ) > > a = f(a) > b = f(b) > c = f(c) > d = f(d) > > I would like to replace : > > a = f(a) > b = f(b) > c = f(c) > d = f(d) This is usually the best thing to do for few variables and simple function calls. If you have many more variables, you should be keeping them in a list or dict instead of individual named variables. If you have a complicated expression, wrap it in a function. You could also do something like this: a,b,c,d = map(f, [a,b,c,d]) but it's harder to understand what is going on that just using four separate lines and no easier to maintain. Don't use eval() or locals(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From reckoner at gmail.com Mon Sep 12 16:12:52 2011 From: reckoner at gmail.com (Reckoner) Date: Mon, 12 Sep 2011 13:12:52 -0700 Subject: [Numpy-discussion] using Matlab compiler to build Python-callable library? Message-ID: Hi, Has anybody ever tried using the Matlab compiler to build a standalone library that would be callable using Python? We have a lot of leftover Matlab code that we are trying to migrate. Thanks! From cjordan1 at uw.edu Mon Sep 12 19:38:13 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Mon, 12 Sep 2011 18:38:13 -0500 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E692C57.2040501@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> Message-ID: I did some timings to see what the advantage would be, in the simplest case possible, of taking multiple lines from the file to process at a time. Assuming the dtype is already known. The code is attached. What I found was I can't use generators to avoid constructing a list and then making a tuple from the list. It appears that the user must create a tuple to place in a numpy record array. (Specifically, if you remove the 'tuple' command from f2 in the attached then you get an error.) Taking multiple lines at a time (using f4) does provide a speed benefit, but it's not very big since Python's re module won't let you capture more than 100 values, and I'm using capturing to extract the values. (This is done because we're allowing the user to use regular expressions to denote delimiters.) In the example it's a bunch of space-delimited integers. f1 splits on the space and uses a list comprehension, f2 splits on the space and uses a generator, f3 uses regular expressions in a manner similar to the current code, and f4 uses regular expressions on multiple lines at once, and f5 uses fromiter. (Though fromiter isn't as useful as I'd hoped because you have to have already parsed out a line, since this is a record array.) f6 and f7 use stripped down versions of Chris Barker's accumulator idea. The difference is that f6 uses resize when expanding the array while f7 uses np.empty followed by np.append. This avoids the penalty from copying data that np.resize imposes. Note that f6 and f7 use the regular expression capturing line by line as in f3. To get a feel for the overheard involved with keeping track of string sizes, f8 is just f3 except with a list for the largest string sizes seen so far. The speeds I get using timeit are f1 : 1.66ms f2 : 2.01ms f3 : 2.35ms f4(2) : 3.02ms (Odd that it starts out worse than f3 when you take 2 lines at a time) f4(5) : 2.25ms f4(10) : 2.02ms f4(15) : 1.93ms f4(20) : error f5 : 2.28ms (As I said, fromiter can't do much when it's just filling in a record array. While it's slightly faster than f3, which it's based on, it also loads all the data as a list before creating a numpy array, which is rather suboptimal.) f6 : 3.26ms f7 : 2.77ms (Apparently it's a lot cheaper to do np.empty followed by append then do to resize) f8 : 3.04ms (Compared to f3, this shows there's a non-trivial performance hit from keeping track of the sizes) It seems like taking multiple lines at once isn't a big gain when we're limited to 100 captured entries at a time. (For Python 2.6, at least.) Especially since taking multiple lines at once would be rather complex since the code must still check each line to see if it's commented out or not. After talking to Chris Farrow, an Enthought developer, and doing some timing on a dataset he was working on, I had loadtable running about 1.7 to 3.3 times as fast as genfromtxt. The catch is that genfromtxt was loading datetimes as strings, while loadtable was loading them as numpy datetimes. The conversion from string to datetime is somewhat expensive, so I think that accounts for some of the extra time. The range of timings--between 1.5 to 3.5 times as slow--reflect how many lines are used to check for sizes and dtypes. As it turns out, checking for those can be quite expensive, and the majority of the time seems to be spent in the regular expression matching. (Though Chris is using a slight variant on my pull request, and I'm getting function times that are not as bad as his.) The cost of the size and type checking was less apparent in the example I have timings on in a previous email because in that case there was a huge cost for converting data with commas instead of decimals and for the datetime conversion. To give some further context, I compared np.genfromtxt and np.loadtable on the same 'pseudo-file' f used in the above tests, when the data is just a bunch of integers. The results were: np.genfromtxt with dtype=None: 4.45 ms np.loadtable with defaults: 5.12ms np.loadtable with check_sizes=False: 3.7ms So it seems that np.loadtable is already competitive with np.genfromtxt other than checking the sizes. And the checking sizes isn't even that huge a penalty compared to genfromtxt. Based on all the above it seems like the accumulator is the most promising way that things could be sped up. But it's not completely clear to me by how much, since we still must keep track of the dtypes and the sizes. Other than possibly changing loadtable to use np.NA instead of masked arrays in the presence of missing data, I'm starting to feel like it's more or less complete for now, and can be left to be improved in the future. Most of the things that have been discussed are either performance trade-offs or somewhat large re-engineering of the internals. -Chris JS On Thu, Sep 8, 2011 at 3:57 PM, Chris.Barker wrote: > On 9/8/11 1:43 PM, Christopher Jordan-Squire wrote: >> I just ran a quick test on my machine of this idea. With >> >> dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)]) >> temp = np.empty((), dtype=dt) >> temp2 = np.zeros(1,dtype=dt) >> >> In [96]: def f(): >> ? ? ?...: ? ? l=[0]*3 >> ? ? ?...: ? ? l[0] = 2.54 >> ? ? ?...: ? ? l[1] = 4 >> ? ? ?...: ? ? l[2] = 2.3645 >> ? ? ?...: ? ? j = tuple(l) >> ? ? ?...: ? ? temp2[0] = j >> >> vs >> >> >> In [97]: def g(): >> ? ? ?...: ? ? temp['x'] = 2.54 >> ? ? ?...: ? ? temp['y'] = 4 >> ? ? ?...: ? ? temp['z'] = 2.3645 >> ? ? ?...: ? ? temp2[0] = temp >> ? ? ?...: >> >> The timing results were 2.73 us for f and 3.43 us for g. So good idea, >> but it doesn't appear to be faster. (Though the difference wasn't >> nearly as dramatic as I thought it would be, based on Pauli's >> comment.) > > my guess is that the lines like: temp['x'] = 2.54 are slower (it > requires a dict lookup, and a conversion from a python type to a "raw" type) > > and > > temp2[0] = temp > > is faster, as that doesn't require any conversion. > > Which means that if you has a larger struct dtype, it would be even > slower, so clearly not the way to go for performance. > > It would be nice to have a higher performing struct dtype scalar -- as > it is ordered, it might be nice to be able to index it with either the > name or an numeric index. > > -Chris > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: txt_time.py Type: text/x-python Size: 4472 bytes Desc: not available URL: From david.froger at gmail.com Tue Sep 13 02:53:48 2011 From: david.froger at gmail.com (David Froger) Date: Tue, 13 Sep 2011 08:53:48 +0200 Subject: [Numpy-discussion] iterate over multiple arrays In-Reply-To: References: <1315810245-sup-1371@david-desktop> Message-ID: <1315896408-sup-2039@david-desktop> Thank you Olivier and Robert for your replies! Some remarks about the dictionnary solution: from numpy import * def f(arr): return arr + 100. arrs = {} arrs['a'] = array( [1,1,1] ) arrs['b'] = array( [2,2,2] ) arrs['c'] = array( [3,3,3] ) arrs['d'] = array( [4,4,4] ) for key,value in arrs.iteritems(): arrs[key] = f(value) 1. about the memory Memory is first allocated with the array functions: arrs['a'] = array( [1,1,1] ) arrs['b'] = array( [2,2,2] ) arrs['c'] = array( [3,3,3] ) arrs['d'] = array( [4,4,4] ) Are there others memory allocations with this assignemnt: arrs[key] = f(value) or is the already allocated memory used to store the result of f(value)? In other words, if I have N arrays of the same shape, each of them costing nbytes of memory, does it use N*nbytes memory, or 2*N*bytes? I think this is well documented on the web and I can find it.... 2. about individual array The problem is that now, if one want to use a individual array, one have now to use: arrs['a'] instead of just: a So I'm sometime tempted to use locals() instead of arrs... -- From robert.kern at gmail.com Tue Sep 13 03:18:55 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 13 Sep 2011 02:18:55 -0500 Subject: [Numpy-discussion] iterate over multiple arrays In-Reply-To: <1315896408-sup-2039@david-desktop> References: <1315810245-sup-1371@david-desktop> <1315896408-sup-2039@david-desktop> Message-ID: On Tue, Sep 13, 2011 at 01:53, David Froger wrote: > > Thank you Olivier and Robert for your replies! > > Some remarks about the dictionnary solution: > > from numpy import * > > def f(arr): > ? ? return arr + 100. > > arrs = {} > arrs['a'] = array( [1,1,1] ) > arrs['b'] = array( [2,2,2] ) > arrs['c'] = array( [3,3,3] ) > arrs['d'] = array( [4,4,4] ) > > for key,value in arrs.iteritems(): > ? ?arrs[key] = f(value) > > 1. about the memory > Memory is first allocated with the array functions: > ? ?arrs['a'] = array( [1,1,1] ) > ? ?arrs['b'] = array( [2,2,2] ) > ? ?arrs['c'] = array( [3,3,3] ) > ? ?arrs['d'] = array( [4,4,4] ) > > Are there others memory allocations with this assignemnt: > ? ?arrs[key] = f(value) > or is the already allocated memory used to store the result of f(value)? > > In other words, if I have N arrays of the same shape, each of them costing > nbytes of memory, does it use N*nbytes memory, or 2*N*bytes? Temporarily, yes, for all of the variations mentioned. When the expression "f(value)" is evaluated, both the result array and the input array will exist simultaneously in memory. Once the assignment happens, the original input array will be destroyed and free up the memory. There is no difference memory-wise between assigning into a dictionary or assigning to a variable name. Sometimes, you can write your f() such that you just need to do f(value) and have the value object modified in-place. In that case, there is no need to reassign the result to a variable or dictionary key. > I think this is well documented on the web and I can find it.... > > 2. about individual array > The problem is that now, if one want to use a individual array, one have now to > use: > ? ?arrs['a'] > instead of just: > ? ?a > So I'm sometime tempted to use locals() instead of arrs... Seriously, don't. It makes your code worse, not better. It's also unreliable. The locals() dictionary is meant to be read-only (and even then for debugger tooling and the like, not regular code), and this is sometimes enforced. If you want to use variable names instead of dictionaries, use them, but write out each assignment statement. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pgmdevlist at gmail.com Tue Sep 13 04:43:38 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 13 Sep 2011 10:43:38 +0200 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> Message-ID: <8B98006A-DA19-46A9-A1DD-73D5F18E3434@gmail.com> On Sep 13, 2011, at 01:38 , Christopher Jordan-Squire wrote: > I did some timings to see what the advantage would be, in the simplest > case possible, of taking multiple lines from the file to process at a > time. Assuming the dtype is already known. The code is attached. What > I found was I can't use generators to avoid constructing a list and > then making a tuple from the list. Still, I think there should be a way to use generators to create the final array (once your dtype is known and assuming you can skip invalid lines)... > The catch is that genfromtxt > was loading datetimes as strings, while loadtable was loading them as > numpy datetimes. The conversion from string to datetime is somewhat > expensive, so I think that accounts for some of the extra time. The > range of timings--between 1.5 to 3.5 times as slow--reflect how many > lines are used to check for sizes and dtypes. As it turns out, > checking for those can be quite expensive, and the majority of the > time seems to be spent in the regular expression matching. (Though > Chris is using a slight variant on my pull request, and I'm getting > function times that are not as bad as his.) The cost of the size and > type checking was less apparent in the example I have timings on in a > previous email because in that case there was a huge cost for > converting data with commas instead of decimals and for the datetime > conversion. The problem with parsing dates with re is that depending on your separator, on your local conventions (e.g., MM-DD-YYYY vs DD/MM/YYYY) and on your string itself, you'll get very different results, not always the ones you want. Hence, I preferred to leave the dates out of the basic convertors and ask the user instead to give her own. If you can provide a functionality in loadtable to that effect, that'd be great. > Other than possibly changing loadtable to use np.NA instead of masked > arrays in the presence of missing data, I'm starting to feel like it's > more or less complete for now, and can be left to be improved in the > future. Most of the things that have been discussed are either > performance trade-offs or somewhat large re-engineering of the > internals. Well, it seems that loadtable doesn't work when you use positions instead of delimiters to separate the fields (e.g. below). What if you want to apply some specific conversion to a column ? e.g., transform a string representing a hexa to a int? Apart from that, I do appreciate the efforts you're putting to improve genfromtxt. It's needed, direly. Sorry that I can't find the time to really work on that (I do need to sleep sometimes)? But chats with Pauli V., Ralf G. among others during EuroScipy lead me to think a basic reorganization of npyio is quite advisable. #C00:07 : YYYYMMDD (8) #C08:15 : HH:mm:SS (8) #C16:18 : XXX (3) #C19:25 : float (7) #C26:32 : float (7) #C27:39 : float (7) # np.genfromtxt('test.txt', delimiter=(8,8,3,7,7,7), usemask=True, dtype=None) 2011010112:34:56AAA001.234005.678010.123999.999 2011010112:34:57BBB001.234005.678010.123999.999 2011010112:34:58CCC001.234005.678010.123999.999 2011010112:34:59 001.234005.678010.123999.999 2011010112:35:00DDD 5.678010.123 2011010112:35:01EEE001.234005.678010.123999.999 From igor.ying at yahoo.com Tue Sep 13 06:38:56 2011 From: igor.ying at yahoo.com (Igor Ying) Date: Tue, 13 Sep 2011 03:38:56 -0700 (PDT) Subject: [Numpy-discussion] Numpy - MKL - build error Message-ID: <1315910336.26287.YahooMailNeo@web120802.mail.ne1.yahoo.com> Hi, ?I am very new to Numpy and? trying to build Numpy 1.6.1 with Intel MKL 10.3.6 and getting the following errors. $python setup.py config --compiler=intel build_clib --compiler=intel build_ext --compiler=intel install Running from numpy source directory.F2PY Version 2 blas_opt_info: blas_mkl_info: ? libraries mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md not found in /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 ? NOT AVAILABLE I have modified the site.cfg as below. [mkl] library_dirs = /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 include_dirs = /opt/intel/composer_xe_2011_sp1/mkl/include mkl_libs = mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md lapack_libs = mkl_lapack_lp64 Environment variables are also set to point to mkl libraries. source /opt/intel/composer_xe_2011_sp1/bin/compilervars.sh intel64 lp64 export OMP_NUM_THREADS=4 LD_LIBRARY_PATH="/opt/intel/composer_xe_2011_sp1/mkl/lib/intel64:${LD_LIBRARY_PATH}"; export LD_LIBRARY_PATH export LD_RUN_PATH=/opt/lib:/opt1/composer_xe_2011_sp1/lib/intel64:/opt/composer_xe_2011_sp1/mkl/lib/intel64 Can anyone help me? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Tue Sep 13 09:27:41 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 13 Sep 2011 08:27:41 -0500 Subject: [Numpy-discussion] iterate over multiple arrays In-Reply-To: <1315896408-sup-2039@david-desktop> References: <1315810245-sup-1371@david-desktop> <1315896408-sup-2039@david-desktop> Message-ID: <4E6F5A4D.4040804@gmail.com> On 09/13/2011 01:53 AM, David Froger wrote: > Thank you Olivier and Robert for your replies! > > Some remarks about the dictionnary solution: > > from numpy import * > > def f(arr): > return arr + 100. > > arrs = {} > arrs['a'] = array( [1,1,1] ) > arrs['b'] = array( [2,2,2] ) > arrs['c'] = array( [3,3,3] ) > arrs['d'] = array( [4,4,4] ) > > for key,value in arrs.iteritems(): > arrs[key] = f(value) > > 1. about the memory > Memory is first allocated with the array functions: > arrs['a'] = array( [1,1,1] ) > arrs['b'] = array( [2,2,2] ) > arrs['c'] = array( [3,3,3] ) > arrs['d'] = array( [4,4,4] ) > > Are there others memory allocations with this assignemnt: > arrs[key] = f(value) > or is the already allocated memory used to store the result of f(value)? > > In other words, if I have N arrays of the same shape, each of them costing > nbytes of memory, does it use N*nbytes memory, or 2*N*bytes? > > I think this is well documented on the web and I can find it.... > > 2. about individual array > The problem is that now, if one want to use a individual array, one have now to > use: > arrs['a'] > instead of just: > a > So I'm sometime tempted to use locals() instead of arrs... > Perhaps not quite what you want but you do have the option of a structured or record array instead of a dict if the individual arrays are the same dimensions. See for example: http://www.scipy.org/RecordArrays >>> import numpy as np >>> img = np.array([(1,2,3,4), (1,2,3,4), (1,2,3,4), (1,2,3,4)], [('a',int),('b',int),('c',int), ('d',int)]) >>> img.dtype.names ('a', 'b', 'c', 'd') >>> img['a'] array([1, 1, 1, 1]) >>> rimg = img.view(np.recarray) #take a view >>> rimg.a array([1, 1, 1, 1]) >>> rimg.b array([2, 2, 2, 2]) Other links: http://docs.scipy.org/doc/numpy/user/basics.rec.html http://www.scipy.org/Cookbook/Recarray Bruce From shish at keba.be Tue Sep 13 09:49:30 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 13 Sep 2011 09:49:30 -0400 Subject: [Numpy-discussion] iterate over multiple arrays In-Reply-To: References: <1315810245-sup-1371@david-desktop> <1315896408-sup-2039@david-desktop> Message-ID: I agree with Robert, don't use locals(). I should have added a disclaimer "this is very hackish and probably not a good idea", sorry ;) (interesting read: http://stackoverflow.com/questions/1450275/modifying-locals-in-python) >From what you said I think what you really want is f to work in-place. This also has the advantage of minimizing memory allocations. If you can't directly modify f, you can also do: def f_inplace(x): x[:] = f(x) then just call map(f_inplace, [a, b, c, d]). However note that there will be some memory temporarily allocated to store the result of f(x) (so it is not as optimal as ensuring f directly works in-place). In addition to the dictionary workaround mentioned by Robert, if it is not practical to have all your variables of interest into a single dictionary, you can instead declare your variables as one-element lists, or use a class with a single field: 1. a = [numpy.array(...)] a[0] = f(a[0]) 2. class ArrayHolder(object): def __init__(self, arr): self.arr = arr a = ArrayHolder(numpy.array(...)) a.arr = f(a.arr) But of course it is not as convenient to write a[0] or a.arr instead of just a. -=- Olivier 2011/9/13 Robert Kern > On Tue, Sep 13, 2011 at 01:53, David Froger > wrote: > > > > Thank you Olivier and Robert for your replies! > > > > Some remarks about the dictionnary solution: > > > > from numpy import * > > > > def f(arr): > > return arr + 100. > > > > arrs = {} > > arrs['a'] = array( [1,1,1] ) > > arrs['b'] = array( [2,2,2] ) > > arrs['c'] = array( [3,3,3] ) > > arrs['d'] = array( [4,4,4] ) > > > > for key,value in arrs.iteritems(): > > arrs[key] = f(value) > > > > 1. about the memory > > Memory is first allocated with the array functions: > > arrs['a'] = array( [1,1,1] ) > > arrs['b'] = array( [2,2,2] ) > > arrs['c'] = array( [3,3,3] ) > > arrs['d'] = array( [4,4,4] ) > > > > Are there others memory allocations with this assignemnt: > > arrs[key] = f(value) > > or is the already allocated memory used to store the result of f(value)? > > > > In other words, if I have N arrays of the same shape, each of them > costing > > nbytes of memory, does it use N*nbytes memory, or 2*N*bytes? > > Temporarily, yes, for all of the variations mentioned. When the > expression "f(value)" is evaluated, both the result array and the > input array will exist simultaneously in memory. Once the assignment > happens, the original input array will be destroyed and free up the > memory. There is no difference memory-wise between assigning into a > dictionary or assigning to a variable name. > > Sometimes, you can write your f() such that you just need to do > > f(value) > > and have the value object modified in-place. In that case, there is no > need to reassign the result to a variable or dictionary key. > > > I think this is well documented on the web and I can find it.... > > > > 2. about individual array > > The problem is that now, if one want to use a individual array, one have > now to > > use: > > arrs['a'] > > instead of just: > > a > > So I'm sometime tempted to use locals() instead of arrs... > > Seriously, don't. It makes your code worse, not better. It's also > unreliable. The locals() dictionary is meant to be read-only (and even > then for debugger tooling and the like, not regular code), and this is > sometimes enforced. If you want to use variable names instead of > dictionaries, use them, but write out each assignment statement. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Tue Sep 13 09:58:27 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 13 Sep 2011 09:58:27 -0400 Subject: [Numpy-discussion] Numpy - MKL - build error In-Reply-To: <1315910336.26287.YahooMailNeo@web120802.mail.ne1.yahoo.com> References: <1315910336.26287.YahooMailNeo@web120802.mail.ne1.yahoo.com> Message-ID: Sorry if it sounds like a stupid question, but are the files listed in the error message present in that directory? If yes, maybe try running the command with sudo, just in case it would be some weird permission issue. -=- Olivier 2011/9/13 Igor Ying > Hi, > > I am very new to Numpy and trying to build Numpy 1.6.1 with Intel MKL > 10.3.6 and getting the following errors. > > $python setup.py config --compiler=intel build_clib --compiler=intel > build_ext --compiler=intel install > > Running from numpy source directory.F2PY Version 2 > blas_opt_info: > blas_mkl_info: > libraries mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md not > found in /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 > NOT AVAILABLE > > I have modified the site.cfg as below. > > [mkl] > library_dirs = /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 > include_dirs = /opt/intel/composer_xe_2011_sp1/mkl/include > mkl_libs = mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md > lapack_libs = mkl_lapack_lp64 > > > Environment variables are also set to point to mkl libraries. > > source /opt/intel/composer_xe_2011_sp1/bin/compilervars.sh intel64 lp64 > > export OMP_NUM_THREADS=4 > LD_LIBRARY_PATH="/opt/intel/composer_xe_2011_sp1/mkl/lib/intel64:${LD_LIBRARY_PATH}"; > > export LD_LIBRARY_PATH > export > LD_RUN_PATH=/opt/lib:/opt1/composer_xe_2011_sp1/lib/intel64:/opt/composer_xe_2011_sp1/mkl/lib/intel64 > > Can anyone help me? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkg at lanl.gov Tue Sep 13 14:15:11 2011 From: tkg at lanl.gov (Thomas K Gamble) Date: Tue, 13 Sep 2011 12:15:11 -0600 Subject: [Numpy-discussion] upgrading to 1.6.1, lapack not used? Message-ID: <201109131215.11545.tkg@lanl.gov> I am trying to upgrade my numpy installation from 1.5.1 to 1.6.1 on Ubuntu 10.04. When I build, it doesn't seem to use the installed lapack libraries, but uses lapack_lite. After installation (which seems to complete without error), importing numpy fails with an error that it can't find libquadmath.so.0. Any ideas what might be wrong? Am I missing a dependent package somewhere? How can I force the build to use the installed lapack stuff? (installed via synaptic from the ubuntu debs) Numpy 1.5.1 builds and installs with no problems. -- Thomas K. Gamble Research Technologist, System/Network Administrator Chemical Diagnostics and Engineering (C-CDE) Los Alamos National Laboratory MS-E543,p:505-665-4323 f:505-665-4267 "There cannot be a crisis next week. My schedule is already full." Henry Kissinger -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Sep 13 14:41:46 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Tue, 13 Sep 2011 11:41:46 -0700 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> Message-ID: <4E6FA3EA.9010306@noaa.gov> On 9/12/11 4:38 PM, Christopher Jordan-Squire wrote: > I did some timings to see what the advantage would be, in the simplest > case possible, of taking multiple lines from the file to process at a > time. Nice work, only a minor comment: > f6 and f7 use stripped down versions of Chris > Barker's accumulator idea. The difference is that f6 uses resize when > expanding the array while f7 uses np.empty followed by np.append. This > avoids the penalty from copying data that np.resize imposes. I don't think it does: """ In [3]: np.append? ---------- arr : array_like Values are appended to a copy of this array. Returns ------- out : ndarray A copy of `arr` with `values` appended to `axis`. Note that `append` does not occur in-place: a new array is allocated and filled. """ There is no getting around the copying. However, I think resize() uses the OS memory re-allocate call, which may, in some instances, have over-allocated the memory in the first place, and thus not require a copy. So I'm pretty sure ndarray.resize is as good as it gets. > f6 : 3.26ms > f7 : 2.77ms (Apparently it's a lot cheaper to do np.empty followed by > append then do to resize) Darn that profiling proving my expectations wrong again! though I'm really confused as to how that could be! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cjordan1 at uw.edu Tue Sep 13 16:01:23 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Tue, 13 Sep 2011 16:01:23 -0400 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E6FA3EA.9010306@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> <4E6FA3EA.9010306@noaa.gov> Message-ID: On Tue, Sep 13, 2011 at 2:41 PM, Chris.Barker wrote: > On 9/12/11 4:38 PM, Christopher Jordan-Squire wrote: >> I did some timings to see what the advantage would be, in the simplest >> case possible, of taking multiple lines from the file to process at a >> time. > > Nice work, only a minor comment: >> f6 and f7 use stripped down versions of Chris >> Barker's accumulator idea. The difference is that f6 uses resize when >> expanding the array while f7 uses np.empty followed by np.append. This >> avoids the penalty from copying data that np.resize imposes. > > I don't think it does: > > """ > In [3]: np.append? > ---------- > arr : array_like > ? ? Values are appended to a copy of this array. > > Returns > ------- > out : ndarray > ? ? A copy of `arr` with `values` appended to `axis`. ?Note that `append` > ? ? does not occur in-place: a new array is allocated and filled. > """ > > There is no getting around the copying. However, I think resize() uses > the OS memory re-allocate call, which may, in some instances, have > over-allocated the memory in the first place, and thus not require a > copy. So I'm pretty sure ndarray.resize is as good as it gets. > >> f6 : 3.26ms >> f7 : 2.77ms (Apparently it's a lot cheaper to do np.empty followed by >> append then do to resize) > > Darn that profiling proving my expectations wrong again! though I'm > really confused as to how that could be! > Sorry, I cheated by reading the docs. :-) """ numpy.resize(a, new_shape) Return a new array with the specified shape. If the new array is larger than the original array, then the new array is filled with repeated copies of a. Note that this behavior is different from a.resize(new_shape) which fills with zeros instead of repeated copies of a. """ The copying I meant was that numpy.resize will fill the resized array with repeated copies of the data. So np.empty followed by np.append avoids that. -Chris > -Chris > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Tue Sep 13 16:34:01 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 13 Sep 2011 22:34:01 +0200 Subject: [Numpy-discussion] Functions for finding the relative extrema of numeric data In-Reply-To: References: Message-ID: Hi Jacob, On Fri, Sep 9, 2011 at 11:57 PM, Jacob Silterra wrote: > Hello all, > > I'd like to see functions for calculating the relative extrema in a set of > data included in numpy. I use that functionality frequently, and always seem > to be writing my own version. It seems like this functionality would be > useful to the community at large, as it's a fairly common operation. > What is your application? > > For numeric data (which is presumably noisy), the definition of a relative > extrema isn't completely obvious. The implementation I am proposing finds a > point in an ndarray along an axis which is larger (or smaller) than it's > `order` nearest neighbors (`order` being an optional parameter, default 1). > This is likely to find more points than may be desired, which I believe is > preferable to the alternative. The output is formatted the same as > numpy.where. > > Code available here: https://github.com/numpy/numpy/pull/154 > > I'm not sure whether this belongs in numpy or scipy, that question is > somewhat debatable. More sophisticated peak-finding functions (in N > dimensions, as opposed to 1) may also be useful to the community, and those > would definitely belong in scipy. > I have the feeling this belongs in scipy. Although if it's just these two functions I'm not sure where exactly to put them. Looking at the functionality, this is quite a simple approach. For any data of the type I'm usually working with it will not be able to find the right local extrema. The same is true for your alternative definition below. A more powerful peak detection function would be a very good addition to scipy imho (perhaps in scipy.interpolate?). See also http://stackoverflow.com/questions/1713335/peak-finding-algorithm-for-python-scipy Cheers, Ralf > An alternative implementation would be to require that function be > continuously descending (or ascending) for `order` points, which would > enforce a minimum width on a peak. > > -Jacob Silterra > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Tue Sep 13 16:44:03 2011 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 13 Sep 2011 15:44:03 -0500 Subject: [Numpy-discussion] Functions for finding the relative extrema of numeric data In-Reply-To: References: Message-ID: On Tue, Sep 13, 2011 at 3:34 PM, Ralf Gommers wrote: > Hi Jacob, > > On Fri, Sep 9, 2011 at 11:57 PM, Jacob Silterra wrote: > >> Hello all, >> >> I'd like to see functions for calculating the relative extrema in a set of >> data included in numpy. I use that functionality frequently, and always seem >> to be writing my own version. It seems like this functionality would be >> useful to the community at large, as it's a fairly common operation. >> > > What is your application? > >> >> For numeric data (which is presumably noisy), the definition of a relative >> extrema isn't completely obvious. The implementation I am proposing finds a >> point in an ndarray along an axis which is larger (or smaller) than it's >> `order` nearest neighbors (`order` being an optional parameter, default 1). >> This is likely to find more points than may be desired, which I believe is >> preferable to the alternative. The output is formatted the same as >> numpy.where. >> >> Code available here: https://github.com/numpy/numpy/pull/154 >> >> I'm not sure whether this belongs in numpy or scipy, that question is >> somewhat debatable. More sophisticated peak-finding functions (in N >> dimensions, as opposed to 1) may also be useful to the community, and those >> would definitely belong in scipy. >> > > I have the feeling this belongs in scipy. Although if it's just these two > functions I'm not sure where exactly to put them. Looking at the > functionality, this is quite a simple approach. For any data of the type I'm > usually working with it will not be able to find the right local extrema. > The same is true for your alternative definition below. > > A more powerful peak detection function would be a very good addition to > scipy imho (perhaps in scipy.interpolate?). See also > http://stackoverflow.com/questions/1713335/peak-finding-algorithm-for-python-scipy > > Cheers, > Ralf > > Actually, such an algorithm would be great to partner with the watershed clustering implementation in ndimage. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Sep 13 17:28:39 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 13 Sep 2011 23:28:39 +0200 Subject: [Numpy-discussion] iterate over multiple arrays In-Reply-To: <1315810245-sup-1371@david-desktop> References: <1315810245-sup-1371@david-desktop> Message-ID: <4E6FCB07.4040408@molden.no> Den 12.09.2011 08:52, skrev David Froger: > Hy everybody, > > I'm wondering what is the (best) way to apply the same function to multiple > arrays. > I tried to experiment a bit with this. Here is from an ipython session: Create some arrays: In [1]: import numpy as np In [2]: a = np.zeros(4) In [3]: b = a+1 In [4]: c = a+2 In [5]: d = a+3 Create an array with dtype=object to store the four arrays a-d: In [6]: e = np.zeros(4, dtype=object) In [7]: e[:] = a,b,c,d In [8]: e Out[8]: array([[ 0. 0. 0. 0.], [ 1. 1. 1. 1.], [ 2. 2. 2. 2.], [ 3. 3. 3. 3.]], dtype=object) Modify array e inplace: In [9]: e += 1 In [10]: e Out[10]: array([[ 1. 1. 1. 1.], [ 2. 2. 2. 2.], [ 3. 3. 3. 3.], [ 4. 4. 4. 4.]], dtype=object) This did not modify arrays a-d though: In [11]: a Out[11]: array([ 0., 0., 0., 0.]) Albeit e[0] was array a prior to the iadd: In [12]: e[:] = a,b,c,d In [13]: e[0] is a Out[13]: True We can apply a function to the arrays in e, getting array([f(a), f(b), f(c), f(d)]): In [14]: np.sum(e) Out[14]: array([ 6., 6., 6., 6.]) Observe that if e was a 2D array, np.sum(e) would have returned a scalar, like so: In [18]: g = np.array((a,b,c,d)) In [19]: g Out[19]: array([[ 0., 0., 0., 0.], [ 1., 1., 1., 1.], [ 2., 2., 2., 2.], [ 3., 3., 3., 3.]]) In [20]: np.sum(g) Out[20]: 24.0 Which means: We can create an array of arrays, and have numpy broadcast an ufunc to multiple arrays. The other obvious way is, as mentioned by others, to keep the arrays in a normal Python container (e.g. list) and use a for loop or functional programming (map, apply, reduce). Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjordan1 at uw.edu Tue Sep 13 19:30:47 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Tue, 13 Sep 2011 18:30:47 -0500 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <8B98006A-DA19-46A9-A1DD-73D5F18E3434@gmail.com> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> <8B98006A-DA19-46A9-A1DD-73D5F18E3434@gmail.com> Message-ID: On Tue, Sep 13, 2011 at 3:43 AM, Pierre GM wrote: > > On Sep 13, 2011, at 01:38 , Christopher Jordan-Squire wrote: > >> I did some timings to see what the advantage would be, in the simplest >> case possible, of taking multiple lines from the file to process at a >> time. Assuming the dtype is already known. The code is attached. What >> I found was I can't use generators to avoid constructing a list and >> then making a tuple from the list. > > Still, I think there should be a way to use generators to create the final array (once your dtype is known and assuming you can skip invalid lines)... > >> ?The catch is that genfromtxt >> was loading datetimes as strings, while loadtable was loading them as >> numpy datetimes. The conversion from string to datetime is somewhat >> expensive, so I think that accounts for some of the extra time. The >> range of timings--between 1.5 to 3.5 times as slow--reflect how many >> lines are used to check for sizes and dtypes. As it turns out, >> checking for those can be quite expensive, and the majority of the >> time seems to be spent in the regular expression matching. (Though >> Chris is using a slight variant on my pull request, and I'm getting >> function times that are not as bad as his.) The cost of the size and >> type checking was less apparent in the example I have timings on in a >> previous email because in that case there was a huge cost for >> converting data with commas instead of decimals and for the datetime >> conversion. > > The problem with parsing dates with re is that depending on your separator, on your local conventions (e.g., MM-DD-YYYY vs DD/MM/YYYY) and on your string itself, you'll get very different results, not always the ones you want. Hence, I preferred to leave the dates out of the basic convertors and ask the user instead to give her own. If you can provide a functionality in loadtable to that effect, that'd be great. > >> Other than possibly changing loadtable to use np.NA instead of masked >> arrays in the presence of missing data, I'm starting to feel like it's >> more or less complete for now, and can be left to be improved in the >> future. Most of the things that have been discussed are either >> performance trade-offs or somewhat large re-engineering of the >> internals. > > Well, ?it seems that loadtable doesn't work when you use positions instead of delimiters to ?separate the fields (e.g. below). > What if you want to apply some specific conversion to a column ? e.g., transform a string representing a hexa to a int? > > Apart from that, I do appreciate the efforts you're putting to improve genfromtxt. It's needed, direly. Sorry that I can't find the time to really work on that (I do need to sleep sometimes)? But chats with Pauli V., Ralf G. among others during EuroScipy lead me to think a basic reorganization of npyio is quite advisable. > > > #C00:07 : YYYYMMDD (8) > #C08:15 : HH:mm:SS (8) > #C16:18 : XXX (3) > #C19:25 : float (7) > #C26:32 : float (7) > #C27:39 : float (7) > # np.genfromtxt('test.txt', delimiter=(8,8,3,7,7,7), usemask=True, dtype=None) > 2011010112:34:56AAA001.234005.678010.123999.999 > 2011010112:34:57BBB001.234005.678010.123999.999 > 2011010112:34:58CCC001.234005.678010.123999.999 > 2011010112:34:59 ? 001.234005.678010.123999.999 > 2011010112:35:00DDD ? ? ? ? 5.678010.123 > 2011010112:35:01EEE001.234005.678010.123999.999 Thanks for mentioning the fixed width file type. I had completely missed genfromtxt allows that. Though, in all honesty, I wasn't really intending that loadtable be a drop-in replacement for genfromtxt. More like a more robust and memory efficient alternative. I think I can add that functionality to loadtable, but it might require adding some special case stuff. Most everything is geared towards delimited text rather than fixed width text. An idea that was floated today when I talked about loadtable at Enthought was refactoring it as a class, and then letting some of the internals that currently aren't exposed to the user be exposed. That way the user could specify their own converters if desired without having to add yet another parameter. In fact, it could make it possible to remove some of the existing parameters by making them instance variables, for example. How do people feel about that? In terms of refactoring numpy io, was there anything concrete or specific discussed? -Chris JS > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From shish at keba.be Tue Sep 13 23:30:09 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 13 Sep 2011 23:30:09 -0400 Subject: [Numpy-discussion] upgrading to 1.6.1, lapack not used? In-Reply-To: <201109131215.11545.tkg@lanl.gov> References: <201109131215.11545.tkg@lanl.gov> Message-ID: Sorry can't really help with the numpy build, but for the "can't find libquadmath.so.0" issue, try to locate this file on your HD, and add the directory that contains it to your LD_LIBRARY_PATH environment variable. -=- Olivier 2011/9/13 Thomas K Gamble > ** > > I am trying to upgrade my numpy installation from 1.5.1 to 1.6.1 on Ubuntu > 10.04. When I build, it doesn't seem to use the installed lapack libraries, > but uses lapack_lite. After installation (which seems to complete without > error), importing numpy fails with an error that it can't find > libquadmath.so.0. Any ideas what might be wrong? Am I missing a dependent > package somewhere? How can I force the build to use the installed lapack > stuff? (installed via synaptic from the ubuntu debs) Numpy 1.5.1 builds and > installs with no problems. > > -- > > Thomas K. Gamble > > Research Technologist, System/Network Administrator > > Chemical Diagnostics and Engineering (C-CDE) > > Los Alamos National Laboratory > > MS-E543,p:505-665-4323 f:505-665-4267 > > "There cannot be a crisis next week. My schedule is already full." > > Henry Kissinger > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.ying at yahoo.com Wed Sep 14 03:37:52 2011 From: igor.ying at yahoo.com (Igor Ying) Date: Wed, 14 Sep 2011 00:37:52 -0700 (PDT) Subject: [Numpy-discussion] Numpy - MKL - build error In-Reply-To: References: Message-ID: <1315985872.15488.YahooMailNeo@web120804.mail.ne1.yahoo.com> Yes, they all are present in that directory.? Also, I tried with root as login. -r-xr-xr-x 1 root root? 26342559 Aug? 9 22:19 libmkl_avx.so -r--r--r-- 1 root root?? 1190224 Aug? 9 22:26 libmkl_blacs_ilp64.a -r--r--r-- 1 root root?? 1191496 Aug? 9 22:25 libmkl_blacs_intelmpi_ilp64.a -r-xr-xr-x 1 root root??? 497597 Aug? 9 22:25 libmkl_blacs_intelmpi_ilp64.so -r--r--r-- 1 root root??? 676206 Aug? 9 22:21 libmkl_blacs_intelmpi_lp64.a -r-xr-xr-x 1 root root??? 267010 Aug? 9 22:21 libmkl_blacs_intelmpi_lp64.so -r--r--r-- 1 root root??? 674926 Aug? 9 22:22 libmkl_blacs_lp64.a -r--r--r-- 1 root root?? 1218290 Aug? 9 22:28 libmkl_blacs_openmpi_ilp64.a -r--r--r-- 1 root root??? 703042 Aug? 9 22:23 libmkl_blacs_openmpi_lp64.a -r--r--r-- 1 root root?? 1191152 Aug? 9 22:29 libmkl_blacs_sgimpt_ilp64.a -r--r--r-- 1 root root??? 675854 Aug? 9 22:23 libmkl_blacs_sgimpt_lp64.a -r--r--r-- 1 root root??? 425802 Aug? 9 20:44 libmkl_blas95_ilp64.a -r--r--r-- 1 root root??? 421410 Aug? 9 20:44 libmkl_blas95_lp64.a -r--r--r-- 1 root root??? 144354 Aug? 9 22:29 libmkl_cdft_core.a -r-xr-xr-x 1 root root??? 115588 Aug? 9 22:29 libmkl_cdft_core.so -r--r--r-- 1 root root 231886824 Aug? 9 22:07 libmkl_core.a -r-xr-xr-x 1 root root? 16730033 Aug? 9 22:18 libmkl_core.so -r-xr-xr-x 1 root root? 21474555 Aug? 9 22:18 libmkl_def.so -r--r--r-- 1 root root? 14974574 Aug? 9 22:06 libmkl_gf_ilp64.a -r-xr-xr-x 1 root root?? 7008828 Aug? 9 22:48 libmkl_gf_ilp64.so -r--r--r-- 1 root root? 15140998 Aug? 9 22:06 libmkl_gf_lp64.a -r-xr-xr-x 1 root root?? 7055304 Aug? 9 22:48 libmkl_gf_lp64.so -r--r--r-- 1 root root? 16435120 Aug? 9 22:07 libmkl_gnu_thread.a -r-xr-xr-x 1 root root?? 9816940 Aug? 9 22:49 libmkl_gnu_thread.so -r--r--r-- 1 root root? 14968130 Aug? 9 22:06 libmkl_intel_ilp64.a -r-xr-xr-x 1 root root?? 7008368 Aug? 9 22:48 libmkl_intel_ilp64.so -r--r--r-- 1 root root? 15134406 Aug? 9 22:06 libmkl_intel_lp64.a -r-xr-xr-x 1 root root?? 7053588 Aug? 9 22:48 libmkl_intel_lp64.so -r--r--r-- 1 root root?? 2472940 Aug? 9 22:07 libmkl_intel_sp2dp.a -r-xr-xr-x 1 root root?? 1191479 Aug? 9 22:20 libmkl_intel_sp2dp.so -r--r--r-- 1 root root? 27642508 Aug? 9 22:07 libmkl_intel_thread.a -r-xr-xr-x 1 root root? 17516608 Aug? 9 22:49 libmkl_intel_thread.so -r--r--r-- 1 root root?? 5350948 Aug? 9 20:44 libmkl_lapack95_ilp64.a -r--r--r-- 1 root root?? 5413476 Aug? 9 20:44 libmkl_lapack95_lp64.a -r-xr-xr-x 1 root root? 29543829 Aug? 9 22:19 libmkl_mc3.so -r-xr-xr-x 1 root root? 25428037 Aug? 9 22:19 libmkl_mc.so -r-xr-xr-x 1 root root? 22888659 Aug? 9 22:18 libmkl_p4n.so -r--r--r-- 1 root root? 19232716 Aug? 9 22:07 libmkl_pgi_thread.a -r-xr-xr-x 1 root root? 12243062 Aug? 9 22:49 libmkl_pgi_thread.so -r-xr-xr-x 1 root root?? 4984870 Aug? 9 22:49 libmkl_rt.so -r--r--r-- 1 root root? 10367758 Aug? 9 22:49 libmkl_scalapack_ilp64.a -r-xr-xr-x 1 root root?? 6574928 Aug? 9 22:50 libmkl_scalapack_ilp64.so -r--r--r-- 1 root root? 10292432 Aug? 9 22:49 libmkl_scalapack_lp64.a -r-xr-xr-x 1 root root?? 6452627 Aug? 9 22:50 libmkl_scalapack_lp64.so -r--r--r-- 1 root root?? 9958444 Aug? 9 22:07 libmkl_sequential.a -r-xr-xr-x 1 root root?? 5926347 Aug? 9 22:48 libmkl_sequential.so -r--r--r-- 1 root root????? 1048 Aug? 9 16:50 libmkl_solver_ilp64.a -r--r--r-- 1 root root????? 1048 Aug? 9 16:50 libmkl_solver_ilp64_sequential.a -r--r--r-- 1 root root????? 1048 Aug? 9 16:50 libmkl_solver_lp64.a -r--r--r-- 1 root root????? 1048 Aug? 9 16:50 libmkl_solver_lp64_sequential.a -r-xr-xr-x 1 root root?? 6711968 Aug? 9 22:48 libmkl_vml_avx.so -r-xr-xr-x 1 root root?? 2795928 Aug? 9 22:47 libmkl_vml_def.so -r-xr-xr-x 1 root root?? 5476786 Aug? 9 22:48 libmkl_vml_mc2.so -r-xr-xr-x 1 root root?? 5778052 Aug? 9 22:48 libmkl_vml_mc3.so -r-xr-xr-x 1 root root?? 5382511 Aug? 9 22:48 libmkl_vml_mc.so -r-xr-xr-x 1 root root?? 4235841 Aug? 9 22:48 libmkl_vml_p4n.so drwxr-xr-x 3 root root????? 4096 Aug 18 11:43 locale Y you can reach the person managing the list at ??? numpy-discussion-owner at scipy.org Message: 1 Date: Tue, 13 Sep 2011 09:58:27 -0400 From: Olivier Delalleau Subject: Re: [Numpy-discussion] Numpy - MKL - build error To: Discussion of Numerical Python Message-ID: ??? Content-Type: text/plain; charset="iso-8859-1" Sorry if it sounds like a stupid question, but are the files listed in the error message present in that directory? If yes, maybe try running the command with sudo, just in case it would be some weird permission issue. -=- Olivier 2011/9/13 Igor Ying > Hi, > >? I am very new to Numpy and? trying to build Numpy 1.6.1 with Intel MKL > 10.3.6 and getting the following errors. > > $python setup.py config --compiler=intel build_clib --compiler=intel > build_ext --compiler=intel install > > Running from numpy source directory.F2PY Version 2 > blas_opt_info: > blas_mkl_info: >? libraries mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md not > found in /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 >? NOT AVAILABLE > > I have modified the site.cfg as below. > > [mkl] > library_dirs = /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 > include_dirs = /opt/intel/composer_xe_2011_sp1/mkl/include > mkl_libs = mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md > lapack_libs = mkl_lapack_lp64 > > > Environment variables are also set to point to mkl libraries. > > source /opt/intel/composer_xe_2011_sp1/bin/compilervars.sh intel64 lp64 > > export OMP_NUM_THREADS=4 > LD_LIBRARY_PATH="/opt/intel/composer_xe_2011_sp1/mkl/lib/intel64:${LD_LIBRARY_PATH}"; > > export LD_LIBRARY_PATH > export > LD_RUN_PATH=/opt/lib:/opt1/composer_xe_2011_sp1/lib/intel64:/opt/composer_xe_2011_sp1/mkl/lib/intel64 > > Can anyone help me? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Sep 14 03:59:30 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 14 Sep 2011 09:59:30 +0200 Subject: [Numpy-discussion] Numpy - MKL - build error In-Reply-To: <1315985872.15488.YahooMailNeo@web120804.mail.ne1.yahoo.com> References: <1315985872.15488.YahooMailNeo@web120804.mail.ne1.yahoo.com> Message-ID: It seems you are missing libiomp5.so, which is sound if you re using the whole Composer package: the needed libs are split in two different locations, and unfortunately, Numpy cannot cope with this last time I checked (I think it was one of the reasons David Cournapeau created numscons and bento). Matthieu 2011/9/14 Igor Ying > Yes, they all are present in that directory. Also, I tried with root as > login. > > -r-xr-xr-x 1 root root 26342559 Aug 9 22:19 libmkl_avx.so > -r--r--r-- 1 root root 1190224 Aug 9 22:26 libmkl_blacs_ilp64.a > -r--r--r-- 1 root root 1191496 Aug 9 22:25 libmkl_blacs_intelmpi_ilp64.a > -r-xr-xr-x 1 root root 497597 Aug 9 22:25 > libmkl_blacs_intelmpi_ilp64.so > -r--r--r-- 1 root root 676206 Aug 9 22:21 libmkl_blacs_intelmpi_lp64.a > -r-xr-xr-x 1 root root 267010 Aug 9 22:21 libmkl_blacs_intelmpi_lp64.so > -r--r--r-- 1 root root 674926 Aug 9 22:22 libmkl_blacs_lp64.a > -r--r--r-- 1 root root 1218290 Aug 9 22:28 libmkl_blacs_openmpi_ilp64.a > -r--r--r-- 1 root root 703042 Aug 9 22:23 libmkl_blacs_openmpi_lp64.a > -r--r--r-- 1 root root 1191152 Aug 9 22:29 libmkl_blacs_sgimpt_ilp64.a > -r--r--r-- 1 root root 675854 Aug 9 22:23 libmkl_blacs_sgimpt_lp64.a > -r--r--r-- 1 root root 425802 Aug 9 20:44 libmkl_blas95_ilp64.a > -r--r--r-- 1 root root 421410 Aug 9 20:44 libmkl_blas95_lp64.a > -r--r--r-- 1 root root 144354 Aug 9 22:29 libmkl_cdft_core.a > -r-xr-xr-x 1 root root 115588 Aug 9 22:29 libmkl_cdft_core.so > -r--r--r-- 1 root root 231886824 Aug 9 22:07 libmkl_core.a > -r-xr-xr-x 1 root root 16730033 Aug 9 22:18 libmkl_core.so > -r-xr-xr-x 1 root root 21474555 Aug 9 22:18 libmkl_def.so > -r--r--r-- 1 root root 14974574 Aug 9 22:06 libmkl_gf_ilp64.a > -r-xr-xr-x 1 root root 7008828 Aug 9 22:48 libmkl_gf_ilp64.so > -r--r--r-- 1 root root 15140998 Aug 9 22:06 libmkl_gf_lp64.a > -r-xr-xr-x 1 root root 7055304 Aug 9 22:48 libmkl_gf_lp64.so > -r--r--r-- 1 root root 16435120 Aug 9 22:07 libmkl_gnu_thread.a > -r-xr-xr-x 1 root root 9816940 Aug 9 22:49 libmkl_gnu_thread.so > -r--r--r-- 1 root root 14968130 Aug 9 22:06 libmkl_intel_ilp64.a > -r-xr-xr-x 1 root root 7008368 Aug 9 22:48 libmkl_intel_ilp64.so > -r--r--r-- 1 root root 15134406 Aug 9 22:06 libmkl_intel_lp64.a > -r-xr-xr-x 1 root root 7053588 Aug 9 22:48 libmkl_intel_lp64.so > -r--r--r-- 1 root root 2472940 Aug 9 22:07 libmkl_intel_sp2dp.a > -r-xr-xr-x 1 root root 1191479 Aug 9 22:20 libmkl_intel_sp2dp.so > -r--r--r-- 1 root root 27642508 Aug 9 22:07 libmkl_intel_thread.a > -r-xr-xr-x 1 root root 17516608 Aug 9 22:49 libmkl_intel_thread.so > -r--r--r-- 1 root root 5350948 Aug 9 20:44 libmkl_lapack95_ilp64.a > -r--r--r-- 1 root root 5413476 Aug 9 20:44 libmkl_lapack95_lp64.a > -r-xr-xr-x 1 root root 29543829 Aug 9 22:19 libmkl_mc3.so > -r-xr-xr-x 1 root root 25428037 Aug 9 22:19 libmkl_mc.so > -r-xr-xr-x 1 root root 22888659 Aug 9 22:18 libmkl_p4n.so > -r--r--r-- 1 root root 19232716 Aug 9 22:07 libmkl_pgi_thread.a > -r-xr-xr-x 1 root root 12243062 Aug 9 22:49 libmkl_pgi_thread.so > -r-xr-xr-x 1 root root 4984870 Aug 9 22:49 libmkl_rt.so > -r--r--r-- 1 root root 10367758 Aug 9 22:49 libmkl_scalapack_ilp64.a > -r-xr-xr-x 1 root root 6574928 Aug 9 22:50 libmkl_scalapack_ilp64.so > -r--r--r-- 1 root root 10292432 Aug 9 22:49 libmkl_scalapack_lp64.a > -r-xr-xr-x 1 root root 6452627 Aug 9 22:50 libmkl_scalapack_lp64.so > -r--r--r-- 1 root root 9958444 Aug 9 22:07 libmkl_sequential.a > -r-xr-xr-x 1 root root 5926347 Aug 9 22:48 libmkl_sequential.so > -r--r--r-- 1 root root 1048 Aug 9 16:50 libmkl_solver_ilp64.a > -r--r--r-- 1 root root 1048 Aug 9 16:50 > libmkl_solver_ilp64_sequential.a > -r--r--r-- 1 root root 1048 Aug 9 16:50 libmkl_solver_lp64.a > -r--r--r-- 1 root root 1048 Aug 9 16:50 > libmkl_solver_lp64_sequential.a > -r-xr-xr-x 1 root root 6711968 Aug 9 22:48 libmkl_vml_avx.so > -r-xr-xr-x 1 root root 2795928 Aug 9 22:47 libmkl_vml_def.so > -r-xr-xr-x 1 root root 5476786 Aug 9 22:48 libmkl_vml_mc2.so > -r-xr-xr-x 1 root root 5778052 Aug 9 22:48 libmkl_vml_mc3.so > -r-xr-xr-x 1 root root 5382511 Aug 9 22:48 libmkl_vml_mc.so > -r-xr-xr-x 1 root root 4235841 Aug 9 22:48 libmkl_vml_p4n.so > drwxr-xr-x 3 root root 4096 Aug 18 11:43 locale > Y > > > you can reach the person managing the list at > numpy-discussion-owner at scipy.org > > > Message: 1 > Date: Tue, 13 Sep 2011 09:58:27 -0400 > From: Olivier Delalleau > Subject: Re: [Numpy-discussion] Numpy - MKL - build error > To: Discussion of Numerical Python > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > > Sorry if it sounds like a stupid question, but are the files listed in the > error message present in that directory? > If yes, maybe try running the command with sudo, just in case it would be > some weird permission issue. > > -=- Olivier > > 2011/9/13 Igor Ying > > > Hi, > > > > I am very new to Numpy and trying to build Numpy 1.6.1 with Intel MKL > > 10.3.6 and getting the following errors. > > > > $python setup.py config --compiler=intel build_clib --compiler=intel > > build_ext --compiler=intel install > > > > Running from numpy source directory.F2PY Version 2 > > blas_opt_info: > > blas_mkl_info: > > libraries mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md not > > found in /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 > > NOT AVAILABLE > > > > I have modified the site.cfg as below. > > > > [mkl] > > library_dirs = /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 > > include_dirs = /opt/intel/composer_xe_2011_sp1/mkl/include > > mkl_libs = mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md > > lapack_libs = mkl_lapack_lp64 > > > > > > Environment variables are also set to point to mkl libraries. > > > > source /opt/intel/composer_xe_2011_sp1/bin/compilervars.sh intel64 lp64 > > > > export OMP_NUM_THREADS=4 > > > LD_LIBRARY_PATH="/opt/intel/composer_xe_2011_sp1/mkl/lib/intel64:${LD_LIBRARY_PATH}"; > > > > export LD_LIBRARY_PATH > > export > > > LD_RUN_PATH=/opt/lib:/opt1/composer_xe_2011_sp1/lib/intel64:/opt/composer_xe_2011_sp1/mkl/lib/intel64 > > > > Can anyone help me? > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.ying at yahoo.com Wed Sep 14 05:41:27 2011 From: igor.ying at yahoo.com (Igor Ying) Date: Wed, 14 Sep 2011 02:41:27 -0700 (PDT) Subject: [Numpy-discussion] Numpy - MKL - build error In-Reply-To: <1315985872.15488.YahooMailNeo@web120804.mail.ne1.yahoo.com> References: <1315985872.15488.YahooMailNeo@web120804.mail.ne1.yahoo.com> Message-ID: <1315993287.70152.YahooMailNeo@web120817.mail.ne1.yahoo.com> My bad, iomp5md is in compiler/lib dir, I copied it to the mkl dir and it worked. ________________________________ From: Igor Ying To: "numpy-discussion at scipy.org" Sent: Wednesday, September 14, 2011 1:07 PM Subject: Re: Numpy - MKL - build error Yes, they all are present in that directory.? Also, I tried with root as login. -r-xr-xr-x 1 root root? 26342559 Aug? 9 22:19 libmkl_avx.so -r--r--r-- 1 root root?? 1190224 Aug? 9 22:26 libmkl_blacs_ilp64.a -r--r--r-- 1 root root?? 1191496 Aug? 9 22:25 libmkl_blacs_intelmpi_ilp64.a -r-xr-xr-x 1 root root??? 497597 Aug? 9 22:25 libmkl_blacs_intelmpi_ilp64.so -r--r--r-- 1 root root??? 676206 Aug? 9 22:21 libmkl_blacs_intelmpi_lp64.a -r-xr-xr-x 1 root root??? 267010 Aug? 9 22:21 libmkl_blacs_intelmpi_lp64.so -r--r--r-- 1 root root??? 674926 Aug? 9 22:22 libmkl_blacs_lp64.a -r--r--r-- 1 root root?? 1218290 Aug? 9 22:28 libmkl_blacs_openmpi_ilp64.a -r--r--r-- 1 root root??? 703042 Aug? 9 22:23 libmkl_blacs_openmpi_lp64.a -r--r--r-- 1 root root?? 1191152 Aug? 9 22:29 libmkl_blacs_sgimpt_ilp64.a -r--r--r-- 1 root root??? 675854 Aug? 9 22:23 libmkl_blacs_sgimpt_lp64.a -r--r--r-- 1 root root??? 425802 Aug? 9 20:44 libmkl_blas95_ilp64.a -r--r--r-- 1 root root??? 421410 Aug? 9 20:44 libmkl_blas95_lp64.a -r--r--r-- 1 root root??? 144354 Aug? 9 22:29 libmkl_cdft_core.a -r-xr-xr-x 1 root root??? 115588 Aug? 9 22:29 libmkl_cdft_core.so -r--r--r-- 1 root root 231886824 Aug? 9 22:07 libmkl_core.a -r-xr-xr-x 1 root root? 16730033 Aug? 9 22:18 libmkl_core.so -r-xr-xr-x 1 root root? 21474555 Aug? 9 22:18 libmkl_def.so -r--r--r-- 1 root root? 14974574 Aug? 9 22:06 libmkl_gf_ilp64.a -r-xr-xr-x 1 root root?? 7008828 Aug? 9 22:48 libmkl_gf_ilp64.so -r--r--r-- 1 root root? 15140998 Aug? 9 22:06 libmkl_gf_lp64.a -r-xr-xr-x 1 root root?? 7055304 Aug? 9 22:48 libmkl_gf_lp64.so -r--r--r-- 1 root root? 16435120 Aug? 9 22:07 libmkl_gnu_thread.a -r-xr-xr-x 1 root root?? 9816940 Aug? 9 22:49 libmkl_gnu_thread.so -r--r--r-- 1 root root? 14968130 Aug? 9 22:06 libmkl_intel_ilp64.a -r-xr-xr-x 1 root root?? 7008368 Aug? 9 22:48 libmkl_intel_ilp64.so -r--r--r-- 1 root root? 15134406 Aug? 9 22:06 libmkl_intel_lp64.a -r-xr-xr-x 1 root root?? 7053588 Aug? 9 22:48 libmkl_intel_lp64.so -r--r--r-- 1 root root?? 2472940 Aug? 9 22:07 libmkl_intel_sp2dp.a -r-xr-xr-x 1 root root?? 1191479 Aug? 9 22:20 libmkl_intel_sp2dp.so -r--r--r-- 1 root root? 27642508 Aug? 9 22:07 libmkl_intel_thread.a -r-xr-xr-x 1 root root? 17516608 Aug? 9 22:49 libmkl_intel_thread.so -r--r--r-- 1 root root?? 5350948 Aug? 9 20:44 libmkl_lapack95_ilp64.a -r--r--r-- 1 root root?? 5413476 Aug? 9 20:44 libmkl_lapack95_lp64.a -r-xr-xr-x 1 root root? 29543829 Aug? 9 22:19 libmkl_mc3.so -r-xr-xr-x 1 root root? 25428037 Aug? 9 22:19 libmkl_mc.so -r-xr-xr-x 1 root root? 22888659 Aug? 9 22:18 libmkl_p4n.so -r--r--r-- 1 root root? 19232716 Aug? 9 22:07 libmkl_pgi_thread.a -r-xr-xr-x 1 root root? 12243062 Aug? 9 22:49 libmkl_pgi_thread.so -r-xr-xr-x 1 root root?? 4984870 Aug? 9 22:49 libmkl_rt.so -r--r--r-- 1 root root? 10367758 Aug? 9 22:49 libmkl_scalapack_ilp64.a -r-xr-xr-x 1 root root?? 6574928 Aug? 9 22:50 libmkl_scalapack_ilp64.so -r--r--r-- 1 root root? 10292432 Aug? 9 22:49 libmkl_scalapack_lp64.a -r-xr-xr-x 1 root root?? 6452627 Aug? 9 22:50 libmkl_scalapack_lp64.so -r--r--r-- 1 root root?? 9958444 Aug? 9 22:07 libmkl_sequential.a -r-xr-xr-x 1 root root?? 5926347 Aug? 9 22:48 libmkl_sequential.so -r--r--r-- 1 root root????? 1048 Aug? 9 16:50 libmkl_solver_ilp64.a -r--r--r-- 1 root root????? 1048 Aug? 9 16:50 libmkl_solver_ilp64_sequential.a -r--r--r-- 1 root root????? 1048 Aug? 9 16:50 libmkl_solver_lp64.a -r--r--r-- 1 root root????? 1048 Aug? 9 16:50 libmkl_solver_lp64_sequential.a -r-xr-xr-x 1 root root?? 6711968 Aug? 9 22:48 libmkl_vml_avx.so -r-xr-xr-x 1 root root?? 2795928 Aug? 9 22:47 libmkl_vml_def.so -r-xr-xr-x 1 root root?? 5476786 Aug? 9 22:48 libmkl_vml_mc2.so -r-xr-xr-x 1 root root?? 5778052 Aug? 9 22:48 libmkl_vml_mc3.so -r-xr-xr-x 1 root root?? 5382511 Aug? 9 22:48 libmkl_vml_mc.so -r-xr-xr-x 1 root root?? 4235841 Aug? 9 22:48 libmkl_vml_p4n.so drwxr-xr-x 3 root root????? 4096 Aug 18 11:43 locale Y you can reach the person managing the list at ??? numpy-discussion-owner at scipy.org Message: 1 Date: Tue, 13 Sep 2011 09:58:27 -0400 From: Olivier Delalleau Subject: Re: [Numpy-discussion] Numpy - MKL - build error To: Discussion of Numerical Python Message-ID: ??? Content-Type: text/plain; charset="iso-8859-1" Sorry if it sounds like a stupid question, but are the files listed in the error message present in that directory? If yes, maybe try running the command with sudo, just in case it would be some weird permission issue. -=- Olivier 2011/9/13 Igor Ying > Hi, > >? I am very new to Numpy and? trying to build Numpy 1.6.1 with Intel MKL > 10.3.6 and getting the following errors. > > $python setup.py config --compiler=intel build_clib --compiler=intel > build_ext --compiler=intel install > > Running from numpy source directory.F2PY Version 2 > blas_opt_info: > blas_mkl_info: >? libraries mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md not > found in /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 >? NOT AVAILABLE > > I have modified the site.cfg as below. > > [mkl] > library_dirs = /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 > include_dirs = /opt/intel/composer_xe_2011_sp1/mkl/include > mkl_libs = mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md > lapack_libs = mkl_lapack_lp64 > > > Environment variables are also set to point to mkl libraries. > > source /opt/intel/composer_xe_2011_sp1/bin/compilervars.sh intel64 lp64 > > export OMP_NUM_THREADS=4 > LD_LIBRARY_PATH="/opt/intel/composer_xe_2011_sp1/mkl/lib/intel64:${LD_LIBRARY_PATH}"; > > export LD_LIBRARY_PATH > export > LD_RUN_PATH=/opt/lib:/opt1/composer_xe_2011_sp1/lib/intel64:/opt/composer_xe_2011_sp1/mkl/lib/intel64 > > Can anyone help me? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davide.lasagna at polito.it Wed Sep 14 08:24:17 2011 From: davide.lasagna at polito.it (Davide) Date: Wed, 14 Sep 2011 14:24:17 +0200 Subject: [Numpy-discussion] behaviour of np.loadtxt Message-ID: <4E709CF1.2010902@polito.it> Dear list, I'm encountering a problem with np.loadtxt. Suppose i have a file containing three columns of data (and 10 rows), like: 0.001 0.003 0.005 0.001 0.003 0.006 0.002 0.004 0.002 0.004 0.002 0.007 0.001 0.003 0.006 0.002 0.004 0.002 0.004 0.002 0.007 0.001 0.003 0.006 0.002 0.004 0.002 0.004 0.002 0.007 If i give: len ( np.loadtxt( filename, unpack=True ) ) i get 3, since i have a three rows array. That's fine. If instead the file has a single column, i.e., 0.004 0.002 0.007 0.004 0.002 0.007 0.004 0.002 0.007 0.004 the command len ( np.loadtxt( filename, unpack=True ) ) returns 10, where i would expect it to return 1, to be consistent with the behaviour when there are multiple columns. Is there a reason for why it is not like that? Cheers Davide Lasagna From derek at astro.physik.uni-goettingen.de Wed Sep 14 11:45:55 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Wed, 14 Sep 2011 17:45:55 +0200 (CEST) Subject: [Numpy-discussion] behaviour of np.loadtxt In-Reply-To: References: <4E709CF1.2010902@polito.it> Message-ID: On Wed, 14 Sep 2011, Davide wrote: > Dear list, > > I'm encountering a problem with np.loadtxt. > > Suppose i have a file containing three columns of data (and 10 rows), like: > > 0.001 0.003 0.005 > 0.001 0.003 0.006 > 0.002 0.004 0.002 > 0.004 0.002 0.007 > 0.001 0.003 0.006 > 0.002 0.004 0.002 > 0.004 0.002 0.007 > 0.001 0.003 0.006 > 0.002 0.004 0.002 > 0.004 0.002 0.007 > > If i give: > > len ( np.loadtxt( filename, unpack=True ) ) > > i get 3, since i have a three rows array. That's fine. > > If instead the file has a single column, i.e., > > 0.004 > 0.002 > 0.007 > 0.004 > 0.002 > 0.007 > 0.004 > 0.002 > 0.007 > 0.004 > > the command > > len ( np.loadtxt( filename, unpack=True ) ) > > returns 10, where i would expect it to return 1, to be consistent with > the behaviour when there are multiple columns. > > Is there a reason for why it is not like that? Yes, the default for loadtxt is to automatically "squeeze" the output to the minimum no. of dimensions required, thus it returns a 1D-array in the latter case. If you are using numpy 1.6 or later, you can ensure to get a consistent shape by passing the "ndmin=2" option. Cheers, Derek -- ---------------------------------------------------------------- Derek Homeier Centre de Recherche Astrophysique de Lyon ENS Lyon 46, All?e d'Italie 69364 Lyon Cedex 07, France +33 1133 47272-8894 ---------------------------------------------------------------- From Chris.Barker at noaa.gov Wed Sep 14 16:01:39 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 14 Sep 2011 13:01:39 -0700 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> <4E6FA3EA.9010306@noaa.gov> Message-ID: <4E710823.30505@noaa.gov> On 9/13/11 1:01 PM, Christopher Jordan-Squire wrote: > Sorry, I cheated by reading the docs. :-) me too... > """ > numpy.resize(a, new_shape) > > Return a new array with the specified shape. > > If the new array is larger than the original array, then the new array > is filled with repeated copies of a. Note that this behavior is > different from a.resize(new_shape) which fills with zeros instead of > repeated copies of a. > """ see the: "this behavior is different from a.resize(new_shape)" so: a.resize(new_shape, refcheck=True) Change shape and size of array in-place. Parameters ---------- new_shape : tuple of ints, or `n` ints Shape of resized array. refcheck : bool, optional If False, reference count will not be checked. Default is True. Returns ------- None > The copying I meant was that numpy.resize will fill the resized array > with repeated copies of the data. So np.empty followed by np.append > avoids that. numpy.ndarray.resize is a different method, and I'm pretty sure it should be as fast or faster that np.empty + np.append. It is often confusing that there is a numpy function and ndarray method with the same name and slightly different usage. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From nwagner at iam.uni-stuttgart.de Wed Sep 14 16:09:31 2011 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Wed, 14 Sep 2011 22:09:31 +0200 Subject: [Numpy-discussion] numpy.test() failure Message-ID: ERROR: test_polyfit (test_polynomial.TestDocs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_polynomial.py", line 106, in test_polyfit weights = arange(8,1,-1)**2/7.0 NameError: global name 'arange' is not defined ====================================================================== FAIL: Tests polyfit ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/tests/test_extras.py", line 622, in test_polyfit assert_almost_equal(a, a_) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", line 155, in assert_almost_equal err_msg=err_msg, verbose=verbose) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", line 221, in assert_array_almost_equal header='Arrays are not almost equal') File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", line 186, in assert_array_compare verbose=verbose, header=header) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", line 677, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 4.25134878, 1.14131297, 0.20519666, 0.01701 ]) y: array([ 1.9345248 , 0.49711011, 0.10202554, 0.00928034]) From scipy at samueljohn.de Wed Sep 14 16:45:25 2011 From: scipy at samueljohn.de (Samuel John) Date: Wed, 14 Sep 2011 22:45:25 +0200 Subject: [Numpy-discussion] numpy.test() failure In-Reply-To: References: Message-ID: <8EF8252C-F1D4-4F24-978D-4BBD9A8CF1FF@samueljohn.de> Hi Nils, which version of numpy, which os? I can infer that you use python 2.6 in 64bit, right? Right after the beginning of the numpy.test() are some crucial information. bests Samuel On 14.09.2011, at 22:09, Nils Wagner wrote: > ERROR: test_polyfit (test_polynomial.TestDocs) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_polynomial.py", > line 106, in test_polyfit > weights = arange(8,1,-1)**2/7.0 > NameError: global name 'arange' is not defined > > ====================================================================== > FAIL: Tests polyfit > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/tests/test_extras.py", > line 622, in test_polyfit > assert_almost_equal(a, a_) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", > line 155, in assert_almost_equal > err_msg=err_msg, verbose=verbose) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", > line 221, in assert_array_almost_equal > header='Arrays are not almost equal') > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", > line 186, in assert_array_compare > verbose=verbose, header=header) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", > line 677, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal > > (mismatch 100.0%) > x: array([ 4.25134878, 1.14131297, 0.20519666, > 0.01701 ]) > y: array([ 1.9345248 , 0.49711011, 0.10202554, > 0.00928034]) > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ralf.gommers at googlemail.com Wed Sep 14 16:50:22 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 14 Sep 2011 22:50:22 +0200 Subject: [Numpy-discussion] numpy.test() failure In-Reply-To: <8EF8252C-F1D4-4F24-978D-4BBD9A8CF1FF@samueljohn.de> References: <8EF8252C-F1D4-4F24-978D-4BBD9A8CF1FF@samueljohn.de> Message-ID: On Wed, Sep 14, 2011 at 10:45 PM, Samuel John wrote: > Hi Nils, > > which version of numpy, which os? > Latest master. Due to https://github.com/numpy/numpy/commit/af22fc43 Travis, did you run the test suite? arange is used but not imported. Ralf > I can infer that you use python 2.6 in 64bit, right? > > Right after the beginning of the numpy.test() are some crucial information. > > bests > Samuel > > On 14.09.2011, at 22:09, Nils Wagner wrote: > > > ERROR: test_polyfit (test_polynomial.TestDocs) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_polynomial.py", > > line 106, in test_polyfit > > weights = arange(8,1,-1)**2/7.0 > > NameError: global name 'arange' is not defined > > > > ====================================================================== > > FAIL: Tests polyfit > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/tests/test_extras.py", > > line 622, in test_polyfit > > assert_almost_equal(a, a_) > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", > > line 155, in assert_almost_equal > > err_msg=err_msg, verbose=verbose) > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", > > line 221, in assert_array_almost_equal > > header='Arrays are not almost equal') > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", > > line 186, in assert_array_compare > > verbose=verbose, header=header) > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", > > line 677, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not almost equal > > > > (mismatch 100.0%) > > x: array([ 4.25134878, 1.14131297, 0.20519666, > > 0.01701 ]) > > y: array([ 1.9345248 , 0.49711011, 0.10202554, > > 0.00928034]) > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 14 16:51:05 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 14 Sep 2011 14:51:05 -0600 Subject: [Numpy-discussion] numpy.test() failure In-Reply-To: <8EF8252C-F1D4-4F24-978D-4BBD9A8CF1FF@samueljohn.de> References: <8EF8252C-F1D4-4F24-978D-4BBD9A8CF1FF@samueljohn.de> Message-ID: On Wed, Sep 14, 2011 at 2:45 PM, Samuel John wrote: > Hi Nils, > > which version of numpy, which os? > I can infer that you use python 2.6 in 64bit, right? > > Right after the beginning of the numpy.test() are some crucial information. > > bests > Samuel > > On 14.09.2011, at 22:09, Nils Wagner wrote: > > > ERROR: test_polyfit (test_polynomial.TestDocs) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_polynomial.py", > > line 106, in test_polyfit > > weights = arange(8,1,-1)**2/7.0 > > NameError: global name 'arange' is not defined > > > > ====================================================================== > > FAIL: Tests polyfit > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/tests/test_extras.py", > > line 622, in test_polyfit > > assert_almost_equal(a, a_) > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", > > line 155, in assert_almost_equal > > err_msg=err_msg, verbose=verbose) > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", > > line 221, in assert_array_almost_equal > > header='Arrays are not almost equal') > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", > > line 186, in assert_array_compare > > verbose=verbose, header=header) > > File > > > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", > > line 677, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not almost equal > > > > (mismatch 100.0%) > > x: array([ 4.25134878, 1.14131297, 0.20519666, > > 0.01701 ]) > > y: array([ 1.9345248 , 0.49711011, 0.10202554, > > 0.00928034]) > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > This is from Travis' push last night adding weights to polyfit. Travis, pretty much everything needs undergo review and testing before commits no matter how trivial it may look. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Sep 14 17:25:13 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 14 Sep 2011 14:25:13 -0700 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E710823.30505@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> <4E6FA3EA.9010306@noaa.gov> <4E710823.30505@noaa.gov> Message-ID: <4E711BB9.8020101@noaa.gov> On 9/14/11 1:01 PM, Christopher Barker wrote: > numpy.ndarray.resize is a different method, and I'm pretty sure it > should be as fast or faster that np.empty + np.append. My profile: In [25]: %timeit f1 # numpy.resize() 10000000 loops, best of 3: 163 ns per loop In [26]: %timeit f2 #numpy.ndarray.resize() 10000000 loops, best of 3: 136 ns per loop In [27]: %timeit f3 # numpy.empty() + append() 10000000 loops, best of 3: 136 ns per loop those last two couldn't b more identical! (though this is an excercise in unrequired optimization!) My test code: #!/usr/bin/env python """ test_resize A test of various numpy re-sizing options """ import numpy def f1(): """ numpy.resize """ l = 100 a = numpy.zeros((l,)) for i in xrange(1000): l += l a = numpy.resize(a, (l,) ) return None def f2(): """ numpy.ndarray.resize """ l = 100 a = numpy.zeros((l,)) for i in xrange(1000): l += l a.resize(a, (l,) ) return None def f3(): """ numpy.empty + append """ l = 100 a = numpy.zeros((l,)) for i in xrange(1000): b = np.empty((l,)) a.append(b) return None -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ben.root at ou.edu Wed Sep 14 17:41:42 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 14 Sep 2011 16:41:42 -0500 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E711BB9.8020101@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> <4E6FA3EA.9010306@noaa.gov> <4E710823.30505@noaa.gov> <4E711BB9.8020101@noaa.gov> Message-ID: On Wed, Sep 14, 2011 at 4:25 PM, Christopher Barker wrote: > On 9/14/11 1:01 PM, Christopher Barker wrote: > > numpy.ndarray.resize is a different method, and I'm pretty sure it > > should be as fast or faster that np.empty + np.append. > > My profile: > > In [25]: %timeit f1 # numpy.resize() > 10000000 loops, best of 3: 163 ns per loop > > In [26]: %timeit f2 #numpy.ndarray.resize() > 10000000 loops, best of 3: 136 ns per loop > > In [27]: %timeit f3 # numpy.empty() + append() > 10000000 loops, best of 3: 136 ns per loop > > > those last two couldn't b more identical! > > (though this is an excercise in unrequired optimization!) > > Are you sure the f2 code works? a.resize() takes only a shape tuple. As coded, you should get an exception. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Sep 14 18:30:11 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 14 Sep 2011 15:30:11 -0700 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> <4E6FA3EA.9010306@noaa.gov> <4E710823.30505@noaa.gov> <4E711BB9.8020101@noaa.gov> Message-ID: <4E712AF3.4020600@noaa.gov> On 9/14/11 2:41 PM, Benjamin Root wrote: > Are you sure the f2 code works? a.resize() takes only a shape tuple. As > coded, you should get an exception. wow, what an idiot! I think I just timed how long it takes to raise that exception... And when I fix that, I get a memory error. When I fix that, I find that f3() wasn't doing the right thing. What an astonishing lack of attention on my part! Here it is again, working, I hope! In [107]: %timeit f1() 10 loops, best of 3: 50.7 ms per loop In [108]: %timeit f2() 1000 loops, best of 3: 719 us per loop In [109]: %timeit f3() 100 loops, best of 3: 19 ms per loop So: numpy.resize() is the slowest numpy.empty+ numpy.append() is faster numpy.ndarray.resize() is the fastest Which matches my expectations, for once! -Chris The code: #!/usr/bin/env python """ test_resize A test of various numpy re-sizing options """ import numpy def f1(): """ numpy.resize """ extra = 100 l = extra a = numpy.zeros((l,)) for i in xrange(100): l += extra a = numpy.resize(a, (l,) ) return a def f2(): """ numpy.ndarray.resize """ extra = 100 l = extra a = numpy.zeros((l,)) for i in xrange(100): l += extra a.resize( (l,) ) return a def f3(): """ numpy.empty + append """ extra = 100 l = extra a = numpy.zeros((l,)) for i in xrange(100): b = numpy.empty((extra,)) a = numpy.append(a, b) return a a1 = f1() a2 = f2() a3 = f3() if a1.shape == a2.shape == a3.shape: print "they are all returning the same size array" else: print "Something is wrong!" -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cjordan1 at uw.edu Wed Sep 14 19:26:43 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Wed, 14 Sep 2011 18:26:43 -0500 Subject: [Numpy-discussion] load from text files Pull Request Review In-Reply-To: <4E712AF3.4020600@noaa.gov> References: <4E5D0DEC.2070507@noaa.gov> <41886BFD-EEF5-4E2B-B3F9-709A6FFDD445@astro.physik.uni-goettingen.de> <4E60FB5E.6010606@noaa.gov> <4E6142A0.4040507@noaa.gov> <4E67CB8C.3000902@noaa.gov> <4E692C57.2040501@noaa.gov> <4E6FA3EA.9010306@noaa.gov> <4E710823.30505@noaa.gov> <4E711BB9.8020101@noaa.gov> <4E712AF3.4020600@noaa.gov> Message-ID: On Wed, Sep 14, 2011 at 5:30 PM, Christopher Barker wrote: > On 9/14/11 2:41 PM, Benjamin Root wrote: >> Are you sure the f2 code works? ?a.resize() takes only a shape tuple. ?As >> coded, you should get an exception. > > wow, what an idiot! > > I think I just timed how long it takes to raise that exception... > > And when I fix that, I get a memory error. > > When I fix that, I find that f3() wasn't doing the right thing. What an > astonishing lack of attention on my part! > > Here it is again, working, I hope! > > In [107]: %timeit f1() > 10 loops, best of 3: 50.7 ms per loop > > In [108]: %timeit f2() > 1000 loops, best of 3: 719 us per loop > > In [109]: %timeit f3() > 100 loops, best of 3: 19 ms per loop > > > So: > numpy.resize() is the slowest > numpy.empty+ numpy.append() is faster > numpy.ndarray.resize() is the fastest > > Which matches my expectations, for once! > Good catch! I didn't think the difference between np.resize and ndarray.resize would matter. (And I was getting inscrutable errors when I called ndarray.resize that told me to use np.resize instead.) -Chris JS > -Chris > The code: > > #!/usr/bin/env python > > """ > test_resize > > A test of various numpy re-sizing options > > """ > > import numpy > > def f1(): > ? ? """ > ? ? numpy.resize > ? ? """ > > ? ? extra = 100 > ? ? l = extra > ? ? a = numpy.zeros((l,)) > ? ? for i in xrange(100): > ? ? ? ? l += extra > ? ? ? ? a = numpy.resize(a, (l,) ) > > ? ? return a > > def f2(): > ? ? """ > ? ? numpy.ndarray.resize > ? ? """ > > ? ? extra = 100 > ? ? l = extra > ? ? a = numpy.zeros((l,)) > ? ? for i in xrange(100): > ? ? ? ? l += extra > ? ? ? ? a.resize( (l,) ) > > ? ? return a > > def f3(): > ? ? """ > ? ? numpy.empty + append > ? ? """ > > ? ? extra = 100 > ? ? l = extra > ? ? a = numpy.zeros((l,)) > ? ? for i in xrange(100): > ? ? ? ? b = numpy.empty((extra,)) > ? ? ? ? a = numpy.append(a, b) > ? ? return a > > a1 = f1() > a2 = f2() > a3 = f3() > > if a1.shape == a2.shape == a3.shape: > ? ? print "they are all returning the same size array" > else: > ? ? print "Something is wrong!" > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From stefan at sun.ac.za Wed Sep 14 20:13:37 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 14 Sep 2011 17:13:37 -0700 Subject: [Numpy-discussion] Polynomial implementation in numpy.ma Message-ID: Hi all, There were some failures in the polynomial tests earlier today, and while investigating I saw that numpy.ma implements its own root finder. It uses inversion of a Van der Monde matrix, which I believe may suffer from some numerical instability problems. Given that Charles has gone to some length to implement good polynomial root finders, I think it would be best to employ those instead, and simply pre-filter the data that comes from the masked array module, if possible. Regards St?fan From teoliphant at gmail.com Wed Sep 14 23:32:36 2011 From: teoliphant at gmail.com (Travis Oliphant) Date: Wed, 14 Sep 2011 22:32:36 -0500 Subject: [Numpy-discussion] 1.7.x release of NumPy Message-ID: <6DD4D224-C350-43C3-BB55-550260042185@gmail.com> Hi all, Has there been a discussion of a 1.7.x release of NumPy? There are a few new features that should go into the 1.x release of NumPy, that don't require the ABI changes of 2.0. I thought I had heard Mark talk in support of such a thing. What are the plans for the release of NumPy 2.0? Between my experiences over the last 4 years of working on real-world projects, the data array summit, SciPy, and conversations with Mark Wiebe, Peter Wang, Wes McKinney and others over the past months, there is a collection of pretty concrete ideas emerging about where the next version of NumPy should go. Some of these can be folded into NumPy 2.0 (and even 1.7), but my mind is racing around NumPy 3.0 at this point. -Travis From ralf.gommers at googlemail.com Thu Sep 15 01:56:03 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 15 Sep 2011 07:56:03 +0200 Subject: [Numpy-discussion] 1.7.x release of NumPy In-Reply-To: <6DD4D224-C350-43C3-BB55-550260042185@gmail.com> References: <6DD4D224-C350-43C3-BB55-550260042185@gmail.com> Message-ID: On Thu, Sep 15, 2011 at 5:32 AM, Travis Oliphant wrote: > Hi all, > > Has there been a discussion of a 1.7.x release of NumPy? There are a > few new features that should go into the 1.x release of NumPy, that don't > require the ABI changes of 2.0. I thought I had heard Mark talk in > support of such a thing. > A little. I think our 6-month release cycle is working pretty well, which means a 1.7.0 release around mid-November. There are definitely enough new features for a solid release already, but I have the impression that it wouldn't hurt for the missing data functionality to get exercised a bit more. > > What are the plans for the release of NumPy 2.0? > > Mark and others have been very good at maintaining ABI compatibility. At the same time the numpy-refactor branch has dropped off the radar (any news/plans?). So I'm wondering what would go into 2.0 that is ready now or soon? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From khamenya at gmail.com Thu Sep 15 06:23:03 2011 From: khamenya at gmail.com (Valery Khamenya) Date: Thu, 15 Sep 2011 12:23:03 +0200 Subject: [Numpy-discussion] PyPy guys deserve some help on micronumpy from you, numpy gurus! Message-ID: Hi (replying please Cc to me) There is a module micronumpy that appeared at PyPy source tree: https://bitbucket.org/pypy/pypy/src/dfae5033127e/pypy/module/micronumpy/ The great contributions of Justin Peel and Ilya Osadchiy to micronumpy module revive step-by-step the functionality of numpy. It would be great if some of numpy-gurus could jump in to assist, contribute some code and also, perhaps, guide a bit where the things go deeply in numpy nature. For those who don't know yet much about PyPy: PyPy is a fast implementation of Python 2.7. As a rule of thumb, PyPy is currently about 4x times faster than CPython. Certain benchmarks taken from the real life show 20x speed-up and more. The successes of PyPy performance are very remarkable http://speed.pypy.org/ best regards -- Valery A.Khamenya -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy-discussion at maubp.freeserve.co.uk Thu Sep 15 06:34:36 2011 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Thu, 15 Sep 2011 11:34:36 +0100 Subject: [Numpy-discussion] PyPy guys deserve some help on micronumpy from you, numpy gurus! In-Reply-To: References: Message-ID: On Thu, Sep 15, 2011 at 11:23 AM, Valery Khamenya wrote: > Hi > (replying please Cc to me) > There is a module micronumpy that appeared at PyPy source tree: > https://bitbucket.org/pypy/pypy/src/dfae5033127e/pypy/module/micronumpy/ > The great contributions of Justin Peel and Ilya?Osadchiy to micronumpy > module revive?step-by-step?the?functionality of numpy. > It would be great if some of numpy-gurus could jump in to assist, contribute > some code and also, perhaps, guide a bit where the things go deeply in numpy > nature. > For those who don't know yet much about PyPy: > PyPy is a fast?implementation of?Python 2.7. > As a rule of thumb, PyPy is currently about 4x times faster than CPython. > Certain benchmarks taken from the real life show 20x speed-up and more. > The successes of PyPy performance are very remarkable?http://speed.pypy.org/ > best regards This may not be the best place to ask, but how should a python script (e.g. setup.py) distinguish between real NumPy and micronumpy? Or should I instead be looking to distinguish PyPy versus another Python implementation? Peter From jsilter at gmail.com Thu Sep 15 09:32:06 2011 From: jsilter at gmail.com (Jacob Silterra) Date: Thu, 15 Sep 2011 09:32:06 -0400 Subject: [Numpy-discussion] Functions for finding the relative extrema of numeric data Message-ID: >What is your application? The most common case is looking at Fourier transforms and identifying spectral peaks. I've also analyzed images looking at 1D slices (usually very regular data) and looked for peaks there. That stackoverflow page had a nice link to a comparison of different algorithms here: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631518/. That paper is focused on mass-spectrometry data, but the approach would generalize to any 1D data set. Unless somebody feels otherwise, I'll close this pull request and start working on an implementation of peak finding via continuous wavelet transform (the best and most computationally intensive approach of those described above). -Jacob ------------------------------ > > Message: 4 > Date: Tue, 13 Sep 2011 22:34:01 +0200 > From: Ralf Gommers > Subject: Re: [Numpy-discussion] Functions for finding the relative > extrema of numeric data > To: Discussion of Numerical Python > Message-ID: > > > Content-Type: text/plain; charset="iso-8859-1" > > Hi Jacob, > > On Fri, Sep 9, 2011 at 11:57 PM, Jacob Silterra wrote: > > > Hello all, > > > > I'd like to see functions for calculating the relative extrema in a set > of > > data included in numpy. I use that functionality frequently, and always > seem > > to be writing my own version. It seems like this functionality would be > > useful to the community at large, as it's a fairly common operation. > > > > What is your application? > > > > > For numeric data (which is presumably noisy), the definition of a > relative > > extrema isn't completely obvious. The implementation I am proposing finds > a > > point in an ndarray along an axis which is larger (or smaller) than it's > > `order` nearest neighbors (`order` being an optional parameter, default > 1). > > This is likely to find more points than may be desired, which I believe > is > > preferable to the alternative. The output is formatted the same as > > numpy.where. > > > > Code available here: https://github.com/numpy/numpy/pull/154 > > > > I'm not sure whether this belongs in numpy or scipy, that question is > > somewhat debatable. More sophisticated peak-finding functions (in N > > dimensions, as opposed to 1) may also be useful to the community, and > those > > would definitely belong in scipy. > > > > I have the feeling this belongs in scipy. Although if it's just these two > functions I'm not sure where exactly to put them. Looking at the > functionality, this is quite a simple approach. For any data of the type > I'm > usually working with it will not be able to find the right local extrema. > The same is true for your alternative definition below. > > A more powerful peak detection function would be a very good addition to > scipy imho (perhaps in scipy.interpolate?). See also > > http://stackoverflow.com/questions/1713335/peak-finding-algorithm-for-python-scipy > > Cheers, > Ralf > > > > An alternative implementation would be to require that function be > > continuously descending (or ascending) for `order` points, which would > > enforce a minimum width on a peak. > > > > -Jacob Silterra > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110913/8bb2e1a5/attachment-0001.html > > ------------------------------ > > Message: 5 > Date: Tue, 13 Sep 2011 15:44:03 -0500 > From: Benjamin Root > Subject: Re: [Numpy-discussion] Functions for finding the relative > extrema of numeric data > To: Discussion of Numerical Python > Message-ID: > > > Content-Type: text/plain; charset="iso-8859-1" > > On Tue, Sep 13, 2011 at 3:34 PM, Ralf Gommers > wrote: > > > Hi Jacob, > > > > On Fri, Sep 9, 2011 at 11:57 PM, Jacob Silterra > wrote: > > > >> Hello all, > >> > >> I'd like to see functions for calculating the relative extrema in a set > of > >> data included in numpy. I use that functionality frequently, and always > seem > >> to be writing my own version. It seems like this functionality would be > >> useful to the community at large, as it's a fairly common operation. > >> > > > > What is your application? > > > >> > >> For numeric data (which is presumably noisy), the definition of a > relative > >> extrema isn't completely obvious. The implementation I am proposing > finds a > >> point in an ndarray along an axis which is larger (or smaller) than it's > >> `order` nearest neighbors (`order` being an optional parameter, default > 1). > >> This is likely to find more points than may be desired, which I believe > is > >> preferable to the alternative. The output is formatted the same as > >> numpy.where. > >> > >> Code available here: https://github.com/numpy/numpy/pull/154 > >> > >> I'm not sure whether this belongs in numpy or scipy, that question is > >> somewhat debatable. More sophisticated peak-finding functions (in N > >> dimensions, as opposed to 1) may also be useful to the community, and > those > >> would definitely belong in scipy. > >> > > > > I have the feeling this belongs in scipy. Although if it's just these two > > functions I'm not sure where exactly to put them. Looking at the > > functionality, this is quite a simple approach. For any data of the type > I'm > > usually working with it will not be able to find the right local extrema. > > The same is true for your alternative definition below. > > > > A more powerful peak detection function would be a very good addition to > > scipy imho (perhaps in scipy.interpolate?). See also > > > http://stackoverflow.com/questions/1713335/peak-finding-algorithm-for-python-scipy > > > > Cheers, > > Ralf > > > > > Actually, such an algorithm would be great to partner with the watershed > clustering implementation in ndimage. > > Ben Root > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjhelmus at gmail.com Thu Sep 15 11:03:23 2011 From: jjhelmus at gmail.com (Jonathan Helmus) Date: Thu, 15 Sep 2011 11:03:23 -0400 Subject: [Numpy-discussion] Functions for finding the relative extrema of numeric data In-Reply-To: References: Message-ID: <4E7213BB.80104@gmail.com> I've written some peak picking functions that work in N dimensions for a module for looking at NMR data in python, http://code.google.com/p/nmrglue/. I'd be glad to polish up the code if people think it would be a useful addition to scipy.ndimage or scipy.interpolate? The methods are not based on any formal algorithms I know of, just some fast and relatively simple methods that I found seem to work decently. The methods are contained in the peakpick.py and segmentation.py files in the analysis directory (specifically see the find_all_connected, find_all_ downward and find_all_thres): http://code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/peakpick.py http://code.google.com/p/nmrglue/source/browse/trunk/nmrglue/analysis/segmentation.py Let me know if there is an interest in including these in scipy or numpy. -Jonathan Helmus Jacob Silterra wrote: > >What is your application? > > The most common case is looking at Fourier transforms and identifying > spectral peaks. I've also analyzed images looking at 1D slices > (usually very regular data) and looked for peaks there. > > That stackoverflow page had a nice link to a comparison of different > algorithms here: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631518/. > That paper is focused on mass-spectrometry data, but the approach > would generalize to any 1D data set. Unless somebody feels otherwise, > I'll close this pull request and start working on an implementation of > peak finding via continuous wavelet transform (the best and most > computationally intensive approach of those described above). > > -Jacob > > ------------------------------ > > Message: 4 > Date: Tue, 13 Sep 2011 22:34:01 +0200 > From: Ralf Gommers > > Subject: Re: [Numpy-discussion] Functions for finding the relative > extrema of numeric data > To: Discussion of Numerical Python > > Message-ID: > > > > Content-Type: text/plain; charset="iso-8859-1" > > Hi Jacob, > > On Fri, Sep 9, 2011 at 11:57 PM, Jacob Silterra > wrote: > > > Hello all, > > > > I'd like to see functions for calculating the relative extrema > in a set of > > data included in numpy. I use that functionality frequently, and > always seem > > to be writing my own version. It seems like this functionality > would be > > useful to the community at large, as it's a fairly common operation. > > > > What is your application? > > > > > For numeric data (which is presumably noisy), the definition of > a relative > > extrema isn't completely obvious. The implementation I am > proposing finds a > > point in an ndarray along an axis which is larger (or smaller) > than it's > > `order` nearest neighbors (`order` being an optional parameter, > default 1). > > This is likely to find more points than may be desired, which I > believe is > > preferable to the alternative. The output is formatted the same as > > numpy.where. > > > > Code available here: https://github.com/numpy/numpy/pull/154 > > > > I'm not sure whether this belongs in numpy or scipy, that > question is > > somewhat debatable. More sophisticated peak-finding functions (in N > > dimensions, as opposed to 1) may also be useful to the > community, and those > > would definitely belong in scipy. > > > > I have the feeling this belongs in scipy. Although if it's just > these two > functions I'm not sure where exactly to put them. Looking at the > functionality, this is quite a simple approach. For any data of > the type I'm > usually working with it will not be able to find the right local > extrema. > The same is true for your alternative definition below. > > A more powerful peak detection function would be a very good > addition to > scipy imho (perhaps in scipy.interpolate?). See also > http://stackoverflow.com/questions/1713335/peak-finding-algorithm-for-python-scipy > > Cheers, > Ralf > > > > An alternative implementation would be to require that function be > > continuously descending (or ascending) for `order` points, which > would > > enforce a minimum width on a peak. > > > > -Jacob Silterra > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110913/8bb2e1a5/attachment-0001.html > > ------------------------------ > > Message: 5 > Date: Tue, 13 Sep 2011 15:44:03 -0500 > From: Benjamin Root > > Subject: Re: [Numpy-discussion] Functions for finding the relative > extrema of numeric data > To: Discussion of Numerical Python > > Message-ID: > > > > Content-Type: text/plain; charset="iso-8859-1" > > On Tue, Sep 13, 2011 at 3:34 PM, Ralf Gommers > >wrote: > > > Hi Jacob, > > > > On Fri, Sep 9, 2011 at 11:57 PM, Jacob Silterra > > wrote: > > > >> Hello all, > >> > >> I'd like to see functions for calculating the relative extrema > in a set of > >> data included in numpy. I use that functionality frequently, > and always seem > >> to be writing my own version. It seems like this functionality > would be > >> useful to the community at large, as it's a fairly common > operation. > >> > > > > What is your application? > > > >> > >> For numeric data (which is presumably noisy), the definition of > a relative > >> extrema isn't completely obvious. The implementation I am > proposing finds a > >> point in an ndarray along an axis which is larger (or smaller) > than it's > >> `order` nearest neighbors (`order` being an optional parameter, > default 1). > >> This is likely to find more points than may be desired, which > I believe is > >> preferable to the alternative. The output is formatted the same as > >> numpy.where. > >> > >> Code available here: https://github.com/numpy/numpy/pull/154 > >> > >> I'm not sure whether this belongs in numpy or scipy, that > question is > >> somewhat debatable. More sophisticated peak-finding functions (in N > >> dimensions, as opposed to 1) may also be useful to the > community, and those > >> would definitely belong in scipy. > >> > > > > I have the feeling this belongs in scipy. Although if it's just > these two > > functions I'm not sure where exactly to put them. Looking at the > > functionality, this is quite a simple approach. For any data of > the type I'm > > usually working with it will not be able to find the right local > extrema. > > The same is true for your alternative definition below. > > > > A more powerful peak detection function would be a very good > addition to > > scipy imho (perhaps in scipy.interpolate?). See also > > > http://stackoverflow.com/questions/1713335/peak-finding-algorithm-for-python-scipy > > > > Cheers, > > Ralf > > > > > Actually, such an algorithm would be great to partner with the > watershed > clustering implementation in ndimage. > > Ben Root > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From davide.lasagna at polito.it Thu Sep 15 11:24:38 2011 From: davide.lasagna at polito.it (Davide) Date: Thu, 15 Sep 2011 17:24:38 +0200 Subject: [Numpy-discussion] Functions for finding the relative extrema of numeric data In-Reply-To: References: Message-ID: <4E7218B6.9070405@polito.it> On 09/15/2011 03:32 PM, Jacob Silterra wrote: > I'll close this pull request and start working on an implementation of > peak finding via continuous wavelet transform (the best and most > computationally intensive approach of those described above). Just for information, which tools are you going to use for the CWT? I may be interested in providing some help. Cheers, Davide Lasagna From scipy at samueljohn.de Thu Sep 15 11:41:51 2011 From: scipy at samueljohn.de (Samuel John) Date: Thu, 15 Sep 2011 17:41:51 +0200 Subject: [Numpy-discussion] Functions for finding the relative extrema of numeric data In-Reply-To: <4E7218B6.9070405@polito.it> References: <4E7218B6.9070405@polito.it> Message-ID: Hi all, I am not sure if this is of help for anyone. I wrote some code to find the relative maxima in a 1D array for my own purpose. Maybe someone is interested or even finds a bug *g*. I post the code here and appreciate any feedback. Even "stop spamming your buggy code" :-) > from numpy import diff, sign, convolve, array, where, around, int32, alen > > def localmaxima_at(x): > '''Returns the indices of local maxima in the 1D array x. > > If several elements in x have the same value, then the > index of the element in the middle is returned. > > If there are two adjacent elements with the same value, > one of them is returned. > > x[0] and x[-1] are never returned as an index for the > local maximum. > > @Author: Samuel John > @copyright: http://creativecommons.org/licenses/by-nc-sa/3.0/ > @todo: unittests > ''' > assert len(x) > 2, "Length of x should be greater than two in order to define a meaningful local maximum." > assert x.ndim == 1, "Expected 1D array." > #print 'x=\n',x > filled=sign(diff(x)).astype(int32) > # fill zeros: > has_zeros = (filled == 0).any() > last = 0 > if has_zeros: > for i in xrange(alen(filled)): > if filled[i] == 0: > filled[i] = last > else: > last = filled[i] > #print 'filled\n',filled > left = where( convolve( > filled, > array([-1,1]), mode='full' ) -2 == 0 )[0] > > if has_zeros: > filled=sign(diff(x)).astype(int32) > last = 0 > for i in reversed(xrange(len(filled))): > if filled[i] == 0: > filled[i] = last > else: > last = filled[i] > > right = where( convolve( > filled, > array([-1,1]), mode='full' ) -2 == 0 )[0] > #print 'left\n',left > #print 'right\n',right > return around( (left + right) / 2.0).astype(int32) > bests, Samuel From Chris.Barker at noaa.gov Thu Sep 15 11:56:09 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 15 Sep 2011 08:56:09 -0700 Subject: [Numpy-discussion] 1.7.x release of NumPy In-Reply-To: <6DD4D224-C350-43C3-BB55-550260042185@gmail.com> References: <6DD4D224-C350-43C3-BB55-550260042185@gmail.com> Message-ID: <4E722019.3060506@noaa.gov> On 9/14/11 8:32 PM, Travis Oliphant wrote: > but my mind is racing around NumPy 3.0 at this point. Travis, could you clarify this point? Do you mean: "There are so many ideas my mind is racing around too much to even think about 3.0?" or "My mind is racing with ideas so much that I want to dive in and get started working on 3.0?" just wondering, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From tmp50 at ukr.net Thu Sep 15 12:21:03 2011 From: tmp50 at ukr.net (Dmitrey) Date: Thu, 15 Sep 2011 19:21:03 +0300 Subject: [Numpy-discussion] [ANN} OpenOpt, FuncDesigner, DerApproximator, SpaceFuncs release 0.36 Message-ID: <44514.1316103663.927787890751766528@ffe8.ukr.net> Hi all, new release of our free soft (OpenOpt, FuncDesigner, DerApproximator, SpaceFuncs) v. 0.36 is out: OpenOpt: > * Now solver interalg can handle all types of constraints and integration problems > * Some minor improvements and code cleanup > FuncDesigner: > * Interval analysis now can involve min, max and 1-d monotone splines R -> R of 1st and 3rd order > * Some bugfixes and improvements > SpaceFuncs: > * Some minor changes > DerApproximator: > * Some improvements for obtaining derivatives in points from R^n where left or right derivative for a variable is absent, especially for stencil > 1 > See http://openopt.org for more details. Regards, D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Thu Sep 15 14:45:20 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 15 Sep 2011 13:45:20 -0500 Subject: [Numpy-discussion] 1.7.x release of NumPy In-Reply-To: References: <6DD4D224-C350-43C3-BB55-550260042185@gmail.com> Message-ID: <4E7247C0.1030502@gmail.com> On 09/15/2011 12:56 AM, Ralf Gommers wrote: > > > On Thu, Sep 15, 2011 at 5:32 AM, Travis Oliphant > wrote: > > Hi all, > > Has there been a discussion of a 1.7.x release of NumPy? There > are a few new features that should go into the 1.x release of > NumPy, that don't require the ABI changes of 2.0. I thought I > had heard Mark talk in support of such a thing. > > > A little. I think our 6-month release cycle is working pretty well, > which means a 1.7.0 release around mid-November. There are definitely > enough new features for a solid release already, but I have the > impression that it wouldn't hurt for the missing data functionality to > get exercised a bit more. > > > What are the plans for the release of NumPy 2.0? > > Mark and others have been very good at maintaining ABI compatibility. > At the same time the numpy-refactor branch has dropped off the radar > (any news/plans?). So I'm wondering what would go into 2.0 that is > ready now or soon? > > Ralf > What is the possibility of adding datarray and carray packages to numpy or scipy in the 'near future'? Or do these need to wait and see what happens to the new masked and NA implementations? (Actually probably a lot of what happens may depend on this.) Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Thu Sep 15 15:49:46 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 15 Sep 2011 14:49:46 -0500 Subject: [Numpy-discussion] can't use argmin for array of timedeltas Message-ID: I encountered something similar back in April or so and that it was fixed. The problem was that the minimum function was implemented as max(0 - a), and so this fails for timedelta objects that can't do this. This was fixed for min(), but apparently not for argmin(). Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 15 16:31:14 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 15 Sep 2011 14:31:14 -0600 Subject: [Numpy-discussion] 1.7.x release of NumPy In-Reply-To: <4E7247C0.1030502@gmail.com> References: <6DD4D224-C350-43C3-BB55-550260042185@gmail.com> <4E7247C0.1030502@gmail.com> Message-ID: On Thu, Sep 15, 2011 at 12:45 PM, Bruce Southey wrote: > ** > On 09/15/2011 12:56 AM, Ralf Gommers wrote: > > > > On Thu, Sep 15, 2011 at 5:32 AM, Travis Oliphant wrote: > >> Hi all, >> >> Has there been a discussion of a 1.7.x release of NumPy? There are a >> few new features that should go into the 1.x release of NumPy, that don't >> require the ABI changes of 2.0. I thought I had heard Mark talk in >> support of such a thing. >> > > A little. I think our 6-month release cycle is working pretty well, which > means a 1.7.0 release around mid-November. There are definitely enough new > features for a solid release already, but I have the impression that it > wouldn't hurt for the missing data functionality to get exercised a bit > more. > >> >> What are the plans for the release of NumPy 2.0? >> >> Mark and others have been very good at maintaining ABI compatibility. At > the same time the numpy-refactor branch has dropped off the radar (any > news/plans?). So I'm wondering what would go into 2.0 that is ready now or > soon? > > Ralf > > What is the possibility of adding datarray and carray packages to numpy or > scipy in the 'near future'? > Or do these need to wait and see what happens to the new masked and NA > implementations? > (Actually probably a lot of what happens may depend on this.) > > I think they would need a good deal of discussion on the list first. I'm not even clear as to how far those packages have been developed and whether or not they touch the c code. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Thu Sep 15 17:09:01 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 15 Sep 2011 14:09:01 -0700 Subject: [Numpy-discussion] PyPy guys deserve some help on micronumpy from you, numpy gurus! In-Reply-To: References: Message-ID: Hi Valery On Thu, Sep 15, 2011 at 3:23 AM, Valery Khamenya wrote: > There is a module micronumpy that appeared at PyPy source tree: > https://bitbucket.org/pypy/pypy/src/dfae5033127e/pypy/module/micronumpy/ > The great contributions of Justin Peel and Ilya?Osadchiy to micronumpy > module revive?step-by-step?the?functionality of numpy. > It would be great if some of numpy-gurus could jump in to assist, contribute > some code and also, perhaps, guide a bit where the things go deeply in numpy > nature. If the micronumpy array object exposes the array interface, it would be easy to intermix standard numpy and PyPy loops on numpy arrays (and then it's probably overkill to redo the whole of NumPy in PyPy?). IIRC, this is the approach we took with the micronumpy GSOC project a year ago, but I haven't been following the project since then. Regards St?fan From ben.root at ou.edu Thu Sep 15 20:52:48 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 15 Sep 2011 19:52:48 -0500 Subject: [Numpy-discussion] can't use argmin for array of timedeltas In-Reply-To: References: Message-ID: On Thu, Sep 15, 2011 at 2:49 PM, Benjamin Root wrote: > I encountered something similar back in April or so and that it was fixed. > The problem was that the minimum function was implemented as max(0 - a), and > so this fails for timedelta objects that can't do this. This was fixed for > min(), but apparently not for argmin(). > > Ben Root > I did a quick fix of this issue and made it available as part of pull request #158: https://github.com/numpy/numpy/pull/158 I simply reused the argmax code, but flipped the arguments for the comparison operations. This did require modifying some header files, so I don't know how big of an issue that will be. Cheers, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy-discussion at maubp.freeserve.co.uk Fri Sep 16 05:49:38 2011 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Fri, 16 Sep 2011 10:49:38 +0100 Subject: [Numpy-discussion] PyPy guys deserve some help on micronumpy from you, numpy gurus! In-Reply-To: References: Message-ID: On Thu, Sep 15, 2011 at 11:34 AM, Peter wrote: > This may not be the best place to ask, but how should a > python script (e.g. setup.py) distinguish between real NumPy > and micronumpy? Or should I instead be looking to distinguish > PyPy versus another Python implementation? For anyone interested, over on the pypy-dev list Alex recommended: import platform; platform.python_implementation == 'PyPy' http://mail.python.org/pipermail/pypy-dev/2011-September/008315.html Peter From ben.root at ou.edu Fri Sep 16 11:11:10 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 16 Sep 2011 10:11:10 -0500 Subject: [Numpy-discussion] Difficulty using special types like timedelta and Decimal Message-ID: Certain numerical types in Python that support accurate fractional calculations such as timedelta and Decimal do not allow for multiplication or division by a floating point number, but do allow for use with an integer. This can cause difficulties with some functions such as np.gradient() which has a division by 2.0 as part of the algorithm. My question is: with Py3k's division always resulting in a floating point result regardless of the denominator's type, would it be possible to change some of these common denominators over to integers in order to better facilitate support for these special types? Obviously, I don't think we could expect full support of these types, but maybe a mechanism could be developed to better handle these? Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin.landenberger at imtek.uni-freiburg.de Fri Sep 16 11:18:11 2011 From: benjamin.landenberger at imtek.uni-freiburg.de (Benjamin Landenberger) Date: Fri, 16 Sep 2011 17:18:11 +0200 Subject: [Numpy-discussion] Indexing by label in 3rd dimension Message-ID: Hello list! I have an array *mask* of shape (a, b) and another array *intensities* of shape (N, a, b), where the values in *mask* range from 0 to N-1. It is somehow similar to label arrays in scipy.ndimage. Now I want to pick those entries from the first dimension of *intensities* which are given by *mask*. The returned array shall again wave shape (a, b). Can this be done with fancy indexing? Thank you, Ben From ben.root at ou.edu Fri Sep 16 11:33:37 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 16 Sep 2011 10:33:37 -0500 Subject: [Numpy-discussion] Indexing by label in 3rd dimension In-Reply-To: References: Message-ID: On Friday, September 16, 2011, Benjamin Landenberger < benjamin.landenberger at imtek.uni-freiburg.de> wrote: > Hello list! > > I have an array *mask* of shape (a, b) and another array *intensities* > of shape (N, a, b), where the values in *mask* range from 0 to N-1. It > is somehow similar to label arrays in scipy.ndimage. > > Now I want to pick those entries from the first dimension of > *intensities* which are given by *mask*. The returned array shall again > wave shape (a, b). > > Can this be done with fancy indexing? > > Thank you, > > Ben > Try intensities[mask, xrange(a), xrange(b)]. Untested, but it would work for 1d, haven't tried for 2d. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Fri Sep 16 11:34:42 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 16 Sep 2011 10:34:42 -0500 Subject: [Numpy-discussion] Indexing by label in 3rd dimension In-Reply-To: References: Message-ID: On Friday, September 16, 2011, Benjamin Root wrote: > > > On Friday, September 16, 2011, Benjamin Landenberger < benjamin.landenberger at imtek.uni-freiburg.de> wrote: >> Hello list! >> >> I have an array *mask* of shape (a, b) and another array *intensities* >> of shape (N, a, b), where the values in *mask* range from 0 to N-1. It >> is somehow similar to label arrays in scipy.ndimage. >> >> Now I want to pick those entries from the first dimension of >> *intensities* which are given by *mask*. The returned array shall again >> wave shape (a, b). >> >> Can this be done with fancy indexing? >> >> Thank you, >> >> Ben >> > > Try intensities[mask, xrange(a), xrange(b)]. > > Untested, but it would work for 1d, haven't tried for 2d. > > Ben Root > Crap, that wouldn't work... Sorry. -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Fri Sep 16 11:43:39 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Fri, 16 Sep 2011 10:43:39 -0500 Subject: [Numpy-discussion] Indexing by label in 3rd dimension In-Reply-To: References: Message-ID: On Fri, Sep 16, 2011 at 10:18 AM, Benjamin Landenberger < benjamin.landenberger at imtek.uni-freiburg.de> wrote: > Hello list! > > I have an array *mask* of shape (a, b) and another array *intensities* > of shape (N, a, b), where the values in *mask* range from 0 to N-1. It > is somehow similar to label arrays in scipy.ndimage. > > Now I want to pick those entries from the first dimension of > *intensities* which are given by *mask*. The returned array shall again > wave shape (a, b). > > Can this be done with fancy indexing? > The choose() function can be used: In [19]: intensities Out[19]: array([[[ 1, 2, 3], [ 4, 5, 6], [ 7, 8, 9]], [[11, 12, 13], [14, 15, 16], [17, 18, 19]], [[21, 22, 23], [24, 25, 26], [27, 28, 29]]]) In [20]: mask Out[20]: array([[0, 0, 1], [1, 1, 1], [1, 2, 2]]) In [21]: choose(mask, intensities) Out[21]: array([[ 1, 2, 13], [14, 15, 16], [17, 28, 29]]) Warren > Thank you, > > Ben > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Sep 16 12:18:57 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 16 Sep 2011 10:18:57 -0600 Subject: [Numpy-discussion] Difficulty using special types like timedelta and Decimal In-Reply-To: References: Message-ID: On Fri, Sep 16, 2011 at 9:11 AM, Benjamin Root wrote: > Certain numerical types in Python that support accurate fractional > calculations such as timedelta and Decimal do not allow for multiplication > or division by a floating point number, but do allow for use with an > integer. This can cause difficulties with some functions such as > np.gradient() which has a division by 2.0 as part of the algorithm. > > My question is: with Py3k's division always resulting in a floating point > result regardless of the denominator's type, would it be possible to change > some of these common denominators over to integers in order to better > facilitate support for these special types? > > Obviously, I don't think we could expect full support of these types, but > maybe a mechanism could be developed to better handle these? > > Timedelta seems to work in master: In [1]: timedelta64(5) Out[1]: numpy.timedelta64(5) In [2]: timedelta64(5)/2.0 Out[2]: numpy.timedelta64(2) In [3]: timedelta64(5)/2 Out[3]: numpy.timedelta64(2) I think making the change to 2 is a reasonable thing to do, but you should put a from __future__ import division at the beginning of the module containing gradient and make sure everything still works. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Fri Sep 16 12:35:03 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 16 Sep 2011 11:35:03 -0500 Subject: [Numpy-discussion] Difficulty using special types like timedelta and Decimal In-Reply-To: References: Message-ID: On Friday, September 16, 2011, Charles R Harris wrote: > > > On Fri, Sep 16, 2011 at 9:11 AM, Benjamin Root wrote: >> >> Certain numerical types in Python that support accurate fractional calculations such as timedelta and Decimal do not allow for multiplication or division by a floating point number, but do allow for use with an integer. This can cause difficulties with some functions such as np.gradient() which has a division by 2.0 as part of the algorithm. >> >> My question is: with Py3k's division always resulting in a floating point result regardless of the denominator's type, would it be possible to change some of these common denominators over to integers in order to better facilitate support for these special types? >> >> Obviously, I don't think we could expect full support of these types, but maybe a mechanism could be developed to better handle these? >> > > Timedelta seems to work in master: > > In [1]: timedelta64(5) > Out[1]: numpy.timedelta64(5) > > In [2]: timedelta64(5)/2.0 > Out[2]: numpy.timedelta64(2) > > In [3]: timedelta64(5)/2 > Out[3]: numpy.timedelta64(2) > > I think making the change to 2 is a reasonable thing to do, but you should put a > > from __future__ import division > > at the beginning of the module containing gradient and make sure everything still works. > > Chuck > > I am referring to a numpy array of python timedeltas, not numpy's. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Fri Sep 16 14:20:41 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 16 Sep 2011 13:20:41 -0500 Subject: [Numpy-discussion] Difficulty using special types like timedelta and Decimal In-Reply-To: References: Message-ID: On Fri, Sep 16, 2011 at 11:35 AM, Benjamin Root wrote: > > > On Friday, September 16, 2011, Charles R Harris > wrote: > > > > > > On Fri, Sep 16, 2011 at 9:11 AM, Benjamin Root wrote: > >> > >> Certain numerical types in Python that support accurate fractional > calculations such as timedelta and Decimal do not allow for multiplication > or division by a floating point number, but do allow for use with an > integer. This can cause difficulties with some functions such as > np.gradient() which has a division by 2.0 as part of the algorithm. > >> > >> My question is: with Py3k's division always resulting in a floating > point result regardless of the denominator's type, would it be possible to > change some of these common denominators over to integers in order to better > facilitate support for these special types? > >> > >> Obviously, I don't think we could expect full support of these types, > but maybe a mechanism could be developed to better handle these? > >> > > > > Timedelta seems to work in master: > > > > In [1]: timedelta64(5) > > Out[1]: numpy.timedelta64(5) > > > > In [2]: timedelta64(5)/2.0 > > Out[2]: numpy.timedelta64(2) > > > > In [3]: timedelta64(5)/2 > > Out[3]: numpy.timedelta64(2) > > > > I think making the change to 2 is a reasonable thing to do, but you > should put a > > > > from __future__ import division > > > > at the beginning of the module containing gradient and make sure > everything still works. > > > > Chuck > > > > > > I am referring to a numpy array of python timedeltas, not numpy's. > > Ben Root A couple of additional bugs I have encountered with np.gradient(). First, the documentation says that the input data can be "array-like", yet the first line of the function calls the shape() method (and later uses the dtype() method). Second, there is a check in the code that sees if the input array is not 'd', 'f', 'D', or 'F'. If it isn't one of those, then it forces the output type to be 'd'. I am not entirely sure I understand the reasoning here, but at the very least, if I pass in an object array, wouldn't it be expected that I get an object array back? As an interesting sidenote, usually, one would expect the same type going in as going out, however, with python datetime objects, datetime goes in and timedeltas come out in functions like np.diff() and (theoretically) np.gradient(). Don't know how much that would impact the prospect of supporting python datetime and/or timedelta objects, but I thought I should mention that. Furthermore, in light of the recent addition of datetime to numpy itself, maybe we ought to consider an automatic coercion of a list of python datetime/timedelta objects into a numpy array of the numpy equivalents? We do this right now with lists of floats and integers. Just a thought. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Fri Sep 16 14:29:12 2011 From: Nicolas.Rougier at inria.fr (Nicolas Rougier) Date: Fri, 16 Sep 2011 20:29:12 +0200 Subject: [Numpy-discussion] [ANN] glumpy 0.2.0 Message-ID: <311E6BAC-FD1A-4DCA-B7B1-964ACFF78F6B@inria.fr> Hi folks, I am pleased to announce a new release of glumpy, a small python library for the (very) fast vizualization of numpy arrays, (mainly two dimensional) that has been designed with efficiency in mind. If you want to draw nice figures for inclusion in a scientific article, you?d better use matplotlib but if you want to have a sense of what?s going on in your simulation while it is running, then maybe glumpy can help you. Download and screenshots at: http://code.google.com/p/glumpy/ Nicolas -------------- next part -------------- An HTML attachment was scrubbed... URL: From scipy at samueljohn.de Fri Sep 16 16:21:00 2011 From: scipy at samueljohn.de (Samuel John) Date: Fri, 16 Sep 2011 22:21:00 +0200 Subject: [Numpy-discussion] [ANN] glumpy 0.2.0 In-Reply-To: <311E6BAC-FD1A-4DCA-B7B1-964ACFF78F6B@inria.fr> References: <311E6BAC-FD1A-4DCA-B7B1-964ACFF78F6B@inria.fr> Message-ID: <9D13DA0B-6777-427A-9435-C992C2342243@samueljohn.de> Hi Nicolas, that looks great. Could you make this available such that `pip install glumpy` would work? cheers, Samuel From ben.root at ou.edu Fri Sep 16 16:28:18 2011 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 16 Sep 2011 15:28:18 -0500 Subject: [Numpy-discussion] type-casting inconsistency with timedelta64 Message-ID: Came across an oddity when experimenting with the datetime64 and timedelta64 objects. a = np.zeros((10,), dtype='l') b = np.datetime64('2010-12-20T14:23:56-0600') c = np.datetime64('2010-12-20T21:27:09-0600') a[0:1] = c - b # This works fine a[0] = c - b # This does not The second assignment throws the following error: TypeError: don't know how to convert scalar number to long Cheers, Ben Root P.S. - np.arange() can make ranges of datetime64 from string inputs, but np.linspace() can not (and does not take a dtype kwarg). -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Sep 16 16:50:59 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 16 Sep 2011 14:50:59 -0600 Subject: [Numpy-discussion] type-casting inconsistency with timedelta64 In-Reply-To: References: Message-ID: On Fri, Sep 16, 2011 at 2:28 PM, Benjamin Root wrote: > Came across an oddity when experimenting with the datetime64 and > timedelta64 objects. > > a = np.zeros((10,), dtype='l') > b = np.datetime64('2010-12-20T14:23:56-0600') > c = np.datetime64('2010-12-20T21:27:09-0600') > > a[0:1] = c - b # This works fine > a[0] = c - b # This does not > > > The second assignment throws the following error: > TypeError: don't know how to convert scalar number to long > > Certainly inconsistent: In [8]: (c - b).astype(long) Out[8]: 25393 In [9]: a[0] = (c - b).astype(long) works. I don't think either should work without an explicit cast because the units get lost. > > P.S. - np.arange() can make ranges of datetime64 from string inputs, but > np.linspace() can not (and does not take a dtype kwarg). > > A dtype keyword would be problematic since linspace needs to have fractions to do the division. On the other hand, having a convenient way to produce equally spaced time sounds useful. Hmm... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Sat Sep 17 08:53:37 2011 From: Nicolas.Rougier at inria.fr (Nicolas Rougier) Date: Sat, 17 Sep 2011 14:53:37 +0200 Subject: [Numpy-discussion] [ANN] glumpy 0.2.0 In-Reply-To: <9D13DA0B-6777-427A-9435-C992C2342243@samueljohn.de> References: <311E6BAC-FD1A-4DCA-B7B1-964ACFF78F6B@inria.fr> <9D13DA0B-6777-427A-9435-C992C2342243@samueljohn.de> Message-ID: <68206BED-2E00-4B95-BB55-2A5A5C237B11@inria.fr> Thanks. I just uploaded it to pypi. Nicolas On Sep 16, 2011, at 22:21 , Samuel John wrote: > Hi Nicolas, > > that looks great. > Could you make this available such that `pip install glumpy` would work? > > cheers, > Samuel > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Sat Sep 17 12:40:22 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 17 Sep 2011 10:40:22 -0600 Subject: [Numpy-discussion] Polynomial implementation in numpy.ma In-Reply-To: References: Message-ID: 2011/9/14 St?fan van der Walt > Hi all, > > There were some failures in the polynomial tests earlier today, and > while investigating I saw that numpy.ma implements its own root > finder. It uses inversion of a Van der Monde matrix, which I believe > may suffer from some numerical instability problems. Given that > Charles has gone to some length to implement good polynomial root > finders, I think it would be best to employ those instead, and simply > pre-filter the data that comes from the masked array module, if > possible. > > The test failure arises because the test compares against np.polyfit where Travis changed the column scaling, which in turn changed the singular values, and it is that comparison that fails. The Vandermonde matrix needs to be used for the fitting so nothing should be changed there. To fix the test failure, either the test could be modified or the ma.polyfit routine can be changed. Probably the latter is the best option, the new scaling code can be brought over and maybe the weight and covariance options added.Travis' changes needs to be audited in any case to make sure all the corner cases are covered. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Sep 17 12:52:18 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 17 Sep 2011 10:52:18 -0600 Subject: [Numpy-discussion] Polynomial implementation in numpy.ma In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 10:40 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > 2011/9/14 St?fan van der Walt > >> Hi all, >> >> There were some failures in the polynomial tests earlier today, and >> while investigating I saw that numpy.ma implements its own root >> finder. It uses inversion of a Van der Monde matrix, which I believe >> may suffer from some numerical instability problems. Given that >> Charles has gone to some length to implement good polynomial root >> finders, I think it would be best to employ those instead, and simply >> pre-filter the data that comes from the masked array module, if >> possible. >> >> > The test failure arises because the test compares against np.polyfit where > Travis changed the column scaling, which in turn changed the singular > values, and it is that comparison that fails. > > The Vandermonde matrix needs to be used for the fitting so nothing should > be changed there. To fix the test failure, either the test could be modified > or the ma.polyfit routine can be changed. Probably the latter is the best > option, the new scaling code can be brought over and maybe the weight and > covariance options added.Travis' changes needs to be audited in any case to > make sure all the corner cases are covered. > Note that the enhanced np.polyfit can be called directly with the weight set to zero for the masked rows, i.e., set w to the compliment of the mask. That should simplify the code a bit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Sep 17 14:52:05 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 17 Sep 2011 12:52:05 -0600 Subject: [Numpy-discussion] lstsq: masked arrays, weights, scaling, and covariance Message-ID: Hi All, I'd like to start a discussion about modifications to lstsq to accommodate the new masked arrays and move weights, scaling, and covariance determination down to a lower common level. This is motivated by Travis' recent changes to polyfit as well as my own various polynomial fits that also allow weights. Also, once these features are pushed down to lstsq, it should be possible to push them down further into a c-wrapper for the LAPACK routines, which is where I really think they belong in the long run. Because missing values will effect the std/var/cov in the same way as weights of zero, I think support for missing values and weights go naturally together. Support for scaling and covariance are less closely tied, but they are both features I use all the time in practice and having them available will be useful. It might also be nice to change the return signature, though this would require a new function. I rather like the idea of returning the coefficients and a dictionary, where everything not a coefficient gets stuffed into the dictionary. In this regard see also Denis Laxalde's proposal , something we might want to be consistent with. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sat Sep 17 15:04:04 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 17 Sep 2011 12:04:04 -0700 Subject: [Numpy-discussion] Polynomial implementation in numpy.ma In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 9:40 AM, Charles R Harris wrote: > The Vandermonde matrix needs to be used for the fitting so nothing should be > changed there. I remember now where I heard this: http://people.maths.ox.ac.uk/trefethen/mythstalk.pdf Specifically, the statement: "Some well-known algorithms take O(n^2) operations (e.g. na?ve Lagrange formula) or are exponentially unstable (e.g. Vandermonde matrices based on monomials x^k )." I wasn't sure whether to interpret that as the Vandermonde-based algorithm being flawed, or the polynomial fitting problem on an equidistant grid being ill-conditioned in general. Regards St?fan From charlesr.harris at gmail.com Sat Sep 17 15:39:10 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 17 Sep 2011 13:39:10 -0600 Subject: [Numpy-discussion] Polynomial implementation in numpy.ma In-Reply-To: References: Message-ID: 2011/9/17 St?fan van der Walt > On Sat, Sep 17, 2011 at 9:40 AM, Charles R Harris > wrote: > > The Vandermonde matrix needs to be used for the fitting so nothing should > be > > changed there. > > I remember now where I heard this: > > http://people.maths.ox.ac.uk/trefethen/mythstalk.pdf > > Specifically, the statement: "Some well-known algorithms take O(n^2) > operations (e.g. na?ve Lagrange formula) > or are exponentially unstable (e.g. Vandermonde matrices based on > monomials x^k )." > > I wasn't sure whether to interpret that as the Vandermonde-based > algorithm being flawed, or the polynomial fitting problem on an > equidistant grid being ill-conditioned in general. > > Equispaced grid points are bad for power series fits -- the Chebyshev points are better -- but power series also tend to have problems even when Chebyshev points are used since they never become particularly orthogonal over the sample points. Alternatives like Bernstein polynomials, Legendre polynomials, and Chebyshev polynomials are much better in that regard but not generally as easy to hand as power series, which one reason I wanted to make available in numpy. However, doing a fit using Chebyshev polynomials and then converting it to a power series throws away the Chebyshev advantage as the Chebyshev series is much less sensitive to numerical errors in the coefficients. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Sep 17 15:40:13 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 17 Sep 2011 15:40:13 -0400 Subject: [Numpy-discussion] lstsq: masked arrays, weights, scaling, and covariance In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 2:52 PM, Charles R Harris wrote: > Hi All, > > I'd like to start a discussion about modifications to lstsq to accommodate > the new masked arrays and move weights, scaling, and covariance > determination down to a lower common level. This is motivated by Travis' > recent changes to polyfit as well as my own various polynomial fits that > also allow weights. Also, once these features are pushed down to lstsq, it > should be possible to push them down further into a c-wrapper for the LAPACK > routines, which is where I really think they belong in the long run. > > Because missing values will effect the std/var/cov in the same way as > weights of zero, I think support for missing values and weights go naturally > together. Support for scaling and covariance are less closely tied, but they > are both features I use all the time in practice and having them available > will be useful.? It might also be nice to change the return signature, > though this would require a new function. I rather like the idea of > returning the coefficients and a dictionary, where everything not a > coefficient gets stuffed into the dictionary. In this regard see also Denis > Laxalde's proposal, something we might want to be consistent with. > > Thoughts? What's the speed penalty if we just want to use numpy/scipy linalg as a library and don't need any of the extra features? As some of the discussions have shown it can be pretty expensive to use linalg in loops. Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Sat Sep 17 15:50:02 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 17 Sep 2011 13:50:02 -0600 Subject: [Numpy-discussion] lstsq: masked arrays, weights, scaling, and covariance In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 1:40 PM, wrote: > On Sat, Sep 17, 2011 at 2:52 PM, Charles R Harris > wrote: > > Hi All, > > > > I'd like to start a discussion about modifications to lstsq to > accommodate > > the new masked arrays and move weights, scaling, and covariance > > determination down to a lower common level. This is motivated by Travis' > > recent changes to polyfit as well as my own various polynomial fits that > > also allow weights. Also, once these features are pushed down to lstsq, > it > > should be possible to push them down further into a c-wrapper for the > LAPACK > > routines, which is where I really think they belong in the long run. > > > > Because missing values will effect the std/var/cov in the same way as > > weights of zero, I think support for missing values and weights go > naturally > > together. Support for scaling and covariance are less closely tied, but > they > > are both features I use all the time in practice and having them > available > > will be useful. It might also be nice to change the return signature, > > though this would require a new function. I rather like the idea of > > returning the coefficients and a dictionary, where everything not a > > coefficient gets stuffed into the dictionary. In this regard see also > Denis > > Laxalde's proposal, something we might want to be consistent with. > > > > Thoughts? > > What's the speed penalty if we just want to use numpy/scipy linalg as > a library and don't need any of the extra features? > > As some of the discussions have shown it can be pretty expensive to > use linalg in loops. > > I think the features should be optional and not the defaults, although masked arrays will always need some special treatment. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Sat Sep 17 16:48:50 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Sat, 17 Sep 2011 16:48:50 -0400 Subject: [Numpy-discussion] Bug in numpy std, etc. with other data structures? Message-ID: Just ran into this. Any objections for having numpy.std and other functions in core/fromnumeric.py call asanyarray before trying to use the array's method? Other data structures like pandas and larry define their own std method, for instance, and this doesn't allow them to pass through. I'm inclined to say that the issue is with numpy, though maybe the data structures shouldn't shadow numpy array methods while altering the signature. I dunno. df = pandas.DataFrame(np.random.random((10,5))) np.std(df,axis=0) TypeError: std() got an unexpected keyword argument 'dtype' np.std(np.asanyarray(df),axis=0) array([ 0.30883352, 0.3133324 , 0.26517361, 0.26389029, 0.20022444]) Though I don't think this would work with larry yet. Pull request: https://github.com/numpy/numpy/pull/160 Skipper From wesmckinn at gmail.com Sat Sep 17 17:12:36 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Sat, 17 Sep 2011 17:12:36 -0400 Subject: [Numpy-discussion] Bug in numpy std, etc. with other data structures? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: > Just ran into this. Any objections for having numpy.std and other > functions in core/fromnumeric.py call asanyarray before trying to use > the array's method? Other data structures like pandas and larry define > their own std method, for instance, and this doesn't allow them to > pass through. I'm inclined to say that the issue is with numpy, though > maybe the data structures shouldn't shadow numpy array methods while > altering the signature. I dunno. > > df = pandas.DataFrame(np.random.random((10,5))) > > np.std(df,axis=0) > > TypeError: std() got an unexpected keyword argument 'dtype' > > np.std(np.asanyarray(df),axis=0) > array([ 0.30883352, ?0.3133324 , ?0.26517361, ?0.26389029, ?0.20022444]) > > Though I don't think this would work with larry yet. > > Pull request: https://github.com/numpy/numpy/pull/160 > > Skipper > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Note I've no real intention of making DataFrame fully ndarray-like-- but it's nice to be able to type: df.std(axis=0) df.std(axis=1) np.sqrt(df) etc. which works the same as ndarray. I suppose the __array__/__array_wrap__ interface is there largely as a convenience. From oliphant at enthought.com Sat Sep 17 18:28:31 2011 From: oliphant at enthought.com (Travis Oliphant) Date: Sat, 17 Sep 2011 17:28:31 -0500 Subject: [Numpy-discussion] lstsq: masked arrays, weights, scaling, and covariance In-Reply-To: References: Message-ID: <79D41148-4895-4CC0-A655-83B2A65DC81E@enthought.com> I think this sounds like a great idea. The lowest level that makes sense is the correct place for them. -Travis On Sep 17, 2011, at 1:52 PM, Charles R Harris wrote: > Hi All, > > I'd like to start a discussion about modifications to lstsq to accommodate the new masked arrays and move weights, scaling, and covariance determination down to a lower common level. This is motivated by Travis' recent changes to polyfit as well as my own various polynomial fits that also allow weights. Also, once these features are pushed down to lstsq, it should be possible to push them down further into a c-wrapper for the LAPACK routines, which is where I really think they belong in the long run. > > Because missing values will effect the std/var/cov in the same way as weights of zero, I think support for missing values and weights go naturally together. Support for scaling and covariance are less closely tied, but they are both features I use all the time in practice and having them available will be useful. It might also be nice to change the return signature, though this would require a new function. I rather like the idea of returning the coefficients and a dictionary, where everything not a coefficient gets stuffed into the dictionary. In this regard see also Denis Laxalde's proposal, something we might want to be consistent with. > > Thoughts? > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Sep 17 20:36:32 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 17 Sep 2011 20:36:32 -0400 Subject: [Numpy-discussion] Bug in numpy std, etc. with other data structures? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 5:12 PM, Wes McKinney wrote: > On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: >> Just ran into this. Any objections for having numpy.std and other >> functions in core/fromnumeric.py call asanyarray before trying to use >> the array's method? Other data structures like pandas and larry define >> their own std method, for instance, and this doesn't allow them to >> pass through. I'm inclined to say that the issue is with numpy, though >> maybe the data structures shouldn't shadow numpy array methods while >> altering the signature. I dunno. >> >> df = pandas.DataFrame(np.random.random((10,5))) >> >> np.std(df,axis=0) >> >> TypeError: std() got an unexpected keyword argument 'dtype' >> >> np.std(np.asanyarray(df),axis=0) >> array([ 0.30883352, ?0.3133324 , ?0.26517361, ?0.26389029, ?0.20022444]) >> >> Though I don't think this would work with larry yet. >> >> Pull request: https://github.com/numpy/numpy/pull/160 >> >> Skipper >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > Note I've no real intention of making DataFrame fully ndarray-like-- > but it's nice to be able to type: > > df.std(axis=0) > df.std(axis=1) > np.sqrt(df) > > etc. which works the same as ndarray. I suppose the > __array__/__array_wrap__ interface is there largely as a convenience. I'm a bit worried about the different ddof defaults in cases like this. Essentially we will not be able to rely on ddof=0 anymore. Different defaults on axis are easy to catch, but having the same function call return sometimes ddof=0 and sometimes ddof=1 might make for some fun debugging. Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From wesmckinn at gmail.com Sat Sep 17 21:11:43 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Sat, 17 Sep 2011 21:11:43 -0400 Subject: [Numpy-discussion] Bug in numpy std, etc. with other data structures? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 8:36 PM, wrote: > On Sat, Sep 17, 2011 at 5:12 PM, Wes McKinney wrote: >> On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: >>> Just ran into this. Any objections for having numpy.std and other >>> functions in core/fromnumeric.py call asanyarray before trying to use >>> the array's method? Other data structures like pandas and larry define >>> their own std method, for instance, and this doesn't allow them to >>> pass through. I'm inclined to say that the issue is with numpy, though >>> maybe the data structures shouldn't shadow numpy array methods while >>> altering the signature. I dunno. >>> >>> df = pandas.DataFrame(np.random.random((10,5))) >>> >>> np.std(df,axis=0) >>> >>> TypeError: std() got an unexpected keyword argument 'dtype' >>> >>> np.std(np.asanyarray(df),axis=0) >>> array([ 0.30883352, ?0.3133324 , ?0.26517361, ?0.26389029, ?0.20022444]) >>> >>> Though I don't think this would work with larry yet. >>> >>> Pull request: https://github.com/numpy/numpy/pull/160 >>> >>> Skipper >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> Note I've no real intention of making DataFrame fully ndarray-like-- >> but it's nice to be able to type: >> >> df.std(axis=0) >> df.std(axis=1) >> np.sqrt(df) >> >> etc. which works the same as ndarray. I suppose the >> __array__/__array_wrap__ interface is there largely as a convenience. > > I'm a bit worried about the different ddof defaults in cases like > this. Essentially we will not be able to rely on ddof=0 anymore. > Different defaults on axis are easy to catch, but having the same > function call return sometimes ddof=0 and sometimes ddof=1 might make > for some fun debugging. > > Josef > > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Can we lobby for default ddof=1 in NumPy 2.0? Breaking with a convention like this doesn't make much sense to me. From bsouthey at gmail.com Sat Sep 17 22:50:56 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Sat, 17 Sep 2011 21:50:56 -0500 Subject: [Numpy-discussion] Bug in numpy std, etc. with other data structures? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 4:12 PM, Wes McKinney wrote: > On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: >> Just ran into this. Any objections for having numpy.std and other >> functions in core/fromnumeric.py call asanyarray before trying to use >> the array's method? Other data structures like pandas and larry define >> their own std method, for instance, and this doesn't allow them to >> pass through. I'm inclined to say that the issue is with numpy, though >> maybe the data structures shouldn't shadow numpy array methods while >> altering the signature. I dunno. >> >> df = pandas.DataFrame(np.random.random((10,5))) >> >> np.std(df,axis=0) >> >> TypeError: std() got an unexpected keyword argument 'dtype' >> >> np.std(np.asanyarray(df),axis=0) >> array([ 0.30883352, ?0.3133324 , ?0.26517361, ?0.26389029, ?0.20022444]) >> >> Though I don't think this would work with larry yet. >> >> Pull request: https://github.com/numpy/numpy/pull/160 >> >> Skipper >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion numpy.std() does accepts array-like which obvious means that np.std([1,2,3,5]) works making asanyarray call a total waste of cpu time. Clearly pandas is not array-like input (as Wes points out below) so an error is correct. Doing this type of 'fix' will have unintended consequences when other non-numpy objects are incorrectly passed to numpy functions. Rather you should determine why 'array-like' failed here IF you think a pandas object is either array-like or a numpy object. > > Note I've no real intention of making DataFrame fully ndarray-like-- > but it's nice to be able to type: > > df.std(axis=0) > df.std(axis=1) > np.sqrt(df) > > etc. which works the same as ndarray. I suppose the > __array__/__array_wrap__ interface is there largely as a convenience. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I consider that the only way pandas or any other numpy-derivative to overcome this is get into numpy/scipy. After all Travis opened the discussion for Numpy 3 which you could still address. Bruce PS Good luck on the ddof thing given the past discussions on it! From robert.kern at gmail.com Sat Sep 17 22:56:25 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 17 Sep 2011 21:56:25 -0500 Subject: [Numpy-discussion] Bug in numpy std, etc. with other data structures? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 21:50, Bruce Southey wrote: > numpy.std() ?does accepts array-like which obvious means that > np.std([1,2,3,5]) works making asanyarray call a total waste of cpu > time. Clearly pandas is not array-like input (as Wes points out below) > so an error is correct. No. Even lists are "array-like" in the terminology of the docstring standard. Anything that np.asarray() or np.asanyarray() can accept is "array-like". Please stop making things up and being sanctimonious about it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From wesmckinn at gmail.com Sat Sep 17 23:00:10 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Sat, 17 Sep 2011 23:00:10 -0400 Subject: [Numpy-discussion] Bug in numpy std, etc. with other data structures? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 10:50 PM, Bruce Southey wrote: > On Sat, Sep 17, 2011 at 4:12 PM, Wes McKinney wrote: >> On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: >>> Just ran into this. Any objections for having numpy.std and other >>> functions in core/fromnumeric.py call asanyarray before trying to use >>> the array's method? Other data structures like pandas and larry define >>> their own std method, for instance, and this doesn't allow them to >>> pass through. I'm inclined to say that the issue is with numpy, though >>> maybe the data structures shouldn't shadow numpy array methods while >>> altering the signature. I dunno. >>> >>> df = pandas.DataFrame(np.random.random((10,5))) >>> >>> np.std(df,axis=0) >>> >>> TypeError: std() got an unexpected keyword argument 'dtype' >>> >>> np.std(np.asanyarray(df),axis=0) >>> array([ 0.30883352, ?0.3133324 , ?0.26517361, ?0.26389029, ?0.20022444]) >>> >>> Though I don't think this would work with larry yet. >>> >>> Pull request: https://github.com/numpy/numpy/pull/160 >>> >>> Skipper >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > numpy.std() ?does accepts array-like which obvious means that > np.std([1,2,3,5]) works making asanyarray call a total waste of cpu > time. Clearly pandas is not array-like input (as Wes points out below) > so an error is correct. Doing this type of 'fix' will have unintended > consequences when other non-numpy objects are incorrectly passed to > numpy functions. Rather you should determine why 'array-like' failed > here IF you think a pandas object is either array-like or a numpy > object. No, the reason it is failing is because np.std takes the EAFP/duck-typing approach: try: std = a.std except AttributeError: return _wrapit(a, 'std', axis, dtype, out, ddof) return std(axis, dtype, out, ddof) Indeed DataFrame has an std method but it doesn't have the same function signature as ndarray.std. > >> >> Note I've no real intention of making DataFrame fully ndarray-like-- >> but it's nice to be able to type: >> >> df.std(axis=0) >> df.std(axis=1) >> np.sqrt(df) >> >> etc. which works the same as ndarray. I suppose the >> __array__/__array_wrap__ interface is there largely as a convenience. >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > I consider that the only way pandas or any other numpy-derivative to > overcome this is get into numpy/scipy. After all Travis opened the > discussion for Numpy 3 which you could still address. > > Bruce > PS Good luck on the ddof thing given the past discussions on it! > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From bsouthey at gmail.com Sat Sep 17 23:11:28 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Sat, 17 Sep 2011 22:11:28 -0500 Subject: [Numpy-discussion] Bug in numpy std, etc. with other data structures? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 10:00 PM, Wes McKinney wrote: > On Sat, Sep 17, 2011 at 10:50 PM, Bruce Southey wrote: >> On Sat, Sep 17, 2011 at 4:12 PM, Wes McKinney wrote: >>> On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: >>>> Just ran into this. Any objections for having numpy.std and other >>>> functions in core/fromnumeric.py call asanyarray before trying to use >>>> the array's method? Other data structures like pandas and larry define >>>> their own std method, for instance, and this doesn't allow them to >>>> pass through. I'm inclined to say that the issue is with numpy, though >>>> maybe the data structures shouldn't shadow numpy array methods while >>>> altering the signature. I dunno. >>>> >>>> df = pandas.DataFrame(np.random.random((10,5))) >>>> >>>> np.std(df,axis=0) >>>> >>>> TypeError: std() got an unexpected keyword argument 'dtype' >>>> >>>> np.std(np.asanyarray(df),axis=0) >>>> array([ 0.30883352, ?0.3133324 , ?0.26517361, ?0.26389029, ?0.20022444]) >>>> >>>> Though I don't think this would work with larry yet. >>>> >>>> Pull request: https://github.com/numpy/numpy/pull/160 >>>> >>>> Skipper >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> numpy.std() ?does accepts array-like which obvious means that >> np.std([1,2,3,5]) works making asanyarray call a total waste of cpu >> time. Clearly pandas is not array-like input (as Wes points out below) >> so an error is correct. Doing this type of 'fix' will have unintended >> consequences when other non-numpy objects are incorrectly passed to >> numpy functions. Rather you should determine why 'array-like' failed >> here IF you think a pandas object is either array-like or a numpy >> object. > > No, the reason it is failing is because np.std takes the > EAFP/duck-typing approach: > > try: > ? ?std = a.std > except AttributeError: > ? ?return _wrapit(a, 'std', axis, dtype, out, ddof) > return std(axis, dtype, out, ddof) > > Indeed DataFrame has an std method but it doesn't have the same > function signature as ndarray.std. > Thanks for the clarification - see Robert I am not making things up! Bruce >> >>> >>> Note I've no real intention of making DataFrame fully ndarray-like-- >>> but it's nice to be able to type: >>> >>> df.std(axis=0) >>> df.std(axis=1) >>> np.sqrt(df) >>> >>> etc. which works the same as ndarray. I suppose the >>> __array__/__array_wrap__ interface is there largely as a convenience. >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> I consider that the only way pandas or any other numpy-derivative to >> overcome this is get into numpy/scipy. After all Travis opened the >> discussion for Numpy 3 which you could still address. >> >> Bruce >> PS Good luck on the ddof thing given the past discussions on it! >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Sat Sep 17 23:20:07 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 17 Sep 2011 22:20:07 -0500 Subject: [Numpy-discussion] Bug in numpy std, etc. with other data structures? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 22:11, Bruce Southey wrote: > On Sat, Sep 17, 2011 at 10:00 PM, Wes McKinney wrote: >> On Sat, Sep 17, 2011 at 10:50 PM, Bruce Southey wrote: >>> On Sat, Sep 17, 2011 at 4:12 PM, Wes McKinney wrote: >>>> On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: >>>>> Just ran into this. Any objections for having numpy.std and other >>>>> functions in core/fromnumeric.py call asanyarray before trying to use >>>>> the array's method? Other data structures like pandas and larry define >>>>> their own std method, for instance, and this doesn't allow them to >>>>> pass through. I'm inclined to say that the issue is with numpy, though >>>>> maybe the data structures shouldn't shadow numpy array methods while >>>>> altering the signature. I dunno. >>>>> >>>>> df = pandas.DataFrame(np.random.random((10,5))) >>>>> >>>>> np.std(df,axis=0) >>>>> >>>>> TypeError: std() got an unexpected keyword argument 'dtype' >>>>> >>>>> np.std(np.asanyarray(df),axis=0) >>>>> array([ 0.30883352, ?0.3133324 , ?0.26517361, ?0.26389029, ?0.20022444]) >>>>> >>>>> Though I don't think this would work with larry yet. >>>>> >>>>> Pull request: https://github.com/numpy/numpy/pull/160 >>>>> >>>>> Skipper >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> numpy.std() ?does accepts array-like which obvious means that >>> np.std([1,2,3,5]) works making asanyarray call a total waste of cpu >>> time. Clearly pandas is not array-like input (as Wes points out below) >>> so an error is correct. Doing this type of 'fix' will have unintended >>> consequences when other non-numpy objects are incorrectly passed to >>> numpy functions. Rather you should determine why 'array-like' failed >>> here IF you think a pandas object is either array-like or a numpy >>> object. >> >> No, the reason it is failing is because np.std takes the >> EAFP/duck-typing approach: >> >> try: >> ? ?std = a.std >> except AttributeError: >> ? ?return _wrapit(a, 'std', axis, dtype, out, ddof) >> return std(axis, dtype, out, ddof) >> >> Indeed DataFrame has an std method but it doesn't have the same >> function signature as ndarray.std. > > Thanks for the clarification - see Robert I am not making things up! I have no doubt that np.std() fails to work as desired. But the fault is with np.std() not living up to the semantics implied by the documentation (or the documentation documenting the wrong semantics), not that DataFrame does not live up to a meaning of "array-like" that you invented. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From keith.hughitt at gmail.com Sun Sep 18 12:09:42 2011 From: keith.hughitt at gmail.com (Keith Hughitt) Date: Sun, 18 Sep 2011 12:09:42 -0400 Subject: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc Message-ID: Hi all, For a ndarray-subclass I'm working on I extended the array_wrap and array_finalize functions so that operations like __sub__() would return a new instance of the subclass, instead of an ndarray (although I still haven't completely figured this out yet .) One side-effect of this, however, is that now other operations which should *not* return a subclass (such as std) are returning one; def __array_finalize__(self, obj): """Finishes instantiation of the new map object""" if obj is None: return if hasattr(obj, 'header'): self.header = obj.header # preserve object properties properties = self.get_properties(obj.header) for attr, value in list(properties.items()): setattr(self, attr, getattr(obj, attr, value)) self.center = obj.center self.scale = obj.scale self.units = obj.units def __array_wrap__(self, out_arr, context=None): """Returns a wrapped instance of a Map object""" return np.ndarray.__array_wrap__(self, out_arr, context) I'm sure it is simply a coding error on my part, but so far I havne't been able to track it down. Any ideas? Thanks, Keith -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sun Sep 18 12:16:54 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 18 Sep 2011 09:16:54 -0700 Subject: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc In-Reply-To: References: Message-ID: On Sun, Sep 18, 2011 at 9:09 AM, Keith Hughitt wrote: > I'm sure it is simply a coding error on my part, but so far I havne't been > able to track it down. I don't think it's something you've done wrong. See e.g.: In [9]: x = np.ma.array([[1,2,3]]) In [10]: np.std(x, axis=0) Out[10]: masked_array(data = [0.0 0.0 0.0], mask = [False False False], fill_value = 1e+20) Regards St?fan From keith.hughitt at gmail.com Sun Sep 18 12:48:45 2011 From: keith.hughitt at gmail.com (Keith Hughitt) Date: Sun, 18 Sep 2011 12:48:45 -0400 Subject: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc In-Reply-To: References: Message-ID: Interesting. It works as expected when called as a method: In [10]: x = np.ma.array([[1,2,3]]) In [11]: x.std() Out[11]: 0.81649658092772603 but not for my class: In [14]: aia.std() Out[14]: AIAMap(292.4342470467856) In [15]: np.std(aia) Out[15]: AIAMap(292.4342470467856) In [16]: np.array(aia).std() Out[16]: 292.43424704678557 Keith 2011/9/18 St?fan van der Walt > On Sun, Sep 18, 2011 at 9:09 AM, Keith Hughitt > wrote: > > I'm sure it is simply a coding error on my part, but so far I havne't > been > > able to track it down. > > I don't think it's something you've done wrong. See e.g.: > > In [9]: x = np.ma.array([[1,2,3]]) > > In [10]: np.std(x, axis=0) > Out[10]: > masked_array(data = [0.0 0.0 0.0], > mask = [False False False], > fill_value = 1e+20) > > Regards > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Sep 18 13:08:09 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 18 Sep 2011 13:08:09 -0400 Subject: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc In-Reply-To: References: Message-ID: On Sun, Sep 18, 2011 at 12:48 PM, Keith Hughitt wrote: > Interesting. It works as expected when called as a method: > > In [10]: x = np.ma.array([[1,2,3]]) > In [11]: x.std() > Out[11]: 0.81649658092772603 That's only the treatment of scalars in std >>> x = np.ma.array(np.random.randn(4,5)) >>> x.std(0) masked_array(data = [0.842985810318 0.423312129749 0.607139983509 0.583121781933 0.681360542632], mask = [False False False False False], fill_value = 1e+20) Josef > > but not for my class: > > In [14]: aia.std() > Out[14]: AIAMap(292.4342470467856) > In [15]: np.std(aia) > Out[15]: AIAMap(292.4342470467856) > In [16]: np.array(aia).std() > Out[16]: 292.43424704678557 > > Keith > 2011/9/18 St?fan van der Walt >> >> On Sun, Sep 18, 2011 at 9:09 AM, Keith Hughitt >> wrote: >> > I'm sure it is simply a coding error on my part, but so far I havne't >> > been >> > able to track it down. >> >> I don't think it's something you've done wrong. ?See e.g.: >> >> In [9]: x = np.ma.array([[1,2,3]]) >> >> In [10]: np.std(x, axis=0) >> Out[10]: >> masked_array(data = [0.0 0.0 0.0], >> ? ? ? ? ? ? mask = [False False False], >> ? ? ? fill_value = 1e+20) >> >> Regards >> St?fan >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From stefan at sun.ac.za Sun Sep 18 15:25:14 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 18 Sep 2011 12:25:14 -0700 Subject: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc In-Reply-To: References: Message-ID: On Sun, Sep 18, 2011 at 9:48 AM, Keith Hughitt wrote: > Interesting. It works as expected when called as a method: > > In [10]: x = np.ma.array([[1,2,3]]) > In [11]: x.std() > Out[11]: 0.81649658092772603 I'm guessing that np.ma.array implements its own std function, but I haven't checked. St?fan From pgmdevlist at gmail.com Sun Sep 18 16:25:31 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 18 Sep 2011 22:25:31 +0200 Subject: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc In-Reply-To: References: Message-ID: <37A2CFB9-0C40-4A29-A146-6874EAF17C8A@gmail.com> On Sep 18, 2011, at 21:25 , St?fan van der Walt wrote: > On Sun, Sep 18, 2011 at 9:48 AM, Keith Hughitt wrote: >> Interesting. It works as expected when called as a method: >> >> In [10]: x = np.ma.array([[1,2,3]]) >> In [11]: x.std() >> Out[11]: 0.81649658092772603 > > I'm guessing that np.ma.array implements its own std function, but I > haven't checked. It does. np.std tries first to call the .std method of the input argument. From wesmckinn at gmail.com Sun Sep 18 16:58:52 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Sun, 18 Sep 2011 16:58:52 -0400 Subject: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc In-Reply-To: <37A2CFB9-0C40-4A29-A146-6874EAF17C8A@gmail.com> References: <37A2CFB9-0C40-4A29-A146-6874EAF17C8A@gmail.com> Message-ID: On Sun, Sep 18, 2011 at 4:25 PM, Pierre GM wrote: > > On Sep 18, 2011, at 21:25 , St?fan van der Walt wrote: > >> On Sun, Sep 18, 2011 at 9:48 AM, Keith Hughitt wrote: >>> Interesting. It works as expected when called as a method: >>> >>> In [10]: x = np.ma.array([[1,2,3]]) >>> In [11]: x.std() >>> Out[11]: 0.81649658092772603 >> >> I'm guessing that np.ma.array implements its own std function, but I >> haven't checked. > > It does. np.std tries first to call the .std method of the input argument. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I thought maybe you can intercept 0-dim objects and return self.item() in array_finalize, but not dice. This is really weird: import numpy as np class example(np.ndarray): def __new__(cls, arr): return np.array(arr).view(cls) def __array_finalize__(self, obj): print 'foo' if self.ndim == 0: return self.item() In [6]: foo = example(np.arange(10)) foo In [7]: foo.std() foo foo foo foo foo foo foo Out[7]: example(2.8722813232690143) From hoytak at stat.washington.edu Sun Sep 18 20:15:16 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Sun, 18 Sep 2011 17:15:16 -0700 Subject: [Numpy-discussion] ANN: PyCPX 0.01, a numpy/cython wrapper for CPlex. Message-ID: Hello, I'm pleased to announce the first release of a wrapper I wrote for IBM's CPlex Optimizer Suite. It focuses on ease of use and seamless integration with numpy, and allows one to naturally specify constrained linear and quadratic optimization problems over real, boolean, and integer variables. The wrapper is written in cython/C++ for speed and released under the LGPL license. For documentation and examples, please see the website: http://www.stat.washington.edu/~hoytak/code/pycpx/index.html. Please send me any comments, questions and suggestions! I am quite open to feedback. Thanks, --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From ben.root at ou.edu Sun Sep 18 20:32:25 2011 From: ben.root at ou.edu (Benjamin Root) Date: Sun, 18 Sep 2011 19:32:25 -0500 Subject: [Numpy-discussion] datetime64 y2k38 bug Message-ID: I was working on adding some test cases in numpy for the argmin/max functions with some datetime64s. I found that on my 32-bit machine, it fails to parse a date past the Y2.038k date. I find this odd because the datetime is supposed to be 64-bits, but I guess there is some arch-dependent code somewhere? Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 18 23:08:49 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 18 Sep 2011 21:08:49 -0600 Subject: [Numpy-discussion] datetime64 y2k38 bug In-Reply-To: References: Message-ID: On Sun, Sep 18, 2011 at 6:32 PM, Benjamin Root wrote: > I was working on adding some test cases in numpy for the argmin/max > functions with some datetime64s. I found that on my 32-bit machine, it > fails to parse a date past the Y2.038k date. I find this odd because the > datetime is supposed to be 64-bits, but I guess there is some arch-dependent > code somewhere? > > I think that is actually POSIX for the time_t structure. Which is not to say it's good ;) Google UNIX Year 2038 problem. ISTR reading recently that there is a movement afoot to fix the time_t structure on 32 bit machines for Linux. You've got to wonder, what were the POSIX people thinking? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 18 23:13:19 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 18 Sep 2011 21:13:19 -0600 Subject: [Numpy-discussion] datetime64 y2k38 bug In-Reply-To: References: Message-ID: On Sun, Sep 18, 2011 at 9:08 PM, Charles R Harris wrote: > > > On Sun, Sep 18, 2011 at 6:32 PM, Benjamin Root wrote: > >> I was working on adding some test cases in numpy for the argmin/max >> functions with some datetime64s. I found that on my 32-bit machine, it >> fails to parse a date past the Y2.038k date. I find this odd because the >> datetime is supposed to be 64-bits, but I guess there is some arch-dependent >> code somewhere? >> >> > I think that is actually POSIX for the time_t structure. Which is not to > say it's good ;) Google UNIX Year 2038 problem. ISTR reading recently that > there is a movement afoot to fix the time_t structure on 32 bit machines for > Linux. You've got to wonder, what were the POSIX people thinking? > > See comments here . Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Sun Sep 18 23:42:02 2011 From: ben.root at ou.edu (Benjamin Root) Date: Sun, 18 Sep 2011 22:42:02 -0500 Subject: [Numpy-discussion] datetime64 y2k38 bug In-Reply-To: References: Message-ID: On Sunday, September 18, 2011, Charles R Harris wrote: > > > On Sun, Sep 18, 2011 at 6:32 PM, Benjamin Root wrote: >> >> I was working on adding some test cases in numpy for the argmin/max functions with some datetime64s. I found that on my 32-bit machine, it fails to parse a date past the Y2.038k date. I find this odd because the datetime is supposed to be 64-bits, but I guess there is some arch-dependent code somewhere? >> > > I think that is actually POSIX for the time_t structure. Which is not to say it's good ;) Google UNIX Year 2038 problem. ISTR reading recently that there is a movement afoot to fix the time_t structure on 32 bit machines for Linux. You've got to wonder, what were the POSIX people thinking? > > Chuck > > Actually, I am quite familiar with the problem, having tried to convince people that there are users of time_t who need it for data archive systems (think digitizing pre-1900 weather obs), but to no avail. There is a time64_t type that explicitly uses 64 bits. Don't know if it is a magic bullet, but given that numpy's type is called datetime64, it might be good to explicitly use time64_t. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Sep 18 23:52:57 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 18 Sep 2011 23:52:57 -0400 Subject: [Numpy-discussion] datetime64 y2k38 bug In-Reply-To: References: Message-ID: On Sun, Sep 18, 2011 at 11:13 PM, Charles R Harris wrote: > > > On Sun, Sep 18, 2011 at 9:08 PM, Charles R Harris > wrote: >> >> >> On Sun, Sep 18, 2011 at 6:32 PM, Benjamin Root wrote: >>> >>> I was working on adding some test cases in numpy for the argmin/max >>> functions with some datetime64s.? I found that on my 32-bit machine, it >>> fails to parse a date past the Y2.038k date.? I find this odd because the >>> datetime is supposed to be 64-bits, but I guess there is some arch-dependent >>> code somewhere? >>> >> >> I think that is actually POSIX for the time_t structure. Which is not to >> say it's good ;) Google UNIX Year 2038 problem. ISTR reading recently that >> there is a movement afoot to fix the time_t structure on 32 bit machines for >> Linux. You've got to wonder, what were the POSIX people thinking? >> > > See comments here. Thanks for the entertaining link " I think it's still perfectly valid to say "you're a moron, and we need to fix it" " (just a quote, doesn't apply to the python community) Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From davide.lasagna at polito.it Mon Sep 19 02:10:31 2011 From: davide.lasagna at polito.it (Davide) Date: Mon, 19 Sep 2011 08:10:31 +0200 Subject: [Numpy-discussion] ANN: PyCPX 0.01, a numpy/cython wrapper for CPlex. In-Reply-To: References: Message-ID: <4E76DCD7.7050807@polito.it> Thanks for your good work! Cheers, Davide On 09/19/2011 02:15 AM, Hoyt Koepke wrote: > Hello, > > I'm pleased to announce the first release of a wrapper I wrote for > IBM's CPlex Optimizer Suite. It focuses on ease of use and seamless > integration with numpy, and allows one to naturally specify > constrained linear and quadratic optimization problems over real, > boolean, and integer variables. The wrapper is written in cython/C++ > for speed and released under the LGPL license. > > For documentation and examples, please see the website: > http://www.stat.washington.edu/~hoytak/code/pycpx/index.html. > > Please send me any comments, questions and suggestions! I am quite > open to feedback. > > Thanks, > --Hoyt > > ++++++++++++++++++++++++++++++++++++++++++++++++ > + Hoyt Koepke > + University of Washington Department of Statistics > + http://www.stat.washington.edu/~hoytak/ > + hoytak at gmail.com > ++++++++++++++++++++++++++++++++++++++++++ > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From benjamin.landenberger at imtek.uni-freiburg.de Mon Sep 19 03:38:27 2011 From: benjamin.landenberger at imtek.uni-freiburg.de (Benjamin Landenberger) Date: Mon, 19 Sep 2011 09:38:27 +0200 Subject: [Numpy-discussion] Indexing by label in 3rd dimension In-Reply-To: References: Message-ID: Great! Thank you! On 16/09/2011 17:43, Warren Weckesser wrote: > > > On Fri, Sep 16, 2011 at 10:18 AM, Benjamin Landenberger > > wrote: > > Hello list! > > I have an array *mask* of shape (a, b) and another array *intensities* > of shape (N, a, b), where the values in *mask* range from 0 to N-1. It > is somehow similar to label arrays in scipy.ndimage. > > Now I want to pick those entries from the first dimension of > *intensities* which are given by *mask*. The returned array shall again > wave shape (a, b). > > Can this be done with fancy indexing? > > > > The choose() function can be used: > > In [19]: intensities > Out[19]: > array([[[ 1, 2, 3], > [ 4, 5, 6], > [ 7, 8, 9]], > > [[11, 12, 13], > [14, 15, 16], > [17, 18, 19]], > > [[21, 22, 23], > [24, 25, 26], > [27, 28, 29]]]) > > In [20]: mask > Out[20]: > array([[0, 0, 1], > [1, 1, 1], > [1, 2, 2]]) > > In [21]: choose(mask, intensities) > Out[21]: > array([[ 1, 2, 13], > [14, 15, 16], > [17, 28, 29]]) > > > Warren > > > > Thank you, > > Ben > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From scipy at samueljohn.de Mon Sep 19 09:03:41 2011 From: scipy at samueljohn.de (Samuel John) Date: Mon, 19 Sep 2011 15:03:41 +0200 Subject: [Numpy-discussion] OS X Lion: llvm: numpy and scipy Message-ID: <4BD955EF-0F08-4979-9A1C-0E52F91D76AB@samueljohn.de> Ahoy numpy gurus :-) Would it be possible to adapt the setup.py and/or numpy/distutils to set the right variables on Mac OS X 10.7? (see below). I have looked a bit into the setup.py and the distutils package of numpy but I am a bit lost. Do you think it's possible to teach numpy to use different CC, CXX? And the FFLAGS and the switch --fcompiler=gnu95 arg? Building scipy on OS X Lion 10.7.x currently fails because of some llvm incompatibilies and the gfortran. While it's easy to get gfortran e.g. via http://mxcl.github.com/homebrew/ it's hard to `pip install scipy` or manual install because you have to: > export CC=gcc-4.2 > export CXX=g++-4.2 > export FFLAGS=-ff2c > python setup.py build --fcompiler=gnu95 This way, numpy and then scipy builds successfully. Scipy uses the distutil settings from numpy -- as far as I get it -- and therefore scipy cannot add these variables. Right? It would be great if numpy and scipy would build right out-of-the-box on OS X, again. I'd love to provide a patch but I am lost in the depth of distutils... Samuel From aronne.merrelli at gmail.com Mon Sep 19 10:26:49 2011 From: aronne.merrelli at gmail.com (Aronne Merrelli) Date: Mon, 19 Sep 2011 09:26:49 -0500 Subject: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc In-Reply-To: References: <37A2CFB9-0C40-4A29-A146-6874EAF17C8A@gmail.com> Message-ID: On Sun, Sep 18, 2011 at 3:58 PM, Wes McKinney wrote: > > I thought maybe you can intercept 0-dim objects and return self.item() > in array_finalize, but not dice. This is really weird: > > import numpy as np > > class example(np.ndarray): > > def __new__(cls, arr): > return np.array(arr).view(cls) > > def __array_finalize__(self, obj): > print 'foo' > if self.ndim == 0: > return self.item() > > > In [6]: foo = example(np.arange(10)) > foo > > In [7]: foo.std() > foo > foo > foo > foo > foo > foo > foo > Out[7]: example(2.8722813232690143) > _______________________________________________ > The documentation on array_wrap/finalize isn't too clear on this point, but I think the return from __array_finalize__ is actually not used in any way. So, in this example, I think it the returned self.item() is discarded. The only important "output" from __array_finalize__ is whatever modification was made to the self object. The 'caller' of __array_finalize__ is (I think) somewhere is the C code, so I can't verify this (I'm not really a C programmer...). The series of "foos" is due to the std calculation, if you change "print 'foo'" to "print 'foo', obj" you get this output: In [71]: a = example([1,2,3]) foo [1 2 3] In [72]: a.std() foo [1 2 3] foo 6.0 foo 2.0 foo [1 2 3] foo [1 2 3] foo [-1. 0. 1.] foo [ 1. 0. 1.] foo 2.0 foo 0.666666666667 Out[72]: example(0.816496580927726) Anyway, back on topic - I'm having similar problems as Keith. It seems like there isn't consistency on how different built-in functions treat array_wrap/finalize/etc, or maybe I'm still confused. At the moment it seems like the only solution is to write a method in the subclass for each case where the return isn't what you want. In this example, to force the result of std() to be an ndarray instead of the subclass, you would need to write something similar to: import numpy as np class example(np.ndarray): def __new__(cls, arr): return np.array(arr).view(cls) def std(self, *args, **kwargs): return np.array(self).std(*args, **kwargs) Is this the best way to handle this problem? It could make the subclass definition quite cluttered, but on the other hand all these extra methods would be pretty straightforward. Thanks, Aronne -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Mon Sep 19 11:06:08 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 19 Sep 2011 17:06:08 +0200 Subject: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc In-Reply-To: References: <37A2CFB9-0C40-4A29-A146-6874EAF17C8A@gmail.com> Message-ID: > > Anyway, back on topic - I'm having similar problems as Keith. It seems like there isn't consistency on how different built-in functions treat array_wrap/finalize/etc, or maybe I'm still confused. Actually, that depends on the np function you need. Functions like np.std? first call the method of your class, and if it fails, call _method._std on your object. The call on __array_finalize__ shows up implicitly in the second case, I think, because of some recent (?) changes. It used to be that the basic std function accepted only ndarrays. Now, _methods._std accepts any kind of subclass (it uses np.asanyarray)? So, np.std on a subclass that does not implement its std will return an instance of the subclass. That's quite silly, IMHO. Why use asanyarray in that case ? If I wanted my subclass to return an instance of itself with std, I'd implement std as a method. Anyhow, Wes, in the current state of things, my understanding is that yes, you have to implement a std method like you suggest. I'd throw a copy=False, subok=False in np.array just in case, to make sure you work with a view of the underlying ndarray. From ralf.gommers at googlemail.com Mon Sep 19 13:05:28 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 19 Sep 2011 19:05:28 +0200 Subject: [Numpy-discussion] OS X Lion: llvm: numpy and scipy In-Reply-To: <4BD955EF-0F08-4979-9A1C-0E52F91D76AB@samueljohn.de> References: <4BD955EF-0F08-4979-9A1C-0E52F91D76AB@samueljohn.de> Message-ID: On Mon, Sep 19, 2011 at 3:03 PM, Samuel John wrote: > Ahoy numpy gurus :-) > > Would it be possible to adapt the setup.py and/or numpy/distutils to set > the right variables on Mac OS X 10.7? (see below). > I have looked a bit into the setup.py and the distutils package of numpy > but I am a bit lost. > > Do you think it's possible to teach numpy to use different CC, CXX? This is possible, but numpy probably shouldn't mess with these variables. As a user you can set them permanently by adding them to your bash_profile for example. And the FFLAGS This needs to be solved. Perhaps the solution involves more wrappers for broken vecLib/Accelerate functions in scipy? Does anyone know which routines are broken on 10.7? For 10.6 I found this discussion helpful: http://www.macresearch.org/lapackblas-fortran-106. It is claimed there that while '-ff2c' fixes complex routines, it breaks SDOT when used with '-m64'. SDOT is used in linalg. > and the switch --fcompiler=gnu95 arg? > This shouldn't be necessary if you only have gfortran installed. Ralf > Building scipy on OS X Lion 10.7.x currently fails because of some llvm > incompatibilies and the gfortran. > While it's easy to get gfortran e.g. via http://mxcl.github.com/homebrew/it's hard to `pip install scipy` or manual install because you have to: > > > export CC=gcc-4.2 > > export CXX=g++-4.2 > > export FFLAGS=-ff2c > > python setup.py build --fcompiler=gnu95 > > This way, numpy and then scipy builds successfully. > Scipy uses the distutil settings from numpy -- as far as I get it -- and > therefore scipy cannot add these variables. Right? > > It would be great if numpy and scipy would build right out-of-the-box on OS > X, again. > > I'd love to provide a patch but I am lost in the depth of distutils... > > > Samuel > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ady.sethi at gmail.com Mon Sep 19 17:38:02 2011 From: ady.sethi at gmail.com (Aditya Sethi) Date: Mon, 19 Sep 2011 17:38:02 -0400 Subject: [Numpy-discussion] Upgrade to 1.6.x: frompyfunc() ufunc casting issue Message-ID: Hi all, I am facing an issue upgrading numpy from 1.5.1 to 1.6.1. In numPy 1.6, the casting behaviour for ufunc has changed and has become stricter. Can someone advise how to implement the below simple example which worked in 1.5.1 but fails in 1.6.1? >>> import numpy as np >>> def add(a,b): ... return (a+b) >>> uadd = np.frompyfunc(add,2,1) >>> uadd >>> uadd.accumulate([1,2,3]) Traceback (most recent call last): File "", line 1, in ValueError: could not find a matching type for add (vectorized).accumulate, requested type has type code 'l' Thanks, Aditya -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Sep 19 18:16:10 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 19 Sep 2011 15:16:10 -0700 Subject: [Numpy-discussion] Upgrade to 1.6.x: frompyfunc() ufunc casting issue In-Reply-To: References: Message-ID: Hi, On Mon, Sep 19, 2011 at 2:38 PM, Aditya Sethi wrote: > I am facing an issue upgrading numpy from 1.5.1 to 1.6.1. > In numPy 1.6, the casting behaviour for ufunc has changed and has become > stricter. > Can someone advise how to implement the below simple example which worked in > 1.5.1 but fails in 1.6.1? >>>> import numpy as np >>>> def add(a,b): > ... ? ?return (a+b) >>>> uadd = np.frompyfunc(add,2,1) >>>> uadd > >>>> uadd.accumulate([1,2,3]) > Traceback (most recent call last): > ? File "", line 1, in > ValueError: could not find a matching type for add (vectorized).accumulate, > requested type has type code 'l' Seems to work ok on the latest version: In [12]: uadd([1,2,3],[1,2,3]) Out[12]: array([2, 4, 6], dtype=object) In [13]: np.__version__ Out[13]: '2.0.0.dev-af22fc4' Cheers St?fan From ady.sethi at gmail.com Mon Sep 19 19:18:29 2011 From: ady.sethi at gmail.com (Aditya Sethi) Date: Mon, 19 Sep 2011 19:18:29 -0400 Subject: [Numpy-discussion] Upgrade to 1.6.x: frompyfunc() ufunc casting issue In-Reply-To: References: Message-ID: Hi Stefan, The problem arises when you try to use methods of the ufunc. So for the custom universal function previously defined, uadd: uadd([1,2,3],[1,2,3]) works fine. But uadd.accumulate(..) or uadd.reduce(..) fail with error: ValueError: could not find a matching type for add (vectorized).accumulate ( or (vectorized).reduce ) Apologies, I should have been more clear before. Thanks, Aditya 2011/9/19 St?fan van der Walt > Hi, > > On Mon, Sep 19, 2011 at 2:38 PM, Aditya Sethi wrote: > > I am facing an issue upgrading numpy from 1.5.1 to 1.6.1. > > In numPy 1.6, the casting behaviour for ufunc has changed and has become > > stricter. > > Can someone advise how to implement the below simple example which worked > in > > 1.5.1 but fails in 1.6.1? > >>>> import numpy as np > >>>> def add(a,b): > > ... return (a+b) > >>>> uadd = np.frompyfunc(add,2,1) > >>>> uadd > > > >>>> uadd.accumulate([1,2,3]) > > Traceback (most recent call last): > > File "", line 1, in > > ValueError: could not find a matching type for add > (vectorized).accumulate, > > requested type has type code 'l' > > Seems to work ok on the latest version: > > In [12]: uadd([1,2,3],[1,2,3]) > Out[12]: array([2, 4, 6], dtype=object) > > In [13]: np.__version__ > Out[13]: '2.0.0.dev-af22fc4' > > Cheers > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Sep 19 19:27:12 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 19 Sep 2011 16:27:12 -0700 Subject: [Numpy-discussion] Upgrade to 1.6.x: frompyfunc() ufunc casting issue In-Reply-To: References: Message-ID: On Mon, Sep 19, 2011 at 4:18 PM, Aditya Sethi wrote: > But uadd.accumulate(..) or uadd.reduce(..) fail with error: > ?ValueError: could not find a matching type for add (vectorized).accumulate > ( or (vectorized).reduce ) > Apologies, I should have been more clear before. In the development version: In [4]: uadd.accumulate([1,2,3]) Out[4]: array([1, 3, 6], dtype=object) In [5]: uadd.reduce([1,2,3]) Out[5]: 6 Regards St?fan From stefan at sun.ac.za Tue Sep 20 02:43:56 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 19 Sep 2011 23:43:56 -0700 Subject: [Numpy-discussion] Fancy indexing with masks Message-ID: Hi all, Matthew Brett showed me an interesting code snippet this evening: # Construct input data In [15]: x Out[15]: array([[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11]]) # Fancy indexing with 1D boolean array In [16]: x[np.array([True, False, True])] Out[16]: array([[0, 1, 2], [6, 7, 8]]) # Fancy indexing with 2D boolean array In [17]: x[np.array([[True, False, True]])] Out[18]: array([0, 2]) I guess it's been a long day, but why does this work at all? I expected the first example to break, because the 1D mask does not match the number of rows in x. In the second example, I expected an error because the 2D mask was not of the same shape as x. But, oddly, both work. There's also no attempt at broadcasting indexes. Any hints would be appreciated! Thanks St?fan From robert.kern at gmail.com Tue Sep 20 03:43:52 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 20 Sep 2011 02:43:52 -0500 Subject: [Numpy-discussion] Fancy indexing with masks In-Reply-To: References: Message-ID: 2011/9/20 St?fan van der Walt : > Hi all, > > Matthew Brett showed me an interesting code snippet this evening: > > # Construct input data > > In [15]: x > Out[15]: > array([[ 0, ?1, ?2], > ? ? ? [ 3, ?4, ?5], > ? ? ? [ 6, ?7, ?8], > ? ? ? [ 9, 10, 11]]) > > # Fancy indexing with 1D boolean array > > In [16]: x[np.array([True, False, True])] > Out[16]: > array([[0, 1, 2], > ? ? ? [6, 7, 8]]) > > # Fancy indexing with 2D boolean array > > In [17]: x[np.array([[True, False, True]])] > Out[18]: array([0, 2]) > > > I guess it's been a long day, but why does this work at all? > > I expected the first example to break, because the 1D mask does not > match the number of rows in x. ?In the second example, I expected an > error because the 2D mask was not of the same shape as x. ?But, oddly, > both work. There's also no attempt at broadcasting indexes. If the array is short in a dimension, it gets implicitly continued with Falses. You can see this in one dimension: [~] |1> x = np.arange(12) [~] |2> x[np.array([True, False, True])] array([0, 2]) I honestly don't know if this is documented or tested anywhere or even if this existed in older versions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From scipy at samueljohn.de Tue Sep 20 05:13:57 2011 From: scipy at samueljohn.de (Samuel John) Date: Tue, 20 Sep 2011 11:13:57 +0200 Subject: [Numpy-discussion] OS X Lion: llvm: numpy and scipy In-Reply-To: References: <4BD955EF-0F08-4979-9A1C-0E52F91D76AB@samueljohn.de> Message-ID: <2055622E-64A8-4116-9FDC-496AAC76917D@samueljohn.de> Ralf, thanks for your answer. However, in short: I want `pip install numpy; pip install scipy` to work on OS X Lion without extra effort :-) On 19.09.2011, at 19:05, Ralf Gommers wrote: > Do you think it's possible to teach numpy to use different CC, CXX? > >> This is possible, but numpy probably shouldn't mess with these variables. As a user you can set them permanently by adding them to your bash_profile for example. The problem is that most things do work fine with the default gcc which has the llvm backend on OS X Lion (10.7) /usr/bin/gcc -> llvm-gcc-4.2. But somehow scipy has problems with that backend. I do not want to set CC CXX permanently. I made a homebrew formula for numpy, which takes care of these things. But the policy of the homebrew team is to avoid duplicates which can be installed via pip. Therefore I am asking for support if someone could point me to the place where the compiler is chosen. I'd propose to add an OS X 10.7 switch there in order to avoid the llvm. >> And the FFLAGS > > This needs to be solved. Perhaps the solution involves more wrappers for broken vecLib/Accelerate functions in scipy? Does anyone know which routines are broken on 10.7? For 10.6 I found this discussion helpful: http://www.macresearch.org/lapackblas-fortran-106. It is claimed there that while '-ff2c' fixes complex routines, it breaks SDOT when used with '-m64'. SDOT is used in linalg. I don't know nothing of such things. :-( If there is really something broken with vecLib/Accelerate, a ticket on Apple's bugtracker rdar should be opened. >> and the switch --fcompiler=gnu95 arg? > > This shouldn't be necessary if you only have gfortran installed. > Ah ok. Thanks! cheers, Samuel > > Building scipy on OS X Lion 10.7.x currently fails because of some llvm incompatibilies and the gfortran. > While it's easy to get gfortran e.g. via http://mxcl.github.com/homebrew/ it's hard to `pip install scipy` or manual install because you have to: > > > export CC=gcc-4.2 > > export CXX=g++-4.2 > > export FFLAGS=-ff2c > > python setup.py build --fcompiler=gnu95 > > This way, numpy and then scipy builds successfully. > Scipy uses the distutil settings from numpy -- as far as I get it -- and therefore scipy cannot add these variables. Right? > > It would be great if numpy and scipy would build right out-of-the-box on OS X, again. > > I'd love to provide a patch but I am lost in the depth of distutils... From ady.sethi at gmail.com Tue Sep 20 08:25:10 2011 From: ady.sethi at gmail.com (Aditya Sethi) Date: Tue, 20 Sep 2011 08:25:10 -0400 Subject: [Numpy-discussion] Upgrade to 1.6.x: frompyfunc() ufunc casting issue In-Reply-To: References: Message-ID: Hi, Stefan, which version of Python and NumPy are you using? I am upgrading Python and NumPy, and would like to get it working on the official releases of Python 2.7.2 + NumPy 1.6.1 In Python 2.6 + NumPy 1.5.1 on win32, it works. In Python 2.7.2 + NumPy 1.6.1 on win32, np.frompyfunc(add,2,1).accumulate definitely gives an error. Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> def add(a,b): ... return (a+b) ... >>> uadd = np.frompyfunc(add,2,1) >>> uadd >>> uadd([1,2,3],[1,2,3]) array([2, 4, 6], dtype=object) >>> >>> uadd.accumulate([1,2,3]) Traceback (most recent call last): File "", line 1, in ValueError: could not find a matching type for add (vectorized).accumulate, requested type has type code 'l' >>> Aditya 2011/9/19 St?fan van der Walt > On Mon, Sep 19, 2011 at 4:18 PM, Aditya Sethi wrote: > > But uadd.accumulate(..) or uadd.reduce(..) fail with error: > > ValueError: could not find a matching type for add > (vectorized).accumulate > > ( or (vectorized).reduce ) > > Apologies, I should have been more clear before. > > In the development version: > > In [4]: uadd.accumulate([1,2,3]) > Out[4]: array([1, 3, 6], dtype=object) > > In [5]: uadd.reduce([1,2,3]) > Out[5]: 6 > > Regards > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Sep 20 08:41:27 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 20 Sep 2011 08:41:27 -0400 Subject: [Numpy-discussion] OS X Lion: llvm: numpy and scipy In-Reply-To: <2055622E-64A8-4116-9FDC-496AAC76917D@samueljohn.de> References: <4BD955EF-0F08-4979-9A1C-0E52F91D76AB@samueljohn.de> <2055622E-64A8-4116-9FDC-496AAC76917D@samueljohn.de> Message-ID: On Tue, Sep 20, 2011 at 5:13 AM, Samuel John wrote: > Ralf, thanks for your answer. > > However, in short: > > ?I want `pip install numpy; pip install scipy` to work on OS X Lion without extra effort :-) > > > On 19.09.2011, at 19:05, Ralf Gommers wrote: >> Do you think it's possible to teach numpy to use different CC, CXX? >> >>> This is possible, but numpy probably shouldn't mess with these variables. As a user you can set them permanently by adding them to your bash_profile for example. > > The problem is that most things do work fine with the default gcc which has the llvm backend on OS X Lion (10.7) /usr/bin/gcc -> llvm-gcc-4.2. But somehow scipy has problems with that backend. > I do not want to set CC CXX permanently. > I made a homebrew formula for numpy, which takes care of these things. But the policy of the homebrew team is to avoid duplicates which can be installed via pip. Therefore I am asking for support if someone could point me to the place where the compiler is chosen. I'd propose to add an OS X 10.7 switch there in order to avoid the llvm. > > >>> And the FFLAGS >> >> This needs to be solved. Perhaps the solution involves more wrappers for broken vecLib/Accelerate functions in scipy? Does anyone know which routines are broken on 10.7? For 10.6 I found this discussion helpful: http://www.macresearch.org/lapackblas-fortran-106. It is claimed there that while '-ff2c' fixes complex routines, it breaks SDOT when used with '-m64'. SDOT is used in linalg. > > I don't know nothing of such things. :-( > If there is really something broken with vecLib/Accelerate, a ticket on Apple's bugtracker rdar should be opened. > > >>> and the switch --fcompiler=gnu95 arg? >> >> This shouldn't be necessary if you only have gfortran installed. >> > > Ah ok. Thanks! I will try to look at this problem next week, when I will receive a new laptop with Lion on it. If I forget about it, please ping me at the end of next week, we need to fix this, cheers, David From shish at keba.be Tue Sep 20 09:18:16 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 20 Sep 2011 09:18:16 -0400 Subject: [Numpy-discussion] Upgrade to 1.6.x: frompyfunc() ufunc casting issue In-Reply-To: References: Message-ID: He is using the development version of Numpy, which is probably the main difference (I tried it too with Numpy 1.6.1 on an x86_64 Linux architecture and got the same bug). If you want to use an official Numpy release you'll probably need to downgrade to 1.5.x and wait until the next Numpy release. -=- Olivier 2011/9/20 Aditya Sethi > Hi, > > Stefan, which version of Python and NumPy are you using? > > I am upgrading Python and NumPy, and would like to get it working on the > official releases of Python 2.7.2 + NumPy 1.6.1 > > In Python 2.6 + NumPy 1.5.1 on win32, it works. > In Python 2.7.2 + NumPy 1.6.1 on win32, np.frompyfunc(add,2,1).accumulate > definitely gives an error. > > Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] > on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy as np > >>> def add(a,b): > ... return (a+b) > ... > >>> uadd = np.frompyfunc(add,2,1) > >>> uadd > > >>> uadd([1,2,3],[1,2,3]) > array([2, 4, 6], dtype=object) > >>> > >>> uadd.accumulate([1,2,3]) > Traceback (most recent call last): > File "", line 1, in > ValueError: could not find a matching type for add (vectorized).accumulate, > requested type has type code 'l' > >>> > > Aditya > > > 2011/9/19 St?fan van der Walt > >> On Mon, Sep 19, 2011 at 4:18 PM, Aditya Sethi >> wrote: >> > But uadd.accumulate(..) or uadd.reduce(..) fail with error: >> > ValueError: could not find a matching type for add >> (vectorized).accumulate >> > ( or (vectorized).reduce ) >> > Apologies, I should have been more clear before. >> >> In the development version: >> >> In [4]: uadd.accumulate([1,2,3]) >> Out[4]: array([1, 3, 6], dtype=object) >> >> In [5]: uadd.reduce([1,2,3]) >> Out[5]: 6 >> >> Regards >> St?fan >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ady.sethi at gmail.com Tue Sep 20 09:40:13 2011 From: ady.sethi at gmail.com (Aditya Sethi) Date: Tue, 20 Sep 2011 09:40:13 -0400 Subject: [Numpy-discussion] Upgrade to 1.6.x: frompyfunc() ufunc casting issue In-Reply-To: References: Message-ID: Understood. Thanks Olivier, Stefan. On Tue, Sep 20, 2011 at 9:18 AM, Olivier Delalleau wrote: > He is using the development version of Numpy, which is probably the main > difference (I tried it too with Numpy 1.6.1 on an x86_64 Linux architecture > and got the same bug). > If you want to use an official Numpy release you'll probably need to > downgrade to 1.5.x and wait until the next Numpy release. > > -=- Olivier > > > 2011/9/20 Aditya Sethi > >> Hi, >> >> Stefan, which version of Python and NumPy are you using? >> >> I am upgrading Python and NumPy, and would like to get it working on the >> official releases of Python 2.7.2 + NumPy 1.6.1 >> >> In Python 2.6 + NumPy 1.5.1 on win32, it works. >> In Python 2.7.2 + NumPy 1.6.1 on win32, np.frompyfunc(add,2,1).accumulate >> definitely gives an error. >> >> Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] >> on win32 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import numpy as np >> >>> def add(a,b): >> ... return (a+b) >> ... >> >>> uadd = np.frompyfunc(add,2,1) >> >>> uadd >> >> >>> uadd([1,2,3],[1,2,3]) >> array([2, 4, 6], dtype=object) >> >>> >> >>> uadd.accumulate([1,2,3]) >> Traceback (most recent call last): >> File "", line 1, in >> ValueError: could not find a matching type for add >> (vectorized).accumulate, requested type has type code 'l' >> >>> >> >> Aditya >> >> >> 2011/9/19 St?fan van der Walt >> >>> On Mon, Sep 19, 2011 at 4:18 PM, Aditya Sethi >>> wrote: >>> > But uadd.accumulate(..) or uadd.reduce(..) fail with error: >>> > ValueError: could not find a matching type for add >>> (vectorized).accumulate >>> > ( or (vectorized).reduce ) >>> > Apologies, I should have been more clear before. >>> >>> In the development version: >>> >>> In [4]: uadd.accumulate([1,2,3]) >>> Out[4]: array([1, 3, 6], dtype=object) >>> >>> In [5]: uadd.reduce([1,2,3]) >>> Out[5]: 6 >>> >>> Regards >>> St?fan >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dipo.elegbede at gmail.com Tue Sep 20 09:42:22 2011 From: dipo.elegbede at gmail.com (DIPO ELEGBEDE) Date: Tue, 20 Sep 2011 14:42:22 +0100 Subject: [Numpy-discussion] Dealing with arrays Message-ID: Hi All, I am extracting a list of lists from an application. In this case, my list has 12 lists. I then coverted each list into a numpy array. This returns 12 different arrays. Very well. The 12 different arrays are like as follows: [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 2] [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] [1 1] [1 1 1 1 1] [1 1 1] [1] [1] [1] [1 1] My next step is to work on individual arrays like count the numbers of 1s,2s,3s in each array, calculate the average count etc. I have been battling this for two days and nothing seems to be working, can anyone put me through? I would be most grateful. -- Dipo Elegbede,OCA 08033299270,08077682428 -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.cottrell at gmail.com Tue Sep 20 09:56:24 2011 From: david.cottrell at gmail.com (David Cottrell) Date: Tue, 20 Sep 2011 09:56:24 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: <919343ED-3F54-4B20-82BB-3267630579E3@samueljohn.de> References: <919343ED-3F54-4B20-82BB-3267630579E3@samueljohn.de> Message-ID: Thanks, just getting back to this. I just checked again, and after setting my LD_LIBRARY_PATH properly, ldd shows _dotblas.so pointing and the sunmath and sunperf libraries. However the test_03.py still runs at about 8-9 seconds ... far too slow. ~/local/lib/python3.1/site-packages/numpy/core $ ldd _dotblas.so | sed -e 's/$me/$USERNAME/g' libsunperf.so.8 => /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunperf.so.8 libsunmath.so.1 => /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunmath.so.1 libgcc_s.so.1 => /usr/sfw/lib/libgcc_s.so.1 libfsu.so.1 => /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfsu.so.1 libfui.so.2 => /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfui.so.2 libpicl.so.1 => /usr/lib/libpicl.so.1 libmtsk.so.1 => /lib/libmtsk.so.1 libm.so.2 => /lib/libm.so.2 libc.so.1 => /lib/libc.so.1 libm.so.1 => /lib/libm.so.1 libdl.so.1 => /lib/libdl.so.1 libdoor.so.1 => /lib/libdoor.so.1 libthread.so.1 => /lib/libthread.so.1 libkstat.so.1 => /lib/libkstat.so.1 libpthread.so.1 => /lib/libpthread.so.1 librt.so.1 => /lib/librt.so.1 libaio.so.1 => /lib/libaio.so.1 libmd.so.1 => /lib/libmd.so.1 /platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1 /platform/SUNW,Sun-Fire-V490/lib/libmd_psr.so.1 ~/local/lib/python3.1/site-packages/numpy/core $ ~/python/numpy/B/test_03.py No ATLAS: 8.49377894402 (1000, 1000) (1000,) (1000, 1000) On Wed, Sep 7, 2011 at 9:08 AM, Samuel John wrote: > > On 06.09.2011, at 22:13, David Cottrell wrote: > >> Thanks, I didn't realize dot was not just calling dgemm or some >> variant which I assume would be reasonably fast. I see dgemm appears >> in the numpy code in various places such as the lapack_lite module. >> >> I ran the svd test on the solaris setup and will check the OSX run >> when back at my laptop. 8.4 seconds is slightly slower than matlab but >> still seems reasonable. >> >> $ ./test_03.py >> No ATLAS: >> (1000, 1000) (1000,) (1000, 1000) >> 8.17235898972 > > I just ran your benchmark code on OSX 10.7.1 on an 2011 MacBook Pro (core-i7) with numpy.version.version '2.0.0.dev-900d82e': > ? Using ATLAS: > ? ((1000, 1000), (1000,), (1000, 1000)) > ? 0.908223152161 > > cheers, > ?Samuel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- David Cottrell +1 416 995 9860 http://ca.linkedin.com/in/dcottrell From shish at keba.be Tue Sep 20 09:56:45 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 20 Sep 2011 09:56:45 -0400 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: References: Message-ID: Let's say your "list of lists" is called my_list, and assuming you are using python 2.7+: import collections import numpy for tmp_list in my_list: array = numpy.array(tmp_list) counts = collections.Counter(array) print counts Note that you don't even need to use numpy for that (you can use Counter on tmp_list directly), but you may want to do other computations that require numpy... Not sure what you mean by "average count", so can't help with that. -=- Olivier 2011/9/20 DIPO ELEGBEDE > Hi All, > > I am extracting a list of lists from an application. In this case, my list > has 12 lists. > I then coverted each list into a numpy array. This returns 12 different > arrays. Very well. The 12 different arrays are like as follows: > [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 2 > 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 > 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 1 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 > 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 > 1 2 2 2 2 2] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] > [1 1] > [1 1 1 1 1] > [1 1 1] > [1] > [1] > [1] > [1 1] > My next step is to work on individual arrays like count the numbers of > 1s,2s,3s in each array, calculate the average count etc. > I have been battling this for two days and nothing seems to be working, can > anyone put me through? > I would be most grateful. > > -- > Dipo Elegbede,OCA > 08033299270,08077682428 > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dipo.elegbede at gmail.com Tue Sep 20 09:58:55 2011 From: dipo.elegbede at gmail.com (dipo.elegbede at gmail.com) Date: Tue, 20 Sep 2011 13:58:55 +0000 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: References: Message-ID: <771550374-1316527133-cardhu_decombobulator_blackberry.rim.net-34530236-@b5.c12.bise7.blackberry> Thanks. There are other computations I have to do with numpy, hence the use. Second, I am on python 2.6. Any help? Sent from my BlackBerry wireless device from MTN -----Original Message----- From: Olivier Delalleau Sender: numpy-discussion-bounces at scipy.org Date: Tue, 20 Sep 2011 09:56:45 To: Discussion of Numerical Python Reply-To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Dealing with arrays _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From chico at cnpmf.embrapa.br Tue Sep 20 10:00:08 2011 From: chico at cnpmf.embrapa.br (chico at cnpmf.embrapa.br) Date: Tue, 20 Sep 2011 11:00:08 -0300 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: References: Message-ID: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> What about: >>> listoflists = [[1,1,1], [1,2,3], [2,2,2]] >>> for eachlist in listoflists: ... print eachlist.count(1), eachlist.count(2), eachlist.count(3) ... 3 0 0 1 1 1 0 3 0 >>> Chico > Hi All, > > I am extracting a list of lists from an application. In this case, my list > has 12 lists. > I then coverted each list into a numpy array. This returns 12 different > arrays. Very well. The 12 different arrays are like as follows: > [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 2 > 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 > 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 1 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 > 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 > 1 2 2 2 2 2] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] > [1 1] > [1 1 1 1 1] > [1 1 1] > [1] > [1] > [1] > [1 1] > My next step is to work on individual arrays like count the numbers of > 1s,2s,3s in each array, calculate the average count etc. > I have been battling this for two days and nothing seems to be working, > can > anyone put me through? > I would be most grateful. > > -- > Dipo Elegbede,OCA > 08033299270,08077682428 > From dipo.elegbede at gmail.com Tue Sep 20 10:06:10 2011 From: dipo.elegbede at gmail.com (dipo.elegbede at gmail.com) Date: Tue, 20 Sep 2011 14:06:10 +0000 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> Message-ID: <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> Hi Chico, my list needs to be converted to a numpy array first before I do the count. Some other calculations rely on the count and its going to be built all on numpy. I have tried the collections.Counter() option but in python 2.6, the module collections has no Counter function. Anymore suggestions please. Sent from my BlackBerry wireless device from MTN -----Original Message----- From: chico at cnpmf.embrapa.br Sender: numpy-discussion-bounces at scipy.org Date: Tue, 20 Sep 2011 11:00:08 To: Discussion of Numerical Python Reply-To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Dealing with arrays What about: >>> listoflists = [[1,1,1], [1,2,3], [2,2,2]] >>> for eachlist in listoflists: ... print eachlist.count(1), eachlist.count(2), eachlist.count(3) ... 3 0 0 1 1 1 0 3 0 >>> Chico > Hi All, > > I am extracting a list of lists from an application. In this case, my list > has 12 lists. > I then coverted each list into a numpy array. This returns 12 different > arrays. Very well. The 12 different arrays are like as follows: > [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 2 > 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 > 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 > 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 1 > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 > 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 > 1 2 2 2 2 2] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 1 1 1 1 1 1 1 1 1] > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] > [1 1] > [1 1 1 1 1] > [1 1 1] > [1] > [1] > [1] > [1 1] > My next step is to work on individual arrays like count the numbers of > 1s,2s,3s in each array, calculate the average count etc. > I have been battling this for two days and nothing seems to be working, > can > anyone put me through? > I would be most grateful. > > -- > Dipo Elegbede,OCA > 08033299270,08077682428 > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From glenn.caltech at gmail.com Tue Sep 20 10:11:17 2011 From: glenn.caltech at gmail.com (G Jones) Date: Tue, 20 Sep 2011 07:11:17 -0700 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> Message-ID: If you know the values that you want to count, you could just do: (data_array == value).sum() to find the number of times that "value" occurs in "data_array". You could use np.unique(data_array) to find the unique values and then count the number of occurrences of each value. On Tue, Sep 20, 2011 at 7:06 AM, wrote: > Hi Chico, my list needs to be converted to a numpy array first before I do > the count. Some other calculations rely on the count and its going to be > built all on numpy. > > I have tried the collections.Counter() option but in python 2.6, the module > collections has no Counter function. > > Anymore suggestions please. > Sent from my BlackBerry wireless device from MTN > > -----Original Message----- > From: chico at cnpmf.embrapa.br > Sender: numpy-discussion-bounces at scipy.org > Date: Tue, 20 Sep 2011 11:00:08 > To: Discussion of Numerical Python > Reply-To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Dealing with arrays > > What about: > > >>> listoflists = [[1,1,1], [1,2,3], [2,2,2]] > >>> for eachlist in listoflists: > ... print eachlist.count(1), eachlist.count(2), eachlist.count(3) > ... > 3 0 0 > 1 1 1 > 0 3 0 > >>> > > Chico > > > > > Hi All, > > > > I am extracting a list of lists from an application. In this case, my > list > > has 12 lists. > > I then coverted each list into a numpy array. This returns 12 different > > arrays. Very well. The 12 different arrays are like as follows: > > [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 > 2 > > 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 > 2 > > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 1 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 > 2 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 > 2 > > 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 > 1 > > 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 > 2 > > 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 > 2 > > 1 2 2 2 2 2] > > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1] > > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1] > > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > > 1 1 1 1 1 1 1 1 1 1] > > [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] > > [1 1] > > [1 1 1 1 1] > > [1 1 1] > > [1] > > [1] > > [1] > > [1 1] > > My next step is to work on individual arrays like count the numbers of > > 1s,2s,3s in each array, calculate the average count etc. > > I have been battling this for two days and nothing seems to be working, > > can > > anyone put me through? > > I would be most grateful. > > > > -- > > Dipo Elegbede,OCA > > 08033299270,08077682428 > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Sep 20 10:11:42 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 20 Sep 2011 10:11:42 -0400 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> Message-ID: On Tue, Sep 20, 2011 at 10:06 AM, wrote: > Hi Chico, my list needs to be converted to a numpy array first before I do the count. Some other calculations rely on the count and its going to be built all on numpy. > > I have tried the collections.Counter() option but in python 2.6, the module collections has no Counter function. > > Anymore suggestions please. > Sent from my BlackBerry wireless device from MTN > > -----Original Message----- > From: chico at cnpmf.embrapa.br > Sender: numpy-discussion-bounces at scipy.org > Date: Tue, 20 Sep 2011 11:00:08 > To: Discussion of Numerical Python > Reply-To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Dealing with arrays > > What about: > >>>> listoflists = [[1,1,1], [1,2,3], [2,2,2]] >>>> for eachlist in listoflists: > ... ? ? print eachlist.count(1), eachlist.count(2), eachlist.count(3) > ... > 3 0 0 > 1 1 1 > 0 3 0 >>>> > > Chico > > > >> Hi All, >> >> I am extracting a list of lists from an application. In this case, my list >> has 12 lists. >> I then coverted each list into a numpy array. This returns 12 different >> arrays. Very well. The 12 different arrays are like as follows: >> [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 >> ?2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 2 >> ?2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 >> ?2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 1 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 >> ?2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 >> ?1 2 2 2 2 2] >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1] >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1] >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1] >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] >> [1 1] >> [1 1 1 1 1] >> [1 1 1] >> [1] >> [1] >> [1] >> [1 1] >> My next step is to work on individual arrays like count the numbers of >> 1s,2s,3s in each array, calculate the average count etc. >> I have been battling this for two days and nothing seems to be working, >> can >> anyone put me through? >> I would be most grateful. for counting integers, np.bincount works well, bincount internally converts list to array >>> import numpy as np >>> >>> lili = [range(i) for i in range(2,5)] >>> for li in lili: count = np.bincount(li) print count, count/float(len(li)) [1 1] [ 0.5 0.5] [1 1 1] [ 0.33333333 0.33333333 0.33333333] [1 1 1 1] [ 0.25 0.25 0.25 0.25] Josef >> >> -- >> Dipo Elegbede,OCA >> 08033299270,08077682428 >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dipo.elegbede at gmail.com Tue Sep 20 10:27:15 2011 From: dipo.elegbede at gmail.com (dipo.elegbede at gmail.com) Date: Tue, 20 Sep 2011 14:27:15 +0000 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br><259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> Message-ID: <469182246-1316528834-cardhu_decombobulator_blackberry.rim.net-1607792423-@b5.c12.bise7.blackberry> Thanks Josef. This seems like a way to go except that it is giving extra values I don't need. For instance, it is also returning a count for 0s and as it were, I don't have zeros at all in my list. Second, I want a situation where, I can have a dictionary that has the figure and number of time it appears. E.g dict={1:69,2:40,3:10} Thanks for the help. More suggestions please. I'm open to inspecting as many options to get this working. Thanks. Sent from my BlackBerry wireless device from MTN -----Original Message----- From: josef.pktd at gmail.com Sender: numpy-discussion-bounces at scipy.org Date: Tue, 20 Sep 2011 10:11:42 To: Discussion of Numerical Python Reply-To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Dealing with arrays On Tue, Sep 20, 2011 at 10:06 AM, wrote: > Hi Chico, my list needs to be converted to a numpy array first before I do the count. Some other calculations rely on the count and its going to be built all on numpy. > > I have tried the collections.Counter() option but in python 2.6, the module collections has no Counter function. > > Anymore suggestions please. > Sent from my BlackBerry wireless device from MTN > > -----Original Message----- > From: chico at cnpmf.embrapa.br > Sender: numpy-discussion-bounces at scipy.org > Date: Tue, 20 Sep 2011 11:00:08 > To: Discussion of Numerical Python > Reply-To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Dealing with arrays > > What about: > >>>> listoflists = [[1,1,1], [1,2,3], [2,2,2]] >>>> for eachlist in listoflists: > ... ? ? print eachlist.count(1), eachlist.count(2), eachlist.count(3) > ... > 3 0 0 > 1 1 1 > 0 3 0 >>>> > > Chico > > > >> Hi All, >> >> I am extracting a list of lists from an application. In this case, my list >> has 12 lists. >> I then coverted each list into a numpy array. This returns 12 different >> arrays. Very well. The 12 different arrays are like as follows: >> [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 >> ?2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 2 >> ?2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 >> ?2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 >> ?2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 1 >> ?2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 >> ?2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 >> ?1 2 2 2 2 2] >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1] >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1] >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> ?1 1 1 1 1 1 1 1 1 1] >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] >> [1 1] >> [1 1 1 1 1] >> [1 1 1] >> [1] >> [1] >> [1] >> [1 1] >> My next step is to work on individual arrays like count the numbers of >> 1s,2s,3s in each array, calculate the average count etc. >> I have been battling this for two days and nothing seems to be working, >> can >> anyone put me through? >> I would be most grateful. for counting integers, np.bincount works well, bincount internally converts list to array >>> import numpy as np >>> >>> lili = [range(i) for i in range(2,5)] >>> for li in lili: count = np.bincount(li) print count, count/float(len(li)) [1 1] [ 0.5 0.5] [1 1 1] [ 0.33333333 0.33333333 0.33333333] [1 1 1 1] [ 0.25 0.25 0.25 0.25] Josef >> >> -- >> Dipo Elegbede,OCA >> 08033299270,08077682428 >> > > >_______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >_______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From yversley at gmail.com Tue Sep 20 10:42:18 2011 From: yversley at gmail.com (Yannick Versley) Date: Tue, 20 Sep 2011 16:42:18 +0200 Subject: [Numpy-discussion] sparse vectors / matrices / tensors Message-ID: Hi all, I've been working quite a lot with sparse vectors and sparse matrices (basically as feature vectors in the context of machine learning), and have noticed that they do crop up in a lot of places (e.g. the CVXOPT library, in scikits, ...) and that people tend to either reinvent the wheel (i.e. implement a complete sparse matrix library) or pretend that no separate data structure is needed (i.e. always passing along pairs of coordinate and data arrays). I do think there would be some benefit to having sparse vectors/matrices or tensors (in parallel to numpy's arrays, which can be vectors or arbitrary-order tensors) with a standardized interface so that different packages (e.g. eigenvalue/SVD computation, least-squares and other QPs, but possibly also things like numpy.bincount as well as computations that are more domain-specific) can be more interoperable than they are now. One problem that I see is that people doing PDE solving usually want banded matrices, whereas other people (including me) do most of their work with coordinate-list or CSR matrices, which normally means some variation in the actual implementations for different domains, and it's also possible that the most convenient interface for a sparse matrix is not the most convenient one for a dense matrix (and vice-versa), but I think it would be nice if there were some kind of standardized data structure or maybe just a standardized vtable-based interface (similar to Python's buffer interface) that would allow all sparse matrix packages to interoperate with each other in some meaningful (even if not most-efficient) way. I'd be willing to adapt the code that I have (C++- and Cython-based) to this kind of interface to provide some kind of 'reference' implementation, but before inventing the N+1th solution to the problem, I'd be curious what other people's opinions are. Yannick -------------- next part -------------- An HTML attachment was scrubbed... URL: From glenn.caltech at gmail.com Tue Sep 20 10:44:59 2011 From: glenn.caltech at gmail.com (G Jones) Date: Tue, 20 Sep 2011 07:44:59 -0700 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: <469182246-1316528834-cardhu_decombobulator_blackberry.rim.net-1607792423-@b5.c12.bise7.blackberry> References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> <469182246-1316528834-cardhu_decombobulator_blackberry.rim.net-1607792423-@b5.c12.bise7.blackberry> Message-ID: data = [[1,2,1,1,4,2,1], [1,2,1,1,4,2,1,2,2,2,1,1,1],[1],[2]] def count_dict(arr): return dict([(x,(arr==x).sum()) for x in np.unique(arr)]) [count_dict(x) for x in data] yields: [{1: 4, 2: 2, 4: 1}, {1: 7, 2: 5, 4: 1}, {1: 1}, {2: 1}] not efficient, but it works On Tue, Sep 20, 2011 at 7:27 AM, wrote: > Thanks Josef. > This seems like a way to go except that it is giving extra values I don't > need. > > For instance, it is also returning a count for 0s and as it were, I don't > have zeros at all in my list. > > Second, I want a situation where, I can have a dictionary that has the > figure and number of time it appears. E.g dict={1:69,2:40,3:10} > > Thanks for the help. More suggestions please. I'm open to inspecting as > many options to get this working. > > Thanks. > Sent from my BlackBerry wireless device from MTN > > -----Original Message----- > From: josef.pktd at gmail.com > Sender: numpy-discussion-bounces at scipy.org > Date: Tue, 20 Sep 2011 10:11:42 > To: Discussion of Numerical Python > Reply-To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Dealing with arrays > > On Tue, Sep 20, 2011 at 10:06 AM, wrote: > > Hi Chico, my list needs to be converted to a numpy array first before I > do the count. Some other calculations rely on the count and its going to be > built all on numpy. > > > > I have tried the collections.Counter() option but in python 2.6, the > module collections has no Counter function. > > > > Anymore suggestions please. > > Sent from my BlackBerry wireless device from MTN > > > > -----Original Message----- > > From: chico at cnpmf.embrapa.br > > Sender: numpy-discussion-bounces at scipy.org > > Date: Tue, 20 Sep 2011 11:00:08 > > To: Discussion of Numerical Python > > Reply-To: Discussion of Numerical Python > > Subject: Re: [Numpy-discussion] Dealing with arrays > > > > What about: > > > >>>> listoflists = [[1,1,1], [1,2,3], [2,2,2]] > >>>> for eachlist in listoflists: > > ... print eachlist.count(1), eachlist.count(2), eachlist.count(3) > > ... > > 3 0 0 > > 1 1 1 > > 0 3 0 > >>>> > > > > Chico > > > > > > > >> Hi All, > >> > >> I am extracting a list of lists from an application. In this case, my > list > >> has 12 lists. > >> I then coverted each list into a numpy array. This returns 12 different > >> arrays. Very well. The 12 different arrays are like as follows: > >> [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 > 2 > >> 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 > 2 > >> 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 1 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 > 2 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 > 2 > >> 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 > 1 > >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 > 2 > >> 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 > 2 > >> 1 2 2 2 2 2] > >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1] > >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1] > >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > 1 > >> 1 1 1 1 1 1 1 1 1 1] > >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] > >> [1 1] > >> [1 1 1 1 1] > >> [1 1 1] > >> [1] > >> [1] > >> [1] > >> [1 1] > >> My next step is to work on individual arrays like count the numbers of > >> 1s,2s,3s in each array, calculate the average count etc. > >> I have been battling this for two days and nothing seems to be working, > >> can > >> anyone put me through? > >> I would be most grateful. > > > for counting integers, np.bincount works well, bincount internally > converts list to array > > >>> import numpy as np > >>> > >>> lili = [range(i) for i in range(2,5)] > >>> for li in lili: > count = np.bincount(li) > print count, count/float(len(li)) > > > [1 1] [ 0.5 0.5] > [1 1 1] [ 0.33333333 0.33333333 0.33333333] > [1 1 1 1] [ 0.25 0.25 0.25 0.25] > > Josef > > >> > >> -- > >> Dipo Elegbede,OCA > >> 08033299270,08077682428 > >> > > > > > >_______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >_______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjordan1 at uw.edu Tue Sep 20 10:51:52 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Tue, 20 Sep 2011 10:51:52 -0400 Subject: [Numpy-discussion] sparse vectors / matrices / tensors In-Reply-To: References: Message-ID: On Tue, Sep 20, 2011 at 10:42 AM, Yannick Versley wrote: > Hi all, > I've been working quite a lot with sparse vectors and sparse matrices > (basically > as feature vectors in the context of machine learning), and have noticed > that they > do crop up in a lot of places (e.g. the CVXOPT library, in scikits, ...) and > that people > tend to either reinvent the wheel (i.e. implement a complete sparse matrix > library) or > pretend that no separate data structure is needed (i.e. always passing along > pairs of > coordinate and data arrays). > I do think there would be some benefit to having sparse vectors/matrices or > tensors > (in parallel to numpy's arrays, which can be vectors or arbitrary-order > tensors) with a > standardized interface so that different packages (e.g. eigenvalue/SVD > computation, > least-squares and other QPs, but possibly also things like numpy.bincount as > well as > computations that are more domain-specific) can be?more interoperable than > they are now. > One problem that I see is that people doing PDE solving usually want banded > matrices, > whereas other people (including me) do most of their work with > coordinate-list or CSR > matrices, which normally means some variation in the actual implementations > for different > domains, and it's also?possible that the most convenient interface for a > sparse matrix is not > the most convenient?one for a dense matrix (and vice-versa), but I think it > would be nice > if there were some kind?of standardized data structure or maybe just a > standardized vtable-based > interface (similar?to Python's buffer interface) that would allow all sparse > matrix packages to > interoperate with?each other in some meaningful (even if not most-efficient) > way. > I'd be willing to adapt the code that I have (C++- and Cython-based) to this > kind of interface > to provide some kind of 'reference' implementation, but before inventing the > N+1th solution > to the problem, I'd be curious what other people's opinions are. > Yannick > This question seems like it's more relevant to the scipy mailing list, since scipy has scipy.sparse. -Chris JS > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From scipy at samueljohn.de Tue Sep 20 10:52:48 2011 From: scipy at samueljohn.de (Samuel John) Date: Tue, 20 Sep 2011 16:52:48 +0200 Subject: [Numpy-discussion] OS X Lion: llvm: numpy and scipy In-Reply-To: References: <4BD955EF-0F08-4979-9A1C-0E52F91D76AB@samueljohn.de> <2055622E-64A8-4116-9FDC-496AAC76917D@samueljohn.de> Message-ID: Hi! On 20.09.2011, at 14:41, David Cournapeau wrote: > On Tue, Sep 20, 2011 at 5:13 AM, Samuel John wrote: >> Ralf, thanks for your answer. >> >> However, in short: >> >> I want `pip install numpy; pip install scipy` to work on OS X Lion without extra effort :-) [...] > I will try to look at this problem next week, when I will receive a > new laptop with Lion on it. If I forget about it, please ping me at > the end of next week, we need to fix this, Congratulation to your new Mac :-) When I download scipy.10.0b2 and get gfortran via homebrew: brew install gfortran (which is not the issue here) cd scipy-0.10.0b2 python setup.py build python setup.py install Then, scipy.test() causes segfaults or malloc errors: > samuel at ubi:~/Downloads/scipy-0.10.0b2 $ cd .. > samuel at ubi:~/Downloads $ ipython > Python 2.7.2 (default, Sep 16 2011, 11:18:55) > Type "copyright", "credits" or "license" for more information. > > IPython 0.11 -- An enhanced Interactive Python. > ? -> Introduction and overview of IPython's features. > %quickref -> Quick reference. > help -> Python's own help system. > object? -> Details about 'object', use 'object??' for extra details. > > In [1]: import scipy > > In [2]: scipy.test() > Running unit tests for scipy > NumPy version 1.6.1 > NumPy is installed in /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy > SciPy version 0.10.0b2 > SciPy is installed in /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy > Python version 2.7.2 (default, Sep 16 2011, 11:18:55) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] > nose version 1.1.2 > ...................................................................................................................................................................................F.FSegmentation fault: 11 > samuel at ubi:~/Downloads $ ipython > Python 2.7.2 (default, Sep 16 2011, 11:18:55) > Type "copyright", "credits" or "license" for more information. > > IPython 0.11 -- An enhanced Interactive Python. > ? -> Introduction and overview of IPython's features. > %quickref -> Quick reference. > help -> Python's own help system. > object? -> Details about 'object', use 'object??' for extra details. > > In [1]: import scipy > > In [2]: scipy.test() > Running unit tests for scipy > NumPy version 1.6.1 > NumPy is installed in /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy > SciPy version 0.10.0b2 > SciPy is installed in /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy > Python version 2.7.2 (default, Sep 16 2011, 11:18:55) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] > nose version 1.1.2 > ...................................................................................................................................................................................F.FFPython(93907,0x7fff7201b960) malloc: *** error for object 0x105ce5630: pointer being freed was not allocated > *** set a breakpoint in malloc_error_break to debug > Abort trap: 6 However, when setting the CC, CXX and FFLAGS explicitly to avoid llvm: export CC=gcc-4.2 export CXX=g++-4.2 export FFLAGS=-ff2c python setup.py build python setup.py install Then scipy.test() works fine: > In [2]: scipy.test() > Running unit tests for scipy > NumPy version 1.6.1 > NumPy is installed in /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy > SciPy version 0.10.0b2 > SciPy is installed in /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy > Python version 2.7.2 (default, Sep 16 2011, 11:18:55) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] > nose version 1.1.2 > ............................................................................................................................................................................................................................K............................................................................................................/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/interpolate/fitpack2.py:674: UserWarning: > The coefficients of the spline returned have been computed as the > minimal norm least-squares solution of a (numerically) rank deficient > system (deficiency=7). If deficiency is large, the results may be > inaccurate. Deficiency may strongly depend on the value of eps. > warnings.warn(message) > ....../usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/interpolate/fitpack2.py:605: UserWarning: > The required storage space exceeds the available storage space: nxest > or nyest too small, or s too small. > The weighted least-squares spline corresponds to the current set of > knots. > warnings.warn(message) > ........................K..K....../usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numeric.py:1920: RuntimeWarning: invalid value encountered in absolute > return all(less_equal(absolute(x-y), atol + rtol * absolute(y))) > ............................................................................................................................................................................................................................................................................................................................................................................................................................................/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/wavfile.py:31: WavFileWarning: Unfamiliar format bytes > warnings.warn("Unfamiliar format bytes", WavFileWarning) > /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/wavfile.py:121: WavFileWarning: chunk not understood > warnings.warn("chunk not understood", WavFileWarning) > ...............................................................................................................................................................................................................................SSSSSS......SSSSSS......SSSS...............................................................................S............................................................................................................................................................................................................................................................K..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SSSSSSSSSSS.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K...............................................................K...........................................................................................................................................................KK.............................................................................................................................................................................................................................................................................................................................................................................................................................................K.K.............................................................................................................................................................................................................................................................................................................................................................................................K........K..............SSSSSSS............................................................................................................................................................................................................................................................................................................................................................................................../usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/stats/distributions.py:4110: RuntimeWarning: overflow encountered in exp > return exp(c*x-exp(x)-gamln(c)) > ................................./usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/stats/distributions.py:4592: RuntimeWarning: invalid value encountered in sqrt > vals = 2*(bt+1.0)*sqrt(b-2.0)/((b-3.0)*sqrt(b)) > .........................................................................................................................................................................................................................................................................................................S................................................................................................................................................................................................/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/stats/distributions.py:5107: RuntimeWarning: divide by zero encountered in divide > return where((lam <= 0) | (abs(x) < 1.0/arr(lam)), Px, 0.0) > ....................................................................................................................................................................................................................................................................................................................................................................................................................... > ---------------------------------------------------------------------- > Ran 5651 tests in 53.627s > > OK (KNOWNFAIL=12, SKIP=36) > Out[2]: Samuel From shish at keba.be Tue Sep 20 11:24:21 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 20 Sep 2011 11:24:21 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: <919343ED-3F54-4B20-82BB-3267630579E3@samueljohn.de> Message-ID: The blas implementation you are using may be slow. Here's my ldd on _dotblas.so, that shows it is using libblas (this is on Ubuntu 11.04): linux-vdso.so.1 => (0x00007fffad5ff000) libblas.so.3gf => /usr/lib/libblas.so.3gf (0x00007fc608ea4000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so. 6 (0x00007fc608b10000) libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007fc60882b000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc6085a6000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc608390000) /lib64/ld-linux-x86-64.so.2 (0x00007fc609352000) My guess is BLAS is implemented in libsunmath.so but is a slow implementation. It's really just a guess though... -=- Olivier 2011/9/20 David Cottrell > Thanks, just getting back to this. I just checked again, and after > setting my LD_LIBRARY_PATH properly, ldd shows _dotblas.so pointing > and the sunmath and sunperf libraries. However the test_03.py still > runs at about 8-9 seconds ... far too slow. > > ~/local/lib/python3.1/site-packages/numpy/core $ ldd _dotblas.so | sed > -e 's/$me/$USERNAME/g' > libsunperf.so.8 => > > /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunperf.so.8 > libsunmath.so.1 => > > /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunmath.so.1 > libgcc_s.so.1 => /usr/sfw/lib/libgcc_s.so.1 > libfsu.so.1 => > > /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfsu.so.1 > libfui.so.2 => > > /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfui.so.2 > libpicl.so.1 => /usr/lib/libpicl.so.1 > libmtsk.so.1 => /lib/libmtsk.so.1 > libm.so.2 => /lib/libm.so.2 > libc.so.1 => /lib/libc.so.1 > libm.so.1 => /lib/libm.so.1 > libdl.so.1 => /lib/libdl.so.1 > libdoor.so.1 => /lib/libdoor.so.1 > libthread.so.1 => /lib/libthread.so.1 > libkstat.so.1 => /lib/libkstat.so.1 > libpthread.so.1 => /lib/libpthread.so.1 > librt.so.1 => /lib/librt.so.1 > libaio.so.1 => /lib/libaio.so.1 > libmd.so.1 => /lib/libmd.so.1 > /platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1 > /platform/SUNW,Sun-Fire-V490/lib/libmd_psr.so.1 > > ~/local/lib/python3.1/site-packages/numpy/core $ > ~/python/numpy/B/test_03.py > No ATLAS: > 8.49377894402 > (1000, 1000) (1000,) (1000, 1000) > > > On Wed, Sep 7, 2011 at 9:08 AM, Samuel John wrote: > > > > On 06.09.2011, at 22:13, David Cottrell wrote: > > > >> Thanks, I didn't realize dot was not just calling dgemm or some > >> variant which I assume would be reasonably fast. I see dgemm appears > >> in the numpy code in various places such as the lapack_lite module. > >> > >> I ran the svd test on the solaris setup and will check the OSX run > >> when back at my laptop. 8.4 seconds is slightly slower than matlab but > >> still seems reasonable. > >> > >> $ ./test_03.py > >> No ATLAS: > >> (1000, 1000) (1000,) (1000, 1000) > >> 8.17235898972 > > > > I just ran your benchmark code on OSX 10.7.1 on an 2011 MacBook Pro > (core-i7) with numpy.version.version '2.0.0.dev-900d82e': > > Using ATLAS: > > ((1000, 1000), (1000,), (1000, 1000)) > > 0.908223152161 > > > > cheers, > > Samuel > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > David Cottrell > +1 416 995 9860 > http://ca.linkedin.com/in/dcottrell > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Tue Sep 20 11:30:59 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 20 Sep 2011 11:30:59 -0400 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: <771550374-1316527133-cardhu_decombobulator_blackberry.rim.net-34530236-@b5.c12.bise7.blackberry> References: <771550374-1316527133-cardhu_decombobulator_blackberry.rim.net-34530236-@b5.c12.bise7.blackberry> Message-ID: Here's another python 2.6 version: import itertools import numpy for tmp_list in my_list: array = numpy.array(tmp_list) counts = dict((x, len(list(c))) for x, c in itertools.groupby(sorted(array))) print counts 2011/9/20 > Thanks. > There are other computations I have to do with numpy, hence the use. > Second, I am on python 2.6. > Any help? > > Sent from my BlackBerry wireless device from MTN > > -----Original Message----- > From: Olivier Delalleau > Sender: numpy-discussion-bounces at scipy.org > Date: Tue, 20 Sep 2011 09:56:45 > To: Discussion of Numerical Python > Reply-To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Dealing with arrays > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yversley at gmail.com Tue Sep 20 11:33:33 2011 From: yversley at gmail.com (Yannick Versley) Date: Tue, 20 Sep 2011 17:33:33 +0200 Subject: [Numpy-discussion] sparse vectors / matrices / tensors In-Reply-To: References: Message-ID: > > This question seems like it's more relevant to the scipy mailing list, > since scipy has scipy.sparse. > My point was that the current situation (i.e., people reinventing the wheel) was a by-product of the fact that there's nothing as standardized as numpy's ndarray or Python's buffer interface, if only for the simple reason that most people only ever work with dense matrices. (scipy.sparse uses several different data structures that are all limited to two axes, and have no common low-level interface). I'll be happy to ask on the scipy list, if that's where the discussion should be. Yannick -------------- next part -------------- An HTML attachment was scrubbed... URL: From dipo.elegbede at gmail.com Tue Sep 20 11:37:48 2011 From: dipo.elegbede at gmail.com (DIPO ELEGBEDE) Date: Tue, 20 Sep 2011 16:37:48 +0100 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> <469182246-1316528834-cardhu_decombobulator_blackberry.rim.net-1607792423-@b5.c12.bise7.blackberry> Message-ID: Thanks all for the very helpful answers. These answers all appear to have helped me in different ways especially in learning something new. I guess I didn't ask the right question after all. What I really want is a way to loop through individual arrays of the 12 arrays that I have. Here are my steps: 1. derive a list of lists. (I have done that) 2. loop through the list to convert each list to a numpy array. (I did that too) Information: There are 12 lists in my list, so I ended up with 12 different arrays. 3. Get each array one after the other and work with it. (This is where the problem is). How can I fetch out array1, then array2 then array3 up to array12? Is it possible to slice? Once I am able to do that, I should be able to dance my way through the rest with the preliminary information you have given earlier. Here is my arrays again [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 2 2] [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1] [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] [1 1] [1 1 1 1 1] [1 1 1] [1] [1] [1] [1 1] Thanks for your kindness. On 20 September 2011 15:44, G Jones wrote: > data = [[1,2,1,1,4,2,1], [1,2,1,1,4,2,1,2,2,2,1,1,1],[1],[2]] > def count_dict(arr): > return dict([(x,(arr==x).sum()) for x in np.unique(arr)]) > > [count_dict(x) for x in data] > > yields: > [{1: 4, 2: 2, 4: 1}, {1: 7, 2: 5, 4: 1}, {1: 1}, {2: 1}] > > not efficient, but it works > > > On Tue, Sep 20, 2011 at 7:27 AM, wrote: > >> Thanks Josef. >> This seems like a way to go except that it is giving extra values I don't >> need. >> >> For instance, it is also returning a count for 0s and as it were, I don't >> have zeros at all in my list. >> >> Second, I want a situation where, I can have a dictionary that has the >> figure and number of time it appears. E.g dict={1:69,2:40,3:10} >> >> Thanks for the help. More suggestions please. I'm open to inspecting as >> many options to get this working. >> >> Thanks. >> Sent from my BlackBerry wireless device from MTN >> >> -----Original Message----- >> From: josef.pktd at gmail.com >> Sender: numpy-discussion-bounces at scipy.org >> Date: Tue, 20 Sep 2011 10:11:42 >> To: Discussion of Numerical Python >> Reply-To: Discussion of Numerical Python >> Subject: Re: [Numpy-discussion] Dealing with arrays >> >> On Tue, Sep 20, 2011 at 10:06 AM, wrote: >> > Hi Chico, my list needs to be converted to a numpy array first before I >> do the count. Some other calculations rely on the count and its going to be >> built all on numpy. >> > >> > I have tried the collections.Counter() option but in python 2.6, the >> module collections has no Counter function. >> > >> > Anymore suggestions please. >> > Sent from my BlackBerry wireless device from MTN >> > >> > -----Original Message----- >> > From: chico at cnpmf.embrapa.br >> > Sender: numpy-discussion-bounces at scipy.org >> > Date: Tue, 20 Sep 2011 11:00:08 >> > To: Discussion of Numerical Python >> > Reply-To: Discussion of Numerical Python >> > Subject: Re: [Numpy-discussion] Dealing with arrays >> > >> > What about: >> > >> >>>> listoflists = [[1,1,1], [1,2,3], [2,2,2]] >> >>>> for eachlist in listoflists: >> > ... print eachlist.count(1), eachlist.count(2), eachlist.count(3) >> > ... >> > 3 0 0 >> > 1 1 1 >> > 0 3 0 >> >>>> >> > >> > Chico >> > >> > >> > >> >> Hi All, >> >> >> >> I am extracting a list of lists from an application. In this case, my >> list >> >> has 12 lists. >> >> I then coverted each list into a numpy array. This returns 12 different >> >> arrays. Very well. The 12 different arrays are like as follows: >> >> [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 1 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 >> 1 2 >> >> 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 1 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 >> 2 2 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 >> 2 2 >> >> 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 >> 2 1 >> >> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 >> 2 2 >> >> 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 1 2 >> 2 2 >> >> 1 2 2 2 2 2] >> >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1] >> >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1] >> >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> 1 1 >> >> 1 1 1 1 1 1 1 1 1 1] >> >> [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1] >> >> [1 1] >> >> [1 1 1 1 1] >> >> [1 1 1] >> >> [1] >> >> [1] >> >> [1] >> >> [1 1] >> >> My next step is to work on individual arrays like count the numbers of >> >> 1s,2s,3s in each array, calculate the average count etc. >> >> I have been battling this for two days and nothing seems to be working, >> >> can >> >> anyone put me through? >> >> I would be most grateful. >> >> >> for counting integers, np.bincount works well, bincount internally >> converts list to array >> >> >>> import numpy as np >> >>> >> >>> lili = [range(i) for i in range(2,5)] >> >>> for li in lili: >> count = np.bincount(li) >> print count, count/float(len(li)) >> >> >> [1 1] [ 0.5 0.5] >> [1 1 1] [ 0.33333333 0.33333333 0.33333333] >> [1 1 1 1] [ 0.25 0.25 0.25 0.25] >> >> Josef >> >> >> >> >> -- >> >> Dipo Elegbede,OCA >> >> 08033299270,08077682428 >> >> >> > >> > >> >_______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >_______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Dipo Elegbede,OCA 08033299270,08077682428 -------------- next part -------------- An HTML attachment was scrubbed... URL: From poust at berkeley.edu Tue Sep 20 11:38:24 2011 From: poust at berkeley.edu (Sean Poust) Date: Tue, 20 Sep 2011 15:38:24 +0000 (UTC) Subject: [Numpy-discussion] Comping Numpy in Cython Message-ID: I keep getting this error: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API" and a bunch of undefined variables when comping cython code with numpy I am new to cython and numpy and I am having trouble getting a *.pyx extension to compile with this in the header: from __future__ import division import numpy as np cimport numpy as np DTYPE = np.int ctypedef np.int_t DTYPE_t I complie to C code using: cython -a montcarl.pyx And it seems to work fine. I then attempt to compile using: gcc -I /Developer/SDKs/MacOSX10.6.sdk/System/Library/Frameworks/ Python.framework/Versions/2.6/include/python2.6 -I /usr/local/lib/ python2.7/site-packages/numpy/core/include -shared -pthread -fPIC - fwrapv -O2 -fno-strict-aliasing -o montcar.so montcarl.c And I get a bunch of undefined variables: spoust-m:old spoust$ gcc $CFLAGS -o montcar.so montcarl.c In file included from /usr/local/lib/python2.7/site-packages/numpy/ core/include/numpy/ndarraytypes.h:1965, from /usr/local/lib/python2.7/site-packages/numpy/ core/include/numpy/ndarrayobject.h:17, from /usr/local/lib/python2.7/site-packages/numpy/ core/include/numpy/arrayobject.h:14, from montcarl.c:225: /usr/local/lib/python2.7/site-packages/numpy/core/include/numpy/ npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API" Undefined symbols: "_PyLong_AsLong", referenced from: ___Pyx_PyInt_AsLong in ccvkIDLQ.o ___Pyx_PyInt_AsLong in ccvkIDLQ.o ... Any ideas on how to get this to compile and how to get rid of these numpy undefined variables? The cython guys directed me here. From stefan at sun.ac.za Tue Sep 20 11:56:31 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 20 Sep 2011 08:56:31 -0700 Subject: [Numpy-discussion] Fancy indexing with masks In-Reply-To: References: Message-ID: On Tue, Sep 20, 2011 at 12:43 AM, Robert Kern wrote: > If the array is short in a dimension, it gets implicitly continued > with Falses. You can see this in one dimension: [...] > I honestly don't know if this is documented or tested anywhere or even > if this existed in older versions. The behaviour is already present in 1.4, so I guess it's too late to insert a shape check now? Regards St?fan From shish at keba.be Tue Sep 20 12:00:51 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 20 Sep 2011 12:00:51 -0400 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> <469182246-1316528834-cardhu_decombobulator_blackberry.rim.net-1607792423-@b5.c12.bise7.blackberry> Message-ID: 2011/9/20 DIPO ELEGBEDE > 3. Get each array one after the other and work with it. (This is where the > problem is). > > How can I fetch out array1, then array2 then array3 up to array12? > > Is it possible to slice? > > Not really sure what your problem is, but since you should have now a list of arrays, you should be able to just loop through this list. for my_array in list_of_arrays: .... # do whatever you want on each array -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From dipo.elegbede at gmail.com Tue Sep 20 12:06:39 2011 From: dipo.elegbede at gmail.com (dipo.elegbede at gmail.com) Date: Tue, 20 Sep 2011 16:06:39 +0000 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br><259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry><469182246-1316528834-cardhu_decombobulator_blackberry.rim.net-1607792423-@b5.c12.bise7.blackberry> Message-ID: <1740140568-1316534797-cardhu_decombobulator_blackberry.rim.net-79683811-@b5.c12.bise7.blackberry> Olivier, the code you sent did so much magic and I'm definitely using that. What baffles me is that looping through the 12 arrays seem impossible. I have tried what you just sent but it won't just work. You could confirm this. Sent from my BlackBerry wireless device from MTN -----Original Message----- From: Olivier Delalleau Sender: numpy-discussion-bounces at scipy.org Date: Tue, 20 Sep 2011 12:00:51 To: Discussion of Numerical Python Reply-To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Dealing with arrays _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From shish at keba.be Tue Sep 20 12:11:46 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 20 Sep 2011 12:11:46 -0400 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: <1740140568-1316534797-cardhu_decombobulator_blackberry.rim.net-79683811-@b5.c12.bise7.blackberry> References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> <469182246-1316528834-cardhu_decombobulator_blackberry.rim.net-1607792423-@b5.c12.bise7.blackberry> <1740140568-1316534797-cardhu_decombobulator_blackberry.rim.net-79683811-@b5.c12.bise7.blackberry> Message-ID: Can you please provide your current code, even if incomplete / not working? It's hard to tell what's wrong without looking at the code. -=- Olivier 2011/9/20 > Olivier, the code you sent did so much magic and I'm definitely using that. > What baffles me is that looping through the 12 arrays seem impossible. > I have tried what you just sent but it won't just work. > You could confirm this. > Sent from my BlackBerry wireless device from MTN > > -----Original Message----- > From: Olivier Delalleau > Sender: numpy-discussion-bounces at scipy.org > Date: Tue, 20 Sep 2011 12:00:51 > To: Discussion of Numerical Python > Reply-To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Dealing with arrays > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dipo.elegbede at gmail.com Tue Sep 20 12:17:13 2011 From: dipo.elegbede at gmail.com (DIPO ELEGBEDE) Date: Tue, 20 Sep 2011 17:17:13 +0100 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> <469182246-1316528834-cardhu_decombobulator_blackberry.rim.net-1607792423-@b5.c12.bise7.blackberry> <1740140568-1316534797-cardhu_decombobulator_blackberry.rim.net-79683811-@b5.c12.bise7.blackberry> Message-ID: I think that should sort it. def p(option_lists): for option_list in option_lists: array = numpy.array(option_list) for i in array: print i the option_lists here is the list of lists looping through it and then converting it into a numpy array is quiet easy. If I try to loop through the array as you adviced, it prints out the whole items in the array. You could also try that out. What I am trying to achieve however is to loop through the 12 arrays and get them one after the other, doing something like: for my_array in list_of_arrays: print my_array[0] Hope this is clear enough. Thanks for your patience. On 20 September 2011 17:11, Olivier Delalleau wrote: > Can you please provide your current code, even if incomplete / not working? > It's hard to tell what's wrong without looking at the code. > > -=- Olivier > > 2011/9/20 > >> Olivier, the code you sent did so much magic and I'm definitely using >> that. >> >> What baffles me is that looping through the 12 arrays seem impossible. >> I have tried what you just sent but it won't just work. >> You could confirm this. >> Sent from my BlackBerry wireless device from MTN >> >> -----Original Message----- >> From: Olivier Delalleau >> Sender: numpy-discussion-bounces at scipy.org >> Date: Tue, 20 Sep 2011 12:00:51 >> To: Discussion of Numerical Python >> Reply-To: Discussion of Numerical Python >> Subject: Re: [Numpy-discussion] Dealing with arrays >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Dipo Elegbede,OCA 08033299270,08077682428 -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Tue Sep 20 12:32:57 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 20 Sep 2011 12:32:57 -0400 Subject: [Numpy-discussion] Dealing with arrays In-Reply-To: References: <0cb34893f97bb3f2223a0570f2d1636f.squirrel@www.cnpmf.embrapa.br> <259242823-1316527568-cardhu_decombobulator_blackberry.rim.net-2135234260-@b5.c12.bise7.blackberry> <469182246-1316528834-cardhu_decombobulator_blackberry.rim.net-1607792423-@b5.c12.bise7.blackberry> <1740140568-1316534797-cardhu_decombobulator_blackberry.rim.net-79683811-@b5.c12.bise7.blackberry> Message-ID: I'm trying to understand what you want to do, but I'm afraid it's still not clear, so I'm sorry if that reply doesn't help either. It looks to me as if you first want to convert your list of lists into a list of arrays, then iterate through this list of arrays. If that's it, then you can just do: def p(option_lists): list_of_arrays = map(numpy.array, option_lists) for my_array in list_of_arrays: ... # Do some operation on array 'my_array' -=- Olivier 2011/9/20 DIPO ELEGBEDE > I think that should sort it. > > def p(option_lists): > for option_list in option_lists: > array = numpy.array(option_list) > for i in array: print i > > the option_lists here is the list of lists > looping through it and then converting it into a numpy array is quiet easy. > > If I try to loop through the array as you adviced, it prints out the whole > items in the array. > > You could also try that out. > > What I am trying to achieve however is to loop through the 12 arrays and > get them one after the other, doing something like: > > for my_array in list_of_arrays: > print my_array[0] > > > Hope this is clear enough. > Thanks for your patience. > > > On 20 September 2011 17:11, Olivier Delalleau wrote: > >> Can you please provide your current code, even if incomplete / not >> working? It's hard to tell what's wrong without looking at the code. >> >> -=- Olivier >> >> 2011/9/20 >> >>> Olivier, the code you sent did so much magic and I'm definitely using >>> that. >>> >>> What baffles me is that looping through the 12 arrays seem impossible. >>> I have tried what you just sent but it won't just work. >>> You could confirm this. >>> Sent from my BlackBerry wireless device from MTN >>> >>> -----Original Message----- >>> From: Olivier Delalleau >>> Sender: numpy-discussion-bounces at scipy.org >>> Date: Tue, 20 Sep 2011 12:00:51 >>> To: Discussion of Numerical Python >>> Reply-To: Discussion of Numerical Python >>> Subject: Re: [Numpy-discussion] Dealing with arrays >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > Dipo Elegbede,OCA > 08033299270,08077682428 > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Sep 20 13:18:25 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 20 Sep 2011 13:18:25 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: <919343ED-3F54-4B20-82BB-3267630579E3@samueljohn.de> Message-ID: On Tue, Sep 20, 2011 at 9:56 AM, David Cottrell wrote: > Thanks, just getting back to this. I just checked again, and after > setting my LD_LIBRARY_PATH properly, ldd shows _dotblas.so pointing > and the sunmath and sunperf libraries. However the test_03.py still > runs at about 8-9 seconds ... far too slow. > > ~/local/lib/python3.1/site-packages/numpy/core $ ldd _dotblas.so | sed > -e 's/$me/$USERNAME/g' > ? ? ? ?libsunperf.so.8 => > /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunperf.so.8 > ? ? ? ?libsunmath.so.1 => > /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunmath.so.1 > ? ? ? ?libgcc_s.so.1 => ? ? ? ? /usr/sfw/lib/libgcc_s.so.1 > ? ? ? ?libfsu.so.1 => > /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfsu.so.1 > ? ? ? ?libfui.so.2 => > /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfui.so.2 > ? ? ? ?libpicl.so.1 => ?/usr/lib/libpicl.so.1 > ? ? ? ?libmtsk.so.1 => ?/lib/libmtsk.so.1 > ? ? ? ?libm.so.2 => ? ? /lib/libm.so.2 > ? ? ? ?libc.so.1 => ? ? /lib/libc.so.1 > ? ? ? ?libm.so.1 => ? ? /lib/libm.so.1 > ? ? ? ?libdl.so.1 => ? ?/lib/libdl.so.1 > ? ? ? ?libdoor.so.1 => ?/lib/libdoor.so.1 > ? ? ? ?libthread.so.1 => ? ? ? ?/lib/libthread.so.1 > ? ? ? ?libkstat.so.1 => ? ? ? ? /lib/libkstat.so.1 > ? ? ? ?libpthread.so.1 => ? ? ? /lib/libpthread.so.1 > ? ? ? ?librt.so.1 => ? ?/lib/librt.so.1 > ? ? ? ?libaio.so.1 => ? /lib/libaio.so.1 > ? ? ? ?libmd.so.1 => ? ?/lib/libmd.so.1 > ? ? ? ?/platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1 > ? ? ? ?/platform/SUNW,Sun-Fire-V490/lib/libmd_psr.so.1 > > ~/local/lib/python3.1/site-packages/numpy/core $ ~/python/numpy/B/test_03.py > No ATLAS: > 8.49377894402 > (1000, 1000) (1000,) (1000, 1000) As mentioned earlier, we currently only optimize dot if you are using ATLAS, not any generic BLAS implementation. Your linear algebra functions will be fast, though, cheers, David From david.cottrell at gmail.com Tue Sep 20 15:18:41 2011 From: david.cottrell at gmail.com (David Cottrell) Date: Tue, 20 Sep 2011 15:18:41 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: <919343ED-3F54-4B20-82BB-3267630579E3@samueljohn.de> Message-ID: The test_03.py was basically a linalg.svd test (which I think is a linalg only routine?"). I guess for linalg checking, I should run ldd on lapack_lite.so? (included below). It's sounding like I need to get ATLAS up and running, but I'm still puzzled as to why the svd routine seems to be so much slower on the solaris install. As a reminder, test_03.py was something like this. It's also somewhere earlier in the thread. import numpy.linalg N = 1000 x = numpy.random.random((N,N)) t = time.time() (U, s, V) = numpy.linalg.svd(x) # S = numpy.matrix(numpy.diag(s)) # y = U * S * V #print(y.shape) print(time.time()-t) print(U.shape, s.shape, V.shape) $ ldd lapack_lite.so libsunmath.so.1 => /opt/SUNWspro11/SUNWspro/lib/libsunmath.so.1 libsunperf.so.6 => /opt/SUNWspro11/SUNWspro/lib/v8plus/libsunperf.so.6 libfsu.so.1 => /opt/SUNWspro11/SUNWspro/lib/v8plus/libfsu.so.1 libm.so.1 => /lib/libm.so.1 libc.so.1 => /lib/libc.so.1 libfsu.so.1 => /opt/SUNWspro11/SUNWspro/prod/lib/v8/../libfsu.so.1 libfui.so.2 => /opt/SUNWspro11/SUNWspro/prod/lib/v8/../libfui.so.2 libmtsk.so.1 => /lib/libmtsk.so.1 libthread.so.1 => /lib/libthread.so.1 libkstat.so.1 => /lib/libkstat.so.1 libdl.so.1 => /lib/libdl.so.1 libpthread.so.1 => /lib/libpthread.so.1 libm.so.2 => /lib/libm.so.2 librt.so.1 => /lib/librt.so.1 libaio.so.1 => /lib/libaio.so.1 libmd.so.1 => /lib/libmd.so.1 /opt/SUNWspro11/SUNWspro/prod/lib/v8/../cpu/sparcv9+vis2/libsunperf_isa.so.6 /opt/SUNWspro11/SUNWspro/prod/lib/v8/../cpu/sparcv8plus+vis/libsunperf_isa.so.6 /opt/SUNWspro11/SUNWspro/prod/lib/v8plus/../cpu/sparcv9+vis2/libfsu_isa.so.1 /platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1 /platform/SUNW,Sun-Fire-V490/lib/libmd_psr.so.1 On Tue, Sep 20, 2011 at 1:18 PM, David Cournapeau wrote: > On Tue, Sep 20, 2011 at 9:56 AM, David Cottrell > wrote: >> Thanks, just getting back to this. I just checked again, and after >> setting my LD_LIBRARY_PATH properly, ldd shows _dotblas.so pointing >> and the sunmath and sunperf libraries. However the test_03.py still >> runs at about 8-9 seconds ... far too slow. >> >> ~/local/lib/python3.1/site-packages/numpy/core $ ldd _dotblas.so | sed >> -e 's/$me/$USERNAME/g' >> ? ? ? ?libsunperf.so.8 => >> /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunperf.so.8 >> ? ? ? ?libsunmath.so.1 => >> /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunmath.so.1 >> ? ? ? ?libgcc_s.so.1 => ? ? ? ? /usr/sfw/lib/libgcc_s.so.1 >> ? ? ? ?libfsu.so.1 => >> /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfsu.so.1 >> ? ? ? ?libfui.so.2 => >> /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfui.so.2 >> ? ? ? ?libpicl.so.1 => ?/usr/lib/libpicl.so.1 >> ? ? ? ?libmtsk.so.1 => ?/lib/libmtsk.so.1 >> ? ? ? ?libm.so.2 => ? ? /lib/libm.so.2 >> ? ? ? ?libc.so.1 => ? ? /lib/libc.so.1 >> ? ? ? ?libm.so.1 => ? ? /lib/libm.so.1 >> ? ? ? ?libdl.so.1 => ? ?/lib/libdl.so.1 >> ? ? ? ?libdoor.so.1 => ?/lib/libdoor.so.1 >> ? ? ? ?libthread.so.1 => ? ? ? ?/lib/libthread.so.1 >> ? ? ? ?libkstat.so.1 => ? ? ? ? /lib/libkstat.so.1 >> ? ? ? ?libpthread.so.1 => ? ? ? /lib/libpthread.so.1 >> ? ? ? ?librt.so.1 => ? ?/lib/librt.so.1 >> ? ? ? ?libaio.so.1 => ? /lib/libaio.so.1 >> ? ? ? ?libmd.so.1 => ? ?/lib/libmd.so.1 >> ? ? ? ?/platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1 >> ? ? ? ?/platform/SUNW,Sun-Fire-V490/lib/libmd_psr.so.1 >> >> ~/local/lib/python3.1/site-packages/numpy/core $ ~/python/numpy/B/test_03.py >> No ATLAS: >> 8.49377894402 >> (1000, 1000) (1000,) (1000, 1000) > > As mentioned earlier, we currently only optimize dot if you are using > ATLAS, not any generic BLAS implementation. Your linear algebra > functions will be fast, though, > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- David Cottrell +1 416 995 9860 http://ca.linkedin.com/in/dcottrell From cournape at gmail.com Tue Sep 20 15:37:29 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 20 Sep 2011 15:37:29 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: <919343ED-3F54-4B20-82BB-3267630579E3@samueljohn.de> Message-ID: On Tue, Sep 20, 2011 at 3:18 PM, David Cottrell wrote: > The test_03.py was basically a linalg.svd test (which I think is a > linalg only routine?"). I guess for linalg checking, I should run ldd > on lapack_lite.so? (included below). > > It's sounding like I need to get ATLAS up and running, but I'm still > puzzled as to why the svd routine seems to be so much slower on the > solaris install. What's the CPU of your solaris machine and macbook ? For macbook, the info as given by the system profiler are enough, and maybe psrinfo -pv is enough on solaris. cheers, David From cournape at gmail.com Tue Sep 20 15:48:47 2011 From: cournape at gmail.com (David Cournapeau) Date: Tue, 20 Sep 2011 15:48:47 -0400 Subject: [Numpy-discussion] numpy blas running slow: how to check that it is properly linked In-Reply-To: References: <919343ED-3F54-4B20-82BB-3267630579E3@samueljohn.de> Message-ID: On Tue, Sep 20, 2011 at 1:18 PM, David Cournapeau wrote: > On Tue, Sep 20, 2011 at 9:56 AM, David Cottrell > wrote: >> Thanks, just getting back to this. I just checked again, and after >> setting my LD_LIBRARY_PATH properly, ldd shows _dotblas.so pointing >> and the sunmath and sunperf libraries. However the test_03.py still >> runs at about 8-9 seconds ... far too slow. >> >> ~/local/lib/python3.1/site-packages/numpy/core $ ldd _dotblas.so | sed >> -e 's/$me/$USERNAME/g' >> ? ? ? ?libsunperf.so.8 => >> /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunperf.so.8 >> ? ? ? ?libsunmath.so.1 => >> /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libsunmath.so.1 >> ? ? ? ?libgcc_s.so.1 => ? ? ? ? /usr/sfw/lib/libgcc_s.so.1 >> ? ? ? ?libfsu.so.1 => >> /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfsu.so.1 >> ? ? ? ?libfui.so.2 => >> /home/$USERNAME/local/archive/SolarisStudio12.2-solaris-sparc-tar-ML/solstudio12.2/lib//libfui.so.2 >> ? ? ? ?libpicl.so.1 => ?/usr/lib/libpicl.so.1 >> ? ? ? ?libmtsk.so.1 => ?/lib/libmtsk.so.1 >> ? ? ? ?libm.so.2 => ? ? /lib/libm.so.2 >> ? ? ? ?libc.so.1 => ? ? /lib/libc.so.1 >> ? ? ? ?libm.so.1 => ? ? /lib/libm.so.1 >> ? ? ? ?libdl.so.1 => ? ?/lib/libdl.so.1 >> ? ? ? ?libdoor.so.1 => ?/lib/libdoor.so.1 >> ? ? ? ?libthread.so.1 => ? ? ? ?/lib/libthread.so.1 >> ? ? ? ?libkstat.so.1 => ? ? ? ? /lib/libkstat.so.1 >> ? ? ? ?libpthread.so.1 => ? ? ? /lib/libpthread.so.1 >> ? ? ? ?librt.so.1 => ? ?/lib/librt.so.1 >> ? ? ? ?libaio.so.1 => ? /lib/libaio.so.1 >> ? ? ? ?libmd.so.1 => ? ?/lib/libmd.so.1 >> ? ? ? ?/platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1 >> ? ? ? ?/platform/SUNW,Sun-Fire-V490/lib/libmd_psr.so.1 If this refers to the model of your solaris machine (SunFire V490, meaning UltraSparc IV), then the numbers are actually not hard to believe. Those are almost a decade-old CPU (from 2003-2004), and SPARC are not speed deamons in the first place. Having them 10 times slower than a recent core i7 (which are very performant for floating points) seems "reasonable". cheers, David From dineshbvadhia at hotmail.com Tue Sep 20 15:53:46 2011 From: dineshbvadhia at hotmail.com (Dinesh B Vadhia) Date: Tue, 20 Sep 2011 12:53:46 -0700 Subject: [Numpy-discussion] Fw: sparse vectors / matrices / tensors Message-ID: Yannick Sounds great. Would cross-post on the Scipy list as that is where the scipy.sparse developers hang out. Dinesh From: Yannick Versley Sent: Tuesday, September 20, 2011 8:33 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] sparse vectors / matrices / tensors This question seems like it's more relevant to the scipy mailing list, since scipy has scipy.sparse. My point was that the current situation (i.e., people reinventing the wheel) was a by-product of the fact that there's nothing as standardized as numpy's ndarray or Python's buffer interface, if only for the simple reason that most people only ever work with dense matrices. (scipy.sparse uses several different data structures that are all limited to two axes, and have no common low-level interface). I'll be happy to ask on the scipy list, if that's where the discussion should be. Yannick -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Wed Sep 21 04:33:28 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 21 Sep 2011 10:33:28 +0200 Subject: [Numpy-discussion] Round table/blog on fixing scientific software distribution Message-ID: <4E79A158.5070104@astro.uio.no> Yet again, the issue of distributing scientific Python software was raised, this time on the mpi4py mailing list. Since that wasn't really the right forum, and we weren't really sure what was the right forum, we started a blog instead. The idea is to get a diverse set of people describe their experiences; something between a brainstorming and a survey. No current solution fits all and the community is fragmented -- perhaps collecting the experiences of different user bases and the solutions they found is helpful at this point. We want your posts! (send me an email to get posting rights) http://fixingscientificsoftwaredistribution.blogspot.com/2011/09/round-table-about-fixing-problem-of.html Dag Sverre From jsseabold at gmail.com Wed Sep 21 12:10:20 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 21 Sep 2011 12:10:20 -0400 Subject: [Numpy-discussion] why doesn't numpy.distutils find ATLAS? Message-ID: I have installed numpy with my own ATLAS, but trying to install PyMC, it can't find the ATLAS libs. I also have an older package that formerly installed but no longer does. I'm being a bit lazy, but am I missing something? Briefly checking, it looks like the conventions/assumptions for site.cfg in an installation as per site.cfg.example are different from those expected of numpy.distutils.system_info. I have the directories specified under [DEFAULT] as per the installation instructions. If I don't have the libraries installed in one of the default folders, it looks like they're not found. [~/] [1]: from numpy.distutils import system_info [~/] [2]: np.show_config() atlas_threads_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/skipper/built/atlas_files/lib'] define_macros = [('ATLAS_INFO', '"\\"3.9.32\\""')] language = f77 include_dirs = ['/home/skipper/built/atlas_files/include'] blas_opt_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/skipper/built/atlas_files/lib'] define_macros = [('ATLAS_INFO', '"\\"3.9.32\\""')] language = c include_dirs = ['/home/skipper/built/atlas_files/include'] atlas_blas_threads_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/skipper/built/atlas_files/lib'] define_macros = [('ATLAS_INFO', '"\\"3.9.32\\""')] language = c include_dirs = ['/home/skipper/built/atlas_files/include'] lapack_opt_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/skipper/built/atlas_files/lib'] define_macros = [('ATLAS_INFO', '"\\"3.9.32\\""')] language = f77 include_dirs = ['/home/skipper/built/atlas_files/include'] lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE [~/] [3]: system_info.get_info('atlas',1) /usr/local/lib/python2.7/dist-packages/numpy/distutils/system_info.py:462: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(self.notfounderror.__doc__) [3]: {} From morph at debian.org Wed Sep 21 16:56:29 2011 From: morph at debian.org (Sandro Tosi) Date: Wed, 21 Sep 2011 22:56:29 +0200 Subject: [Numpy-discussion] How to run unittests at build time? Message-ID: Hello, at (Debian) package build time, it would be really important to run the tests suite, but I'm not getting how to do that. The simple approach doesn't work: >>> $ PYTHONPATH=build/lib.linux-x86_64-2.6 python -c "import numpy; numpy.test()" Traceback (most recent call last): File "", line 1, in File "numpy/__init__.py", line 127, in raise ImportError(msg) ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python intepreter from there. <<< because '.' is placed first in the sys.path list, so it gets used the numpy/ dir in . instead the one in build/lib... . Ok, so let's tune sys.path: >>> $ python -c "import sys ; sys.path.insert(0, 'build/lib.linux-x86_64-2.6') ; import numpy; numpy.test()" Running unit tests for numpy NumPy version 1.6.1 NumPy is installed in build/lib.linux-x86_64-2.6/numpy Python version 2.6.7 (r267:88850, Aug 3 2011, 11:33:52) [GCC 4.6.1] nose version 1.0.0 EEEEEEE ====================================================================== ERROR: Failure: NameError (name 'numeric' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "build/lib.linux-x86_64-2.6/numpy/core/__init__.py", line 30, in __all__ += numeric.__all__ NameError: name 'numeric' is not defined ====================================================================== ERROR: Failure: ImportError (cannot import name log) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "build/lib.linux-x86_64-2.6/numpy/distutils/fcompiler/__init__.py", line 37, in from numpy.distutils import log ImportError: cannot import name log ====================================================================== ERROR: Failure: NameError (name 'type_check' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "build/lib.linux-x86_64-2.6/numpy/lib/__init__.py", line 23, in __all__ += type_check.__all__ NameError: name 'type_check' is not defined ====================================================================== ERROR: Failure: NameError (name 'numeric' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "build/lib.linux-x86_64-2.6/numpy/ma/testutils.py", line 17, in import numpy.core.umath as umath File "build/lib.linux-x86_64-2.6/numpy/core/__init__.py", line 30, in __all__ += numeric.__all__ NameError: name 'numeric' is not defined ====================================================================== ERROR: Failure: NameError (name 'defmatrix' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "build/lib.linux-x86_64-2.6/numpy/matrixlib/__init__.py", line 4, in __all__ = defmatrix.__all__ NameError: name 'defmatrix' is not defined ====================================================================== ERROR: Failure: NameError (name 'numeric' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "build/lib.linux-x86_64-2.6/numpy/oldnumeric/__init__.py", line 14, in from compat import * File "build/lib.linux-x86_64-2.6/numpy/oldnumeric/compat.py", line 13, in import numpy.core.multiarray as multiarray File "build/lib.linux-x86_64-2.6/numpy/core/__init__.py", line 30, in __all__ += numeric.__all__ NameError: name 'numeric' is not defined ====================================================================== ERROR: Failure: NameError (name 'type_check' is not defined) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "build/lib.linux-x86_64-2.6/numpy/polynomial/__init__.py", line 28, in from numpy.lib import deprecate File "build/lib.linux-x86_64-2.6/numpy/lib/__init__.py", line 23, in __all__ += type_check.__all__ NameError: name 'type_check' is not defined ---------------------------------------------------------------------- Ran 7 tests in 0.004s FAILED (errors=7) <<< Not nice either, let's try another way, importing * from numpy: $ python -c "import sys ; sys.path.insert(0, 'build/lib.linux-x86_64-2.6') ; import numpy; from numpy import * ; numpy.test()" same result as above. So, what's the proper way to run the tests suite at build time? Thanks in advance, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From robert.kern at gmail.com Wed Sep 21 17:09:46 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 21 Sep 2011 16:09:46 -0500 Subject: [Numpy-discussion] How to run unittests at build time? In-Reply-To: References: Message-ID: On Wed, Sep 21, 2011 at 15:56, Sandro Tosi wrote: > Hello, > at (Debian) package build time, it would be really important to run > the tests suite, but I'm not getting how to do that. The way that nosetests imports packages in order to navigate to find test files is a bit wonky. It doesn't respect proper package semantics. Or perhaps just the behavior that we see from normal imports. In particular, many of our __init__.py files do something like this: from foo import ** __all__ = [] __all__.extend(foo.__all__) The reason why this normally works is that once you do any kind of import from the foo submodule, it gets added to the namespace of the package's actual module object. However, nosetests is manually "importing" the __init__.py file and bypassing the normal import mechanism, so it misses some of these semantics. I suspect nosetests may be doing something different to load files that have been installed into site-packages rather than other locations. Unfortunately, I don't know a workaround other than to modify all of our __init__.py file to do more explicit imports rather than relying on the implicit behavior of the full Python import mechanism. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ocefpaf at gmail.com Wed Sep 21 17:15:14 2011 From: ocefpaf at gmail.com (Filipe Pires Alvarenga Fernandes) Date: Wed, 21 Sep 2011 17:15:14 -0400 Subject: [Numpy-discussion] How to run unittests at build time? In-Reply-To: References: Message-ID: Don't know about about Debian, but this works OK for RPM distros: PYTHONPATH="%{buildroot}%{py_sitedir}" %{__python} -c \ "import pkg_resources, numpy ; numpy.test()" The only thing you are missing is the "import pkg_resources", let me know if that works for you. -Filipe On Wed, Sep 21, 2011 at 16:56, Sandro Tosi wrote: > Hello, > at (Debian) package build time, it would be really important to run > the tests suite, but I'm not getting how to do that. > > The simple approach doesn't work: > >>>> > $ PYTHONPATH=build/lib.linux-x86_64-2.6 python -c "import numpy; numpy.test()" > Traceback (most recent call last): > ?File "", line 1, in > ?File "numpy/__init__.py", line 127, in > ? ?raise ImportError(msg) > ImportError: Error importing numpy: you should not try to import numpy from > ? ? ? ?its source directory; please exit the numpy source tree, and relaunch > ? ? ? ?your python intepreter from there. > <<< > > because '.' is placed first in the sys.path list, so it gets used the > numpy/ dir in . instead the one in build/lib... . Ok, so let's tune > sys.path: > >>>> > $ python -c "import sys ; sys.path.insert(0, > 'build/lib.linux-x86_64-2.6') ; import numpy; numpy.test()" > Running unit tests for numpy > NumPy version 1.6.1 > NumPy is installed in build/lib.linux-x86_64-2.6/numpy > Python version 2.6.7 (r267:88850, Aug ?3 2011, 11:33:52) [GCC 4.6.1] > nose version 1.0.0 > EEEEEEE > ====================================================================== > ERROR: Failure: NameError (name 'numeric' is not defined) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in > loadTestsFromName > ? ?addr.filename, addr.module) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in > importFromPath > ? ?return self.importFromDir(dir_path, fqname) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 84, in > importFromDir > ? ?mod = load_module(part_fqname, fh, filename, desc) > ?File "build/lib.linux-x86_64-2.6/numpy/core/__init__.py", line 30, in > ? ?__all__ += numeric.__all__ > NameError: name 'numeric' is not defined > > ====================================================================== > ERROR: Failure: ImportError (cannot import name log) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in > loadTestsFromName > ? ?addr.filename, addr.module) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in > importFromPath > ? ?return self.importFromDir(dir_path, fqname) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in > importFromDir > ? ?mod = load_module(part_fqname, fh, filename, desc) > ?File "build/lib.linux-x86_64-2.6/numpy/distutils/fcompiler/__init__.py", > line 37, in > ? ?from numpy.distutils import log > ImportError: cannot import name log > > ====================================================================== > ERROR: Failure: NameError (name 'type_check' is not defined) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in > loadTestsFromName > ? ?addr.filename, addr.module) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in > importFromPath > ? ?return self.importFromDir(dir_path, fqname) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 84, in > importFromDir > ? ?mod = load_module(part_fqname, fh, filename, desc) > ?File "build/lib.linux-x86_64-2.6/numpy/lib/__init__.py", line 23, in > ? ?__all__ += type_check.__all__ > NameError: name 'type_check' is not defined > > ====================================================================== > ERROR: Failure: NameError (name 'numeric' is not defined) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in > loadTestsFromName > ? ?addr.filename, addr.module) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in > importFromPath > ? ?return self.importFromDir(dir_path, fqname) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in > importFromDir > ? ?mod = load_module(part_fqname, fh, filename, desc) > ?File "build/lib.linux-x86_64-2.6/numpy/ma/testutils.py", line 17, in > ? ?import numpy.core.umath as umath > ?File "build/lib.linux-x86_64-2.6/numpy/core/__init__.py", line 30, in > ? ?__all__ += numeric.__all__ > NameError: name 'numeric' is not defined > > ====================================================================== > ERROR: Failure: NameError (name 'defmatrix' is not defined) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in > loadTestsFromName > ? ?addr.filename, addr.module) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in > importFromPath > ? ?return self.importFromDir(dir_path, fqname) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 84, in > importFromDir > ? ?mod = load_module(part_fqname, fh, filename, desc) > ?File "build/lib.linux-x86_64-2.6/numpy/matrixlib/__init__.py", line > 4, in > ? ?__all__ = defmatrix.__all__ > NameError: name 'defmatrix' is not defined > > ====================================================================== > ERROR: Failure: NameError (name 'numeric' is not defined) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in > loadTestsFromName > ? ?addr.filename, addr.module) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in > importFromPath > ? ?return self.importFromDir(dir_path, fqname) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in > importFromDir > ? ?mod = load_module(part_fqname, fh, filename, desc) > ?File "build/lib.linux-x86_64-2.6/numpy/oldnumeric/__init__.py", line > 14, in > ? ?from compat import * > ?File "build/lib.linux-x86_64-2.6/numpy/oldnumeric/compat.py", line > 13, in > ? ?import numpy.core.multiarray as multiarray > ?File "build/lib.linux-x86_64-2.6/numpy/core/__init__.py", line 30, in > ? ?__all__ += numeric.__all__ > NameError: name 'numeric' is not defined > > ====================================================================== > ERROR: Failure: NameError (name 'type_check' is not defined) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib/pymodules/python2.6/nose/loader.py", line 390, in > loadTestsFromName > ? ?addr.filename, addr.module) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in > importFromPath > ? ?return self.importFromDir(dir_path, fqname) > ?File "/usr/lib/pymodules/python2.6/nose/importer.py", line 84, in > importFromDir > ? ?mod = load_module(part_fqname, fh, filename, desc) > ?File "build/lib.linux-x86_64-2.6/numpy/polynomial/__init__.py", line > 28, in > ? ?from numpy.lib import deprecate > ?File "build/lib.linux-x86_64-2.6/numpy/lib/__init__.py", line 23, in > ? ?__all__ += type_check.__all__ > NameError: name 'type_check' is not defined > > ---------------------------------------------------------------------- > Ran 7 tests in 0.004s > > FAILED (errors=7) > <<< > > Not nice either, let's try another way, importing * from numpy: > > $ python -c "import sys ; sys.path.insert(0, > 'build/lib.linux-x86_64-2.6') ; import numpy; from numpy import * ; > numpy.test()" > > same result as above. > > So, what's the proper way to run the tests suite at build time? > > Thanks in advance, > -- > Sandro Tosi (aka morph, morpheus, matrixhasu) > My website: http://matrixhasu.altervista.org/ > Me at Debian: http://wiki.debian.org/SandroTosi > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Wed Sep 21 17:20:50 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 21 Sep 2011 16:20:50 -0500 Subject: [Numpy-discussion] How to run unittests at build time? In-Reply-To: References: Message-ID: On Wed, Sep 21, 2011 at 16:15, Filipe Pires Alvarenga Fernandes wrote: > Don't know about about Debian, but this works OK for RPM distros: > > PYTHONPATH="%{buildroot}%{py_sitedir}" %{__python} -c \ > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?"import pkg_resources, numpy ; numpy.test()" > > The only thing you are missing is the "import pkg_resources", let me > know if that works for you. What does py_sitedir expand to, usually? I don't think pkg_resources has anything to do with it. numpy doesn't use it, and I don't think nosetests does either (well, only for its plugin mechanism, but that's neither here nor there). In any case, I've tried importing pkg_resources while running from the build/lib.*/ directory and still get the same errors as reported by Sandro. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From morph at debian.org Wed Sep 21 17:26:04 2011 From: morph at debian.org (Sandro Tosi) Date: Wed, 21 Sep 2011 23:26:04 +0200 Subject: [Numpy-discussion] How to run unittests at build time? In-Reply-To: References: Message-ID: On Wed, Sep 21, 2011 at 23:15, Filipe Pires Alvarenga Fernandes wrote: > Don't know about about Debian, but this works OK for RPM distros: > > PYTHONPATH="%{buildroot}%{py_sitedir}" %{__python} -c \ > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?"import pkg_resources, numpy ; numpy.test()" > > The only thing you are missing is the "import pkg_resources", let me > know if that works for you. Ok it works: I have to run tests from the installed location and not from the build/ subdir; Thanks!! Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From morph at debian.org Wed Sep 21 17:29:40 2011 From: morph at debian.org (Sandro Tosi) Date: Wed, 21 Sep 2011 23:29:40 +0200 Subject: [Numpy-discussion] How to run unittests at build time? In-Reply-To: References: Message-ID: On Wed, Sep 21, 2011 at 23:20, Robert Kern wrote: > On Wed, Sep 21, 2011 at 16:15, Filipe Pires Alvarenga Fernandes > wrote: >> Don't know about about Debian, but this works OK for RPM distros: >> >> PYTHONPATH="%{buildroot}%{py_sitedir}" %{__python} -c \ >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?"import pkg_resources, numpy ; numpy.test()" >> >> The only thing you are missing is the "import pkg_resources", let me >> know if that works for you. > > What does py_sitedir expand to, usually? I suppose the site-packages directory location, so something like /usr/lib/python2.6/site-packages/ > I don't think pkg_resources > has anything to do with it. numpy doesn't use it, and I don't think > nosetests does either (well, only for its plugin mechanism, but that's > neither here nor there). ack, I didn't use it in my tests: it works fine the python -c "import numpy; numpy.test()" > In any case, I've tried importing > pkg_resources while running from the build/lib.*/ directory and still > get the same errors as reported by Sandro. Yep, the different with my approach and Felipe's one is I ran from the build directory Felipe from the installation directory. It works, so I'll use the latter Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From ocefpaf at gmail.com Wed Sep 21 19:08:21 2011 From: ocefpaf at gmail.com (Filipe Pires Alvarenga Fernandes) Date: Wed, 21 Sep 2011 19:08:21 -0400 Subject: [Numpy-discussion] How to run unittests at build time? In-Reply-To: References: Message-ID: On Wed, Sep 21, 2011 at 17:29, Sandro Tosi wrote: > On Wed, Sep 21, 2011 at 23:20, Robert Kern wrote: >> On Wed, Sep 21, 2011 at 16:15, Filipe Pires Alvarenga Fernandes >> wrote: >>> Don't know about about Debian, but this works OK for RPM distros: >>> >>> PYTHONPATH="%{buildroot}%{py_sitedir}" %{__python} -c \ >>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?"import pkg_resources, numpy ; numpy.test()" >>> >>> The only thing you are missing is the "import pkg_resources", let me >>> know if that works for you. >> >> What does py_sitedir expand to, usually? > > I suppose the site-packages directory location, so something like > /usr/lib/python2.6/site-packages/ Yep. That's correct. If you want to look at the whole spec file: https://build.opensuse.org/package/view_file?file=python-numpy.spec&package=python-numpy&project=home%3Aocefpaf&srcmd5=ffa9c6e1b9a6f715f0101eb334c5b5a8 >> I don't think pkg_resources >> has anything to do with it. numpy doesn't use it, and I don't think >> nosetests does either (well, only for its plugin mechanism, but that's >> neither here nor there). > > ack, I didn't use it in my tests: it works fine the python -c "import > numpy; numpy.test()" > >> In any case, I've tried importing >> pkg_resources while running from the build/lib.*/ directory and still >> get the same errors as reported by Sandro. You are right, I just removed pkg_resources and things still work OK. > Yep, the different with my approach and Felipe's one is I ran from the > build directory Felipe from the installation directory. It works, so > I'll use the latter > > Cheers, > -- > Sandro Tosi (aka morph, morpheus, matrixhasu) > My website: http://matrixhasu.altervista.org/ > Me at Debian: http://wiki.debian.org/SandroTosi > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From hangenuit at gmail.com Thu Sep 22 02:12:12 2011 From: hangenuit at gmail.com (Han Genuit) Date: Thu, 22 Sep 2011 08:12:12 +0200 Subject: [Numpy-discussion] Fancy indexing with masks In-Reply-To: References: Message-ID: 2011/9/20 St?fan van der Walt > On Tue, Sep 20, 2011 at 12:43 AM, Robert Kern > wrote: > > If the array is short in a dimension, it gets implicitly continued > > with Falses. You can see this in one dimension: > > [...] > > > I honestly don't know if this is documented or tested anywhere or even > > if this existed in older versions. > > The behaviour is already present in 1.4, so I guess it's too late to > insert a shape check now? > > There already is a shape check present in the development version[1]: >>> a = np.arange(10) >>> b = np.array([False, True, False]) >>> a[b] Traceback (most recent call last): File "", line 1, in ValueError: operands could not be broadcast together with shapes (10) (3) But it does not seem to work on multidimensional arrays: >>> c = np.arange(12).reshape((4,3)) >>> c[b] array([[3, 4, 5]]) I also noticed that it does strange things when using a list: >>> c[[True, False, True]] array([[3, 4, 5], [0, 1, 2], [3, 4, 5]]) Regards, Han [1] See also: http://mail.scipy.org/pipermail/numpy-discussion/2011-July/057870.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Sep 22 06:09:26 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 22 Sep 2011 10:09:26 +0000 (UTC) Subject: [Numpy-discussion] Fancy indexing with masks References: Message-ID: Thu, 22 Sep 2011 08:12:12 +0200, Han Genuit wrote: [clip] > I also noticed that it does strange things when using a list: > >>>> c[[True, False, True]] > array([[3, 4, 5], > [0, 1, 2], > [3, 4, 5]]) It casts the list with booleans to an integer array. Probably shouldn't work like that... Pauli From m.verkerk at erasmusmc.nl Thu Sep 22 07:09:51 2011 From: m.verkerk at erasmusmc.nl (Marijn Verkerk) Date: Thu, 22 Sep 2011 13:09:51 +0200 Subject: [Numpy-discussion] ImportError: libptf77blas.so.3: cannot open shared object file: No such file or directory Message-ID: <4E7B177F.1040107@erasmusmc.nl> Dear all, Maybe someone has experienced this problem before. I was able to build and install numpy in a seperate python 2.7 folder on our cluster. When I try to import numpy I get the following error: $ /software/Python-2.7.2/bin/python -c 'import numpy; numpy.test()' Traceback (most recent call last): File "", line 1, in File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/__init__.py", line 137, in import add_newdocs File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/add_newdocs.py", line 9, in from numpy.lib import add_newdoc File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/lib/__init__.py", line 13, in from polynomial import * File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/lib/polynomial.py", line 17, in from numpy.linalg import eigvals, lstsq File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/linalg/__init__.py", line 48, in from linalg import * File "/illumina/software/Python-2.7.2/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 23, in from numpy.linalg import lapack_lite ImportError: libptf77blas.so.3: cannot open shared object file: No such file or directory In the __config.py file the folder where libptf77 should be is present. Any suggestions? Regards, Marijn From pav at iki.fi Thu Sep 22 07:39:57 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 22 Sep 2011 11:39:57 +0000 (UTC) Subject: [Numpy-discussion] ImportError: libptf77blas.so.3: cannot open shared object file: No such file or directory References: <4E7B177F.1040107@erasmusmc.nl> Message-ID: Hi, Thu, 22 Sep 2011 13:09:51 +0200, Marijn Verkerk wrote: [clip] > ImportError: libptf77blas.so.3: cannot open shared object file: No such > file or directory > > In the __config.py file the folder where libptf77 should be is present. > > Any suggestions? You need to make the dynamic linker able to find this file, e.g., by setting the LD_LIBRARY_PATH environment variable to a correct value. Test with "ldd lapack_lite.so" (The stuff in __config.py is unrelated.) From morph at debian.org Thu Sep 22 15:44:53 2011 From: morph at debian.org (Sandro Tosi) Date: Thu, 22 Sep 2011 21:44:53 +0200 Subject: [Numpy-discussion] Debug memory block at address p=0x2ba5928: API 'm' Message-ID: Hello, I'm running tests for numpy 1.6.1 on 2.7 -dbg flavor and I got this: .........................Debug memory block at address p=0x2ba5928: API 'm' 8 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail=0x2ba5930 are FORBIDDENBYTE, as expected. The block was made by call #1253108 to debug malloc/realloc. Data at p: a3 03 00 00 00 00 00 00 Fatal Python error: bad ID: Allocated using API 'm', verified using API 'o' Running unit tests for numpy NumPy version 1.6.1 and it got aborted. Running the tests with 2.6-dbg doesn't show this behavior. How can I help you debug it? Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From bsouthey at gmail.com Thu Sep 22 15:49:51 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 22 Sep 2011 14:49:51 -0500 Subject: [Numpy-discussion] Debug memory block at address p=0x2ba5928: API 'm' In-Reply-To: References: Message-ID: <4E7B915F.3020302@gmail.com> On 09/22/2011 02:44 PM, Sandro Tosi wrote: > Hello, > I'm running tests for numpy 1.6.1 on 2.7 -dbg flavor and I got this: > > .........................Debug memory block at address p=0x2ba5928: API 'm' > 8 bytes originally requested > The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. > The 8 pad bytes at tail=0x2ba5930 are FORBIDDENBYTE, as expected. > The block was made by call #1253108 to debug malloc/realloc. > Data at p: a3 03 00 00 00 00 00 00 > Fatal Python error: bad ID: Allocated using API 'm', verified using API 'o' > Running unit tests for numpy > NumPy version 1.6.1 > > and it got aborted. > > Running the tests with 2.6-dbg doesn't show this behavior. How can I > help you debug it? > > Cheers, All that I can tell you is that this is Ticket 1578: http://projects.scipy.org/numpy/ticket/1578 Bruce From morph at debian.org Thu Sep 22 15:53:19 2011 From: morph at debian.org (Sandro Tosi) Date: Thu, 22 Sep 2011 21:53:19 +0200 Subject: [Numpy-discussion] Debug memory block at address p=0x2ba5928: API 'm' In-Reply-To: <4E7B915F.3020302@gmail.com> References: <4E7B915F.3020302@gmail.com> Message-ID: On Thu, Sep 22, 2011 at 21:49, Bruce Southey wrote: > All that I can tell you is that this is Ticket 1578: > http://projects.scipy.org/numpy/ticket/1578 Ah, I could have probably checked it before sending this email, but now that webpage gives me 500 - it's spreading! :) -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From hniksic at gmail.com Fri Sep 23 04:33:58 2011 From: hniksic at gmail.com (=?UTF-8?B?SHJ2b2plIE5pa8WhacSH?=) Date: Fri, 23 Sep 2011 10:33:58 +0200 Subject: [Numpy-discussion] Ticket review process In-Reply-To: References: Message-ID: Hello! About a month ago I filed ticket #1943 (http://projects.scipy.org/numpy/ticket/1943), which describes a remaining unfixed item in #1388. Several days ago I also attached a patch and moved the ticket to the "needs review" state. Is there anything else I should or could do to get the patch reviewed and, if passing, applied? The bug can be worked around, but requires workarounds in all code that needs to use frombuffer with potentially empty buffers. From matthew.brett at gmail.com Fri Sep 23 13:48:27 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 23 Sep 2011 10:48:27 -0700 Subject: [Numpy-discussion] Ticket review process In-Reply-To: References: Message-ID: Hi, On Fri, Sep 23, 2011 at 1:33 AM, Hrvoje Nik?i? wrote: > Hello! > > About a month ago I filed ticket #1943 > (http://projects.scipy.org/numpy/ticket/1943), which describes a > remaining unfixed item in #1388. Several days ago I also attached a > patch and moved the ticket to the "needs review" state. > > Is there anything else I should or could do to get the patch reviewed > and, if passing, applied? The bug can be worked around, but requires > workarounds in all code that needs to use frombuffer with potentially > empty buffers. I might be wrong, but I have the impression that the most effective route for patches is via a pull request: http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html#asking-for-your-changes-to-be-merged-with-the-main-repo - especially if your changes include a test which will fail for the broken behavior. Best, Matthew From charlesr.harris at gmail.com Fri Sep 23 13:57:15 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 23 Sep 2011 11:57:15 -0600 Subject: [Numpy-discussion] Ticket review process In-Reply-To: References: Message-ID: On Fri, Sep 23, 2011 at 11:48 AM, Matthew Brett wrote: > Hi, > > On Fri, Sep 23, 2011 at 1:33 AM, Hrvoje Nik?i? wrote: > > Hello! > > > > About a month ago I filed ticket #1943 > > (http://projects.scipy.org/numpy/ticket/1943), which describes a > > remaining unfixed item in #1388. Several days ago I also attached a > > patch and moved the ticket to the "needs review" state. > > > > Is there anything else I should or could do to get the patch reviewed > > and, if passing, applied? The bug can be worked around, but requires > > workarounds in all code that needs to use frombuffer with potentially > > empty buffers. > > I might be wrong, but I have the impression that the most effective > route for patches is via a pull request: > > > http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html#asking-for-your-changes-to-be-merged-with-the-main-repo > > - especially if your changes include a test which will fail for the > broken behavior. > > Yep, We're falling behind. The real world is impacting hard at the moment. Matthew's suggestion is a good one. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Sep 23 14:37:34 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 23 Sep 2011 18:37:34 +0000 (UTC) Subject: [Numpy-discussion] Ticket review process References: Message-ID: On Fri, 23 Sep 2011 11:57:15 -0600, Charles R Harris wrote: [clip] > We're falling behind. The real world is impacting hard at the moment. > Matthew's suggestion is a good one. Should we stop recommending using Trac for contributions, and just say "use git"? (I think we recommend it somewhere.) I'd rather drown in pull requests than have a sea of patches hidden from view in Trac... Pauli From antonio.valentino at tiscali.it Fri Sep 23 14:39:52 2011 From: antonio.valentino at tiscali.it (Antonio Valentino) Date: Fri, 23 Sep 2011 20:39:52 +0200 Subject: [Numpy-discussion] ANN: PyTables 2.3 released Message-ID: <4E7CD278.20100@tiscali.it> =========================== Announcing PyTables 2.3 =========================== We are happy to announce PyTables 2.3. This release comes after about 10 months of development and after that Francesc Altet, the creator of PyTables, ceased activities with the project. Thank you Francesc. Also the project has been moved to GitHub: http://github.com/PyTables/PyTables. What's new ========== The main new features in 2.3 series are: * PyTables now includes the codebase of PyTables Pro (now release under open source license) gaining a lot of performance improvements and some new features like: - the new and powerful indexing engine: OPSI - a fine-tuned LRU cache for both metadata (nodes) and regular data * The entire documentation set has been converted to ReStructuredTest and Sphinx As always, a large amount of bugs have been addressed and squashed too. In case you want to know more in detail what has changed in this version, have a look at: http://pytables.github.com/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/2.3 For an on-line version of the manual, visit: http://pytables.github.com/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than 1 tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team From cournape at gmail.com Fri Sep 23 14:47:37 2011 From: cournape at gmail.com (David Cournapeau) Date: Fri, 23 Sep 2011 14:47:37 -0400 Subject: [Numpy-discussion] Ticket review process In-Reply-To: References: Message-ID: On Fri, Sep 23, 2011 at 2:37 PM, Pauli Virtanen wrote: > On Fri, 23 Sep 2011 11:57:15 -0600, Charles R Harris wrote: > [clip] >> We're falling behind. The real world is impacting hard at the moment. >> Matthew's suggestion is a good one. > > Should we stop recommending using Trac for contributions, and just > say "use git"? (I think we recommend it somewhere.) Github in theory prevents us from doing stuff that could be useful like assigning mentor to patches. But realistically, at least for now, it seems that we barely have enough resources to deal with pull requests only, so I would concur. cheers, David From shish at keba.be Fri Sep 23 16:52:02 2011 From: shish at keba.be (Olivier Delalleau) Date: Fri, 23 Sep 2011 16:52:02 -0400 Subject: [Numpy-discussion] Weird upcast behavior with 1.6.x, working as intended? In-Reply-To: References: Message-ID: NB: I opened a ticket (http://projects.scipy.org/numpy/ticket/1949) about this, in case it would help getting some attention on this issue. Besides this, I've been experimenting with the cast mechanisms of mixed scalar / array operations in numpy 1.6.1 on a Linux x86_64 architecture, and I can't make sense out of the current behavior. Here are some experiments adding a two-element array to a scalar (both of integer types): (1) [0 0] (int8) + 0 (int32) -> [0 0] (int8) (2) [0 0] (int8) + 127 (int32) -> [127 127] (int16) (3) [0 0] (int8) + -128 (int32) -> [-128 -128] (int8) (4) [0 0] (int8) + 2147483647 (int32) -> [2147483647 2147483647] (int32) (5) [1 1] (int8) + 127 (int32) -> [128 128] (int16) (6) [1 1] (int8) + 2147483647 (int32) -> [-2147483648 -2147483648] (int32) (7) [127 127] (int8) + 1 (int32) -> [-128 -128] (int8) (8) [127 127] (int8) + 127 (int32) -> [254 254] (int16) Here are some examples of things that confuse me: - Output dtype in (2) is int16 while in (3) it is int8, although both results can be written as int8 - Adding a number that would cause an overflow causes the output dtype to be upgraded to a dtype that can hold the result in (5), but not in (6) - Adding a small int32 in (7) that causes an overflow makes it keep the base int8 dtype, but a bigger int32 (although still representable as an int8) in (8) makes it switch to int16 (if someone wonders, adding 126 instead of 127 in (8) would result in [-3 -3] (int8), so 127 is special for some reason). My feeling is actually that the logic is to try to downcast the scalar as much as possible without changing its value, but with a bug that 127 is not downcasted to int8, and remains int16 (!). Some more behavior that puzzles me, this time comparing + vs -: (9) [0 0] (uint32) + -1 (int32) -> [-1 -1] (int64) (10) [0 0] (uint32) - 1 (int32) -> [4294967295 4294967295] (uint32) Here I would expect that adding -1 would be the same as subtracting 1, but that is not the case. Is there anyone with intimate knowledge of the numpy casting behavior for mixed scalar / array operations who could explain what are the rules governing it? Thanks, -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdu.xiaojf at gmail.com Sat Sep 24 00:24:28 2011 From: fdu.xiaojf at gmail.com (fdu.xiaojf at gmail.com) Date: Sat, 24 Sep 2011 12:24:28 +0800 Subject: [Numpy-discussion] How to plot heatmap in matplotlib? Message-ID: <4E7D5B7C.1080906@gmail.com> Dear all, Heatmap (like those on the page http://www2.warwick.ac.uk/fac/sci/moac/students/peter_cock/r/heatmap/) is a frequently used type of image in microarray data analysis. However, it seems there are no convenient functions in matplotlib to plot heatmap (please correct me if I was wrong), so I'm planning to write my own. Let me take the heatmap by the link http://www2.warwick.ac.uk/fac/sci/moac/students/peter_cock/r/heatmap/scaled_color_key.png as an example, which is produced by R. With my limited knowledge and expertise of matplotlib, I have the following questions and I hope you guys could help me. 1) I tend to use pcolor to draw the colormap in the central area. However, I've seen a lot of examples draw colormap with imshow. What's the difference between pcolor and imshow? Shall I use pcolor or imshow to produce the heatmap in the link above? 2) How to draw the dendrograms on the top and left of the colormap? I got hints from http://matplotlib.sourceforge.net/examples/axes_grid/scatter_hist.html on how to append axes to current plot, but I still have now idea how to draw the dengrograms. 3) How to draw the column side colormap (the smaller one) between the top dendrogram and the big colormap? 4) I can use colorbar to draw a colorbar, but how to place the colorbar on the topleft of the image just as the R heatmap does? 5) Any other suggestions on how to draw the heatmap? Thanks and any help will be greatly appreciated. Regards, Jianfeng -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Sat Sep 24 10:36:07 2011 From: shish at keba.be (Olivier Delalleau) Date: Sat, 24 Sep 2011 10:36:07 -0400 Subject: [Numpy-discussion] How to plot heatmap in matplotlib? In-Reply-To: <4E7D5B7C.1080906@gmail.com> References: <4E7D5B7C.1080906@gmail.com> Message-ID: I'm sorry I can't help on your questions, but maybe you should ask on matplotlib-users instead ( https://lists.sourceforge.net/lists/listinfo/matplotlib-users). -=- Olivier 2011/9/24 fdu.xiaojf at gmail.com > ** > Dear all, > > Heatmap (like those on the page > http://www2.warwick.ac.uk/fac/sci/moac/students/peter_cock/r/heatmap/) is > a frequently used type of image in microarray data analysis. However, it > seems there are no convenient functions in matplotlib to plot heatmap > (please correct me if I was wrong), so I'm planning to write my own. > > Let me take the heatmap by the link > http://www2.warwick.ac.uk/fac/sci/moac/students/peter_cock/r/heatmap/scaled_color_key.pngas an example, which is produced by R. > > With my limited knowledge and expertise of matplotlib, I have the following > questions and I hope you guys could help me. > > 1) I tend to use pcolor to draw the colormap in the central area. However, > I've seen a lot of examples draw colormap with imshow. > > What's the difference between pcolor and imshow? > Shall I use pcolor or imshow to produce the heatmap in the link above? > > 2) How to draw the dendrograms on the top and left of the colormap? > > I got hints from > http://matplotlib.sourceforge.net/examples/axes_grid/scatter_hist.html on > how to append axes to current plot, but I still have now idea how to draw > the dengrograms. > > 3) How to draw the column side colormap (the smaller one) between the top > dendrogram and the big colormap? > > 4) I can use colorbar to draw a colorbar, but how to place the colorbar on > the topleft of the image just as the R heatmap does? > > 5) Any other suggestions on how to draw the heatmap? > > Thanks and any help will be greatly appreciated. > > Regards, > Jianfeng > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hniksic at gmail.com Sun Sep 25 09:22:22 2011 From: hniksic at gmail.com (=?UTF-8?B?SHJ2b2plIE5pa8WhacSH?=) Date: Sun, 25 Sep 2011 15:22:22 +0200 Subject: [Numpy-discussion] Ticket review process In-Reply-To: References: Message-ID: Thanks for the info. I've now created a pull request for https://github.com/hniksic/numpy/commit/88971ce96e7137b17fd93cf5a3e6409f91d4d2ff Hrvoje From oc-spam66 at laposte.net Sun Sep 25 12:23:05 2011 From: oc-spam66 at laposte.net (OC) Date: Sun, 25 Sep 2011 18:23:05 +0200 Subject: [Numpy-discussion] array.tofile() refuses to write into a StringIO Message-ID: <4E7F5569.3050403@laposte.net> Hello, as said in the subject, the following code produces an error. Is it normal ? ********************************************************** A = r_[1] file_buffer = StringIO() A.tofile(file_buffer) IOError: first argument must be a string or open file ********************************************************** From pav at iki.fi Sun Sep 25 17:33:10 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 25 Sep 2011 23:33:10 +0200 Subject: [Numpy-discussion] array.tofile() refuses to write into a StringIO In-Reply-To: <4E7F5569.3050403@laposte.net> References: <4E7F5569.3050403@laposte.net> Message-ID: On 25.09.2011 18:23, OC wrote: > as said in the subject, the following code produces an error. Is it normal ? > > ********************************************************** > A = r_[1] > file_buffer = StringIO() > A.tofile(file_buffer) > > IOError: first argument must be a string or open file > ********************************************************** Yes, this is a known shortcoming of .tofile(). -- Pauli Virtanen From Chris.Barker at noaa.gov Sun Sep 25 19:57:01 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Sun, 25 Sep 2011 16:57:01 -0700 Subject: [Numpy-discussion] array.tofile() refuses to write into a StringIO In-Reply-To: References: <4E7F5569.3050403@laposte.net> Message-ID: <4E7FBFCD.2090902@noaa.gov> > On 25.09.2011 18:23, OC wrote: >> as said in the subject, the following code produces an error. Is it normal ? >> >> ********************************************************** >> A = r_[1] >> file_buffer = StringIO() >> A.tofile(file_buffer) >> >> IOError: first argument must be a string or open file >> ********************************************************** if you want to write to a string, why not use .tostring()? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From keith.hughitt at gmail.com Mon Sep 26 10:19:15 2011 From: keith.hughitt at gmail.com (Keith Hughitt) Date: Mon, 26 Sep 2011 10:19:15 -0400 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance Message-ID: Hi all, Myself and several colleagues have recently started work on a Python library for solar physics , in order to provide an alternative to the current mainstay for solar physics, which is written in IDL. One of the first steps we have taken is to create a Python portof a popular benchmark for IDL (time_test3) which measures performance for a variety of (primarily matrix) operations. In our initial attempt, however, Python performs significantly poorer than IDL for several of the tests. I have attached a graph which shows the results for one machine: the x-axis is the test # being compared, and the y-axis is the time it took to complete the test, in milliseconds. While it is possible that this is simply due to limitations in Python/Numpy, I suspect that this is due at least in part to our lack in familiarity with NumPy and SciPy. So my question is, does anyone see any places where we are doing things very inefficiently in Python? In order to try and ensure a fair comparison between IDL and Python there are some things (e.g. the style of timing and output) which we have deliberately chosen to do a certain way. In other cases, however, it is likely that we just didn't know a better method. Any feedback or suggestions people have would be greatly appreciated. Unfortunately, due to the proprietary nature of IDL, we cannot share the original version of time_test3, but hopefully the comments in time_test3.py will be clear enough. Thanks! Keith -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sunpy_time_test3_idl_python_2011-09-26.png Type: image/png Size: 10272 bytes Desc: not available URL: From johann.cohentanugi at gmail.com Mon Sep 26 10:38:39 2011 From: johann.cohentanugi at gmail.com (Johann Cohen-Tanugi) Date: Mon, 26 Sep 2011 16:38:39 +0200 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: References: Message-ID: <4E808E6F.6010605@gmail.com> hi Keith, I do not think that your primary concern should be with this kind of speed test at this stage : 1/ rest assured that this sort of tests have been performed in other contexts, and you can always do some hard work on high level computing languages like IDL and python to improve performance 2/ "early optimization is the root of all evil" (Knuth?) 3/ I believe that your primary motivation is to provide an alternative library to a proprietary software. If this is so, then your effort is most welcome and I would suggest first to port an interesting but small piece of the IDL solar physics lib and then study the path to speed improvements on such a concrete use case. As for your python time_test3, if it is a benchmark code proprietary to the IDL codebas, there is no wonder it performs well there! :) At any rate, I would suggest simplifying your code with ipython : In [1]: import numpy as np In [2]: a = np.zeros([512, 512], dtype=np.uint8) In [3]: a[200:250, 200:250] = 10 In [4]: from scipy import ndimage In [5]: %timeit ndimage.filters.median_filter(a, size=(5, 5)) 10 loops, best of 3: 98 ms per loop I am not sure what unit is your vertical axis.... best, Johann On 09/26/2011 04:19 PM, Keith Hughitt wrote: > Hi all, > > Myself and several colleagues have recently started work on a Python > library for solar physics , in order to provide > an alternative to the current mainstay for solar physics > , which is written in IDL. > > One of the first steps we have taken is to create a Python port > > of a popular benchmark for IDL (time_test3) which measures performance > for a variety of (primarily matrix) operations. In our initial > attempt, however, Python performs significantly poorer than IDL for > several of the tests. I have attached a graph which shows the results > for one machine: the x-axis is the test # being compared, and the > y-axis is the time it took to complete the test, in milliseconds. > While it is possible that this is simply due to limitations in > Python/Numpy, I suspect that this is due at least in part to our lack > in familiarity with NumPy and SciPy. > > So my question is, does anyone see any places where we are doing > things very inefficiently in Python? > > In order to try and ensure a fair comparison between IDL and Python > there are some things (e.g. the style of timing and output) which we > have deliberately chosen to do a certain way. In other cases, however, > it is likely that we just didn't know a better method. > > Any feedback or suggestions people have would be greatly appreciated. > Unfortunately, due to the proprietary nature of IDL, we cannot share > the original version of time_test3, but hopefully the comments in > time_test3.py will be clear enough. > > Thanks! > Keith > > -- > This message has been scanned for viruses and > dangerous content by *MailScanner* , and is > believed to be clean. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy-discussion at maubp.freeserve.co.uk Mon Sep 26 11:12:35 2011 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Mon, 26 Sep 2011 16:12:35 +0100 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: References: Message-ID: On Mon, Sep 26, 2011 at 3:19 PM, Keith Hughitt wrote: > Hi all, > Myself and several colleagues have recently started work on a Python library > for solar physics, in order to?provide an alternative to the current > mainstay for solar physics, which is written in IDL. > One of the first steps we have taken is to create a Python port of a popular > benchmark for IDL (time_test3) which measures performance for a variety of > (primarily matrix) operations. In our initial attempt, however, Python > performs significantly poorer than IDL for several of the tests. I have > attached a graph which shows the results for one machine: the x-axis is the > test # being compared, and the y-axis is the time it took to complete the > test, in milliseconds. While it is possible that this is simply due to > limitations in Python/Numpy, I suspect that this is due at least in part to > our lack in familiarity with NumPy and SciPy. > > So my question is, does anyone see any places where we are doing things very > inefficiently in Python? Looking at the plot there are five stand out tests, 1,2,3, 6 and 21. Tests 1, 2 and 3 are testing Python itself (no numpy or scipy), but are things you should be avoiding when using numpy anyway (don't use loops, use vectorised calculations etc). This is test 6, #Test 6 - Shift 512 by 512 byte and store nrep = 300 * scale_factor for i in range(nrep): c = np.roll(np.roll(b, 10, axis=0), 10, axis=1) #pylint: disable=W0612 timer.log('Shift 512 by 512 byte and store, %d times.' % nrep) The precise contents of b are determined by the previous tests (is that deliberate - it makes testing it in isolation hard). I'm unsure what you are trying to do and if it is the best way. This is test 21, which is just calling a scipy function repeatedly. Questions about this might be better directed to the scipy mailing list - also check what version of SciPy etc you have. n = 2**(17 * scale_factor) a = np.arange(n, dtype=np.float32) ... #Test 21 - Smooth 512 by 512 byte array, 5x5 boxcar for i in range(nrep): b = scipy.ndimage.filters.median_filter(a, size=(5, 5)) timer.log('Smooth 512 by 512 byte array, 5x5 boxcar, %d times' % nrep) After than, tests 10, 15 and 18 stand out. Test 10 is another use of roll, so whatever advice you get on test 6 may apply. Test 10: #Test 10 - Shift 512 x 512 array nrep = 60 * scale_factor for i in range(nrep): c = np.roll(np.roll(b, 10, axis=0), 10, axis=1) #for i in range(nrep): c = d.rotate( timer.log('Shift 512 x 512 array, %d times' % nrep) Test 15 is a loop based version of 16, where Python wins. Test 18 is a loop based version of 19 (log), where the difference is small. So in terms of numpy speed, your question just seems to be about numpy.roll and how else one might achieve this result? Peter From charlesr.harris at gmail.com Mon Sep 26 11:21:32 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Sep 2011 09:21:32 -0600 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: References: Message-ID: On Mon, Sep 26, 2011 at 8:19 AM, Keith Hughitt wrote: > Hi all, > > Myself and several colleagues have recently started work on a Python > library for solar physics , in order to provide an > alternative to the current mainstay for solar physics, > which is written in IDL. > > One of the first steps we have taken is to create a Python portof a popular benchmark for IDL (time_test3) which measures performance for a > variety of (primarily matrix) operations. In our initial attempt, however, > Python performs significantly poorer than IDL for several of the tests. I > have attached a graph which shows the results for one machine: the x-axis is > the test # being compared, and the y-axis is the time it took to complete > the test, in milliseconds. While it is possible that this is simply due to > limitations in Python/Numpy, I suspect that this is due at least in part to > our lack in familiarity with NumPy and SciPy. > > So my question is, does anyone see any places where we are doing things > very inefficiently in Python? > > In order to try and ensure a fair comparison between IDL and Python there > are some things (e.g. the style of timing and output) which we have > deliberately chosen to do a certain way. In other cases, however, it is > likely that we just didn't know a better method. > > Any feedback or suggestions people have would be greatly appreciated. > Unfortunately, due to the proprietary nature of IDL, we cannot share the > original version of time_test3, but hopefully the comments in time_test3.py > will be clear enough. > > The first three tests are of Python loops over python lists, so I'm not much surprised at the results. Number 6 uses numpy roll, which is not implemented in a particularly efficient way, so could use some improvement. I haven't looked at the rest of the results, but I suspect they are similar. So in some cases I think the benchmark isn't particularly useful, but in a few others numpy could be improved. It would be interesting to see which features are actually widely used in IDL code and weight them accordingly. In general, for loops are to be avoided, but if some numpy routine is a bottleneck we should fix it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Mon Sep 26 11:24:01 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 26 Sep 2011 11:24:01 -0400 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: References: Message-ID: Hello Keith, While I also echo Johann's points about the arbitrariness and non-utility of benchmarking I'll briefly comment just on just a few tests to help out with getting things into idiomatic python/numpy: Tests 1 and 2 are fairly pointless (empty for loop and empty procedure) that won't actually influence the running time of well-written non-pathological code. Test 3: #Test 3 - Add 200000 scalar ints nrep = 2000000 * scale_factor for i in range(nrep): a = i + 1 well, python looping is slow... one doesn't do such loops in idiomatic code if the underlying intent can be re-cast into array operations in numpy. But here the test is on such a simple operation that it's not clear how to recast in a way that would remain reasonable. Ideally you'd test something like: i = numpy.arange(200000) for j in range(scale_factor): a = i + 1 but that sort of changes what the test is testing. Finally, test 21: #Test 21 - Smooth 512 by 512 byte array, 5x5 boxcar for i in range(nrep): b = scipy.ndimage.filters.median_filter(a, size=(5, 5)) timer.log('Smooth 512 by 512 byte array, 5x5 boxcar, %d times' % nrep) A median filter is definitely NOT a boxcar filter! You want "uniform_filter": In [4]: a = numpy.empty((1000,1000)) In [5]: timeit scipy.ndimage.filters.median_filter(a, size=(5, 5)) 10 loops, best of 3: 93.2 ms per loop In [6]: timeit scipy.ndimage.filters.uniform_filter(a, size=(5, 5)) 10 loops, best of 3: 27.7 ms per loop Zach On Sep 26, 2011, at 10:19 AM, Keith Hughitt wrote: > Hi all, > > Myself and several colleagues have recently started work on a Python library for solar physics, in order to provide an alternative to the current mainstay for solar physics, which is written in IDL. > > One of the first steps we have taken is to create a Python port of a popular benchmark for IDL (time_test3) which measures performance for a variety of (primarily matrix) operations. In our initial attempt, however, Python performs significantly poorer than IDL for several of the tests. I have attached a graph which shows the results for one machine: the x-axis is the test # being compared, and the y-axis is the time it took to complete the test, in milliseconds. While it is possible that this is simply due to limitations in Python/Numpy, I suspect that this is due at least in part to our lack in familiarity with NumPy and SciPy. > > So my question is, does anyone see any places where we are doing things very inefficiently in Python? > > In order to try and ensure a fair comparison between IDL and Python there are some things (e.g. the style of timing and output) which we have deliberately chosen to do a certain way. In other cases, however, it is likely that we just didn't know a better method. > > Any feedback or suggestions people have would be greatly appreciated. Unfortunately, due to the proprietary nature of IDL, we cannot share the original version of time_test3, but hopefully the comments in time_test3.py will be clear enough. > > Thanks! > Keith > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From njs at pobox.com Mon Sep 26 11:37:34 2011 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 26 Sep 2011 08:37:34 -0700 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: References: Message-ID: On Mon, Sep 26, 2011 at 8:24 AM, Zachary Pincus wrote: > Test 3: > ? ?#Test 3 - Add 200000 scalar ints > ? ?nrep = 2000000 * scale_factor > ? ?for i in range(nrep): > ? ? ? ?a = i + 1 > > well, python looping is slow... one doesn't do such loops in idiomatic code if the underlying intent can be re-cast into array operations in numpy. Also, in this particular case, what you're mostly measuring is how much time it takes to allocate a giant list of integers by calling 'range'. Using 'xrange' instead speeds things up by a factor of two: def f(): nrep = 2000000 for i in range(nrep): a = i + 1 def g(): nrep = 2000000 for i in xrange(nrep): a = i + 1 In [8]: timeit f() 10 loops, best of 3: 138 ms per loop In [9]: timeit g() 10 loops, best of 3: 72.1 ms per loop Usually I don't worry about the difference between xrange and range -- it doesn't really matter for small loops or loops that are doing more work inside each iteration -- and that's every loop I actually write in practice :-). And if I really did need to write a loop like this (lots of iterations with a small amount of work in each and speed is critical) then I'd use cython. But, you might as well get in the habit of using 'xrange'; it won't hurt and occasionally will help. -- Nathaniel From helmrp at yahoo.com Mon Sep 26 11:43:22 2011 From: helmrp at yahoo.com (The Helmbolds) Date: Mon, 26 Sep 2011 08:43:22 -0700 (PDT) Subject: [Numpy-discussion] Trouble installing numpy Message-ID: <1317051802.3868.YahooMailNeo@web31802.mail.mud.yahoo.com> Using Source Forge download of NumPy installer package: ? numpy-1.6.1-win32-superpack-python 2.7.exe. ? Installation Wizard starts, but then?installation fails with error message: ? Python version 2.7 required, which was not found in the registry ? Idle says it's using: ? Python 2.7.2 64 bit AMD64 on Win 32 ? So what's holding up the installation, and what do I need to do to install numpy? The Helmbolds 2645 E Southern Ave A241 Tempe AZ 85282 Email: helmrp at yahoo.com VOX: 480-831-3611 CELL: 602-568-6948 (but not often turned on) -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Mon Sep 26 11:52:11 2011 From: shish at keba.be (Olivier Delalleau) Date: Mon, 26 Sep 2011 11:52:11 -0400 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: References: Message-ID: One minor thing is you should use xrange rather than range. Although it will probably only make a difference for the empty loop ;) Otherwise, from what I can see, tests where numpy is really much worse are: - 1, 2, 3, 15, 18: Not numpy but Python related: for loops are not efficient - 6, 10: Maybe numpy.roll is indeed not efficiently implemented - 21: Same for this scipy function -=- Olivier 2011/9/26 Keith Hughitt > Hi all, > > Myself and several colleagues have recently started work on a Python > library for solar physics , in order to provide an > alternative to the current mainstay for solar physics, > which is written in IDL. > > One of the first steps we have taken is to create a Python portof a popular benchmark for IDL (time_test3) which measures performance for a > variety of (primarily matrix) operations. In our initial attempt, however, > Python performs significantly poorer than IDL for several of the tests. I > have attached a graph which shows the results for one machine: the x-axis is > the test # being compared, and the y-axis is the time it took to complete > the test, in milliseconds. While it is possible that this is simply due to > limitations in Python/Numpy, I suspect that this is due at least in part to > our lack in familiarity with NumPy and SciPy. > > So my question is, does anyone see any places where we are doing things > very inefficiently in Python? > > In order to try and ensure a fair comparison between IDL and Python there > are some things (e.g. the style of timing and output) which we have > deliberately chosen to do a certain way. In other cases, however, it is > likely that we just didn't know a better method. > > Any feedback or suggestions people have would be greatly appreciated. > Unfortunately, due to the proprietary nature of IDL, we cannot share the > original version of time_test3, but hopefully the comments in time_test3.py > will be clear enough. > > Thanks! > Keith > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 26 12:14:16 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Sep 2011 10:14:16 -0600 Subject: [Numpy-discussion] Trouble installing numpy In-Reply-To: <1317051802.3868.YahooMailNeo@web31802.mail.mud.yahoo.com> References: <1317051802.3868.YahooMailNeo@web31802.mail.mud.yahoo.com> Message-ID: On Mon, Sep 26, 2011 at 9:43 AM, The Helmbolds wrote: > Using Source Forge download of NumPy installer package: > numpy-1.6.1-win32-superpack-python 2.7.exe. > > Installation Wizard starts, but then installation fails with error message: > Python version 2.7 required, which was not found in the registry > > Idle says it's using: > Python 2.7.2 64 bit AMD64 on Win 32 > > So what's holding up the installation, and what do I need to do to install > numpy? > > Your Python is 64 bits, the numpy package is 32 bits and needs 32 bit Python. If you need free 64 bit numpy on windows your best bet is probably here . Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Mon Sep 26 12:32:08 2011 From: shish at keba.be (Olivier Delalleau) Date: Mon, 26 Sep 2011 12:32:08 -0400 Subject: [Numpy-discussion] Trouble installing numpy In-Reply-To: <1317051802.3868.YahooMailNeo@web31802.mail.mud.yahoo.com> References: <1317051802.3868.YahooMailNeo@web31802.mail.mud.yahoo.com> Message-ID: You are probably trying to install the 32 bit version of numpy on your 64 bit Python. Either switch to 64 bit numpy or 32 bit Python. -=- Olivier 2011/9/26 The Helmbolds > Using Source Forge download of NumPy installer package: > numpy-1.6.1-win32-superpack-python 2.7.exe. > > Installation Wizard starts, but then installation fails with error message: > Python version 2.7 required, which was not found in the registry > > Idle says it's using: > Python 2.7.2 64 bit AMD64 on Win 32 > > So what's holding up the installation, and what do I need to do to install > numpy? > > The Helmbolds > 2645 E Southern Ave A241 > Tempe AZ 85282 > Email: helmrp at yahoo.com > VOX: 480-831-3611 > CELL: 602-568-6948 (but not often turned on) > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From helmrp at yahoo.com Mon Sep 26 16:02:19 2011 From: helmrp at yahoo.com (The Helmbolds) Date: Mon, 26 Sep 2011 13:02:19 -0700 (PDT) Subject: [Numpy-discussion] Trouble installing numpy In-Reply-To: References: Message-ID: <1317067339.6349.YahooMailNeo@web31804.mail.mud.yahoo.com> Thanks. The error message did not refer to the difference between 32-bit and 64-bit, only to the supposed absence of Python 2.7. And Idle says it's using?Python 2.7.2 64 bit AMD64 on Win 32, which confused me.? ? However, I see that the suggested web page contains the following four options, each of which apparently offers a NumPy-like Windows installation package for 64-bit windows machines: ??Bottleneck-0.5.0.win-amd64-py2.7.exe ? numpy-MKL-1.6.1.win-amd64-py2.7.exe? [*-see comment below] ? numpy-unoptimized-1.6.1.win-amd64-py2.7.exe ? numscons-012.0.win-amd64-py2.7.exe [*-coment] This item is described as "not compatible with the official SciPy distributions." Any comments on which ones I should try first? The Helmbolds 2645 E Southern Ave A241 Tempe AZ 85282 Email: helmrp at yahoo.com VOX: 480-831-3611 CELL: 602-568-6948 (but not often turned on) >________________________________ >From: "numpy-discussion-request at scipy.org" >To: numpy-discussion at scipy.org >Sent: Monday, September 26, 2011 10:00 AM >Subject: NumPy-Discussion Digest, Vol 60, Issue 55 > >Send NumPy-Discussion mailing list submissions to >??? numpy-discussion at scipy.org > >To subscribe or unsubscribe via the World Wide Web, visit >??? http://mail.scipy.org/mailman/listinfo/numpy-discussion >or, via email, send a message with subject or body 'help' to >??? numpy-discussion-request at scipy.org > >You can reach the person managing the list at >??? numpy-discussion-owner at scipy.org > >When replying, please edit your Subject line so it is more specific >than "Re: Contents of NumPy-Discussion digest..." > > >Today's Topics: > >? 1. Re: Trouble installing numpy (Charles R Harris) >? 2. Re: Trouble installing numpy (Olivier Delalleau) > > >---------------------------------------------------------------------- > >Message: 1 >Date: Mon, 26 Sep 2011 10:14:16 -0600 >From: Charles R Harris >Subject: Re: [Numpy-discussion] Trouble installing numpy >To: Discussion of Numerical Python >Message-ID: >??? >Content-Type: text/plain; charset="iso-8859-1" > >On Mon, Sep 26, 2011 at 9:43 AM, The Helmbolds wrote: > >> Using Source Forge download of NumPy installer package: >>? numpy-1.6.1-win32-superpack-python 2.7.exe. >> >> Installation Wizard starts, but then installation fails with error message: >>? Python version 2.7 required, which was not found in the registry >> >> Idle says it's using: >>? Python 2.7.2 64 bit AMD64 on Win 32 >> >> So what's holding up the installation, and what do I need to do to install >> numpy? >> >> > >Your Python is 64 bits, the numpy package is 32 bits and needs 32 bit >Python. If you need free 64 bit numpy on windows your best bet is probably >here . > >Chuck >-------------- next part -------------- >An HTML attachment was scrubbed... >URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110926/2ea1a5ec/attachment-0001.html > >------------------------------ > >Message: 2 >Date: Mon, 26 Sep 2011 12:32:08 -0400 >From: Olivier Delalleau >Subject: Re: [Numpy-discussion] Trouble installing numpy >To: Discussion of Numerical Python >Message-ID: >??? >Content-Type: text/plain; charset="iso-8859-1" > >You are probably trying to install the 32 bit version of numpy on your 64 >bit Python. Either switch to 64 bit numpy or 32 bit Python. > >-=- Olivier > >2011/9/26 The Helmbolds > >> Using Source Forge download of NumPy installer package: >>? numpy-1.6.1-win32-superpack-python 2.7.exe. >> >> Installation Wizard starts, but then installation fails with error message: >>? Python version 2.7 required, which was not found in the registry >> >> Idle says it's using: >>? Python 2.7.2 64 bit AMD64 on Win 32 >> >> So what's holding up the installation, and what do I need to do to install >> numpy? >> >> The Helmbolds >> 2645 E Southern Ave A241 >> Tempe AZ 85282 >> Email: helmrp at yahoo.com >> VOX: 480-831-3611 >> CELL: 602-568-6948 (but not often turned on) >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >-------------- next part -------------- >An HTML attachment was scrubbed... >URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110926/ce5633df/attachment-0001.html > >------------------------------ > >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion > > >End of NumPy-Discussion Digest, Vol 60, Issue 55 >************************************************ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Mon Sep 26 17:28:00 2011 From: tmp50 at ukr.net (Dmitrey) Date: Tue, 27 Sep 2011 00:28:00 +0300 Subject: [Numpy-discussion] =?windows-1251?b?W0FOTl0gT0RFIGR5L2R0ID0gZih0?= =?windows-1251?q?=29_solver_with_guaranteed_speficiable_accuracy?= Message-ID: <96683.1317072480.9969521652689403904@ffe6.ukr.net> hi all, now free solver interalg from OpenOpt framework (based on interval analysis) can solve ODE dy/dt = f(t) with guaranteed specifiable accuracy. See the ODE webpage for more details, there is an example of comparison with scipy.integrate.odeint, where latter fails to solve a problem. Future plans include solving of some general ODE systems dy/dt = f(y, t). Regards, D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From highegg at gmail.com Tue Sep 27 04:40:31 2011 From: highegg at gmail.com (Jaroslav Hajek) Date: Tue, 27 Sep 2011 10:40:31 +0200 Subject: [Numpy-discussion] factorization updates in numpy Message-ID: hello, I've been looking into options to do fast low-rank dense matrix factorization updates (aka Matlab's qrupdate) with NumPy, but couldn't find basically anything. Did I miss something? A few years ago I wrote the qrupdate library http://sourceforge.net/projects/qrupdate/, and subsequently wrapped it for Octave, giving it a broad spectrum of efficient factorization updating capabilities - see http://www.gnu.org/software/octave/doc/interpreter/Matrix-Factorizations.html#doc%2dqrupdate, functions qrupdate, qrinsert, qrdelete, qrshift, cholupdate, cholinsert, choldelete, cholshift, luupdate. Would there be interest in having similar functions in NumPy? Would NumPy or SciPy be a better place? best regards -- Jaroslav Hajek Site Reliability Engineer Google Switzerland GmbH From oc-spam66 at laposte.net Tue Sep 27 05:14:31 2011 From: oc-spam66 at laposte.net (oc-spam66) Date: Tue, 27 Sep 2011 11:14:31 +0200 Subject: [Numpy-discussion] array.tofile() refuses to write into a StringIO In-Reply-To: <4E7FBFCD.2090902@noaa.gov> References: <4E7FBFCD.2090902@noaa.gov> Message-ID: <4E8193F7.1070908@laposte.net> > if you want to write to a string, why not use .tostring()? Because A.tostring() returns the binary data, while I would like the text representation. More precisely, I would like to use A.tofile(sep="\t"). > Yes, this is a known shortcoming of .tofile(). Is it worth filing a bug report ? -- O.C. From pav at iki.fi Tue Sep 27 05:27:07 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 27 Sep 2011 11:27:07 +0200 Subject: [Numpy-discussion] factorization updates in numpy In-Reply-To: References: Message-ID: 27.09.2011 10:40, Jaroslav Hajek kirjoitti: > I've been looking into options to do fast low-rank dense matrix > factorization updates (aka Matlab's qrupdate) with NumPy, but couldn't > find basically anything. Did I miss something? At the moment, we do not have equivalents to these routines. However, such additions would be very welcome! If you wish to contribute here, we would be interested in adding routines for doing these operations. There are already some places inside Scipy where such routines could be used. > A few years ago I wrote the qrupdate library > http://sourceforge.net/projects/qrupdate/, and subsequently wrapped it > for Octave, giving it a broad spectrum of efficient factorization > updating capabilities - see > http://www.gnu.org/software/octave/doc/interpreter/Matrix-Factorizations.html#doc%2dqrupdate, > functions qrupdate, qrinsert, qrdelete, qrshift, cholupdate, > cholinsert, choldelete, cholshift, luupdate. A > Would there be interest in having similar functions in NumPy? Would > NumPy or SciPy be a better place? These routines probably fit in the scope of Scipy: the linear algebra package in Numpy is kept intentionally bare-bones. There is one question, though: your qrupdate code is licensed under the GPL. To include it in Scipy, we need to have it distributable under a BSD-compatible license. Would you be willing to relicense it? -- Pauli Virtanen From oc-spam66 at laposte.net Tue Sep 27 06:03:46 2011 From: oc-spam66 at laposte.net (oc-spam66) Date: Tue, 27 Sep 2011 12:03:46 +0200 Subject: [Numpy-discussion] array.tofile() refuses to write into a StringIO In-Reply-To: <4E7FBFCD.2090902@noaa.gov> References: <4E7FBFCD.2090902@noaa.gov> Message-ID: <4E819F82.8080204@laposte.net> Ah, I found a workaround: savetxt() can work with a StringIO -> savetxt(file_buffer, A) This is only a workaround. I still think A.tofile() should be capable of writing into a StringIO. -- O.C. From zbyszek at in.waw.pl Tue Sep 27 10:15:41 2011 From: zbyszek at in.waw.pl (=?UTF-8?B?WmJpZ25pZXcgSsSZZHJ6ZWpld3NraS1Tem1law==?=) Date: Tue, 27 Sep 2011 16:15:41 +0200 Subject: [Numpy-discussion] Fancy indexing with masks In-Reply-To: References: Message-ID: <4E81DA8D.3080002@in.waw.pl> On 09/22/2011 12:09 PM, Pauli Virtanen wrote: > Thu, 22 Sep 2011 08:12:12 +0200, Han Genuit wrote: > [clip] >> I also noticed that it does strange things when using a list: >> >>>>> c[[True, False, True]] >> array([[3, 4, 5], >> [0, 1, 2], >> [3, 4, 5]]) > > It casts the list with booleans to an integer array. Probably shouldn't > work like that... Changing that would require looping over the list first to check if everything is an boolean, or maybe just looking at the first few elements. Either way pretty ugly. So I guess that the current (slightly surprising) behaviour has to stay. Zbyszek From dsdale24 at gmail.com Tue Sep 27 11:41:54 2011 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 27 Sep 2011 11:41:54 -0400 Subject: [Numpy-discussion] replacing the mechanism for dispatching ufuncs In-Reply-To: References: <87hb7i3uzx.fsf@ginnungagap.bsc.es> Message-ID: What is the status of this proposal? On Wed, Jun 22, 2011 at 6:56 PM, Mark Wiebe wrote: > On Wed, Jun 22, 2011 at 4:57 PM, Darren Dale wrote: >> >> On Wed, Jun 22, 2011 at 1:31 PM, Mark Wiebe wrote: >> > On Wed, Jun 22, 2011 at 7:34 AM, Llu?s wrote: >> >> >> >> Darren Dale writes: >> >> >> >> > On Tue, Jun 21, 2011 at 1:57 PM, Mark Wiebe >> >> > wrote: >> >> >> On Tue, Jun 21, 2011 at 12:36 PM, Charles R Harris >> >> >> wrote: >> >> >>> How does the ufunc get called so it doesn't get caught in an >> >> >>> endless >> >> >>> loop? >> >> >> >> > [...] >> >> >> >> >> The function being called needs to ensure this, either by extracting >> >> >> a >> >> >> raw >> >> >> ndarray from instances of its class, or adding a 'subok = False' >> >> >> parameter >> >> >> to the kwargs. >> >> >> >> > I didn't understand, could you please expand on that or show an >> >> > example? >> >> >> >> As I understood the initial description and examples, the ufunc >> >> overload >> >> will keep being used as long as its arguments are of classes that >> >> declare ufunc overrides (i.e., classes with the "_numpy_ufunc_" >> >> attribute). >> >> >> >> Thus Mark's comment saying that you have to either transform the >> >> arguments into raw ndarrays (either by creating new ones or passing a >> >> view) or use the "subok = False" kwarg parameter to break a possible >> >> overloading loop. >> > >> > The sequence of events is something like this: >> > 1. You call np.sin(x) >> > 2. The np.sin ufunc looks at x, sees the _numpy_ufunc_ attribute, and >> > calls >> > x._numpy_ufunc_(np.sin, x) >> > 3. _numpy_ufunc_ uses np.sin.name (which is "sin") to get the correct >> > my_sin >> > function to call >> > 4A. If my_sin called np.sin(x), we would go back to 1. and get an >> > infinite >> > loop >> > 4B. If x is a subclass of ndarray, my_sin can call np.sin(x, >> > subok=False), >> > as this disables the subclass overloading mechanism. >> > 4C. If x is not a subclass of ndarray, x needs to produce an ndarray, >> > for >> > instance it might have an x.arr property. Then it can call np.sin(x.arr) >> >> Ok, that seems straightforward and, for what its worth, it looks like >> it would meet my needs. However, I wonder if the _numpy_func_ >> mechanism is the best approach. This is a bit sketchy, but why not do >> something like: >> >> class Proxy: >> >> ? ?def __init__(self, ufunc, *args): >> ? ? ? ?self._ufunc = ufunc >> ? ? ? ?self._args = args >> >> ? ?def __call__(self, func): >> ? ? ? ?self._ufunc._registry[tuple(type(arg) for arg in self._args)] = >> func >> ? ? ? ?return func >> >> >> class UfuncObject: >> >> ? ?... >> >> ? ?def __call__(self, *args, **kwargs): >> ? ? ? ?func = self._registry.get(tuple(type(arg) for arg in args), None) >> ? ? ? ?if func is None: >> ? ? ? ? ? ?raise TypeError >> ? ? ? ?return func(*args, **kwargs) >> >> ? ?def register(self, *args): >> ? ? ? ?return Proxy(self, *args) >> >> >> @np.sin.register(Quantity) >> def sin(pq): >> ? ?if pq.units != degrees: >> ? ? ? ?pq = pq.rescale(degrees) >> ? ?temp = np.sin(pq.view(np.ndarray)) >> ? ?return Quantity(temp, copy=False) >> >> This way, classes don't have to implement special methods to support >> ufunc registration, special attributes to claim primacy in ufunc >> registration lookup, special versions of the functions for each numpy >> ufunc, *and* the logic to determine whether the combination of >> arguments is supported. By that I mean, if I call np.sum with a >> quantity and a masked array, and Quantity wins the __array_priority__ >> competition, then I also need to check that my special sum function(s) >> know how to operate on that combination of inputs. With the decorator >> approach, I just need to implement the special versions of the ufuncs, >> and the decorators handle the logic of knowing what combinations of >> arguments are supported. >> >> It might be worth considering using ABCs for registration and have >> UfuncObject use isinstance to determine the appropriate special >> function to call. > > The thing I'm not sure about with this idea is how to do something general > to modify all ufuncs for a particular class in one swoop. Having to > individually override everything would be a hassle, I think. I also don't > think this approach saves very much effort compared to the _numpy_ufunc_ > call approach, checking the types and raising NotImplemented if they aren't > what's wanted is pretty much the only difference, and that check could still > be handled by a decorator like you're showing. > -Mark > >> >> Darren >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From Chris.Barker at noaa.gov Tue Sep 27 11:48:38 2011 From: Chris.Barker at noaa.gov (Chris.Barker) Date: Tue, 27 Sep 2011 08:48:38 -0700 Subject: [Numpy-discussion] array.tofile() refuses to write into a StringIO In-Reply-To: <4E8193F7.1070908@laposte.net> References: <4E7FBFCD.2090902@noaa.gov> <4E8193F7.1070908@laposte.net> Message-ID: <4E81F056.8030104@noaa.gov> On 9/27/11 2:14 AM, oc-spam66 wrote: >> if you want to write to a string, why not use .tostring()? > > Because A.tostring() returns the binary data, while I would like the text representation. > More precisely, I would like to use A.tofile(sep="\t"). I see -- I've always thought mingling binary and text reading and writing was an API mistake. > Ah, I found a workaround: savetxt() can work with a StringIO > -> savetxt(file_buffer, A) > > This is only a workaround. I still think A.tofile() should be capable of > writing into a StringIO. or .tostring() should support text (but that's what savetxt is for) Anyway, the text behavior of fromfile() and tofile() is broken in many ways -- it really need a major refactor,and yes,this is one issue that it would be nice to address. Or just get deprecate the text functionality. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From shish at keba.be Tue Sep 27 12:32:08 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 27 Sep 2011 12:32:08 -0400 Subject: [Numpy-discussion] Fancy indexing with masks In-Reply-To: <4E81DA8D.3080002@in.waw.pl> References: <4E81DA8D.3080002@in.waw.pl> Message-ID: 2011/9/27 Zbigniew J?drzejewski-Szmek > On 09/22/2011 12:09 PM, Pauli Virtanen wrote: > > Thu, 22 Sep 2011 08:12:12 +0200, Han Genuit wrote: > > [clip] > >> I also noticed that it does strange things when using a list: > >> > >>>>> c[[True, False, True]] > >> array([[3, 4, 5], > >> [0, 1, 2], > >> [3, 4, 5]]) > > > > It casts the list with booleans to an integer array. Probably shouldn't > > work like that... > Changing that would require looping over the list first to check if > everything is an boolean, or maybe just looking at the first few > elements. Either way pretty ugly. So I guess that the current (slightly > surprising) behaviour has to stay. > > Ugly implementation is better than ugly behavior IMO. -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From hangenuit at gmail.com Tue Sep 27 14:42:43 2011 From: hangenuit at gmail.com (Han Genuit) Date: Tue, 27 Sep 2011 20:42:43 +0200 Subject: [Numpy-discussion] Fancy indexing with masks In-Reply-To: References: <4E81DA8D.3080002@in.waw.pl> Message-ID: 2011/9/27 Olivier Delalleau > 2011/9/27 Zbigniew J?drzejewski-Szmek > >> On 09/22/2011 12:09 PM, Pauli Virtanen wrote: >> > Thu, 22 Sep 2011 08:12:12 +0200, Han Genuit wrote: >> > [clip] >> >> I also noticed that it does strange things when using a list: >> >> >> >>>>> c[[True, False, True]] >> >> array([[3, 4, 5], >> >> [0, 1, 2], >> >> [3, 4, 5]]) >> > >> > It casts the list with booleans to an integer array. Probably shouldn't >> > work like that... >> Changing that would require looping over the list first to check if >> everything is an boolean, or maybe just looking at the first few >> elements. Either way pretty ugly. So I guess that the current (slightly >> surprising) behaviour has to stay. >> >> > Ugly implementation is better than ugly behavior IMO. > > -=- Olivier > > It should also be possible to convert the list to a NumPy array before doing the actual indexing, then you won't lose consistency. -------------- next part -------------- An HTML attachment was scrubbed... URL: From grove.steyn at gmail.com Wed Sep 28 12:15:44 2011 From: grove.steyn at gmail.com (=?utf-8?b?R3JvdsOp?=) Date: Wed, 28 Sep 2011 16:15:44 +0000 (UTC) Subject: [Numpy-discussion] Problem with importing csv into datetime64 Message-ID: Hi, I am trying out the latest version of numpy 2.0 dev: np.__version__ Out[44]: '2.0.0.dev-aded70c' I am trying to import CSV data that looks like this: date,system,pumping,rgt,agt,sps,eskom_import,temperature,wind,pressure,weather 2007-01-01 00:30,481.9,,,,,481.9,15,SW,1040,Fine 2007-01-01 01:00,471.9,,,,,471.9,15,SW,1040,Fine 2007-01-01 01:30,455.9,,,,,455.9,,,, etc. by using the following code: convertdict = {0: lambda s: np.datetime64(s, 'm'), 1: lambda s: float(s or 0), 2: lambda s: float(s or 0), 3: lambda s: float(s or 0), 4: lambda s: float(s or 0), 5: lambda s: float(s or 0), 6: lambda s: float(s or 0), 7: lambda s: float(s or 0), 8: str, 9: str, 10: str} dt = [('date', np.datetime64),('system', float), ('pumping', float),('rgt', float), ('agt', float), ('sps', float) ,('eskom_import', float), ('temperature', float), ('wind', str), ('pressure', float), ('weather', str)] a = np.recfromcsv(fp, dtype=dt, converters=convertdict, usecols=range(0-11), names=True) The dtype it generates for a.date is 'object': array([2007-01-01T00:30+0200, 2007-01-01T01:00+0200, 2007-01-01T01:30+0200, ..., 2007-12-31T23:00+0200, 2007-12-31T23:30+0200, 2008-01-01T00:00+0200], dtype=object) But I need it to be datetime64, like in this example (but including hrs and minutes): array(['2011-07-11', '2011-07-12', '2011-07-13', '2011-07-14', '2011-07-15', '2011-07-16', '2011-07-17'], dtype='datetime64[D]') It seems that the CSV import creates an embedded object datetype for 'date' rather than a datetime64 data type. Any ideas on how to fix this? Grov? From alan.isaac at gmail.com Wed Sep 28 23:09:35 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Wed, 28 Sep 2011 23:09:35 -0400 Subject: [Numpy-discussion] should the return type of matlib.reshape be ndarray or matrix? Message-ID: <4E83E16F.7050304@gmail.com> Is this the intended behavior? >>> from numpy import matlib >>> m = matlib.reshape([1,2],(2,1)) >>> type(m) For any 2d shape, I expected a matrix. (And probably an exception if the shape is not 2d.) Thanks, Alan Isaac From cjordan1 at uw.edu Thu Sep 29 02:51:33 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Wed, 28 Sep 2011 23:51:33 -0700 Subject: [Numpy-discussion] Problem with importing csv into datetime64 In-Reply-To: References: Message-ID: On Wed, Sep 28, 2011 at 9:15 AM, Grov? wrote: > Hi, > > I am trying out the latest version of numpy 2.0 dev: > > np.__version__ > Out[44]: '2.0.0.dev-aded70c' > > I am trying to import CSV data that looks like this: > > date,system,pumping,rgt,agt,sps,eskom_import,temperature,wind,pressure,weather > 2007-01-01 00:30,481.9,,,,,481.9,15,SW,1040,Fine > 2007-01-01 01:00,471.9,,,,,471.9,15,SW,1040,Fine > 2007-01-01 01:30,455.9,,,,,455.9,,,, > etc. > > by using the following code: > > convertdict = {0: lambda s: np.datetime64(s, 'm'), 1: lambda s: float(s or 0), > 2: lambda s: float(s or 0), 3: lambda s: float(s or 0), 4: lambda s: float(s or > 0), 5: lambda s: float(s or 0), 6: lambda s: float(s or 0), 7: lambda s: float(s > or 0), 8: str, 9: str, 10: str} > dt = [('date', np.datetime64),('system', float), ('pumping', float),('rgt', > float), ('agt', float), ('sps', float) ,('eskom_import', float), ('temperature', > float), ('wind', str), ('pressure', float), ('weather', str)] > a = np.recfromcsv(fp, dtype=dt, converters=convertdict, usecols=range(0-11), > names=True) > > The dtype it generates for a.date is 'object': > > array([2007-01-01T00:30+0200, 2007-01-01T01:00+0200, 2007-01-01T01:30+0200, > ? ? ? ..., 2007-12-31T23:00+0200, 2007-12-31T23:30+0200, > ? ? ? 2008-01-01T00:00+0200], dtype=object) > > But I need it to be datetime64, like in this example (but including hrs and > minutes): > > array(['2011-07-11', '2011-07-12', '2011-07-13', '2011-07-14', > ? ? ? '2011-07-15', '2011-07-16', '2011-07-17'], dtype='datetime64[D]') > > It seems that the CSV import creates an embedded object datetype for 'date' > rather than a datetime64 data type. ?Any ideas on how to fix this? > > Grov? > Not sure how big your file is, but you might take a look at the loadtable branch on my numpy fork: https://github.com/chrisjordansquire/numpy. It has a function loadtable, with some docs and tests. It currently only loads dates, but you could likely modify it to handle datetimes as well without too much trouble. (Well, it should be pretty simple once you kinda grok how the function works. Unfortunately it's somewhat large and complicated, so it might not be what you want if you just want to load your date quick and be done with it.) -Chris JS > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From grove.steyn at gmail.com Thu Sep 29 10:27:59 2011 From: grove.steyn at gmail.com (=?utf-8?b?R3JvdsOp?=) Date: Thu, 29 Sep 2011 14:27:59 +0000 (UTC) Subject: [Numpy-discussion] datetimes with date vs time units, local time, and time zones References: Message-ID: Hi Mark Did you ever get to write: date_as_datetime(datearray, hour, minute, second, microsecond, timezone='local', unit=None, out=None) and datetime_as_date(datetimearray, timezone='local', out=None) ? I am looking for an easy way of using datetime[m] data to test for business days and do half hourly comparisons. I am using: In [181]: np.__version__ Out[181]: '2.0.0.dev-aded70c' Regards Grov? Steyn From keith.hughitt at gmail.com Thu Sep 29 12:11:57 2011 From: keith.hughitt at gmail.com (Keith Hughitt) Date: Thu, 29 Sep 2011 12:11:57 -0400 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: References: Message-ID: Thank you all for the comments and suggestions. First off, I would like to say that I entirely agree with people's suggestions about lack of objectiveness in the test design, and the caveat about optimizing early. The main reason we put together the Python version of the benchmark was as a quick "sanity check" to make sure that there are no major show-stoppers before we began work on the library. We also wanted to put together something to show other people who are firmly in the IDL camp that this is a viable option. We did in fact put together another short test-suite (test_testr.py & time_testr.pro ) which consists of operations that would are frequently used by us, but it also is testing a very small portion of the kinds of things our library will eventually do. That said, I made a few small changes to the original benchmark, based on people's feedback, and put together a new plot. The changes made include: 1. Using xrange instead of range 2. Using uniform filter instead of median filter 3. Fixed a typo for tests 2 & 3 which resulted in slower Python results Again, note that some of the tests are testing non-numpy functionality. Several of the results still stand out, but overall the results are much more reasonable than before. Cheers, Keith -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: time_test3_idl_vs_python.png Type: image/png Size: 54787 bytes Desc: not available URL: From zachary.pincus at yale.edu Thu Sep 29 12:18:44 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 29 Sep 2011 12:18:44 -0400 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: References: Message-ID: <545CBEFF-FEF6-4C15-B53F-0DF1A7A820C1@yale.edu> I think the remaining delta between the integer and float "boxcar" smoothing is that the integer version (test 21) still uses median_filter(), while the float one (test 22) is using uniform_filter(), which is a boxcar. Other than that and the slow roll() implementation in numpy, things look pretty solid, yes? Zach On Sep 29, 2011, at 12:11 PM, Keith Hughitt wrote: > Thank you all for the comments and suggestions. > > First off, I would like to say that I entirely agree with people's suggestions about lack of objectiveness in the test design, and the caveat about optimizing early. The main reason we put together the Python version of the benchmark was as a quick "sanity check" to make sure that there are no major show-stoppers before we began work on the library. We also wanted to put together something to show other people who are firmly in the IDL camp that this is a viable option. > > We did in fact put together another short test-suite (test_testr.py & time_testr.pro) which consists of operations that would are frequently used by us, but it also is testing a very small portion of the kinds of things our library will eventually do. > > That said, I made a few small changes to the original benchmark, based on people's feedback, and put together a new plot. > > The changes made include: > > 1. Using xrange instead of range > 2. Using uniform filter instead of median filter > 3. Fixed a typo for tests 2 & 3 which resulted in slower Python results > > Again, note that some of the tests are testing non-numpy functionality. Several of the results still stand out, but overall the results are much more reasonable than before. > > Cheers, > Keith > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From keith.hughitt at gmail.com Thu Sep 29 14:46:51 2011 From: keith.hughitt at gmail.com (Keith Hughitt) Date: Thu, 29 Sep 2011 14:46:51 -0400 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: <545CBEFF-FEF6-4C15-B53F-0DF1A7A820C1@yale.edu> References: <545CBEFF-FEF6-4C15-B53F-0DF1A7A820C1@yale.edu> Message-ID: Ah. Thanks for catching that! Otherwise though I think everything looks pretty good. Thanks all, Keith On Thu, Sep 29, 2011 at 12:18 PM, Zachary Pincus wrote: > I think the remaining delta between the integer and float "boxcar" > smoothing is that the integer version (test 21) still uses median_filter(), > while the float one (test 22) is using uniform_filter(), which is a boxcar. > > Other than that and the slow roll() implementation in numpy, things look > pretty solid, yes? > > Zach > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.verelst at gmail.com Fri Sep 30 09:00:21 2011 From: david.verelst at gmail.com (David Verelst) Date: Fri, 30 Sep 2011 15:00:21 +0200 Subject: [Numpy-discussion] Comparing NumPy/IDL Performance In-Reply-To: References: <545CBEFF-FEF6-4C15-B53F-0DF1A7A820C1@yale.edu> Message-ID: Just want to point to some excellent material that was recently presented at the course Advanced Scientific Programming in Pythonat St Andrews. Day 3 was titled "The Quest for Speed" (see https://python.g-node.org/wiki/schedule) and might interest you as well. Regards, David On 29 September 2011 20:46, Keith Hughitt wrote: > Ah. Thanks for catching that! > > Otherwise though I think everything looks pretty good. > > Thanks all, > Keith > > On Thu, Sep 29, 2011 at 12:18 PM, Zachary Pincus wrote: > >> I think the remaining delta between the integer and float "boxcar" >> smoothing is that the integer version (test 21) still uses median_filter(), >> while the float one (test 22) is using uniform_filter(), which is a boxcar. >> >> Other than that and the slow roll() implementation in numpy, things look >> pretty solid, yes? >> >> Zach >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Fri Sep 30 11:03:40 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Fri, 30 Sep 2011 08:03:40 -0700 Subject: [Numpy-discussion] nditer: possible to manually handle dimensions with different lengths? Message-ID: Using nditer, is it possible to manually handle dimensions with different lengths? For example, lets say I had an array A[5, 100] and I wanted to sample every 10 along the second axis so I would end up with an array B[5,10]. Is it possible to do this with nditer, handling the iteration over the second axis manually of course (probably in cython)? I want something like this (modified from http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#putting-the-inner-loop-in-cython ) @cython.boundscheck(False) def sum_squares_cy(arr): cdef np.ndarray[double] x cdef np.ndarray[double] y cdef int size cdef double value cdef int j axeslist = list(arr.shape) axeslist[1] = -1 out = zeros((arr.shape[0], 10)) it = np.nditer([arr, out], flags=['reduce_ok', 'external_loop', 'buffered', 'delay_bufalloc'], op_flags=[['readonly'], ['readwrite', 'no_broadcast']], op_axes=[None, axeslist], op_dtypes=['float64', 'float64']) it.operands[1][...] = 0 it.reset() for xarr, yarr in it: x = xarr y = yarr size = x.shape[0] j = 0 for i in range(size): #some magic here involving indexing into x[i] and y[j] return it.operands[1] Does this make sense? Is it possible to do? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsilter at gmail.com Fri Sep 30 13:52:04 2011 From: jsilter at gmail.com (Jacob Silterra) Date: Fri, 30 Sep 2011 13:52:04 -0400 Subject: [Numpy-discussion] Functions for finding the relative extrema of numeric data Message-ID: Hello all, I just opened a pull request to add scipy.signal._peak_finding.find_peaks, which finds the relative maxima in a 1d ndarray. The algorithm behind this function has been discussed on the scipy list, but here's a quick recap: The idea is basically convolve the function with a wavelet of varying widths (default is ricker, or "mexican hat" wavelet) and identify as peaks those locations where there are relative maxima (meaning they are larger than their nearest neighbor on each side) at each of several widths, and have a sufficiently high SNR. More details are provided in the docstrings of the functions. Pull request at https://github.com/scipy/scipy/pull/85. I believe the code quality/docstrings/test coverage are pretty good, however it would be of great benefit for someone with experience with similar algorithms to review the code. -Jacob PS I've sent this to both the scipy and numpy lists, apologies to those who get it twice. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Fri Sep 30 19:03:44 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 30 Sep 2011 16:03:44 -0700 Subject: [Numpy-discussion] datetimes with date vs time units, local time, and time zones In-Reply-To: References: Message-ID: 2011/9/29 Grov? > > Hi Mark > > Did you ever get to write: > > date_as_datetime(datearray, hour, minute, second, microsecond, > timezone='local', unit=None, out=None) > and > datetime_as_date(datetimearray, timezone='local', out=None) > ? > I never got to these functions, no. I am looking for an easy way of using datetime[m] data to test for business > days > and do half hourly comparisons. > > I am using: > > In [181]: np.__version__ > Out[181]: '2.0.0.dev-aded70c' > Here's a stopgap solution for converting to dates, that works for a single np.datetime64: def my_datetime_as_date(dt, timezone = 'local'): s = np.datetime_as_string(np.datetime64(dt), timezone=timezone) e = s.find('T') if e != -1: s = s[:e] return np.datetime64(s, 'D') >>> my_datetime_as_date('now') numpy.datetime64('2011-09-30') >>> my_datetime_as_date('2011-03-13T00:30Z') numpy.datetime64('2011-03-12') >>> my_datetime_as_date('2011-03-13T00:30') numpy.datetime64('2011-03-13') >>> my_datetime_as_date('2011-03-13T00:30Z', timezone='UTC') numpy.datetime64('2011-03-13') Cheers, Mark > > > Regards > > Grov? Steyn > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Fri Sep 30 19:21:09 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 30 Sep 2011 16:21:09 -0700 Subject: [Numpy-discussion] Weird upcast behavior with 1.6.x, working as intended? In-Reply-To: References: Message-ID: On Fri, Sep 23, 2011 at 1:52 PM, Olivier Delalleau wrote: > NB: I opened a ticket (http://projects.scipy.org/numpy/ticket/1949) about > this, in case it would help getting some attention on this issue. > A lot of what you're seeing here is due to changes I did for 1.6. I generally made the casting mechanism symmetric (before it could give different types depending on the order of the input arguments), and added a little bit of value-based casting for scalars to reduce some of the overflow that could happen. Before, it always downcast to the smallest-size type regardless of the value in the scalar. > Besides this, I've been experimenting with the cast mechanisms of mixed > scalar / array operations in numpy 1.6.1 on a Linux x86_64 architecture, and > I can't make sense out of the current behavior. Here are some experiments > adding a two-element array to a scalar (both of integer types): > > (1) [0 0] (int8) + 0 (int32) -> [0 0] (int8) > (2) [0 0] (int8) + 127 (int32) -> [127 127] (int16) > (3) [0 0] (int8) + -128 (int32) -> [-128 -128] (int8) > (4) [0 0] (int8) + 2147483647 (int32) -> [2147483647 2147483647] (int32) > (5) [1 1] (int8) + 127 (int32) -> [128 128] (int16) > (6) [1 1] (int8) + 2147483647 (int32) -> [-2147483648 -2147483648] (int32) > (7) [127 127] (int8) + 1 (int32) -> [-128 -128] (int8) > (8) [127 127] (int8) + 127 (int32) -> [254 254] (int16) > > Here are some examples of things that confuse me: > - Output dtype in (2) is int16 while in (3) it is int8, although both > results can be written as int8 > Here would be the cause of it: https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/convert_datatype.c#L1098 It should be a <= instead of a <, to include the value 127. > - Adding a number that would cause an overflow causes the output dtype to > be upgraded to a dtype that can hold the result in (5), but not in (6) > Actually, it's upgraded because of the previous point, not because of the overflow. With the change to <= above, this would produce int8 > - Adding a small int32 in (7) that causes an overflow makes it keep the > base int8 dtype, but a bigger int32 (although still representable as an > int8) in (8) makes it switch to int16 (if someone wonders, adding 126 > instead of 127 in (8) would result in [-3 -3] (int8), so 127 is special for > some reason). > > My feeling is actually that the logic is to try to downcast the scalar as > much as possible without changing its value, but with a bug that 127 is not > downcasted to int8, and remains int16 (!). > > Some more behavior that puzzles me, this time comparing + vs -: > (9) [0 0] (uint32) + -1 (int32) -> [-1 -1] (int64) > (10) [0 0] (uint32) - 1 (int32) -> [4294967295 4294967295] (uint32) > > Here I would expect that adding -1 would be the same as subtracting 1, but > that is not the case. > In the second case, it's equivalent to np.subtract(np.array([0, 0], np.uint32), np.int32(1)). The scalar 1 fits into the uint32, so the result type of the subtraction is uint32. In the first case, the scalar -1 does not fit into the uint32, so it is upgraded to int64. > > Is there anyone with intimate knowledge of the numpy casting behavior for > mixed scalar / array operations who could explain what are the rules > governing it? > Hopefully my explanations help a bit. I think this situation is less than ideal, and it would be better to do something more automatic, like doing an up-conversion on overflow. This would more closely emulate Python's behavior of integers never overflowing, at least until 64 bits. This kind of change would be a fair bit of work, and would likely reduce the performance of NumPy slightly. Cheers, Mark > > Thanks, > > -=- Olivier > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Fri Sep 30 19:34:47 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 30 Sep 2011 16:34:47 -0700 Subject: [Numpy-discussion] Comping Numpy in Cython In-Reply-To: References: Message-ID: On Tue, Sep 20, 2011 at 8:38 AM, Sean Poust wrote: > I keep getting this error: #warning "Using deprecated NumPy > API, disable it by #defining NPY_NO_DEPRECATED_API" and a bunch of > undefined variables when comping cython code with numpy > This is a warning introduced in current NumPy master. It's just a warning, so it's not the cause of the errors. > > I am new to cython and numpy and I am having trouble getting a *.pyx > extension > to compile with this in the header: > > from __future__ import division > import numpy as np > cimport numpy as np > DTYPE = np.int > ctypedef np.int_t DTYPE_t > > I complie to C code using: > cython -a montcarl.pyx > > And it seems to work fine. > > I then attempt to compile using: > > gcc -I /Developer/SDKs/MacOSX10.6.sdk/System/Library/Frameworks/ > Python.framework/Versions/2.6/include/python2.6 -I /usr/local/lib/ > python2.7/site-packages/numpy/core/include -shared -pthread -fPIC - > fwrapv -O2 -fno-strict-aliasing > -o montcar.so montcarl.c > > And I get a bunch of undefined variables: > > spoust-m:old spoust$ gcc $CFLAGS -o montcar.so montcarl.c > In file included from /usr/local/lib/python2.7/site-packages/numpy/ > core/include/numpy/ndarraytypes.h:1965, > from /usr/local/lib/python2.7/site-packages/numpy/ > core/include/numpy/ndarrayobject.h:17, > from /usr/local/lib/python2.7/site-packages/numpy/ > core/include/numpy/arrayobject.h:14, > from montcarl.c:225: > /usr/local/lib/python2.7/site-packages/numpy/core/include/numpy/ > npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy > API, disable it by #defining NPY_NO_DEPRECATED_API" > Undefined symbols: > "_PyLong_AsLong", referenced from: > ___Pyx_PyInt_AsLong in ccvkIDLQ.o > ___Pyx_PyInt_AsLong in ccvkIDLQ.o > ... > > Any ideas on how to get this to compile and how to get rid of these > numpy undefined variables? The cython guys directed me here. > PyLong_AsLong is a CPython API call: http://docs.python.org/c-api/long.html#PyLong_AsLong Looks like it's a problem linking with CPython, nothing to do with NumPy. I would recommend looking at the Python version numbers of all the parts in your build command line, I'm pretty sure that's where you will find your problem. ;) -Mark > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Fri Sep 30 19:38:03 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 30 Sep 2011 16:38:03 -0700 Subject: [Numpy-discussion] datetime64 y2k38 bug In-Reply-To: References: Message-ID: On Sun, Sep 18, 2011 at 8:52 PM, wrote: > On Sun, Sep 18, 2011 at 11:13 PM, Charles R Harris > wrote: > > > > > > On Sun, Sep 18, 2011 at 9:08 PM, Charles R Harris > > wrote: > >> > >> > >> On Sun, Sep 18, 2011 at 6:32 PM, Benjamin Root wrote: > >>> > >>> I was working on adding some test cases in numpy for the argmin/max > >>> functions with some datetime64s. I found that on my 32-bit machine, it > >>> fails to parse a date past the Y2.038k date. I find this odd because > the > >>> datetime is supposed to be 64-bits, but I guess there is some > arch-dependent > >>> code somewhere? > >>> > >> > >> I think that is actually POSIX for the time_t structure. Which is not to > >> say it's good ;) Google UNIX Year 2038 problem. ISTR reading recently > that > >> there is a movement afoot to fix the time_t structure on 32 bit machines > for > >> Linux. You've got to wonder, what were the POSIX people thinking? > >> > > > > See comments here. > > > > Thanks for the entertaining link > > " > I think it's still perfectly valid to say "you're a moron, > and we need to fix it" > " > (just a quote, doesn't apply to the python community) > > Josef > I've added a hack to try and work around this problem to the datetime-cleanup pull request: https://github.com/numpy/numpy/pull/161 Basically, for years >= 2038, it uses the year 2036 or 2037 (depending on whether it's a leap year), then adds the year offset back on. Everything already worked fine for me on my 64-bit platform, so it needs testing to confirm the fix works. -Mark > > > > > Chuck > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Fri Sep 30 20:04:41 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Fri, 30 Sep 2011 17:04:41 -0700 Subject: [Numpy-discussion] nditer: possible to manually handle dimensions with different lengths? In-Reply-To: References: Message-ID: On Fri, Sep 30, 2011 at 8:03 AM, John Salvatier wrote: > Using nditer, is it possible to manually handle dimensions with different > lengths? > > For example, lets say I had an array A[5, 100] and I wanted to sample every > 10 along the second axis so I would end up with an array B[5,10]. Is it > possible to do this with nditer, handling the iteration over the second axis > manually of course (probably in cython)? > > I want something like this (modified from > http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#putting-the-inner-loop-in-cython > ) > > @cython.boundscheck(False) > def sum_squares_cy(arr): > cdef np.ndarray[double] x > cdef np.ndarray[double] y > cdef int size > cdef double value > cdef int j > > axeslist = list(arr.shape) > axeslist[1] = -1 > > out = zeros((arr.shape[0], 10)) > it = np.nditer([arr, out], flags=['reduce_ok', 'external_loop', > 'buffered', 'delay_bufalloc'], > op_flags=[['readonly'], ['readwrite', 'no_broadcast']], > op_axes=[None, axeslist], > op_dtypes=['float64', 'float64']) > it.operands[1][...] = 0 > it.reset() > for xarr, yarr in it: > x = xarr > y = yarr > size = x.shape[0] > j = 0 > for i in range(size): > #some magic here involving indexing into x[i] and y[j] > return it.operands[1] > > Does this make sense? Is it possible to do? > I'm not sure I understand precisely what you're asking. Maybe you could reshape A to have shape [5, 10, 10], so that one of those 10's can match up with the 10 in B, perhaps with the op_axes? -Mark > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Fri Sep 30 23:58:35 2011 From: shish at keba.be (Olivier Delalleau) Date: Fri, 30 Sep 2011 23:58:35 -0400 Subject: [Numpy-discussion] Weird upcast behavior with 1.6.x, working as intended? In-Reply-To: References: Message-ID: 2011/9/30 Mark Wiebe > On Fri, Sep 23, 2011 at 1:52 PM, Olivier Delalleau wrote: > >> NB: I opened a ticket (http://projects.scipy.org/numpy/ticket/1949) about >> this, in case it would help getting some attention on this issue. >> > > A lot of what you're seeing here is due to changes I did for 1.6. I > generally made the casting mechanism symmetric (before it could give > different types depending on the order of the input arguments), and added a > little bit of value-based casting for scalars to reduce some of the overflow > that could happen. Before, it always downcast to the smallest-size type > regardless of the value in the scalar. > > >> Besides this, I've been experimenting with the cast mechanisms of mixed >> scalar / array operations in numpy 1.6.1 on a Linux x86_64 architecture, and >> I can't make sense out of the current behavior. Here are some experiments >> adding a two-element array to a scalar (both of integer types): >> >> (1) [0 0] (int8) + 0 (int32) -> [0 0] (int8) >> (2) [0 0] (int8) + 127 (int32) -> [127 127] (int16) >> (3) [0 0] (int8) + -128 (int32) -> [-128 -128] (int8) >> (4) [0 0] (int8) + 2147483647 (int32) -> [2147483647 2147483647] (int32) >> (5) [1 1] (int8) + 127 (int32) -> [128 128] (int16) >> (6) [1 1] (int8) + 2147483647 (int32) -> [-2147483648 -2147483648] >> (int32) >> (7) [127 127] (int8) + 1 (int32) -> [-128 -128] (int8) >> (8) [127 127] (int8) + 127 (int32) -> [254 254] (int16) >> >> Here are some examples of things that confuse me: >> - Output dtype in (2) is int16 while in (3) it is int8, although both >> results can be written as int8 >> > > Here would be the cause of it: > > > https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/convert_datatype.c#L1098 > > It should be a <= instead of a <, to include the value 127. > > >> - Adding a number that would cause an overflow causes the output dtype to >> be upgraded to a dtype that can hold the result in (5), but not in (6) >> > > Actually, it's upgraded because of the previous point, not because of the > overflow. With the change to <= above, this would produce int8 > > >> - Adding a small int32 in (7) that causes an overflow makes it keep the >> base int8 dtype, but a bigger int32 (although still representable as an >> int8) in (8) makes it switch to int16 (if someone wonders, adding 126 >> instead of 127 in (8) would result in [-3 -3] (int8), so 127 is special for >> some reason). >> >> My feeling is actually that the logic is to try to downcast the scalar as >> much as possible without changing its value, but with a bug that 127 is not >> downcasted to int8, and remains int16 (!). >> >> Some more behavior that puzzles me, this time comparing + vs -: >> (9) [0 0] (uint32) + -1 (int32) -> [-1 -1] (int64) >> (10) [0 0] (uint32) - 1 (int32) -> [4294967295 4294967295] (uint32) >> >> Here I would expect that adding -1 would be the same as subtracting 1, but >> that is not the case. >> > > In the second case, it's equivalent to np.subtract(np.array([0, 0], > np.uint32), np.int32(1)). The scalar 1 fits into the uint32, so the result > type of the subtraction is uint32. In the first case, the scalar -1 does not > fit into the uint32, so it is upgraded to int64. > > >> >> Is there anyone with intimate knowledge of the numpy casting behavior for >> mixed scalar / array operations who could explain what are the rules >> governing it? >> > > Hopefully my explanations help a bit. I think this situation is less than > ideal, and it would be better to do something more automatic, like doing an > up-conversion on overflow. This would more closely emulate Python's behavior > of integers never overflowing, at least until 64 bits. This kind of change > would be a fair bit of work, and would likely reduce the performance of > NumPy slightly. > > Cheers, > Mark > > Thanks! It's re-assuring to hear that part of it is caused by a bug, and the other part has some logic behind it (even though it leads to surprising results). I appreciate you taking the time to clear it up for me :) -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: