From hangenuit at gmail.com Mon Oct 1 03:19:59 2012 From: hangenuit at gmail.com (Han Genuit) Date: Mon, 1 Oct 2012 09:19:59 +0200 Subject: [Numpy-discussion] Reductions with nditer working only with the last axis In-Reply-To: References: Message-ID: On Thu, Sep 27, 2012 at 6:08 PM, Sergio Pascual wrote: > Hello, I'm trying to understand how to work with nditer to do a > reduction, in my case converting a 3d array into a 2d array. > > I followed the help here > http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html and > managed to create a function that applies reduction over the last axis > of the input. With this function > > def nditer_sum(data, red_axes): > it = numpy.nditer([data, None], > flags=['reduce_ok', 'external_loop'], > op_flags=[['readonly'], ['readwrite', 'allocate']], > op_axes=[None, red_axes]) > it.operands[1][...] = 0 > > for x, y in it: > y[...] = x.sum() > > return it.operands[1] > > I can get something equivalent to data.sum(axis=2) > >>>> data = numpy.arange(2*3*4).reshape((2,3,4)) >>>> nditer_sum(data, [0, 1, -1]) > [[ 6 22 38] > [54 70 86]] >>>> data.sum(axis=2) > [[ 6 22 38] > [54 70 86]] > > So to get something equivalent to data.sum(axis=0) I though that it > was enough to change the argument red_axes to [-1, 0,1] > But the result is quite different. > >>>> data = numpy.arange(2*3*4).reshape((2,3,4)) >>>> data.sum(axis=0) > [[12 14 16 18] > [20 22 24 26] > [28 30 32 34]] >>>> nditer_sum(data, [-1, 0, 1]) > [[210 210 210 210] > [210 210 210 210] > [210 210 210 210]] > > In the for loop inside nditer_sum (for x,y in it:), the iterator is > looping 2 times and giving an array of length 12 each time, instead of > looping 12 times and giving an array of length 2 each time. I have > read the numpy documentation several times and googled about this to > no avail. > > Does anybody have an example of a reduction in the first axis of an > array using nditer? Is this a bug? > > > Regards, Sergio > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion The example from the link it shows how to do a reduction with y[...] += x. If you would replace x.sum() by += x, then it works. >>> nditer_sum(data, [-1,0,1]) array([[12, 14, 16, 18], [20, 22, 24, 26], [28, 30, 32, 34]]) >>> data.sum(axis=0) array([[12, 14, 16, 18], [20, 22, 24, 26], [28, 30, 32, 34]]) >>> nditer_sum(data, [0,-1,1]) array([[12, 15, 18, 21], [48, 51, 54, 57]]) >>> data.sum(axis=1) array([[12, 15, 18, 21], [48, 51, 54, 57]]) I think that is because sum() already reduces all axis by default. Regards, Han From pierre.haessig at crans.org Mon Oct 1 04:04:32 2012 From: pierre.haessig at crans.org (Pierre Haessig) Date: Mon, 01 Oct 2012 10:04:32 +0200 Subject: [Numpy-discussion] silently ignored size mismatch (bug??) In-Reply-To: References: Message-ID: <50694E90.3040503@crans.org> Hi, Le 28/09/2012 21:02, Neal Becker a ?crit : > In [19]: u = np.arange (10) > > In [20]: v = np.arange (10) > > In [21]: u[v] = u > > In [22]: u[v] = np.arange(11) > > silence... I've same behavior with my numpy 1.6.2. It indeed looks strange that the end of the data vector is dropped in silence. Best, Pierre -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From matthew.brett at gmail.com Mon Oct 1 06:29:21 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 1 Oct 2012 11:29:21 +0100 Subject: [Numpy-discussion] silently ignored size mismatch (bug??) In-Reply-To: <50694E90.3040503@crans.org> References: <50694E90.3040503@crans.org> Message-ID: Hi, On Mon, Oct 1, 2012 at 9:04 AM, Pierre Haessig wrote: > Hi, > > Le 28/09/2012 21:02, Neal Becker a ?crit : >> In [19]: u = np.arange (10) >> >> In [20]: v = np.arange (10) >> >> In [21]: u[v] = u >> >> In [22]: u[v] = np.arange(11) >> >> silence... > I've same behavior with my numpy 1.6.2. > > It indeed looks strange that the end of the data vector is dropped in > silence. Same on numpy 1.4.1. I also would have predicted an error in that situation. For example in matlab: >> a = 1:10 a = 1 2 3 4 5 6 7 8 9 10 >> b = 1:10 b = 1 2 3 4 5 6 7 8 9 10 >> a(b) = 11:20 a = 11 12 13 14 15 16 17 18 19 20 >> a(b) = 11:21 ??? In an assignment A(I) = B, the number of elements in B and I must be the same. Is there any reason not to do that? Best, Matthew From njs at pobox.com Mon Oct 1 08:20:32 2012 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 1 Oct 2012 13:20:32 +0100 Subject: [Numpy-discussion] Behavior of .base In-Reply-To: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> References: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> Message-ID: On Sun, Sep 30, 2012 at 8:59 PM, Travis Oliphant wrote: > Hey all, > > In a github-discussion with Gael and Nathaniel, we came up with a proposal for .base that we should put before this list. Traditionally, .base has always pointed to None for arrays that owned their own memory and to the "most immediate" array object parent for arrays that did not own their own memory. There was a long-standing issue related to running out of stack space that this behavior created. To be *completely* accurate, I'd say that they've always pointed to some object that owned the underlying memory. Usually that's an ndarray, but sometimes that's a thing exposing the buffer interface, sometimes it's a thing exposing __array_interface__, sometimes it's a mmap object, sometimes it's some random ad hoc C-level wrapper object[1], etc. [1] e.g. https://github.com/njsmith/scikits-sparse/blob/master/scikits/sparse/cholmod.pyx#L225 > Recently this behavior was altered so that .base always points to "the original" object holding the memory (something exposing the buffer interface). This created some problems for users who relied on the fact that most of the time .base pointed to an instance of an array object. > > The proposal here is to change the behavior of .base for arrays that don't own their own memory so that the .base attribute of an array points to "the most original object" that is still an instance of the type of the array. This would go into the 1.7.0 release so as to correct the issues reported. > > What are reactions to this proposal? As a band-aid to avoid breaking some code in 1.7, it seems reasonable to me. I was actually considering proposing basically the same idea. But it's only a band-aid; the larger problem is that we don't *know* what semantics people are relying on for "base" (and probably aren't implementing the ones people think we are, either before or after this change). As an example of how messy this is: do you know whether Gael's code will still work, after we make this fix, if someone uses as_strided() on a (view of a) memmap array? Answer: as_strided() creates an ndarray view on an ad-hoc object with __array_interface__ attribute, and this dummy object ends up as the returned ndarray's .base. According to the proposed rule, the .base chain collapsing will stop at this point. So it isn't true that an array that is ultimately backed by mmap will have a .memmap() array as its .base. However, if you read stride_tricks.py, it turns out the dummy object as_strided makes does happen to use the name ".base" for its attribute holding the original array, so Gael's code will work correctly in this case iff he keeps the .base walking code in place (which would otherwise serve no purpose after Travis' change). Anyway, my point is: If we have to carefully analyze interactions between code in numpy.lib.stride_tricks, numpy.core.memmap, and a third-party library, just to figure out which sorts of reference-counting changes are correct in the core ndarray object, then we have a problem. This is horrible cross-coupling, the sort of thing that, if allowed to proliferate, makes it impossible to ever know whether code is correct or not. So even if we put in a band-aid for 1.7, we really don't want to be guaranteeing this kind of stuff forever, and should aggressively encourage people to stop using .base in these ways. The mmap thing should really switch to something more reliable and less tightly coupled to the rest of the code all over numpy, like I described here: http://mail.scipy.org/pipermail/numpy-discussion/2012-September/064003.html How can we discourage people from doing this in the future? Can we make .base write-only from the Python level (with suitable deprecation period)? Rename it to ._base (likewise) so that it's still possible to peek under the covers but we remind people that it's really an implementation detail with poorly defined semantics that might change? -n From njs at pobox.com Mon Oct 1 08:35:56 2012 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 1 Oct 2012 13:35:56 +0100 Subject: [Numpy-discussion] Memory order of array copies In-Reply-To: <20120930182259.GC15857@phare.normalesup.org> References: <20120930182259.GC15857@phare.normalesup.org> Message-ID: On Sun, Sep 30, 2012 at 7:22 PM, Gael Varoquaux wrote: > On Sun, Sep 30, 2012 at 07:17:42PM +0100, Nathaniel Smith wrote: >> Is there anything better to do than simply revert np.copy() to its >> traditional behaviour and accept that np.copy(a) and a.copy() will >> continue to have different semantics indefinitely? > > Have np.copy take an 'order=None', which would translate to 'K'. Detect > 'None' as a sentinel that order as not been specified. If the order is > not specified, raise a FutureWarning that np.copy will change semantics > in 2 releases. In two releases, do the change. I'm actually suggesting that arr.copy() should change, rather than np.copy() (the opposite of the change currently in master), but that aside: the problem with this approach is that the vast majority of calls to these functions don't care at all about this order thing, so spewing warnings all over the place is pretty useless at helping people actually detect and fix their code. Compare to the np.diagonal() change in 1.7, where we don't issue a warning when np.diagonal() is called, but wait until people write to the array. -n From talljimbo at gmail.com Mon Oct 1 10:11:07 2012 From: talljimbo at gmail.com (Jim Bosch) Date: Mon, 01 Oct 2012 10:11:07 -0400 Subject: [Numpy-discussion] Behavior of .base In-Reply-To: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> References: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> Message-ID: <5069A47B.50803@gmail.com> On 09/30/2012 03:59 PM, Travis Oliphant wrote: > Hey all, > > In a github-discussion with Gael and Nathaniel, we came up with a proposal for .base that we should put before this list. Traditionally, .base has always pointed to None for arrays that owned their own memory and to the "most immediate" array object parent for arrays that did not own their own memory. There was a long-standing issue related to running out of stack space that this behavior created. > > Recently this behavior was altered so that .base always points to "the original" object holding the memory (something exposing the buffer interface). This created some problems for users who relied on the fact that most of the time .base pointed to an instance of an array object. > > The proposal here is to change the behavior of .base for arrays that don't own their own memory so that the .base attribute of an array points to "the most original object" that is still an instance of the type of the array. This would go into the 1.7.0 release so as to correct the issues reported. > > What are reactions to this proposal? > In the past, I've relied on putting arbitrary Python objects in .base in my C++ to NumPy conversion code to make sure reference counting for array memory works properly. In particular, I've used Python CObjects that hold boost::shared_ptrs, which don't even have a buffer interface. So it sounds like I may be a few steps behind on the rules of what actually should go in .base. I'm very concerned that if we do demand that .base always point to a NumPy array (rather than an arbitrary Python object or even just one with a buffer interface), there's no longer any way for a NumPy array to hold data allocated by something other than NumPy. If I want to put external memory in a NumPy array and indicate that it's owned by some non-NumPy Python object, what is the recommended way to do that? Thanks! Jim Bosch From charlesr.harris at gmail.com Mon Oct 1 10:12:55 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 1 Oct 2012 08:12:55 -0600 Subject: [Numpy-discussion] Behavior of .base In-Reply-To: References: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> Message-ID: On Mon, Oct 1, 2012 at 6:20 AM, Nathaniel Smith wrote: > On Sun, Sep 30, 2012 at 8:59 PM, Travis Oliphant > wrote: > > Hey all, > > > > In a github-discussion with Gael and Nathaniel, we came up with a > proposal for .base that we should put before this list. Traditionally, > .base has always pointed to None for arrays that owned their own memory and > to the "most immediate" array object parent for arrays that did not own > their own memory. There was a long-standing issue related to running out > of stack space that this behavior created. > > To be *completely* accurate, I'd say that they've always pointed to > some object that owned the underlying memory. Usually that's an > ndarray, but sometimes that's a thing exposing the buffer interface, > sometimes it's a thing exposing __array_interface__, sometimes it's a > mmap object, sometimes it's some random ad hoc C-level wrapper > object[1], etc. > > [1] e.g. > https://github.com/njsmith/scikits-sparse/blob/master/scikits/sparse/cholmod.pyx#L225 > > > Recently this behavior was altered so that .base always points to "the > original" object holding the memory (something exposing the buffer > interface). This created some problems for users who relied on the fact > that most of the time .base pointed to an instance of an array object. > > > > The proposal here is to change the behavior of .base for arrays that > don't own their own memory so that the .base attribute of an array points > to "the most original object" that is still an instance of the type of the > array. This would go into the 1.7.0 release so as to correct the > issues reported. > > > > What are reactions to this proposal? > > As a band-aid to avoid breaking some code in 1.7, it seems reasonable > to me. I was actually considering proposing basically the same idea. > But it's only a band-aid; the larger problem is that we don't *know* > what semantics people are relying on for "base" (and probably aren't > implementing the ones people think we are, either before or after this > change). > > As an example of how messy this is: do you know whether Gael's code > will still work, after we make this fix, if someone uses as_strided() > on a (view of a) memmap array? > > Answer: as_strided() creates an ndarray view on an ad-hoc object with > __array_interface__ attribute, and this dummy object ends up as the > returned ndarray's .base. According to the proposed rule, the .base > chain collapsing will stop at this point. So it isn't true that an > array that is ultimately backed by mmap will have a .memmap() array as > its .base. However, if you read stride_tricks.py, it turns out the > dummy object as_strided makes does happen to use the name ".base" for > its attribute holding the original array, so Gael's code will work > correctly in this case iff he keeps the .base walking code in place > (which would otherwise serve no purpose after Travis' change). > > Anyway, my point is: If we have to carefully analyze interactions > between code in numpy.lib.stride_tricks, numpy.core.memmap, and a > third-party library, just to figure out which sorts of > reference-counting changes are correct in the core ndarray object, > then we have a problem. This is horrible cross-coupling, the sort of > thing that, if allowed to proliferate, makes it impossible to ever > know whether code is correct or not. > > So even if we put in a band-aid for 1.7, we really don't want to be > guaranteeing this kind of stuff forever, and should aggressively > encourage people to stop using .base in these ways. The mmap thing > should really switch to something more reliable and less tightly > coupled to the rest of the code all over numpy, like I described here: > > http://mail.scipy.org/pipermail/numpy-discussion/2012-September/064003.html > > How can we discourage people from doing this in the future? Can we > make .base write-only from the Python level (with suitable deprecation > period)? Rename it to ._base (likewise) so that it's still possible to > peek under the covers but we remind people that it's really an > implementation detail with poorly defined semantics that might change? > > Well said. This reminds me of the fellow who used genetic programming to design an algorithm for a signal processing chip and discovered that the result was making use of some stray capacitance present on the chip. Here users such as Gael are the genetic programmers and .base is the stray capacitance. I tend to the ._base idea, but I think this needs to be addressed in detail. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From thouis at gmail.com Mon Oct 1 10:14:46 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Mon, 1 Oct 2012 10:14:46 -0400 Subject: [Numpy-discussion] Memory order of array copies In-Reply-To: References: <20120930182259.GC15857@phare.normalesup.org> Message-ID: On Mon, Oct 1, 2012 at 8:35 AM, Nathaniel Smith wrote: > On Sun, Sep 30, 2012 at 7:22 PM, Gael Varoquaux > wrote: >> On Sun, Sep 30, 2012 at 07:17:42PM +0100, Nathaniel Smith wrote: >>> Is there anything better to do than simply revert np.copy() to its >>> traditional behaviour and accept that np.copy(a) and a.copy() will >>> continue to have different semantics indefinitely? >> >> Have np.copy take an 'order=None', which would translate to 'K'. Detect >> 'None' as a sentinel that order as not been specified. If the order is >> not specified, raise a FutureWarning that np.copy will change semantics >> in 2 releases. In two releases, do the change. > > I'm actually suggesting that arr.copy() should change, rather than > np.copy() (the opposite of the change currently in master), but that > aside: the problem with this approach is that the vast majority of > calls to these functions don't care at all about this order thing, so > spewing warnings all over the place is pretty useless at helping > people actually detect and fix their code. Compare to the > np.diagonal() change in 1.7, where we don't issue a warning when > np.diagonal() is called, but wait until people write to the array. I'm +1 on changing a.copy() to be more consistent with np.copy()/np.array(copy=True), but would push it to 2.0 (or whatever release API changes are next expected). I would not worry about a FutureWarning, for this reason. From thouis at gmail.com Mon Oct 1 10:40:52 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Mon, 1 Oct 2012 10:40:52 -0400 Subject: [Numpy-discussion] Behavior of .base In-Reply-To: References: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> Message-ID: On Mon, Oct 1, 2012 at 8:20 AM, Nathaniel Smith wrote: > [...] > How can we discourage people from doing this in the future? Can we > make .base write-only from the Python level (with suitable deprecation > period)? Rename it to ._base (likewise) so that it's still possible to > peek under the covers but we remind people that it's really an > implementation detail with poorly defined semantics that might change? Could we use the simpler .base behavior (fully collapsing the .base chain), but be more aggressive about propagating information like address/filename/offset for np.arrays that are created by slicing, asarray(), etc.? Ray (Sorry if I'm missing some context that makes this suggestion idiotic. I'm still trying to catch back up on the list and may have missed relevant discussion on other threads.) From charlesr.harris at gmail.com Mon Oct 1 10:56:48 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 1 Oct 2012 08:56:48 -0600 Subject: [Numpy-discussion] Behavior of .base In-Reply-To: References: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> Message-ID: On Mon, Oct 1, 2012 at 8:40 AM, Thouis (Ray) Jones wrote: > On Mon, Oct 1, 2012 at 8:20 AM, Nathaniel Smith wrote: > > [...] > > How can we discourage people from doing this in the future? Can we > > make .base write-only from the Python level (with suitable deprecation > > period)? Rename it to ._base (likewise) so that it's still possible to > > peek under the covers but we remind people that it's really an > > implementation detail with poorly defined semantics that might change? > > Could we use the simpler .base behavior (fully collapsing the .base > chain), but be more aggressive about propagating information like > address/filename/offset for np.arrays that are created by slicing, > asarray(), etc.? > > Ray > (Sorry if I'm missing some context that makes this suggestion idiotic. > I'm still trying to catch back up on the list and may have missed > relevant discussion on other threads.) > It might be productive to step back a bit and ask if this is a memmap problem or a workflow problem. My impression is that pickling memmaps is a solution to a higher level problem in Scikits.learn workflow and I'd like more details on what that problem is. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Oct 1 11:18:09 2012 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 1 Oct 2012 16:18:09 +0100 Subject: [Numpy-discussion] Behavior of .base In-Reply-To: References: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> Message-ID: On Mon, Oct 1, 2012 at 3:40 PM, Thouis (Ray) Jones wrote: > On Mon, Oct 1, 2012 at 8:20 AM, Nathaniel Smith wrote: >> [...] >> How can we discourage people from doing this in the future? Can we >> make .base write-only from the Python level (with suitable deprecation >> period)? Rename it to ._base (likewise) so that it's still possible to >> peek under the covers but we remind people that it's really an >> implementation detail with poorly defined semantics that might change? > > Could we use the simpler .base behavior (fully collapsing the .base > chain), but be more aggressive about propagating information like > address/filename/offset for np.arrays that are created by slicing, > asarray(), etc.? There are definitely other solutions to the memmap pickling problem that don't rely on the semantics of .base at all, yeah. I don't particularly like the one you suggest, because it requires that many pieces of code in many places all be careful to preserve this information, whereas keeping a global table of the process's active memory maps requires only a single piece of new code in the memmap module (and will be more reliable to boot). But strictly speaking the details here are irrelevant to the discussion about .base itself. There are two questions about .base: 1) Is it okay to break people's code when we release 1.7, given that they have relied on this behaviour that they probably ought not to have? 2) Can we untangle things enough that making changes to .base *won't* break people's code, so that we don't end up having to ask question (1) again in the future? I'm totally happy with deciding that we need to band-aid the 1.7 release b/c working code is working code and breaking it isn't okay, so long as we also address question (2). -n From jay.bourque at continuum.io Mon Oct 1 11:36:58 2012 From: jay.bourque at continuum.io (Jay Bourque) Date: Mon, 1 Oct 2012 10:36:58 -0500 Subject: [Numpy-discussion] ufuncs for structured arrays Message-ID: All, I've submitted the following pull request for NumPy: https://github.com/numpy/numpy/pull/462 This change allows ufuncs to be registered for structured arrays by using a new API method PyUFunc_RegisterLoopForStructType. For example, a ufunc could be registered to take two arrays of type 'u8,u8,u8' and return an array of type 'u8,u8,u8'. I have a trivial example of this included in my pull request, along with further details of my changes. I suspect there might be a better way to do this, so any suggestions for improvements would be welcome. Thanks, -Jay continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Mon Oct 1 12:09:55 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 01 Oct 2012 18:09:55 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.7.0b2 release In-Reply-To: References: Message-ID: <1349107795.17769.4.camel@sebastian-laptop> Hey, About the imaginary part being ignored for all/any function... > The all method fails also. > > In [1]: a = zeros(5, complex) > > In [2]: a.imag = 1 > > In [3]: a.all() > Out[3]: False > > Chuck > I believe this diff fixes the issue (also posted on Tracker), I doubt its the best way to fix the issue, but if anyone who knows this code wants to look into it its good to know what is broken. Note that also a.astype(bool) fails: --- a/numpy/core/src/multiarray/lowlevel_strided_loops.c.src +++ b/numpy/core/src/multiarray/lowlevel_strided_loops.c.src @@ -811,9 +811,17 @@ static void dst_value[0] = _CONVERT_FN(src_value[0]); dst_value[1] = _CONVERT_FN(src_value[1]); # elif !@aligned@ - dst_value = _CONVERT_FN(src_value[0]); +# if @is_bool2@ + dst_value = _CONVERT_FN(src_value[0]) || _CONVERT_FN(src_value[1]); +# else + dst_value = _CONVERT_FN(src_value[0]); +# endif # else - *(_TYPE2 *)dst = _CONVERT_FN(src_value[0]); +# if @is_bool2@ + *(_TYPE2 *)dst = _CONVERT_FN(src_value[0]) || _CONVERT_FN(src_value[1]); +# else + *(_TYPE2 *)dst = _CONVERT_FN(src_value[0]); +# endif # endif #else # if @is_complex2@ Regards, Sebastian > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Mon Oct 1 12:59:00 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 1 Oct 2012 10:59:00 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.7.0b2 release In-Reply-To: <1349107795.17769.4.camel@sebastian-laptop> References: <1349107795.17769.4.camel@sebastian-laptop> Message-ID: On Mon, Oct 1, 2012 at 10:09 AM, Sebastian Berg wrote: > Hey, > > About the imaginary part being ignored for all/any function... > > > > > > The all method fails also. > > > > In [1]: a = zeros(5, complex) > > > > In [2]: a.imag = 1 > > > > In [3]: a.all() > > Out[3]: False > > > > Chuck > > > I believe this diff fixes the issue (also posted on Tracker), I doubt > its the best way to fix the issue, but if anyone who knows this code > wants to look into it its good to know what is broken. Note that also > a.astype(bool) fails: > > --- a/numpy/core/src/multiarray/lowlevel_strided_loops.c.src > +++ b/numpy/core/src/multiarray/lowlevel_strided_loops.c.src > @@ -811,9 +811,17 @@ static void > dst_value[0] = _CONVERT_FN(src_value[0]); > dst_value[1] = _CONVERT_FN(src_value[1]); > # elif !@aligned@ > - dst_value = _CONVERT_FN(src_value[0]); > +# if @is_bool2@ > + dst_value = _CONVERT_FN(src_value[0]) || _CONVERT_FN(src_value[1]); > +# else > + dst_value = _CONVERT_FN(src_value[0]); > +# endif > # else > - *(_TYPE2 *)dst = _CONVERT_FN(src_value[0]); > +# if @is_bool2@ > + *(_TYPE2 *)dst = _CONVERT_FN(src_value[0]) || > _CONVERT_FN(src_value[1]); > +# else > + *(_TYPE2 *)dst = _CONVERT_FN(src_value[0]); > +# endif > # endif > #else > # if @is_complex2@ > > > Hey, I think you are onto something. I'm I correct in assuming the this also fixes astype(bool)? In any case, it would be nice if you submitted this as an ordinary PR and added some tests. That would be the fastest route to getting it reviewed and committed. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Mon Oct 1 13:22:29 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 01 Oct 2012 19:22:29 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.7.0b2 release In-Reply-To: References: <1349107795.17769.4.camel@sebastian-laptop> Message-ID: <1349112149.17769.6.camel@sebastian-laptop> On Mon, 2012-10-01 at 10:59 -0600, Charles R Harris wrote: > > > On Mon, Oct 1, 2012 at 10:09 AM, Sebastian Berg > wrote: > Hey, > > About the imaginary part being ignored for all/any function... > > > > > > The all method fails also. > > > > In [1]: a = zeros(5, complex) > > > > In [2]: a.imag = 1 > > > > In [3]: a.all() > > Out[3]: False > > > > Chuck > > > I believe this diff fixes the issue (also posted on Tracker), > I doubt > its the best way to fix the issue, but if anyone who knows > this code > wants to look into it its good to know what is broken. Note > that also > a.astype(bool) fails: > > --- a/numpy/core/src/multiarray/lowlevel_strided_loops.c.src > +++ b/numpy/core/src/multiarray/lowlevel_strided_loops.c.src > @@ -811,9 +811,17 @@ static void > dst_value[0] = _CONVERT_FN(src_value[0]); > dst_value[1] = _CONVERT_FN(src_value[1]); > # elif !@aligned@ > - dst_value = _CONVERT_FN(src_value[0]); > +# if @is_bool2@ > + dst_value = _CONVERT_FN(src_value[0]) || > _CONVERT_FN(src_value[1]); > +# else > + dst_value = _CONVERT_FN(src_value[0]); > +# endif > # else > - *(_TYPE2 *)dst = _CONVERT_FN(src_value[0]); > +# if @is_bool2@ > + *(_TYPE2 *)dst = _CONVERT_FN(src_value[0]) || > _CONVERT_FN(src_value[1]); > +# else > + *(_TYPE2 *)dst = _CONVERT_FN(src_value[0]); > +# endif > # endif > #else > # if @is_complex2@ > > > > Hey, I think you are onto something. I'm I correct in assuming the > this also fixes astype(bool)? In any case, it would be nice if you > submitted this as an ordinary PR and added some tests. That would be > the fastest route to getting it reviewed and committed. > Yes, it fixes that too of course, just noted it because then it may be obvious why to change the code there for you guys knowing numpy well :). I will do a PR, then can see if this is the best fix. > Chuck > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ndbecker2 at gmail.com Mon Oct 1 13:51:13 2012 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 01 Oct 2012 13:51:13 -0400 Subject: [Numpy-discussion] silently ignored size mismatch (bug??) References: <50694E90.3040503@crans.org> Message-ID: Sounds like I'm not the only one surprised then: http://projects.scipy.org/numpy/ticket/2220 Matthew Brett wrote: > Hi, > > On Mon, Oct 1, 2012 at 9:04 AM, Pierre Haessig > wrote: >> Hi, >> >> Le 28/09/2012 21:02, Neal Becker a ?crit : >>> In [19]: u = np.arange (10) >>> >>> In [20]: v = np.arange (10) >>> >>> In [21]: u[v] = u >>> >>> In [22]: u[v] = np.arange(11) >>> >>> silence... >> I've same behavior with my numpy 1.6.2. >> >> It indeed looks strange that the end of the data vector is dropped in >> silence. > > Same on numpy 1.4.1. I also would have predicted an error in that > situation. For example in matlab: > >>> a = 1:10 > > a = > > 1 2 3 4 5 6 7 8 9 10 > >>> b = 1:10 > > b = > > 1 2 3 4 5 6 7 8 9 10 > >>> a(b) = 11:20 > > a = > > 11 12 13 14 15 16 17 18 19 20 > >>> a(b) = 11:21 > ??? In an assignment A(I) = B, the number of elements in B and > I must be the same. > > Is there any reason not to do that? > > Best, > > Matthew From chris.barker at noaa.gov Mon Oct 1 15:07:19 2012 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 1 Oct 2012 12:07:19 -0700 Subject: [Numpy-discussion] memory-efficient loadtxt In-Reply-To: References: Message-ID: Paul, Nice to see someone working on these issues, but: I'm not sure the problem you are trying to solve -- accumulating in a list is pretty efficient anyway -- not a whole lot overhead. But if you do want to improve that, it may be better to change the accumulating method, rather than doing the double-read thing. I"ve written, and posted here, code that provides an Acumulator that uses numpy internally, so not much memory overhead. In the end, it's not any faster than accumulating in a list and then converting to an array, but it does use less memory. I also have a Cython version that is not quite done (darn regular job getting in the way) that is both faster and more memory efficient. Also, frankly, just writing the array pre-allocation and re-sizeing code into loadtxt would not be a whole lot of code either, and would be both fast and memory efficient. Let mw know if you want any of my code to play with. > However, I got the impression that someone was > working on a More Advanced (TM) C-based file reader, which will > replace loadtxt; yes -- I wonder what happened with that? Anyone? -CHB this patch is intended as a useful thing to have > while we're waiting for that to appear. > > The patch passes all tests in the test suite, and documentation for > the kwarg has been added. I've modified all tests to include the > seekable kwarg, but that was mostly to check that all tests are passed > also with this kwarg. I guess it's bit too late for 1.7.0 though? > > Should I make a pull request? I'm happy to take any and all > suggestions before I do. > > Cheers > Paul > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From chris.barker at noaa.gov Mon Oct 1 15:30:24 2012 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 1 Oct 2012 12:30:24 -0700 Subject: [Numpy-discussion] Making numpy sensible: backward compatibility please In-Reply-To: References: <20120928212327.GA22708@phare.normalesup.org> Message-ID: On Fri, Sep 28, 2012 at 3:11 PM, Charles R Harris wrote: > If the behaviour is not specified and tested, there is no guarantee that it > will continue. This is an open-source project - there is no guarantee of ANYTHING. But that being said, the specification and testing of numpy is quite weak -- we have no choice but to use unspecified and untested (and how do we even know what is tested?) features. So, be definition, the current code base IS the specification -- if it changes that is a change in the spec. Travis is right -- we users are the test cases -- which is better than most code, frankly. Perhaps more effort could be put into considering the impact that internal changes have, and announcing them more aggressively? So we in the user community have a better idea what to look for when adopting/testing a new release (candidate). Another thought -- perhaps we're being too incremental, and a py3k type change is in order -- do a while bunch at once. Though, as we've seen with py3k, that would mean maintaining two version for a good long while... Also -- last time I tried adding some tests, I found it to be very hard to figure out where and how to put them in. Granted, I only poked at it for an hour or two -- but if users could add tests in less than an hour, we'd get a lot more tests. Maybe it just need more documentation, or maybe I just didn't know nose at the time -- I don't remember, but I do know I was discouraged, and haven't tried since. Maybe a test change log of some sort would be helpful: easy access to know what tests had to be changed/added to accommodate change behavior (i.e any test that is passing with version i that would not have passed version i-1) - looking over that might be a good way to figure out if your own code is likely to be affected. -Chris > >> >> >> I think that this is a cultural issue: priority is not given to stability >> and backward compatibility. I think that this culture is very much >> ingrained in the Python world, that likes iteratively cleaning its >> software design. For instance, I have the feeling that in the >> scikit-learn, we probably fall in the same trap. That said, such a >> behavior cannot fare well for a base scientific environment. People tell >> me that if they take old matlab code, the odds that it will still works >> is much higher than with Python code. As a geek, I tend to reply that we >> get a lot out of this mobility, because we accumulate less cruft. >> However, in research settings, for reproducibility reasons, ones need to >> be able to pick up an old codebase and trust its results without knowing >> its intricacies. > > > Bitch, bitch, bitch. Look, I know you are pissed and venting a bit, but this > problem could have been detected and reported 6 months ago, that is, unless > it is new due to development on your end. > >> >> >> >From a practical standpoint, I believe that people implementing large >> changes to the numpy codebase, or any other core scipy package, should >> think really hard about their impact. I do realise that the changes are >> discussed on the mailing lists, but there is a lot of activity to follow >> and I don't believe that it is possible for many of us to monitor the >> discussions. Also, putting more emphasis on backward compatibility is >> possible. For instance, the 'order' parameter added to np.copy could have >> defaulted to the old behavior, 'K', for a year, with a >> DeprecationWarning, same thing for the casting rules. >> >> Thank you for reading this long email. I don't mean it to be a complaint >> about the past, but more a suggestion on something to keep in mind when >> making changes to core projects. >> > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From chris.barker at noaa.gov Mon Oct 1 15:32:07 2012 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 1 Oct 2012 12:32:07 -0700 Subject: [Numpy-discussion] Making numpy sensible: backward compatibility please In-Reply-To: <20120929091600.GA13677@phare.normalesup.org> References: <20120928212327.GA22708@phare.normalesup.org> <20120929091600.GA13677@phare.normalesup.org> Message-ID: On Sat, Sep 29, 2012 at 2:16 AM, Gael Varoquaux wrote: > Next time I see you, I owe you a beer for making you cross :). If I curse at you, will I get a beer too? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From matthew.brett at gmail.com Mon Oct 1 16:42:40 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 1 Oct 2012 21:42:40 +0100 Subject: [Numpy-discussion] numpy distutils log error with easy_install Message-ID: Hi, One of our kind users pointed out an error when using easy_install to install our package nipy. I've reproduced it now on a bare package using numpy distutils and having a trivial extension: https://github.com/matthew-brett/apkg To reproduce: git clone git://github.com/mathew-brett/apkg.git easy_install apkg You should get something like this: Processing apkg Running setup.py -q bdist_egg --dist-dir /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB Appending apkg configuration to Ignoring attempt to set 'name' (from '' to 'apkg') zip_safe flag not set; analyzing archive contents... Adding apkg 0.1 to easy-install.pth file Installed /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg Processing dependencies for apkg==0.1 Finished processing dependencies for apkg==0.1 /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: RuntimeWarning: Parent module 'numpy.distutils' not found while handling absolute import from numpy.distutils import log Note the last error. All of nipy, scikits-learn and scikits-timeseries (that use numpy distutils) give this same error on easy_install. The error requires the use of an extension in the package, as in: https://github.com/matthew-brett/apkg/blob/master/apkg/setup.py#L4 It appears to be benign but it certainly confuses people doing easy_install. This is running current numpy master, but the same occurs with numpy 1.5.1 that I have to hand. Any ideas what could be causing this? Cheers, Matthew From matthew.brett at gmail.com Mon Oct 1 16:47:28 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 1 Oct 2012 21:47:28 +0100 Subject: [Numpy-discussion] numpy distutils log error with easy_install In-Reply-To: References: Message-ID: Hi, On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett wrote: > Hi, > > One of our kind users pointed out an error when using easy_install to > install our package nipy. I've reproduced it now on a bare package > using numpy distutils and having a trivial extension: > > https://github.com/matthew-brett/apkg > > To reproduce: > > git clone git://github.com/mathew-brett/apkg.git > > easy_install apkg > > You should get something like this: > > Processing apkg > Running setup.py -q bdist_egg --dist-dir > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB > Appending apkg configuration to > Ignoring attempt to set 'name' (from '' to 'apkg') > zip_safe flag not set; analyzing archive contents... > Adding apkg 0.1 to easy-install.pth file > > Installed /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg > Processing dependencies for apkg==0.1 > Finished processing dependencies for apkg==0.1 > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: > RuntimeWarning: Parent module 'numpy.distutils' not found while > handling absolute import > from numpy.distutils import log > > Note the last error. Sorry, correcting myself - it's (obviously) a Warning rather than an error, but still distracting, and it would be good to avoid it if possible... Best, Matthew From scopatz at gmail.com Mon Oct 1 23:05:48 2012 From: scopatz at gmail.com (Anthony Scopatz) Date: Mon, 1 Oct 2012 22:05:48 -0500 Subject: [Numpy-discussion] Making numpy sensible: backward compatibility please In-Reply-To: References: <20120928212327.GA22708@phare.normalesup.org> <20120929091600.GA13677@phare.normalesup.org> Message-ID: On Mon, Oct 1, 2012 at 2:32 PM, Chris Barker wrote: > On Sat, Sep 29, 2012 at 2:16 AM, Gael Varoquaux > wrote: > > Next time I see you, I owe you a beer for making you cross :). > > If I curse at you, will I get a beer too? > Wow! This is taking a very Pavlovian turn... Be Well Anthony > > -Chris > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergiopr at fis.ucm.es Mon Oct 1 18:01:05 2012 From: sergiopr at fis.ucm.es (Sergio Pascual) Date: Tue, 2 Oct 2012 00:01:05 +0200 Subject: [Numpy-discussion] Reductions with nditer working only with the last axis In-Reply-To: References: Message-ID: Perhaps sum wasn't the best function for this example. I'm going to rework the code with other function Consider a function that operates on an array and returns a number def myfunc(data): return data.min() + 2 * data.max() The function with nditer is: def nditer_fun(data, axes): it = numpy.nditer([data, None], flags=['reduce_ok', 'external_loop'], op_flags=[['readonly'], ['readwrite', 'allocate']], op_axes=[None, axes]) it.operands[1][...] = 0 for x, y in it: y[...] = myfun(x) return it.operands[1] We can compara the result of this function with the result of numpy.apply_along_axis With the data data = numpy.arange(2*3*4).reshape((2,3,4)) the values obtained with axis 0 ara numpy.apply_along_axis(myfun, 0 ,data) [[24 27 30 33] [36 39 42 45] [48 51 54 57]] nditer_fun(data, [-1, 0, 1]) [[58 58 58 58] [58 58 58 58] [58 58 58 58]] Only along the last axis both functions give the same result 2012/10/1 Han Genuit : > On Thu, Sep 27, 2012 at 6:08 PM, Sergio Pascual wrote: >> Hello, I'm trying to understand how to work with nditer to do a >> reduction, in my case converting a 3d array into a 2d array. >> >> I followed the help here >> http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html and >> managed to create a function that applies reduction over the last axis >> of the input. With this function >> >> def nditer_sum(data, red_axes): >> it = numpy.nditer([data, None], >> flags=['reduce_ok', 'external_loop'], >> op_flags=[['readonly'], ['readwrite', 'allocate']], >> op_axes=[None, red_axes]) >> it.operands[1][...] = 0 >> >> for x, y in it: >> y[...] = x.sum() >> >> return it.operands[1] >> >> I can get something equivalent to data.sum(axis=2) >> >>>>> data = numpy.arange(2*3*4).reshape((2,3,4)) >>>>> nditer_sum(data, [0, 1, -1]) >> [[ 6 22 38] >> [54 70 86]] >>>>> data.sum(axis=2) >> [[ 6 22 38] >> [54 70 86]] >> >> So to get something equivalent to data.sum(axis=0) I though that it >> was enough to change the argument red_axes to [-1, 0,1] >> But the result is quite different. >> >>>>> data = numpy.arange(2*3*4).reshape((2,3,4)) >>>>> data.sum(axis=0) >> [[12 14 16 18] >> [20 22 24 26] >> [28 30 32 34]] >>>>> nditer_sum(data, [-1, 0, 1]) >> [[210 210 210 210] >> [210 210 210 210] >> [210 210 210 210]] >> >> In the for loop inside nditer_sum (for x,y in it:), the iterator is >> looping 2 times and giving an array of length 12 each time, instead of >> looping 12 times and giving an array of length 2 each time. I have >> read the numpy documentation several times and googled about this to >> no avail. >> >> Does anybody have an example of a reduction in the first axis of an >> array using nditer? Is this a bug? >> >> >> Regards, Sergio >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > The example from the link it shows how to do a reduction with y[...] > += x. If you would replace x.sum() by += x, then it works. > >>>> nditer_sum(data, [-1,0,1]) > array([[12, 14, 16, 18], > [20, 22, 24, 26], > [28, 30, 32, 34]]) >>>> data.sum(axis=0) > array([[12, 14, 16, 18], > [20, 22, 24, 26], > [28, 30, 32, 34]]) > >>>> nditer_sum(data, [0,-1,1]) > array([[12, 15, 18, 21], > [48, 51, 54, 57]]) >>>> data.sum(axis=1) > array([[12, 15, 18, 21], > [48, 51, 54, 57]]) > > I think that is because sum() already reduces all axis by default. > > Regards, > Han > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sergio Pascual http://guaix.fis.ucm.es/~spr +34 91 394 5018 gpg fingerprint: 5203 B42D 86A0 5649 410A F4AC A35F D465 F263 BCCC Departamento de Astrof?sica -- Universidad Complutense de Madrid (Spain) From scopatz at gmail.com Mon Oct 1 23:32:05 2012 From: scopatz at gmail.com (Anthony Scopatz) Date: Mon, 1 Oct 2012 22:32:05 -0500 Subject: [Numpy-discussion] ufuncs for structured arrays In-Reply-To: References: Message-ID: Hello Jay, Cool idea! I like to see work on structured arrays. Just a couple of questions: - Since there are already have ufuncs for primitive dtypes (int, float, etc), and you are just acting columnwise here, can't you write a single function which interprets the dtypes, gathers the approriate scalar ufuncs, and applies those down each field? Yes registering individual functions using your mechanism will be faster, but it will also be more work. - I didn't try this out myself, but will the normal numpy broadcasting rules apply here? Be Well Anthony On Mon, Oct 1, 2012 at 10:36 AM, Jay Bourque wrote: > All, > > I've submitted the following pull request for NumPy: > > https://github.com/numpy/numpy/pull/462 > > This change allows ufuncs to be registered for structured arrays by using > a new API method PyUFunc_RegisterLoopForStructType. For example, a ufunc > could be registered to take two arrays of type 'u8,u8,u8' and return an > array of type 'u8,u8,u8'. I have a trivial example of this included in my > pull request, along with further details of my changes. I suspect there > might be a better way to do this, so any suggestions for improvements would > be welcome. > > Thanks, > -Jay > continuum.io > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Oct 2 01:51:42 2012 From: travis at continuum.io (Travis Oliphant) Date: Tue, 2 Oct 2012 00:51:42 -0500 Subject: [Numpy-discussion] Behavior of .base In-Reply-To: <5069A47B.50803@gmail.com> References: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> <5069A47B.50803@gmail.com> Message-ID: <8FB3B29F-D6BD-4AD4-85CD-CACEFBA8DE19@continuum.io> On Oct 1, 2012, at 9:11 AM, Jim Bosch wrote: > On 09/30/2012 03:59 PM, Travis Oliphant wrote: >> Hey all, >> >> In a github-discussion with Gael and Nathaniel, we came up with a proposal for .base that we should put before this list. Traditionally, .base has always pointed to None for arrays that owned their own memory and to the "most immediate" array object parent for arrays that did not own their own memory. There was a long-standing issue related to running out of stack space that this behavior created. >> >> Recently this behavior was altered so that .base always points to "the original" object holding the memory (something exposing the buffer interface). This created some problems for users who relied on the fact that most of the time .base pointed to an instance of an array object. >> >> The proposal here is to change the behavior of .base for arrays that don't own their own memory so that the .base attribute of an array points to "the most original object" that is still an instance of the type of the array. This would go into the 1.7.0 release so as to correct the issues reported. >> >> What are reactions to this proposal? >> > > In the past, I've relied on putting arbitrary Python objects in .base in > my C++ to NumPy conversion code to make sure reference counting for > array memory works properly. In particular, I've used Python CObjects > that hold boost::shared_ptrs, which don't even have a buffer interface. > So it sounds like I may be a few steps behind on the rules of what > actually should go in .base. > This should still work, nothing has been proposed to change this use-case. > I'm very concerned that if we do demand that .base always point to a > NumPy array (rather than an arbitrary Python object or even just one > with a buffer interface), there's no longer any way for a NumPy array to > hold data allocated by something other than NumPy. I don't recall a suggestion to demand that .base always point to a NumPy array. The suggestion is that a view of a view of an array that has your boost::shared_ptr as a PyCObject pointed to by base will have it's base point to the first array instead of the PyCObject (as the recent change made). > > If I want to put external memory in a NumPy array and indicate that it's > owned by some non-NumPy Python object, what is the recommended way to do > that? The approach you took is still the way I would recommend doing that. There may be other suggestions. -Travis From d.s.seljebotn at astro.uio.no Tue Oct 2 03:38:11 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Tue, 02 Oct 2012 09:38:11 +0200 Subject: [Numpy-discussion] Behavior of .base In-Reply-To: References: <15149FE7-75A4-4335-912C-92433F0AC98A@continuum.io> Message-ID: <506A99E3.3050006@astro.uio.no> On 10/01/2012 04:56 PM, Charles R Harris wrote: > > > On Mon, Oct 1, 2012 at 8:40 AM, Thouis (Ray) Jones > wrote: > > On Mon, Oct 1, 2012 at 8:20 AM, Nathaniel Smith > wrote: > > [...] > > How can we discourage people from doing this in the future? Can we > > make .base write-only from the Python level (with suitable > deprecation > > period)? Rename it to ._base (likewise) so that it's still > possible to > > peek under the covers but we remind people that it's really an > > implementation detail with poorly defined semantics that might > change? > > Could we use the simpler .base behavior (fully collapsing the .base > chain), but be more aggressive about propagating information like > address/filename/offset for np.arrays that are created by slicing, > asarray(), etc.? > > Ray > (Sorry if I'm missing some context that makes this suggestion idiotic. > I'm still trying to catch back up on the list and may have missed > relevant discussion on other threads.) > > > It might be productive to step back a bit and ask if this is a memmap > problem or a workflow problem. My impression is that pickling memmaps is > a solution to a higher level problem in Scikits.learn workflow and I'd > like more details on what that problem is. I'm not scikits-learn, but I'm pretty sure this is about wanting to use multiprocessing to parallelise code. You send pickled views of arrays, but the memory is shared amongst all processes (using either a file, or process shared memory). It would be cool to have some support for this in NumPy itself. The scikits-learn people should chime in here, but a suggestion: # pickles by reference to process-shared memory, or raises an exception # if memory can't be process-shared s = dumps(arr.byref) # in another process: arr = loads(s) Of course, *real* fixes would be to remove the GIL, or push forward the work in CPython on multiple independent interpreters in the same process. But that's rather more difficult. Dag Sverre From nouiz at nouiz.org Tue Oct 2 10:34:47 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Tue, 2 Oct 2012 10:34:47 -0400 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail Message-ID: Hi, I don't know if that was raised or not, but in np1.7b2 doing this fail with this error message: PyArray_BYTES(obj)=ptr file:lne_number:offset: error: lvalue required as left operatnd of assignment. I tried with PyArray_DATA(obj)=ptr and this also fail. Do you want to remove this feature now? I would think this change will be done at the same time as the one related to the macro NPY_NO_DEPRECATED_API. If I missed the discussion about this, tell me. thanks Fred From charlesr.harris at gmail.com Tue Oct 2 11:33:20 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 2 Oct 2012 09:33:20 -0600 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail In-Reply-To: References: Message-ID: On Tue, Oct 2, 2012 at 8:34 AM, Fr?d?ric Bastien wrote: > Hi, > > I don't know if that was raised or not, but in np1.7b2 doing this fail > with this error message: > > PyArray_BYTES(obj)=ptr > > file:lne_number:offset: error: lvalue required as left operatnd of > assignment. > > I tried with PyArray_DATA(obj)=ptr and this also fail. > > Do you want to remove this feature now? I would think this change will > be done at the same time as the one related to the macro > NPY_NO_DEPRECATED_API. > > If I missed the discussion about this, tell me. > > f2py wants to do the same thing, i.e., change the data pointer of an existing array, which is why NPY_NO_DEPRECATED_API is not defined in that module. I had some discussion off list with Pearu about that but it petered out. I think for this sort of thing is a new function needs to be implemented. What precisely is your application? IIRC, Pearu was using it to exchange pointers between two arrays to avoid a copy. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Tue Oct 2 11:45:50 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Tue, 2 Oct 2012 11:45:50 -0400 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail In-Reply-To: References: Message-ID: We implement our own subtensor(x[...], where ... can be index or slice) c code due to the way Theano work. I can probably change the logic, but if you plan to revert this interface changes, I prefer to wait for this change we someone else is doing other changes that would conflict. Also, I did a Theano release candidate and I really would like the final version to work with the next release of NumPy. thanks. Fred On Tue, Oct 2, 2012 at 11:33 AM, Charles R Harris wrote: > > > On Tue, Oct 2, 2012 at 8:34 AM, Fr?d?ric Bastien wrote: >> >> Hi, >> >> I don't know if that was raised or not, but in np1.7b2 doing this fail >> with this error message: >> >> PyArray_BYTES(obj)=ptr >> >> file:lne_number:offset: error: lvalue required as left operatnd of >> assignment. >> >> I tried with PyArray_DATA(obj)=ptr and this also fail. >> >> Do you want to remove this feature now? I would think this change will >> be done at the same time as the one related to the macro >> NPY_NO_DEPRECATED_API. >> >> If I missed the discussion about this, tell me. >> > > f2py wants to do the same thing, i.e., change the data pointer of an > existing array, which is why NPY_NO_DEPRECATED_API is not defined in that > module. I had some discussion off list with Pearu about that but it petered > out. I think for this sort of thing is a new function needs to be > implemented. What precisely is your application? IIRC, Pearu was using it to > exchange pointers between two arrays to avoid a copy. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Tue Oct 2 13:18:24 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 2 Oct 2012 11:18:24 -0600 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail In-Reply-To: References: Message-ID: On Tue, Oct 2, 2012 at 9:45 AM, Fr?d?ric Bastien wrote: > We implement our own subtensor(x[...], where ... can be index or > slice) c code due to the way Theano work. > > I can probably change the logic, but if you plan to revert this > interface changes, I prefer to wait for this change we someone else is > doing other changes that would conflict. Also, I did a Theano release > candidate and I really would like the final version to work with the > next release of NumPy. > Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't you can access the array as before using the macros even for future versions of numpy. The only way that could cause a problems is if the array structure is rearranged and I don't think that will happen anytime soon. On that account there has not been any discussion of reverting the changes. However, I'd like f2py generated modules to use the new functions at some point and in order to do that numpy needs to supply some extra functionality, I'm just not sure of the best way to do it at the moment. If I had a good idea of what you want to do it would help in deciding what numpy should provide. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Tue Oct 2 13:30:08 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Tue, 2 Oct 2012 13:30:08 -0400 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail In-Reply-To: References: Message-ID: On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris wrote: > > > On Tue, Oct 2, 2012 at 9:45 AM, Fr?d?ric Bastien wrote: >> >> We implement our own subtensor(x[...], where ... can be index or >> slice) c code due to the way Theano work. >> >> I can probably change the logic, but if you plan to revert this >> interface changes, I prefer to wait for this change we someone else is >> doing other changes that would conflict. Also, I did a Theano release >> candidate and I really would like the final version to work with the >> next release of NumPy. > > > Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't you can > access the array as before using the macros even for future versions of > numpy. The only way that could cause a problems is if the array structure is > rearranged and I don't think that will happen anytime soon. On that account > there has not been any discussion of reverting the changes. However, I'd > like f2py generated modules to use the new functions at some point and in > order to do that numpy needs to supply some extra functionality, I'm just > not sure of the best way to do it at the moment. If I had a good idea of > what you want to do it would help in deciding what numpy should provide. I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about that. That is the problem I have. What I do is close to that: alloc a new ndarray that point to the start of the ndarray I want to view with the right number of output dimensions. Compute the new dimensions/strides and data ptr. Then set the data ptr to what I computed. Then set the base ptr. I can reverse this and create the ndarray only at the end, but as this change break existing code here, it can break more code. That is why I wrote to the list. doing "PyArray_BASE(xview) = ptr" work when I don't define NPY_NO_DEPRECATED_API, but do not work when I define NPY_NO_DEPRECATED_API. I would have expected the same for PyArray_BYTES/DATA. Do this explain clearly the problem I saw? Fred From charlesr.harris at gmail.com Tue Oct 2 13:43:39 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 2 Oct 2012 11:43:39 -0600 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail In-Reply-To: References: Message-ID: On Tue, Oct 2, 2012 at 11:30 AM, Fr?d?ric Bastien wrote: > On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris > wrote: > > > > > > On Tue, Oct 2, 2012 at 9:45 AM, Fr?d?ric Bastien > wrote: > >> > >> We implement our own subtensor(x[...], where ... can be index or > >> slice) c code due to the way Theano work. > >> > >> I can probably change the logic, but if you plan to revert this > >> interface changes, I prefer to wait for this change we someone else is > >> doing other changes that would conflict. Also, I did a Theano release > >> candidate and I really would like the final version to work with the > >> next release of NumPy. > > > > > > Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't you > can > > access the array as before using the macros even for future versions of > > numpy. The only way that could cause a problems is if the array > structure is > > rearranged and I don't think that will happen anytime soon. On that > account > > there has not been any discussion of reverting the changes. However, I'd > > like f2py generated modules to use the new functions at some point and in > > order to do that numpy needs to supply some extra functionality, I'm just > > not sure of the best way to do it at the moment. If I had a good idea of > > what you want to do it would help in deciding what numpy should provide. > > I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about > that. That is the problem I have. > > What I do is close to that: > > alloc a new ndarray that point to the start of the ndarray I want to > view with the right number of output dimensions. > > Compute the new dimensions/strides and data ptr. Then set the data ptr > to what I computed. Then set the base ptr. > > I can reverse this and create the ndarray only at the end, but as this > change break existing code here, it can break more code. That is why I > wrote to the list. > > doing "PyArray_BASE(xview) = ptr" work when I don't define > NPY_NO_DEPRECATED_API, but do not work when I define > NPY_NO_DEPRECATED_API. I would have expected the same for > PyArray_BYTES/DATA. > > Do this explain clearly the problem I saw? > > Yes, thanks. I see in ndarraytypes.h #define PyArray_DATA(obj) ((void *)(((PyArrayObject_fields *)(obj))->data)) I wonder if the cast to void* is causing a problem? Could you try char* instead? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Oct 2 13:47:51 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 2 Oct 2012 11:47:51 -0600 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail In-Reply-To: References: Message-ID: On Tue, Oct 2, 2012 at 11:43 AM, Charles R Harris wrote: > > > On Tue, Oct 2, 2012 at 11:30 AM, Fr?d?ric Bastien wrote: > >> On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris >> wrote: >> > >> > >> > On Tue, Oct 2, 2012 at 9:45 AM, Fr?d?ric Bastien >> wrote: >> >> >> >> We implement our own subtensor(x[...], where ... can be index or >> >> slice) c code due to the way Theano work. >> >> >> >> I can probably change the logic, but if you plan to revert this >> >> interface changes, I prefer to wait for this change we someone else is >> >> doing other changes that would conflict. Also, I did a Theano release >> >> candidate and I really would like the final version to work with the >> >> next release of NumPy. >> > >> > >> > Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't >> you can >> > access the array as before using the macros even for future versions of >> > numpy. The only way that could cause a problems is if the array >> structure is >> > rearranged and I don't think that will happen anytime soon. On that >> account >> > there has not been any discussion of reverting the changes. However, I'd >> > like f2py generated modules to use the new functions at some point and >> in >> > order to do that numpy needs to supply some extra functionality, I'm >> just >> > not sure of the best way to do it at the moment. If I had a good idea of >> > what you want to do it would help in deciding what numpy should provide. >> >> I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about >> that. That is the problem I have. >> >> What I do is close to that: >> >> alloc a new ndarray that point to the start of the ndarray I want to >> view with the right number of output dimensions. >> >> Compute the new dimensions/strides and data ptr. Then set the data ptr >> to what I computed. Then set the base ptr. >> >> I can reverse this and create the ndarray only at the end, but as this >> change break existing code here, it can break more code. That is why I >> wrote to the list. >> >> doing "PyArray_BASE(xview) = ptr" work when I don't define >> NPY_NO_DEPRECATED_API, but do not work when I define >> NPY_NO_DEPRECATED_API. I would have expected the same for >> PyArray_BYTES/DATA. >> >> Do this explain clearly the problem I saw? >> >> > Yes, thanks. I see in ndarraytypes.h > > #define PyArray_DATA(obj) ((void *)(((PyArrayObject_fields *)(obj))->data)) > > I wonder if the cast to void* is causing a problem? Could you try char* > instead? > Oops, the problem is that you need a pointer to the slot, not the pointer in the slot. That is, you need a different macro/function. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Oct 2 14:21:19 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 2 Oct 2012 12:21:19 -0600 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail In-Reply-To: References: Message-ID: On Tue, Oct 2, 2012 at 11:47 AM, Charles R Harris wrote: > > > On Tue, Oct 2, 2012 at 11:43 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Tue, Oct 2, 2012 at 11:30 AM, Fr?d?ric Bastien wrote: >> >>> On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris >>> wrote: >>> > >>> > >>> > On Tue, Oct 2, 2012 at 9:45 AM, Fr?d?ric Bastien >>> wrote: >>> >> >>> >> We implement our own subtensor(x[...], where ... can be index or >>> >> slice) c code due to the way Theano work. >>> >> >>> >> I can probably change the logic, but if you plan to revert this >>> >> interface changes, I prefer to wait for this change we someone else is >>> >> doing other changes that would conflict. Also, I did a Theano release >>> >> candidate and I really would like the final version to work with the >>> >> next release of NumPy. >>> > >>> > >>> > Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't >>> you can >>> > access the array as before using the macros even for future versions of >>> > numpy. The only way that could cause a problems is if the array >>> structure is >>> > rearranged and I don't think that will happen anytime soon. On that >>> account >>> > there has not been any discussion of reverting the changes. However, >>> I'd >>> > like f2py generated modules to use the new functions at some point and >>> in >>> > order to do that numpy needs to supply some extra functionality, I'm >>> just >>> > not sure of the best way to do it at the moment. If I had a good idea >>> of >>> > what you want to do it would help in deciding what numpy should >>> provide. >>> >>> I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about >>> that. That is the problem I have. >>> >>> What I do is close to that: >>> >>> alloc a new ndarray that point to the start of the ndarray I want to >>> view with the right number of output dimensions. >>> >>> Compute the new dimensions/strides and data ptr. Then set the data ptr >>> to what I computed. Then set the base ptr. >>> >>> I can reverse this and create the ndarray only at the end, but as this >>> change break existing code here, it can break more code. That is why I >>> wrote to the list. >>> >>> doing "PyArray_BASE(xview) = ptr" work when I don't define >>> NPY_NO_DEPRECATED_API, but do not work when I define >>> NPY_NO_DEPRECATED_API. I would have expected the same for >>> PyArray_BYTES/DATA. >>> >>> Do this explain clearly the problem I saw? >>> >>> >> Yes, thanks. I see in ndarraytypes.h >> >> #define PyArray_DATA(obj) ((void *)(((PyArrayObject_fields >> *)(obj))->data)) >> >> I wonder if the cast to void* is causing a problem? Could you try char* >> instead? >> > > Oops, the problem is that you need a pointer to the slot, not the pointer > in the slot. That is, you need a different macro/function. > And what I was thinking of for this was something like a PyArray_SWAP, that simply swaps the contents of two array structures. However, that needs to be analysed closely for nasty side effects beyond the surprise of an array unexpectedly changing all of its characteristics underneath its pointer. It's inherently dangerous but probably essential for some things. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Tue Oct 2 15:44:03 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Tue, 2 Oct 2012 15:44:03 -0400 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail In-Reply-To: References: Message-ID: With numpy 1.6.2, it is working. So this is an interface change. Are you sure you want this? This break existing code. I do not understand what you mean by slot? I'm not sure is the PyArray_SWAP is a good long term idea. I would not make it if it is only for temporarily. To set the base ptr, there is PyArray_SetBaseObject() fct that is new in 1.7. Is a similar fct useful in the long term for numpy? In the case where we implement differently the ndarray object, I think it won't be useful. We will also need to know how the memory is laid out by numpy for performance critical code. We we will need an attribute that tell the intern structure used. So do you want to force this interface change in numpy 1.7 so that I need to write codes now or can I wait to do it when you force the new interface? Currently the used code for PyArray_BYTES is: #define PyArray_BYTES(obj) ((char *)(((PyArrayObject_fields *)(obj))->data)) if I change it to #define PyArray_BYTES(obj) ((((PyArrayObject_fields *)(obj))->data)) it work! I don't understand why removing the case make it work. the data field is already an (char*), so this should not make a difference to my underderstanding. But I'm missing something here, do someone know? Fred On Tue, Oct 2, 2012 at 1:47 PM, Charles R Harris wrote: > > > On Tue, Oct 2, 2012 at 11:43 AM, Charles R Harris > wrote: >> >> >> >> On Tue, Oct 2, 2012 at 11:30 AM, Fr?d?ric Bastien wrote: >>> >>> On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris >>> wrote: >>> > >>> > >>> > On Tue, Oct 2, 2012 at 9:45 AM, Fr?d?ric Bastien >>> > wrote: >>> >> >>> >> We implement our own subtensor(x[...], where ... can be index or >>> >> slice) c code due to the way Theano work. >>> >> >>> >> I can probably change the logic, but if you plan to revert this >>> >> interface changes, I prefer to wait for this change we someone else is >>> >> doing other changes that would conflict. Also, I did a Theano release >>> >> candidate and I really would like the final version to work with the >>> >> next release of NumPy. >>> > >>> > >>> > Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't >>> > you can >>> > access the array as before using the macros even for future versions of >>> > numpy. The only way that could cause a problems is if the array >>> > structure is >>> > rearranged and I don't think that will happen anytime soon. On that >>> > account >>> > there has not been any discussion of reverting the changes. However, >>> > I'd >>> > like f2py generated modules to use the new functions at some point and >>> > in >>> > order to do that numpy needs to supply some extra functionality, I'm >>> > just >>> > not sure of the best way to do it at the moment. If I had a good idea >>> > of >>> > what you want to do it would help in deciding what numpy should >>> > provide. >>> >>> I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about >>> that. That is the problem I have. >>> >>> What I do is close to that: >>> >>> alloc a new ndarray that point to the start of the ndarray I want to >>> view with the right number of output dimensions. >>> >>> Compute the new dimensions/strides and data ptr. Then set the data ptr >>> to what I computed. Then set the base ptr. >>> >>> I can reverse this and create the ndarray only at the end, but as this >>> change break existing code here, it can break more code. That is why I >>> wrote to the list. >>> >>> doing "PyArray_BASE(xview) = ptr" work when I don't define >>> NPY_NO_DEPRECATED_API, but do not work when I define >>> NPY_NO_DEPRECATED_API. I would have expected the same for >>> PyArray_BYTES/DATA. >>> >>> Do this explain clearly the problem I saw? >>> >> >> Yes, thanks. I see in ndarraytypes.h >> >> #define PyArray_DATA(obj) ((void *)(((PyArrayObject_fields >> *)(obj))->data)) >> >> I wonder if the cast to void* is causing a problem? Could you try char* >> instead? > > > Oops, the problem is that you need a pointer to the slot, not the pointer in > the slot. That is, you need a different macro/function. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Tue Oct 2 18:18:58 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 2 Oct 2012 16:18:58 -0600 Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail In-Reply-To: References: Message-ID: On Tue, Oct 2, 2012 at 1:44 PM, Fr?d?ric Bastien wrote: > With numpy 1.6.2, it is working. So this is an interface change. Are > you sure you want this? This break existing code. > > I do not understand what you mean by slot? > Pythonese for structure member ;) > > I'm not sure is the PyArray_SWAP is a good long term idea. I would not > make it if it is only for temporarily. > The C++ stdlib provides something similar for std::vector. One common use case would be to pass in a vector by reference that gets swapped with one on the stack. When the function exits the one on the stack is cleaned up and the vector that was passed in has the new data, but it has to be the same type. For PyArray_SWAP I was thinking of swapping everything: type, dims, strides, data, etc. That is what f2py does. > To set the base ptr, there is PyArray_SetBaseObject() fct that is new > in 1.7. Is a similar fct useful in the long term for numpy? In the > case where we implement differently the ndarray object, I think it > won't be useful. We will also need to know how the memory is laid out > by numpy for performance critical code. We we will need an attribute > that tell the intern structure used. > > So do you want to force this interface change in numpy 1.7 so that I > need to write codes now or can I wait to do it when you force the new > interface? > > Well, no we don't want to force you to use the new interface. If you don't define NPY_NO_DEPRECATED_API things should still work. Although if it is defined the function returns an rvalue, so some other method needs to be provided for what you are doing. > Currently the used code for PyArray_BYTES is: > > #define PyArray_BYTES(obj) ((char *)(((PyArrayObject_fields > *)(obj))->data)) > > if I change it to > > #define PyArray_BYTES(obj) ((((PyArrayObject_fields *)(obj))->data)) > > it work! I don't understand why removing the case make it work. the > data field is already an (char*), so this should not make a difference > to my underderstanding. But I'm missing something here, do someone > know? > What I find strange is that it is the same macro in 1.7 and 1.6, only the name of the structure was changed. Hmm... This looks almost like some compiler subtlety, I wonder if the compiler version/optimization flags have changed? In any case, I think the second form would be more correct for the lvalue since the structure member is, as you say, already a char*. We want things to work for you as they should, so we need to understand this and fix it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Oct 3 07:26:24 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 3 Oct 2012 12:26:24 +0100 Subject: [Numpy-discussion] tests for casting table? (was: Numpy 1.7b1 API change cause big trouble) In-Reply-To: References: <2F489A53-EE68-4E62-9DB2-5224FA4EEB14@continuum.io> Message-ID: Hi, On Fri, Sep 21, 2012 at 1:05 AM, Fr?d?ric Bastien wrote: > Hi, > > Finally, the change about the casting rule was done in NumPy 1.6. It > is our test that checked specifically for numpy 1.6 behavior. But > adding the test to make sure it don't change is an excellent idea. Sorry - family reunion and Starbuck's-coffee-on-laptop action combined to prevent me working on this until now. Do you want a PR against 1.7.x or trunk? See you, Matthew From wesmckinn at gmail.com Wed Oct 3 11:48:53 2012 From: wesmckinn at gmail.com (Wes McKinney) Date: Wed, 3 Oct 2012 11:48:53 -0400 Subject: [Numpy-discussion] memory-efficient loadtxt In-Reply-To: References: Message-ID: On Monday, October 1, 2012, Chris Barker wrote: > Paul, > > Nice to see someone working on these issues, but: > > I'm not sure the problem you are trying to solve -- accumulating in a > list is pretty efficient anyway -- not a whole lot overhead. > > But if you do want to improve that, it may be better to change the > accumulating method, rather than doing the double-read thing. I"ve > written, and posted here, code that provides an Acumulator that uses > numpy internally, so not much memory overhead. In the end, it's not > any faster than accumulating in a list and then converting to an > array, but it does use less memory. > > I also have a Cython version that is not quite done (darn regular job > getting in the way) that is both faster and more memory efficient. > > Also, frankly, just writing the array pre-allocation and re-sizeing > code into loadtxt would not be a whole lot of code either, and would > be both fast and memory efficient. > > Let mw know if you want any of my code to play with. > > > However, I got the impression that someone was > > working on a More Advanced (TM) C-based file reader, which will > > replace loadtxt; > > yes -- I wonder what happened with that? Anyone? > > -CHB > > > > this patch is intended as a useful thing to have > > while we're waiting for that to appear. > > > > The patch passes all tests in the test suite, and documentation for > > the kwarg has been added. I've modified all tests to include the > > seekable kwarg, but that was mostly to check that all tests are passed > > also with this kwarg. I guess it's bit too late for 1.7.0 though? > > > > Should I make a pull request? I'm happy to take any and all > > suggestions before I do. > > > > Cheers > > Paul > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I've finally built a new, very fast C-based tokenizer/parser with type inference, NA-handling, etc. for pandas sporadically over the last month-- it's almost ready to ship. It's roughly an order of magnitude faster than loadtxt and uses very little temporary space. Should be easy to push upstream into NumPy to replace the innards of np.loadtxt if I can get a bit of help with the plumbing (it already yields structured arrays in addition to pandas DataFrames so there isn't a great deal that needs doing). Blog post with CPU and memory benchmarks to follow-- will post a link here. - Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Wed Oct 3 11:58:42 2012 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 3 Oct 2012 17:58:42 +0200 Subject: [Numpy-discussion] memory-efficient loadtxt In-Reply-To: References: Message-ID: <1D47FCFA-654F-4FEC-80A7-A390E3FBE50B@gmail.com> On 3. okt. 2012, at 17:48, Wes McKinney wrote: > On Monday, October 1, 2012, Chris Barker wrote: > Paul, > > Nice to see someone working on these issues, but: > > I'm not sure the problem you are trying to solve -- accumulating in a > list is pretty efficient anyway -- not a whole lot overhead. > > But if you do want to improve that, it may be better to change the > accumulating method, rather than doing the double-read thing. I"ve > written, and posted here, code that provides an Acumulator that uses > numpy internally, so not much memory overhead. In the end, it's not > any faster than accumulating in a list and then converting to an > array, but it does use less memory. > > I also have a Cython version that is not quite done (darn regular job > getting in the way) that is both faster and more memory efficient. > > Also, frankly, just writing the array pre-allocation and re-sizeing > code into loadtxt would not be a whole lot of code either, and would > be both fast and memory efficient. > > Let mw know if you want any of my code to play with. > > > However, I got the impression that someone was > > working on a More Advanced (TM) C-based file reader, which will > > replace loadtxt; > > yes -- I wonder what happened with that? Anyone? > > -CHB > > > > this patch is intended as a useful thing to have > > while we're waiting for that to appear. > > > > The patch passes all tests in the test suite, and documentation for > > the kwarg has been added. I've modified all tests to include the > > seekable kwarg, but that was mostly to check that all tests are passed > > also with this kwarg. I guess it's bit too late for 1.7.0 though? > > > > Should I make a pull request? I'm happy to take any and all > > suggestions before I do. > > > > Cheers > > Paul > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > I've finally built a new, very fast C-based tokenizer/parser with type inference, NA-handling, etc. for pandas sporadically over the last month-- it's almost ready to ship. It's roughly an order of magnitude faster than loadtxt and uses very little temporary space. Should be easy to push upstream into NumPy to replace the innards of np.loadtxt if I can get a bit of help with the plumbing (it already yields structured arrays in addition to pandas DataFrames so there isn't a great deal that needs doing). > > Blog post with CPU and memory benchmarks to follow-- will post a link here. > > - Wes > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion So Chris, looks like Wes has us beaten in every conceivable way. Hey, that's a good thing :) I suppose the thing to do now is to make sure Wes' function tackles the loadtxt test suite? Paul From paul.anton.letnes at gmail.com Wed Oct 3 12:05:49 2012 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 3 Oct 2012 18:05:49 +0200 Subject: [Numpy-discussion] memory-efficient loadtxt In-Reply-To: References: Message-ID: <29329AEB-C362-4712-8AD2-B7F2860FEC4B@gmail.com> On 1. okt. 2012, at 21:07, Chris Barker wrote: > Paul, > > Nice to see someone working on these issues, but: > > I'm not sure the problem you are trying to solve -- accumulating in a > list is pretty efficient anyway -- not a whole lot overhead. Oh, there's significant overhead, since we're not talking of a list - we're talking of a list-of-lists. My guesstimate from my hacking session (off the top of my head - I don't have my benchmarks in working memory :) is around 3-5 times more memory with the list-of-lists approach for a single column / 1D array, which presumably is the worst case (a length 1 list for each line of input). Hence, if you want to load a 2 GB file into RAM on a machine with 4 GB of the stuff, you're screwed with the old approach and a happy camper with mine. > But if you do want to improve that, it may be better to change the > accumulating method, rather than doing the double-read thing. I"ve > written, and posted here, code that provides an Acumulator that uses > numpy internally, so not much memory overhead. In the end, it's not > any faster than accumulating in a list and then converting to an > array, but it does use less memory. I see your point - but if you're to return a single array, and the file is close to the total system memory, you've still got a factor of 2 issue when shuffling the binary data from the accumulator into the result array. That is, unless I'm missong something here? Cheers Paul From chris.barker at noaa.gov Wed Oct 3 12:22:35 2012 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 3 Oct 2012 09:22:35 -0700 Subject: [Numpy-discussion] memory-efficient loadtxt In-Reply-To: <29329AEB-C362-4712-8AD2-B7F2860FEC4B@gmail.com> References: <29329AEB-C362-4712-8AD2-B7F2860FEC4B@gmail.com> Message-ID: On Wed, Oct 3, 2012 at 9:05 AM, Paul Anton Letnes wrote: >> I'm not sure the problem you are trying to solve -- accumulating in a >> list is pretty efficient anyway -- not a whole lot overhead. > > Oh, there's significant overhead, since we're not talking of a list - we're talking of a list-of-lists. hmm, a list of nupy scalers (custom dtype) would be a better option, though maybe not all that much better -- still an extra pointer and pyton object for each row. > I see your point - but if you're to return a single array, and the file is close to the total system memory, you've still got a factor of 2 issue when shuffling the binary data from the accumulator into the result array. That is, unless I'm missong something here? Indeed, I think that's how my current accumulator works -- the __array__() method returns a copy of the data buffer, so that you won't accidentally re-allocate it under the hood later and screw up the returned version. But it is indeed accumulating in a numpy array, so it should be pretty possible, maybe even easy to turn it into a regular array without a data copy. You'd just have to destroy the original somehow (or mark it as never-resize) so you wouldn't have the clash. messing wwith the OWNDATA flags might take care of that. But it seems Wes has a better solution. One other note, though -- if you have arrays that are that close to max system memory, you are very likely to have other trouble anyway -- numpy does make a lot of copies! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From paul.anton.letnes at gmail.com Wed Oct 3 12:36:28 2012 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 3 Oct 2012 18:36:28 +0200 Subject: [Numpy-discussion] memory-efficient loadtxt In-Reply-To: References: <29329AEB-C362-4712-8AD2-B7F2860FEC4B@gmail.com> Message-ID: <9911D65D-A0D0-43AC-A408-27C9C6461FCE@gmail.com> On 3. okt. 2012, at 18:22, Chris Barker wrote: > On Wed, Oct 3, 2012 at 9:05 AM, Paul Anton Letnes > wrote: > >>> I'm not sure the problem you are trying to solve -- accumulating in a >>> list is pretty efficient anyway -- not a whole lot overhead. >> >> Oh, there's significant overhead, since we're not talking of a list - we're talking of a list-of-lists. > > hmm, a list of nupy scalers (custom dtype) would be a better option, > though maybe not all that much better -- still an extra pointer and > pyton object for each row. > > >> I see your point - but if you're to return a single array, and the file is close to the total system memory, you've still got a factor of 2 issue when shuffling the binary data from the accumulator into the result array. That is, unless I'm missong something here? > > Indeed, I think that's how my current accumulator works -- the > __array__() method returns a copy of the data buffer, so that you > won't accidentally re-allocate it under the hood later and screw up > the returned version. > > But it is indeed accumulating in a numpy array, so it should be pretty > possible, maybe even easy to turn it into a regular array without a > data copy. You'd just have to destroy the original somehow (or mark it > as never-resize) so you wouldn't have the clash. messing wwith the > OWNDATA flags might take care of that. > > But it seems Wes has a better solution. Indeed. > One other note, though -- if you have arrays that are that close to > max system memory, you are very likely to have other trouble anyway -- > numpy does make a lot of copies! That's true. Now, I'm not worried about this myself, but several people have complained about this on the mailing list, and it seemed like an easy fix. Oh well, it's too late for it now, anyways. Paul From lee.will at gmail.com Wed Oct 3 13:58:04 2012 From: lee.will at gmail.com (Will Lee) Date: Wed, 3 Oct 2012 12:58:04 -0500 Subject: [Numpy-discussion] dtype '|S0' not understood In-Reply-To: <113F0316-56E1-4403-B967-EC06847B8251@cs.toronto.edu> References: <113F0316-56E1-4403-B967-EC06847B8251@cs.toronto.edu> Message-ID: This seems to be a old problem but I've recently hit with this in a very random way (I'm using numpy 1.6.1). There seems to be a ticket (1239) but it seems the issue is unscheduled. Can somebody tell me if this is fixed? In particular, it makes for a very unstable behavior when you try to reference something from a string array and pickle it across the wire. For example: In [1]: import numpy In [2]: a = numpy.array(['a', '', 'b']) In [3]: import cPickle In [4]: s = cPickle.dumps(a[1]) In [5]: cPickle.loads(s) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /auto/cnvtvws/wlee/fstrat/src/ in () ----> 1 cPickle.loads(s) TypeError: ('data type not understood', , ('S0', 0, 1)) Note that if you reference a[0] and a[2], it would work, so you're in the case where sometimes it'd work but sometimes it won't. Checking for this case in the code and work around it would really be a pain. Thanks! Will On Thu, Sep 24, 2009 at 7:03 PM, David Warde-Farley wrote: > On 23-Sep-09, at 7:55 PM, David Warde-Farley wrote: > > > It seems that either dtype(str) should do something more sensible than > > zero-length string, or it should be possible to create it with > > dtype('| > > S0'). Which should it be? > > > Since there wasn't any response I went ahead and fixed it by making > str and unicode dtypes allow a size of 0 when constructed with > protocol type codes. Either S0 and U0 should be constructable with > typecodes or they shouldn't be allowed at all; I opted for the latter > since a) it was simple and b) I don't know what a sensible default for > dtype(str) would be (length 1? length 10?). > > Patch is at: > > http://projects.scipy.org/numpy/ticket/1239 > > Review away! > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Oct 3 17:17:50 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 3 Oct 2012 23:17:50 +0200 Subject: [Numpy-discussion] numpy distutils log error with easy_install In-Reply-To: References: Message-ID: On Mon, Oct 1, 2012 at 10:47 PM, Matthew Brett wrote: > Hi, > > On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett > wrote: > > Hi, > > > > One of our kind users pointed out an error when using easy_install to > > install our package nipy. I've reproduced it now on a bare package > > using numpy distutils and having a trivial extension: > > > > https://github.com/matthew-brett/apkg > > > > To reproduce: > > > > git clone git://github.com/mathew-brett/apkg.git > > > > easy_install apkg > > > > You should get something like this: > > > > Processing apkg > > Running setup.py -q bdist_egg --dist-dir > > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB > > Appending apkg configuration to > > Ignoring attempt to set 'name' (from '' to 'apkg') > > zip_safe flag not set; analyzing archive contents... > > Adding apkg 0.1 to easy-install.pth file > > > > Installed > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg > > Processing dependencies for apkg==0.1 > > Finished processing dependencies for apkg==0.1 > > > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: > > RuntimeWarning: Parent module 'numpy.distutils' not found while > > handling absolute import > > from numpy.distutils import log > > > > Note the last error. > > Sorry, correcting myself - it's (obviously) a Warning rather than an > error, but still distracting, and it would be good to avoid it if > possible... > The combination of two or all of atexit.register, easy_install and virtualenv seems to be causing this. Unless someone feels like digging into that (I certainly don't), there are two easy solutions: 1. Silence the warning. 2. Remove the offending import and the logging. This will only remove the line "removing: _configtest.c _configtest.o" from the build log (x20). Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Oct 3 17:29:21 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 3 Oct 2012 22:29:21 +0100 Subject: [Numpy-discussion] numpy distutils log error with easy_install In-Reply-To: References: Message-ID: Hi, On Wed, Oct 3, 2012 at 10:17 PM, Ralf Gommers wrote: > > > On Mon, Oct 1, 2012 at 10:47 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett >> wrote: >> > Hi, >> > >> > One of our kind users pointed out an error when using easy_install to >> > install our package nipy. I've reproduced it now on a bare package >> > using numpy distutils and having a trivial extension: >> > >> > https://github.com/matthew-brett/apkg >> > >> > To reproduce: >> > >> > git clone git://github.com/mathew-brett/apkg.git >> > >> > easy_install apkg >> > >> > You should get something like this: >> > >> > Processing apkg >> > Running setup.py -q bdist_egg --dist-dir >> > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB >> > Appending apkg configuration to >> > Ignoring attempt to set 'name' (from '' to 'apkg') >> > zip_safe flag not set; analyzing archive contents... >> > Adding apkg 0.1 to easy-install.pth file >> > >> > Installed >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg >> > Processing dependencies for apkg==0.1 >> > Finished processing dependencies for apkg==0.1 >> > >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: >> > RuntimeWarning: Parent module 'numpy.distutils' not found while >> > handling absolute import >> > from numpy.distutils import log >> > >> > Note the last error. >> >> Sorry, correcting myself - it's (obviously) a Warning rather than an >> error, but still distracting, and it would be good to avoid it if >> possible... > > > The combination of two or all of atexit.register, easy_install and > virtualenv seems to be causing this. Unless someone feels like digging into > that (I certainly don't), there are two easy solutions: > 1. Silence the warning. Sorry - I am not sure what you mean. The problem here is the user who assumes that something bad happened when running easy_install - which is what happened in the case of nipy. Is there some way of silencing this (specific) warning from within setup.py? > 2. Remove the offending import and the logging. This will only remove the > line "removing: _configtest.c _configtest.o" from the build log (x20). Which import did you mean? I think I need all the imports I'm using in the example minimal package. I'm not explicitly importing logging for example. Thanks for looking into it... Cheers, Matthew From ralf.gommers at gmail.com Wed Oct 3 17:51:25 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 3 Oct 2012 23:51:25 +0200 Subject: [Numpy-discussion] numpy distutils log error with easy_install In-Reply-To: References: Message-ID: On Wed, Oct 3, 2012 at 11:29 PM, Matthew Brett wrote: > Hi, > > On Wed, Oct 3, 2012 at 10:17 PM, Ralf Gommers > wrote: > > > > > > On Mon, Oct 1, 2012 at 10:47 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett > >> wrote: > >> > Hi, > >> > > >> > One of our kind users pointed out an error when using easy_install to > >> > install our package nipy. I've reproduced it now on a bare package > >> > using numpy distutils and having a trivial extension: > >> > > >> > https://github.com/matthew-brett/apkg > >> > > >> > To reproduce: > >> > > >> > git clone git://github.com/mathew-brett/apkg.git > >> > > >> > easy_install apkg > >> > > >> > You should get something like this: > >> > > >> > Processing apkg > >> > Running setup.py -q bdist_egg --dist-dir > >> > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB > >> > Appending apkg configuration to > >> > Ignoring attempt to set 'name' (from '' to 'apkg') > >> > zip_safe flag not set; analyzing archive contents... > >> > Adding apkg 0.1 to easy-install.pth file > >> > > >> > Installed > >> > > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg > >> > Processing dependencies for apkg==0.1 > >> > Finished processing dependencies for apkg==0.1 > >> > > >> > > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: > >> > RuntimeWarning: Parent module 'numpy.distutils' not found while > >> > handling absolute import > >> > from numpy.distutils import log > >> > > >> > Note the last error. > >> > >> Sorry, correcting myself - it's (obviously) a Warning rather than an > >> error, but still distracting, and it would be good to avoid it if > >> possible... > > > > > > The combination of two or all of atexit.register, easy_install and > > virtualenv seems to be causing this. Unless someone feels like digging > into > > that (I certainly don't), there are two easy solutions: > > 1. Silence the warning. > > Sorry - I am not sure what you mean. The problem here is the user who > assumes that something bad happened when running easy_install - which > is what happened in the case of nipy. Is there some way of silencing > this (specific) warning from within setup.py? > > > 2. Remove the offending import and the logging. This will only remove the > > line "removing: _configtest.c _configtest.o" from the build log (x20). > > Which import did you mean? I think I need all the imports I'm using > in the example minimal package. I'm not explicitly importing logging > for example. > The import that's indicated in the warning, on line 252 of numpy/distutils/misc_util.py. Relevant code: _temporary_directory = None def clean_up_temporary_directory(): from numpy.distutils import log # CAUSES RUNTIME WARNING global _temporary_directory if not _temporary_directory: return log.debug('removing %s', _temporary_directory) try: shutil.rmtree(_temporary_directory) except OSError: pass _temporary_directory = None def make_temp_file(suffix='', prefix='', text=True): global _temporary_directory if not _temporary_directory: _temporary_directory = tempfile.mkdtemp() atexit.register(clean_up_temporary_directory) Ralf > Thanks for looking into it... > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Oct 3 18:14:55 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 3 Oct 2012 23:14:55 +0100 Subject: [Numpy-discussion] numpy distutils log error with easy_install In-Reply-To: References: Message-ID: Hi, On Wed, Oct 3, 2012 at 10:51 PM, Ralf Gommers wrote: > > > On Wed, Oct 3, 2012 at 11:29 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Wed, Oct 3, 2012 at 10:17 PM, Ralf Gommers >> wrote: >> > >> > >> > On Mon, Oct 1, 2012 at 10:47 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett >> >> wrote: >> >> > Hi, >> >> > >> >> > One of our kind users pointed out an error when using easy_install to >> >> > install our package nipy. I've reproduced it now on a bare package >> >> > using numpy distutils and having a trivial extension: >> >> > >> >> > https://github.com/matthew-brett/apkg >> >> > >> >> > To reproduce: >> >> > >> >> > git clone git://github.com/mathew-brett/apkg.git >> >> > >> >> > easy_install apkg >> >> > >> >> > You should get something like this: >> >> > >> >> > Processing apkg >> >> > Running setup.py -q bdist_egg --dist-dir >> >> > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB >> >> > Appending apkg configuration to >> >> > Ignoring attempt to set 'name' (from '' to 'apkg') >> >> > zip_safe flag not set; analyzing archive contents... >> >> > Adding apkg 0.1 to easy-install.pth file >> >> > >> >> > Installed >> >> > >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg >> >> > Processing dependencies for apkg==0.1 >> >> > Finished processing dependencies for apkg==0.1 >> >> > >> >> > >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: >> >> > RuntimeWarning: Parent module 'numpy.distutils' not found while >> >> > handling absolute import >> >> > from numpy.distutils import log >> >> > >> >> > Note the last error. >> >> >> >> Sorry, correcting myself - it's (obviously) a Warning rather than an >> >> error, but still distracting, and it would be good to avoid it if >> >> possible... >> > >> > >> > The combination of two or all of atexit.register, easy_install and >> > virtualenv seems to be causing this. Unless someone feels like digging >> > into >> > that (I certainly don't), there are two easy solutions: >> > 1. Silence the warning. >> >> Sorry - I am not sure what you mean. The problem here is the user who >> assumes that something bad happened when running easy_install - which >> is what happened in the case of nipy. Is there some way of silencing >> this (specific) warning from within setup.py? >> >> > 2. Remove the offending import and the logging. This will only remove >> > the >> > line "removing: _configtest.c _configtest.o" from the build log (x20). >> >> Which import did you mean? I think I need all the imports I'm using >> in the example minimal package. I'm not explicitly importing logging >> for example. > > > The import that's indicated in the warning, on line 252 of > numpy/distutils/misc_util.py. Relevant code: > > _temporary_directory = None > def clean_up_temporary_directory(): > from numpy.distutils import log # CAUSES RUNTIME WARNING > global _temporary_directory > if not _temporary_directory: > return > log.debug('removing %s', _temporary_directory) > try: > shutil.rmtree(_temporary_directory) > except OSError: > pass > _temporary_directory = None > > def make_temp_file(suffix='', prefix='', text=True): > global _temporary_directory > if not _temporary_directory: > _temporary_directory = tempfile.mkdtemp() > atexit.register(clean_up_temporary_directory) Sorry - I still don't understand. You mean I should (in my package - say nipy or 'apkg') monkey-patch numpy distutils.misc_util ? Another option would be to move the import outside the callback function thus: diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py index 2e4ed27..e00d924 100644 --- a/numpy/distutils/misc_util.py +++ b/numpy/distutils/misc_util.py @@ -18,6 +18,7 @@ except NameError: from sets import Set as set from numpy.distutils.compat import get_exception +from numpy.distutils.log import debug as log_debug __all__ = ['Configuration', 'get_numpy_include_dirs', 'default_config_dict', 'dict_append', 'appendpath', 'generate_config_py', @@ -249,11 +250,10 @@ def gpaths(paths, local_path='', include_non_existing=True): _temporary_directory = None def clean_up_temporary_directory(): - from numpy.distutils import log global _temporary_directory if not _temporary_directory: return - log.debug('removing %s', _temporary_directory) + log_debug('removing %s', _temporary_directory) try: shutil.rmtree(_temporary_directory) except OSError: Do you happen to know if that will break anything? Setup install runs for me... Cheers, Matthew From ralf.gommers at gmail.com Thu Oct 4 02:35:54 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 4 Oct 2012 08:35:54 +0200 Subject: [Numpy-discussion] numpy distutils log error with easy_install In-Reply-To: References: Message-ID: On Thu, Oct 4, 2012 at 12:14 AM, Matthew Brett wrote: > Hi, > > On Wed, Oct 3, 2012 at 10:51 PM, Ralf Gommers > wrote: > > > > > > On Wed, Oct 3, 2012 at 11:29 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Wed, Oct 3, 2012 at 10:17 PM, Ralf Gommers > >> wrote: > >> > > >> > > >> > On Mon, Oct 1, 2012 at 10:47 PM, Matthew Brett < > matthew.brett at gmail.com> > >> > wrote: > >> >> > >> >> Hi, > >> >> > >> >> On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett < > matthew.brett at gmail.com> > >> >> wrote: > >> >> > Hi, > >> >> > > >> >> > One of our kind users pointed out an error when using easy_install > to > >> >> > install our package nipy. I've reproduced it now on a bare > package > >> >> > using numpy distutils and having a trivial extension: > >> >> > > >> >> > https://github.com/matthew-brett/apkg > >> >> > > >> >> > To reproduce: > >> >> > > >> >> > git clone git://github.com/mathew-brett/apkg.git > >> >> > > >> >> > easy_install apkg > >> >> > > >> >> > You should get something like this: > >> >> > > >> >> > Processing apkg > >> >> > Running setup.py -q bdist_egg --dist-dir > >> >> > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB > >> >> > Appending apkg configuration to > >> >> > Ignoring attempt to set 'name' (from '' to 'apkg') > >> >> > zip_safe flag not set; analyzing archive contents... > >> >> > Adding apkg 0.1 to easy-install.pth file > >> >> > > >> >> > Installed > >> >> > > >> >> > > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg > >> >> > Processing dependencies for apkg==0.1 > >> >> > Finished processing dependencies for apkg==0.1 > >> >> > > >> >> > > >> >> > > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: > >> >> > RuntimeWarning: Parent module 'numpy.distutils' not found while > >> >> > handling absolute import > >> >> > from numpy.distutils import log > >> >> > > >> >> > Note the last error. > >> >> > >> >> Sorry, correcting myself - it's (obviously) a Warning rather than an > >> >> error, but still distracting, and it would be good to avoid it if > >> >> possible... > >> > > >> > > >> > The combination of two or all of atexit.register, easy_install and > >> > virtualenv seems to be causing this. Unless someone feels like digging > >> > into > >> > that (I certainly don't), there are two easy solutions: > >> > 1. Silence the warning. > >> > >> Sorry - I am not sure what you mean. The problem here is the user who > >> assumes that something bad happened when running easy_install - which > >> is what happened in the case of nipy. Is there some way of silencing > >> this (specific) warning from within setup.py? > >> > >> > 2. Remove the offending import and the logging. This will only remove > >> > the > >> > line "removing: _configtest.c _configtest.o" from the build log (x20). > >> > >> Which import did you mean? I think I need all the imports I'm using > >> in the example minimal package. I'm not explicitly importing logging > >> for example. > > > > > > The import that's indicated in the warning, on line 252 of > > numpy/distutils/misc_util.py. Relevant code: > > > > _temporary_directory = None > > def clean_up_temporary_directory(): > > from numpy.distutils import log # CAUSES RUNTIME WARNING > > global _temporary_directory > > if not _temporary_directory: > > return > > log.debug('removing %s', _temporary_directory) > > try: > > shutil.rmtree(_temporary_directory) > > except OSError: > > pass > > _temporary_directory = None > > > > def make_temp_file(suffix='', prefix='', text=True): > > global _temporary_directory > > if not _temporary_directory: > > _temporary_directory = tempfile.mkdtemp() > > atexit.register(clean_up_temporary_directory) > > Sorry - I still don't understand. You mean I should (in my package - > say nipy or 'apkg') monkey-patch numpy distutils.misc_util ? > No, I was proposing to fix this in numpy.distutils.misc_util directly. Sorry for not being clear. > > Another option would be to move the import outside the callback function > thus: > Should work too I guess, but since I don't understand why the import is where it is now I'm not 100% sure. Removing the import on line 252 and the log.debug call would be the safest fix. Ralf > diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py > index 2e4ed27..e00d924 100644 > --- a/numpy/distutils/misc_util.py > +++ b/numpy/distutils/misc_util.py > @@ -18,6 +18,7 @@ except NameError: > from sets import Set as set > > from numpy.distutils.compat import get_exception > +from numpy.distutils.log import debug as log_debug > > __all__ = ['Configuration', 'get_numpy_include_dirs', > 'default_config_dict', > 'dict_append', 'appendpath', 'generate_config_py', > @@ -249,11 +250,10 @@ def gpaths(paths, local_path='', > include_non_existing=True): > > _temporary_directory = None > def clean_up_temporary_directory(): > - from numpy.distutils import log > global _temporary_directory > if not _temporary_directory: > return > - log.debug('removing %s', _temporary_directory) > + log_debug('removing %s', _temporary_directory) > try: > shutil.rmtree(_temporary_directory) > except OSError: > > > Do you happen to know if that will break anything? Setup install runs for > me... > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Oct 4 05:29:33 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 4 Oct 2012 10:29:33 +0100 Subject: [Numpy-discussion] numpy distutils log error with easy_install In-Reply-To: References: Message-ID: Hi, On Thu, Oct 4, 2012 at 7:35 AM, Ralf Gommers wrote: > > > On Thu, Oct 4, 2012 at 12:14 AM, Matthew Brett > wrote: >> >> Hi, >> >> On Wed, Oct 3, 2012 at 10:51 PM, Ralf Gommers >> wrote: >> > >> > >> > On Wed, Oct 3, 2012 at 11:29 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Wed, Oct 3, 2012 at 10:17 PM, Ralf Gommers >> >> wrote: >> >> > >> >> > >> >> > On Mon, Oct 1, 2012 at 10:47 PM, Matthew Brett >> >> > >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett >> >> >> >> >> >> wrote: >> >> >> > Hi, >> >> >> > >> >> >> > One of our kind users pointed out an error when using easy_install >> >> >> > to >> >> >> > install our package nipy. I've reproduced it now on a bare >> >> >> > package >> >> >> > using numpy distutils and having a trivial extension: >> >> >> > >> >> >> > https://github.com/matthew-brett/apkg >> >> >> > >> >> >> > To reproduce: >> >> >> > >> >> >> > git clone git://github.com/mathew-brett/apkg.git >> >> >> > >> >> >> > easy_install apkg >> >> >> > >> >> >> > You should get something like this: >> >> >> > >> >> >> > Processing apkg >> >> >> > Running setup.py -q bdist_egg --dist-dir >> >> >> > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB >> >> >> > Appending apkg configuration to >> >> >> > Ignoring attempt to set 'name' (from '' to 'apkg') >> >> >> > zip_safe flag not set; analyzing archive contents... >> >> >> > Adding apkg 0.1 to easy-install.pth file >> >> >> > >> >> >> > Installed >> >> >> > >> >> >> > >> >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg >> >> >> > Processing dependencies for apkg==0.1 >> >> >> > Finished processing dependencies for apkg==0.1 >> >> >> > >> >> >> > >> >> >> > >> >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: >> >> >> > RuntimeWarning: Parent module 'numpy.distutils' not found while >> >> >> > handling absolute import >> >> >> > from numpy.distutils import log >> >> >> > >> >> >> > Note the last error. >> >> >> >> >> >> Sorry, correcting myself - it's (obviously) a Warning rather than an >> >> >> error, but still distracting, and it would be good to avoid it if >> >> >> possible... >> >> > >> >> > >> >> > The combination of two or all of atexit.register, easy_install and >> >> > virtualenv seems to be causing this. Unless someone feels like >> >> > digging >> >> > into >> >> > that (I certainly don't), there are two easy solutions: >> >> > 1. Silence the warning. >> >> >> >> Sorry - I am not sure what you mean. The problem here is the user who >> >> assumes that something bad happened when running easy_install - which >> >> is what happened in the case of nipy. Is there some way of silencing >> >> this (specific) warning from within setup.py? >> >> >> >> > 2. Remove the offending import and the logging. This will only remove >> >> > the >> >> > line "removing: _configtest.c _configtest.o" from the build log >> >> > (x20). >> >> >> >> Which import did you mean? I think I need all the imports I'm using >> >> in the example minimal package. I'm not explicitly importing logging >> >> for example. >> > >> > >> > The import that's indicated in the warning, on line 252 of >> > numpy/distutils/misc_util.py. Relevant code: >> > >> > _temporary_directory = None >> > def clean_up_temporary_directory(): >> > from numpy.distutils import log # CAUSES RUNTIME WARNING >> > global _temporary_directory >> > if not _temporary_directory: >> > return >> > log.debug('removing %s', _temporary_directory) >> > try: >> > shutil.rmtree(_temporary_directory) >> > except OSError: >> > pass >> > _temporary_directory = None >> > >> > def make_temp_file(suffix='', prefix='', text=True): >> > global _temporary_directory >> > if not _temporary_directory: >> > _temporary_directory = tempfile.mkdtemp() >> > atexit.register(clean_up_temporary_directory) >> >> Sorry - I still don't understand. You mean I should (in my package - >> say nipy or 'apkg') monkey-patch numpy distutils.misc_util ? > > > No, I was proposing to fix this in numpy.distutils.misc_util directly. Sorry > for not being clear. > >> >> >> Another option would be to move the import outside the callback function >> thus: > > > Should work too I guess, but since I don't understand why the import is > where it is now I'm not 100% sure. Removing the import on line 252 and the > log.debug call would be the safest fix. The import is in the function because the following outside the function (at the top of misc_util): from numpy.distutils import log leads to the following on `python setup.py install`: Running from numpy source directory. Traceback (most recent call last): File "setup.py", line 214, in setup_package() File "setup.py", line 191, in setup_package from numpy.distutils.core import setup File "/home/mb312/dev_trees/numpy/numpy/distutils/__init__.py", line 7, in import ccompiler File "/home/mb312/dev_trees/numpy/numpy/distutils/ccompiler.py", line 14, in from numpy.distutils import log File "/home/mb312/dev_trees/numpy/numpy/distutils/log.py", line 9, in from misc_util import red_text, default_text, cyan_text, green_text, is_sequence, is_string File "/home/mb312/dev_trees/numpy/numpy/distutils/misc_util.py", line 21, in from numpy.distutils import log ImportError: cannot import name log I'm guessing this is a problem of circular imports, because in numpy.distutils.log we see that numpy distutils is importing misc_util. It so happens that: from numpy.distutils.log import debug as log_debug works in this situation, I don't understand why. I suppose also that it may rely on something complex and / or undocumented in the details of python's import mechanism, and so might be better avoided. So, is the 'remove log call in function' reasonable for a patch to trunk? For the upcoming release? Cheers, Matthew From opossumnano at gmail.com Thu Oct 4 09:26:13 2012 From: opossumnano at gmail.com (Tiziano Zito) Date: Thu, 4 Oct 2012 15:26:13 +0200 (CEST) Subject: [Numpy-discussion] =?utf-8?q?=5BANN=5D_MDP-3=2E3_released!?= Message-ID: <20121004132613.C46E612E00A9@comms.bccn-berlin.de> We are glad to announce release 3.3 of the Modular toolkit for Data Processing (MDP). This a bug-fix release, all current users are invited to upgrade. MDP is a Python library of widely used data processing algorithms that can be combined according to a pipeline analogy to build more complex data processing software. The base of available algorithms includes signal processing methods (Principal Component Analysis, Independent Component Analysis, Slow Feature Analysis), manifold learning methods ([Hessian] Locally Linear Embedding), several classifiers, probabilistic methods (Factor Analysis, RBM), data pre-processing methods, and many others. What's new in version 3.3? -------------------------- - support sklearn versions up to 0.12 - cleanly support reload - fail gracefully if pp server does not start - several bug-fixes and improvements Resources --------- Download: http://sourceforge.net/projects/mdp-toolkit/files Homepage: http://mdp-toolkit.sourceforge.net Mailing list: http://lists.sourceforge.net/mailman/listinfo/mdp-toolkit-users Acknowledgments --------------- We thank the contributors to this release: Philip DeBoer, Yaroslav Halchenko. The MDP developers, Pietro Berkes Zbigniew J?drzejewski-Szmek Rike-Benjamin Schuppner Niko Wilbert Tiziano Zito From andrew.collette at gmail.com Thu Oct 4 11:20:20 2012 From: andrew.collette at gmail.com (Andrew Collette) Date: Thu, 4 Oct 2012 09:20:20 -0600 Subject: [Numpy-discussion] ANN: HDF5 for Python (h5py) 2.1.0-final Message-ID: Announcing HDF5 for Python (h5py) 2.1.0 ======================================= We are proud to announce the availability of HDF5 for Python (h5py) 2.1.0! This release has been a long time coming. Thanks to everyone who contributed code and filed bug reports! What's new in h5py 2.1 ----------------------- * The HDF5 Dimension Scales API is now available, along with high-level integration with Dataset objects. Thanks to D. Dale for implementing this. * Unicode scalar strings can now be stored in attributes. * Dataset objects now expose a .size property giving the total number of elements. * Many performance improvements and bug fixes About the project ----------------------- HDF5 for Python (h5py) is a general-purpose Python interface to the Hierarchical Data Format library, version 5. HDF5 is a mature scientific software library originally developed at NCSA, designed for the fast, flexible storage of enormous amounts of data. >From a Python programmer's perspective, HDF5 provides a robust way to store data, organized by name in a tree-like fashion. You can create datasets (arrays on disk) hundreds of gigabytes in size, and perform random-access I/O on desired sections. Datasets are organized in a filesystem-like hierarchy using containers called "groups", and accessed using the traditional POSIX /path/to/resource syntax. Downloads, FAQ and bug tracker are available at Google Code: * Google code site: http://h5py.googlecode.com Documentation is available at Alfven.org: * http://h5py.alfven.org From jay.bourque at continuum.io Thu Oct 4 13:23:01 2012 From: jay.bourque at continuum.io (Jay Bourque) Date: Thu, 4 Oct 2012 12:23:01 -0500 Subject: [Numpy-discussion] ufuncs for structured arrays In-Reply-To: References: Message-ID: Hey Anthony, thanks for the comments. On Mon, Oct 1, 2012 at 10:32 PM, Anthony Scopatz wrote: > Hello Jay, > > Cool idea! I like to see work on structured arrays. Just a couple of > questions: > > - Since there are already have ufuncs for primitive dtypes (int, > float, etc), and you are just acting columnwise here, can't you write a > single function which interprets the dtypes, gathers the approriate scalar > ufuncs, and applies those down each field? Yes registering individual > functions using your mechanism will be faster, but it will also be more > work. > > I hadn't thought of this approach, but I suppose that would work okay too for certain dtypes. You'd lose some flexibility with writing a ufunc for a specific struct dtype though (for example maybe the ufunc shouldn't be applied to all the fields in a record). Maybe this approach would be a good default to use when a ufunc hasn't been registered for a struct dtype? > > - I didn't try this out myself, but will the normal numpy broadcasting > rules apply here? > > I need to do more testing for this, but yes, I believe normal broadcasting rules should work fine. > Be Well > Anthony > > On Mon, Oct 1, 2012 at 10:36 AM, Jay Bourque wrote: > >> All, >> >> I've submitted the following pull request for NumPy: >> >> https://github.com/numpy/numpy/pull/462 >> >> This change allows ufuncs to be registered for structured arrays by using >> a new API method PyUFunc_RegisterLoopForStructType. For example, a ufunc >> could be registered to take two arrays of type 'u8,u8,u8' and return an >> array of type 'u8,u8,u8'. I have a trivial example of this included in my >> pull request, along with further details of my changes. I suspect there >> might be a better way to do this, so any suggestions for improvements would >> be welcome. >> >> Thanks, >> -Jay >> continuum.io >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scopatz at gmail.com Thu Oct 4 16:53:05 2012 From: scopatz at gmail.com (Anthony Scopatz) Date: Thu, 4 Oct 2012 15:53:05 -0500 Subject: [Numpy-discussion] ufuncs for structured arrays In-Reply-To: References: Message-ID: On Thu, Oct 4, 2012 at 12:23 PM, Jay Bourque wrote: > Hey Anthony, thanks for the comments. > > On Mon, Oct 1, 2012 at 10:32 PM, Anthony Scopatz wrote: > >> Hello Jay, >> >> Cool idea! I like to see work on structured arrays. Just a couple of >> questions: >> >> - Since there are already have ufuncs for primitive dtypes (int, >> float, etc), and you are just acting columnwise here, can't you write a >> single function which interprets the dtypes, gathers the approriate scalar >> ufuncs, and applies those down each field? Yes registering individual >> functions using your mechanism will be faster, but it will also be more >> work. >> >> I hadn't thought of this approach, but I suppose that would work okay too > for certain dtypes. You'd lose some flexibility with writing a ufunc for a > specific struct dtype though (for example maybe the ufunc shouldn't be > applied to all the fields in a record). Maybe this approach would be a good > default to use when a ufunc hasn't been registered for a struct dtype? > A sensible default was use case that I had in mind. I think that it is cool to be register your own ufuncs and I like what you have done there. So what I suggested is more of an addition than a strict criticism of work to date. > > >> >> - I didn't try this out myself, but will the normal numpy >> broadcasting rules apply here? >> >> I need to do more testing for this, but yes, I believe normal > broadcasting rules should work fine. > That'd be great! Be Well Anthony > > >> Be Well >> Anthony >> >> On Mon, Oct 1, 2012 at 10:36 AM, Jay Bourque wrote: >> >>> All, >>> >>> I've submitted the following pull request for NumPy: >>> >>> https://github.com/numpy/numpy/pull/462 >>> >>> This change allows ufuncs to be registered for structured arrays by >>> using a new API method PyUFunc_RegisterLoopForStructType. For example, a >>> ufunc could be registered to take two arrays of type 'u8,u8,u8' and return >>> an array of type 'u8,u8,u8'. I have a trivial example of this included in >>> my pull request, along with further details of my changes. I suspect there >>> might be a better way to do this, so any suggestions for improvements would >>> be welcome. >>> >>> Thanks, >>> -Jay >>> continuum.io >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.raybaut at gmail.com Fri Oct 5 04:44:30 2012 From: pierre.raybaut at gmail.com (Pierre Raybaut) Date: Fri, 5 Oct 2012 10:44:30 +0200 Subject: [Numpy-discussion] ANN: WinPython v2.7.3.1 Message-ID: Hi all, WinPython v2.7.3.1 has been released and is available for 32-bit and 64-bit Windows platforms: http://code.google.com/p/winpython/ WinPython is a free open-source portable distribution of Python for Windows, designed for scientists. It is a full-featured (see http://code.google.com/p/winpython/wiki/PackageIndex) Python-based scientific environment: * Designed for scientists (thanks to the integrated libraries NumPy, SciPy, Matplotlib, guiqwt, etc.: * Regular *scientific users*: interactive data processing and visualization using Python with Spyder * *Advanced scientific users and software developers*: Python applications development with Spyder, version control with Mercurial and other development tools (like gettext) * *Portable*: preconfigured, it should run out of the box on any machine under Windows (without any installation requirements) and the folder containing WinPython can be moved to any location (local, network or removable drive) * *Flexible*: one can install (or should I write "use" as it's portable) as many WinPython versions as necessary (like isolated and self-consistent environments), even if those versions are running different versions of Python (2.7, 3.x in the near future) or different architectures (32bit or 64bit) on the same machine * *Customizable*: using the integrated package manager (wppm, as WinPython Package Manager), it's possible to install, uninstall or upgrade Python packages (see http://code.google.com/p/winpython/wiki/WPPM for more details on supported package formats). *WinPython is not an attempt to replace Python(x,y)*, this is just something different (see http://code.google.com/p/winpython/wiki/Roadmap): more flexible, easier to maintain, movable and less invasive for the OS, but certainly less user-friendly, with less packages/contents and without any integration to Windows explorer [*]. [*] Actually there is an optional integration into Windows explorer, providing the same features as the official Python installer regarding file associations and context menu entry (this option may be activated through the WinPython Control Panel). Enjoy! From dg.gmane at thesamovar.net Fri Oct 5 11:17:46 2012 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Fri, 05 Oct 2012 17:17:46 +0200 Subject: [Numpy-discussion] set_printoptions precision and single floats Message-ID: Hi, numpy.set_printoptions(precision=...) doesn't affect single floats, even if they are numpy floats rather than Python floats. Is this a bug or is there some reason for this behaviour? I ask because I have a class that derives from numpy.float64 and adds some extra information, and I'd like to be able to control the precision. I could fix it to use the precision set by numpy.set_printoptions, but then it would be inconsistent with how numpy itself handles precision. Thoughts? Dan From sextonhadoop at gmail.com Fri Oct 5 16:45:10 2012 From: sextonhadoop at gmail.com (Ed Sexton) Date: Fri, 5 Oct 2012 13:45:10 -0700 Subject: [Numpy-discussion] numpy-1.6.2: python setup.py build bdist_rpm fails In-Reply-To: References: Message-ID: Hello Numpy Group- I have a working installation of Numpy-1.6.2 on a RHEL 6.3 host, now I have to create an RPM to push out to 600+ hosts. 1. I cannot get the RPM build command to work, error listed below. 2. Another issue, how do you pass --fcompiler=gnu95 to bdlist_rpm? I have everything compiled under fcompiler. # python setup.py build_bdist Tail of the error: RPM build errors: File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/array_from_pyobj/wrapmodule.c File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/foo_free.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/foo_mod.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/foo_use.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/precision.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/kind/foo.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/mixed/foo.f File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/mixed/foo_fixed.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/mixed/foo_free.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/size/foo.f90 Installed (but unpackaged) file(s) found: /usr/lib64/python2.6/site-packages/numpy/tests/test_matlib.pyc /usr/lib64/python2.6/site-packages/numpy/tests/test_matlib.pyo error: command 'rpmbuild' failed with exit status 1 Full Error on Pastebin: http://paste.ubuntu.com/1262671/ Thanks for your ideas/suggestions. Sincerely, Ed -- Sincerely, Ed Sexton e: sextonhadoop at gmail.com gv: (408) 475-2358 -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Oct 6 07:22:31 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 6 Oct 2012 12:22:31 +0100 Subject: [Numpy-discussion] numpy distutils log error with easy_install In-Reply-To: References: Message-ID: Hi, On Thu, Oct 4, 2012 at 10:29 AM, Matthew Brett wrote: > Hi, > > On Thu, Oct 4, 2012 at 7:35 AM, Ralf Gommers wrote: >> >> >> On Thu, Oct 4, 2012 at 12:14 AM, Matthew Brett >> wrote: >>> >>> Hi, >>> >>> On Wed, Oct 3, 2012 at 10:51 PM, Ralf Gommers >>> wrote: >>> > >>> > >>> > On Wed, Oct 3, 2012 at 11:29 PM, Matthew Brett >>> > wrote: >>> >> >>> >> Hi, >>> >> >>> >> On Wed, Oct 3, 2012 at 10:17 PM, Ralf Gommers >>> >> wrote: >>> >> > >>> >> > >>> >> > On Mon, Oct 1, 2012 at 10:47 PM, Matthew Brett >>> >> > >>> >> > wrote: >>> >> >> >>> >> >> Hi, >>> >> >> >>> >> >> On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett >>> >> >> >>> >> >> wrote: >>> >> >> > Hi, >>> >> >> > >>> >> >> > One of our kind users pointed out an error when using easy_install >>> >> >> > to >>> >> >> > install our package nipy. I've reproduced it now on a bare >>> >> >> > package >>> >> >> > using numpy distutils and having a trivial extension: >>> >> >> > >>> >> >> > https://github.com/matthew-brett/apkg >>> >> >> > >>> >> >> > To reproduce: >>> >> >> > >>> >> >> > git clone git://github.com/mathew-brett/apkg.git >>> >> >> > >>> >> >> > easy_install apkg >>> >> >> > >>> >> >> > You should get something like this: >>> >> >> > >>> >> >> > Processing apkg >>> >> >> > Running setup.py -q bdist_egg --dist-dir >>> >> >> > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB >>> >> >> > Appending apkg configuration to >>> >> >> > Ignoring attempt to set 'name' (from '' to 'apkg') >>> >> >> > zip_safe flag not set; analyzing archive contents... >>> >> >> > Adding apkg 0.1 to easy-install.pth file >>> >> >> > >>> >> >> > Installed >>> >> >> > >>> >> >> > >>> >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg >>> >> >> > Processing dependencies for apkg==0.1 >>> >> >> > Finished processing dependencies for apkg==0.1 >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: >>> >> >> > RuntimeWarning: Parent module 'numpy.distutils' not found while >>> >> >> > handling absolute import >>> >> >> > from numpy.distutils import log >>> >> >> > >>> >> >> > Note the last error. >>> >> >> >>> >> >> Sorry, correcting myself - it's (obviously) a Warning rather than an >>> >> >> error, but still distracting, and it would be good to avoid it if >>> >> >> possible... >>> >> > >>> >> > >>> >> > The combination of two or all of atexit.register, easy_install and >>> >> > virtualenv seems to be causing this. Unless someone feels like >>> >> > digging >>> >> > into >>> >> > that (I certainly don't), there are two easy solutions: >>> >> > 1. Silence the warning. >>> >> >>> >> Sorry - I am not sure what you mean. The problem here is the user who >>> >> assumes that something bad happened when running easy_install - which >>> >> is what happened in the case of nipy. Is there some way of silencing >>> >> this (specific) warning from within setup.py? >>> >> >>> >> > 2. Remove the offending import and the logging. This will only remove >>> >> > the >>> >> > line "removing: _configtest.c _configtest.o" from the build log >>> >> > (x20). >>> >> >>> >> Which import did you mean? I think I need all the imports I'm using >>> >> in the example minimal package. I'm not explicitly importing logging >>> >> for example. >>> > >>> > >>> > The import that's indicated in the warning, on line 252 of >>> > numpy/distutils/misc_util.py. Relevant code: >>> > >>> > _temporary_directory = None >>> > def clean_up_temporary_directory(): >>> > from numpy.distutils import log # CAUSES RUNTIME WARNING >>> > global _temporary_directory >>> > if not _temporary_directory: >>> > return >>> > log.debug('removing %s', _temporary_directory) >>> > try: >>> > shutil.rmtree(_temporary_directory) >>> > except OSError: >>> > pass >>> > _temporary_directory = None >>> > >>> > def make_temp_file(suffix='', prefix='', text=True): >>> > global _temporary_directory >>> > if not _temporary_directory: >>> > _temporary_directory = tempfile.mkdtemp() >>> > atexit.register(clean_up_temporary_directory) >>> >>> Sorry - I still don't understand. You mean I should (in my package - >>> say nipy or 'apkg') monkey-patch numpy distutils.misc_util ? >> >> >> No, I was proposing to fix this in numpy.distutils.misc_util directly. Sorry >> for not being clear. >> >>> >>> >>> Another option would be to move the import outside the callback function >>> thus: >> >> >> Should work too I guess, but since I don't understand why the import is >> where it is now I'm not 100% sure. Removing the import on line 252 and the >> log.debug call would be the safest fix. > > The import is in the function because the following outside the > function (at the top of misc_util): > > from numpy.distutils import log > > leads to the following on `python setup.py install`: > > Running from numpy source directory. > Traceback (most recent call last): > File "setup.py", line 214, in > setup_package() > File "setup.py", line 191, in setup_package > from numpy.distutils.core import setup > File "/home/mb312/dev_trees/numpy/numpy/distutils/__init__.py", line > 7, in > import ccompiler > File "/home/mb312/dev_trees/numpy/numpy/distutils/ccompiler.py", > line 14, in > from numpy.distutils import log > File "/home/mb312/dev_trees/numpy/numpy/distutils/log.py", line 9, in > from misc_util import red_text, default_text, cyan_text, > green_text, is_sequence, is_string > File "/home/mb312/dev_trees/numpy/numpy/distutils/misc_util.py", > line 21, in > from numpy.distutils import log > ImportError: cannot import name log > > I'm guessing this is a problem of circular imports, because in > numpy.distutils.log we see that numpy distutils is importing > misc_util. > > It so happens that: > > from numpy.distutils.log import debug as log_debug > > works in this situation, I don't understand why. I suppose also that > it may rely on something complex and / or undocumented in the details > of python's import mechanism, and so might be better avoided. > > So, is the 'remove log call in function' reasonable for a patch to > trunk? For the upcoming release? Ralf's suggestion as pull request: https://github.com/numpy/numpy/pull/480 Best, Matthew From ralf.gommers at gmail.com Sat Oct 6 07:56:37 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 6 Oct 2012 13:56:37 +0200 Subject: [Numpy-discussion] numpy-1.6.2: python setup.py build bdist_rpm fails In-Reply-To: References: Message-ID: On Fri, Oct 5, 2012 at 10:45 PM, Ed Sexton wrote: > > Hello Numpy Group- > > I have a working installation of Numpy-1.6.2 on a RHEL 6.3 host, now I > have to create an RPM to push out to 600+ hosts. > > 1. I cannot get the RPM build command to work, error listed below. > > 2. Another issue, how do you pass --fcompiler=gnu95 to bdlist_rpm? I > have everything compiled under fcompiler. > > # python setup.py build_bdist > > Tail of the error: > > > > RPM build errors: > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/array_from_pyobj/wrapmodule.c > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/foo_free.f90 > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/foo_mod.f90 > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/foo_use.f90 > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/precision.f90 > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/kind/foo.f90 > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/mixed/foo.f > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/mixed/foo_fixed.f90 > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/mixed/foo_free.f90 > File listed twice: > /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/size/foo.f90 > Installed (but unpackaged) file(s) found: > > > /usr/lib64/python2.6/site-packages/numpy/tests/test_matlib.pyc > /usr/lib64/python2.6/site-packages/numpy/tests/test_matlib.pyo > error: command 'rpmbuild' failed with exit status 1 > > Full Error on Pastebin: > http://paste.ubuntu.com/1262671/ > > > Thanks for your ideas/suggestions. > No idea why that happens, but I think you could work around it by fixing up the list of installed files. 1. `python setup.py install bdist_rpm --spec-only` gives you the spec file. Looks like: %install python setup.py install -O1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES %clean rm -rf $RPM_BUILD_ROOT %files -f INSTALLED_FILES %defattr(-,root,root) 2. Write a script to fix up INSTALLED_FILES by removing duplicates and adding the missing files. Insert a call to that script after the clean step in the spec file. 3. Run: python setup.py install bdist_rpm --source-only python setup.py install bdist_rpm --binary-only Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Oct 6 10:14:14 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 6 Oct 2012 16:14:14 +0200 Subject: [Numpy-discussion] specifying numpy as dependency in your project, install_requires In-Reply-To: References: <9644D8AB-2BA7-4537-989B-F1B80DF73155@continuum.io> Message-ID: On Sun, Sep 23, 2012 at 10:20 PM, Nathaniel Smith wrote: > On Sat, Sep 22, 2012 at 1:18 PM, Ralf Gommers > wrote: > > On Fri, Sep 21, 2012 at 11:39 PM, Nathaniel Smith wrote: > >> So the question is, how do we get a .egg-info? For the specific case > >> Ralf ran into, I'm pretty sure the solution is just that if you're > >> clever enough to do an in-place build and add it to your PYTHONPATH, > >> you should be clever enough to also run 'python setupegg.py egg_info' > >> which will create a .egg-info to go with your in-place build and > >> everything will be fine. > > > > That command first starts rebuilding numpy. > > No, it just seems to run the config and source-generation bits, not > build anything. It also leaves the .egg-info in the source directory, > which is what you want. > You're right, sorry. I saw output like "building extension "numpy.core._dotblas" sources" scrolling by and hit Ctrl-C. > > > >> P.S.: yeah the thing where pip decides to upgrade the world is REALLY > >> OBNOXIOUS. It also appears to be on the list to be fixed in the next > >> release or the next release+1, so I guess there's hope?: > >> https://github.com/pypa/pip/pull/571 > > > > Good to know. Let's hope that does make it in. Given it's development > model, > > I'm less optimistic that easy_install will receive the same fix though > .... > > Yeah, easy_install is abandoned and bit-rotting, which is why people > usually recommend pip :-). But in this case, I thought that > easy_install already doesn't upgrade the world when it runs? Is there > something to fix here? > It does, as Josef said above. It has the same -U and --no-deps flags. > > Until both pip and easy_install are fixed, this alone should be enough > for > > the advice to be "don't use install_requires". It's not like my > alternative > > suggestion takes away any information or valuable functionality. > > pandas, for example, requires several other packages, and I found it > quite convenient the other day when I wanted to try out a new version > and pip automatically took care of setting all that up for me. It even > correctly upgraded numpy, since the virtualenv I was using for testing > had inherited my system-installed 1.5.2, but this was the first > version of pandas that needed 1.6. > So this saved you from reading "pandas requires numpy >= 1.6.1" and typing "pip install -U numpy". Not my definition of valuable functionality, and certainly not worth the risk of upgrading numpy silently for users. Python packaging tools make me feel grumpy and traumatized too but I > don't see how the solution is to just give up on computer-readable > dependency-tracking altogether. > Proper dependency tracking would be preferable, but none at all is better than the current situation imho. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Oct 6 12:17:00 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 6 Oct 2012 18:17:00 +0200 Subject: [Numpy-discussion] set_printoptions precision and single floats In-Reply-To: References: Message-ID: On Fri, Oct 5, 2012 at 5:17 PM, Dan Goodman wrote: > Hi, > > numpy.set_printoptions(precision=...) doesn't affect single floats, even > if they are numpy floats rather than Python floats. Is this a bug or is > there some reason for this behaviour? I ask because I have a class that > derives from numpy.float64 and adds some extra information, and I'd like > to be able to control the precision. I could fix it to use the precision > set by numpy.set_printoptions, but then it would be inconsistent with > how numpy itself handles precision. Thoughts? > Do you mean scalars or arrays? For me set_printoptions only affects arrays and not scalars. Both float32 and float64 arrays work as advertised: In [28]: np.set_printoptions(precision=4) In [29]: np.array([np.float32(1.234567891011011101111012345679)]) Out[29]: array([ 1.2346], dtype=float32) In [30]: np.array([np.float64(1.234567891011011101111012345679)]) Out[30]: array([ 1.2346]) In [31]: np.set_printoptions(precision=8) In [32]: np.array([np.float32(1.234567891011011101111012345679)]) Out[32]: array([ 1.23456788], dtype=float32) In [33]: np.array([np.float64(1.234567891011011101111012345679)]) Out[33]: array([ 1.23456789]) But for scalars it doesn't work: In [34]: np.float32(1.234567891011011101111012345679) Out[34]: 1.2345679 In [35]: np.float64(1.234567891011011101111012345679) Out[35]: 1.2345678910110112 In [36]: np.set_printoptions(precision=4) In [37]: np.float32(1.234567891011011101111012345679) Out[37]: 1.2345679 In [38]: np.float64(1.234567891011011101111012345679) Out[38]: 1.2345678910110112 Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Sat Oct 6 12:26:22 2012 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sat, 6 Oct 2012 12:26:22 -0400 Subject: [Numpy-discussion] set_printoptions precision and single floats In-Reply-To: References: Message-ID: On Sat, Oct 6, 2012 at 12:17 PM, Ralf Gommers wrote: > > > On Fri, Oct 5, 2012 at 5:17 PM, Dan Goodman wrote: > >> Hi, >> >> numpy.set_printoptions(precision=...) doesn't affect single floats, even >> if they are numpy floats rather than Python floats. Is this a bug or is >> there some reason for this behaviour? I ask because I have a class that >> derives from numpy.float64 and adds some extra information, and I'd like >> to be able to control the precision. I could fix it to use the precision >> set by numpy.set_printoptions, but then it would be inconsistent with >> how numpy itself handles precision. Thoughts? >> > > Do you mean scalars or arrays? For me set_printoptions only affects arrays > and not scalars. Both float32 and float64 arrays work as advertised: > > In [28]: np.set_printoptions(precision=4) > > In [29]: np.array([np.float32(1.234567891011011101111012345679)]) > Out[29]: array([ 1.2346], dtype=float32) > > In [30]: np.array([np.float64(1.234567891011011101111012345679)]) > Out[30]: array([ 1.2346]) > > In [31]: np.set_printoptions(precision=8) > > In [32]: np.array([np.float32(1.234567891011011101111012345679)]) > Out[32]: array([ 1.23456788], dtype=float32) > > In [33]: np.array([np.float64(1.234567891011011101111012345679)]) > Out[33]: array([ 1.23456789]) > > > But for scalars it doesn't work: > > In [34]: np.float32(1.234567891011011101111012345679) > Out[34]: 1.2345679 > > In [35]: np.float64(1.234567891011011101111012345679) > Out[35]: 1.2345678910110112 > > In [36]: np.set_printoptions(precision=4) > > In [37]: np.float32(1.234567891011011101111012345679) > Out[37]: 1.2345679 > > In [38]: np.float64(1.234567891011011101111012345679) > Out[38]: 1.2345678910110112 > > > Ralf > It also does not affect zero-dimensional (i.e. scalar) arrays (e.g. array(1.2345)): In [1]: x = array(1./3) In [2]: x Out[2]: array(0.3333333333333333) In [3]: set_printoptions(precision=3) In [4]: x Out[4]: array(0.3333333333333333) In [5]: type(x) Out[5]: numpy.ndarray `y` is a 1-d array, so this works as expected: In [6]: y = array([1./3]) In [7]: y Out[7]: array([ 0.333]) Warren > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wardefar at iro.umontreal.ca Sun Oct 7 00:35:02 2012 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Sun, 7 Oct 2012 00:35:02 -0400 Subject: [Numpy-discussion] dtype '|S0' not understood In-Reply-To: References: <113F0316-56E1-4403-B967-EC06847B8251@cs.toronto.edu> Message-ID: On Wed, Oct 3, 2012 at 1:58 PM, Will Lee wrote: > This seems to be a old problem but I've recently hit with this in a very > random way (I'm using numpy 1.6.1). There seems to be a ticket (1239) but > it seems the issue is unscheduled. Can somebody tell me if this is fixed? > > In particular, it makes for a very unstable behavior when you try to > reference something from a string array and pickle it across the wire. For > example: > > In [1]: import numpy > In [2]: a = numpy.array(['a', '', 'b']) > In [3]: import cPickle > In [4]: s = cPickle.dumps(a[1]) > In [5]: cPickle.loads(s) > --------------------------------------------------------------------------- > TypeError Traceback (most recent call last) > /auto/cnvtvws/wlee/fstrat/src/ in () > ----> 1 cPickle.loads(s) > TypeError: ('data type not understood', , ('S0', 0, 1)) > > Note that if you reference a[0] and a[2], it would work, so you're in the > case where sometimes it'd work but sometimes it won't. Checking for this > case in the code and work around it would really be a pain. Hi Will, Yes, this has been waiting on the bug tracker for a long while. I'll resubmit my patch as a pull request to see if we can't get this fixed up soon... David From jianbao.tao at gmail.com Sun Oct 7 02:41:12 2012 From: jianbao.tao at gmail.com (Jianbao Tao) Date: Sat, 6 Oct 2012 23:41:12 -0700 Subject: [Numpy-discussion] How to copy data from a C array to a numpy array efficiently? Message-ID: Hi, I am developing a Python wrapper of the NASA CDF C library in Cython. I have got a working version of it now, but it is slower than the counterpart in IDL. For loading the same file, mine takes about 400 ms, whereas the IDL version takes about 290 ms. The main overhead in my code is caused by a for-loop of element-by-element copying. Here is the relevant code in cython: #-------------------------------- code ----------------------------------------------------- #-- double realData = numpy.zeros(lenData, np_dtype) dblEntry = malloc(lenData * sizeof(double)) status = CDFlib( SELECT_, zVAR_RECCOUNT_, numRecs, NULL_) status = CDFlib( GET_, zVAR_HYPERDATA_, dblEntry, NULL_) for ii in range(lenData): realData[ii] = dblEntry[ii] realData.shape = np_shape free(dblEntry) #--------------------------------- end of code ------------------------------------------- The time-consuming for-loop is highlighted in red. If I change range(lenData) to range(lenData/2), the cpu time will be down from 400 ms to 230 ms for the case I mentioned above. Because the element-by-element copying for-loop seems pretty naive to me, I am wondering if there is a better way to copy data from the C array, dblEntry, to the numpy array, realData. I tried the numpy C API PyArray_NewFromDescr with flag NPY_ENSURECOPY, but didn't get any luck. On the one hand, the flag above didn't seem to work as I expected, because I got memory deallocation failure error messages when I quitted ipython, where I tested my code, which I don't get if I use the naive for-loop. On the other hand, I can't figure out how to use PyArray_NewFromDescr correctly because the loaded data I got were not correct. Anyway, here is how I used PyArray_NewFromDescr: #----------------------------------------- code ------------------------------------------ cdef np.npy_intp dims[1] dims[0] = lenData realData = PyArray_NewFromDescr(numpy.ndarray, numpy.dtype(np_dtype), 1, dims, NULL, dblEntry, NPY_CARRAY|NPY_ENSURECOPY, None) free(dblEntry) #-------------------------------------- end of code -------------------------------------- BTW, it can be compiled successfully by cython, in case you are wondering if the code had all the necessary pieces, Thank you very much for reading. :-) Cheers, Jianbao -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Sun Oct 7 02:48:10 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Sun, 07 Oct 2012 08:48:10 +0200 Subject: [Numpy-discussion] How to copy data from a C array to a numpy array efficiently? In-Reply-To: References: Message-ID: <507125AA.8040108@astro.uio.no> On 10/07/2012 08:41 AM, Jianbao Tao wrote: > Hi, > > I am developing a Python wrapper of the NASA CDF C library in Cython. I > have got a working version of it now, but it is slower than the > counterpart in IDL. For loading the same file, mine takes about 400 ms, > whereas the IDL version takes about 290 ms. > > The main overhead in my code is caused by a for-loop of > element-by-element copying. Here is the relevant code in cython: > #-------------------------------- code > ----------------------------------------------------- > #-- double > realData = numpy.zeros(lenData, np_dtype) > > dblEntry = malloc(lenData * sizeof(double)) > status = CDFlib( > SELECT_, zVAR_RECCOUNT_, numRecs, > NULL_) > status = CDFlib( > GET_, zVAR_HYPERDATA_, dblEntry, > NULL_) > for ii in range(lenData): > realData[ii] = dblEntry[ii] > realData.shape = np_shape > free(dblEntry) You don't say what np_dtype is here (or the Cython variable declaration for it). Assuming it is np.double and "cdef np.ndarray[double] realData", what you should do is simple pass the buffer of realData to the CDFlib function: status = CDFlib(GET_, ..., &realData[0], NULL) Then there's no need for copying. This is really what you should do anyway, then if the dtype is different leave it to the "astype" function (but then comparisons with IDL should take into account the dtype conversion). Dag Sverre From d.s.seljebotn at astro.uio.no Sun Oct 7 02:49:07 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Sun, 07 Oct 2012 08:49:07 +0200 Subject: [Numpy-discussion] How to copy data from a C array to a numpy array efficiently? In-Reply-To: <507125AA.8040108@astro.uio.no> References: <507125AA.8040108@astro.uio.no> Message-ID: <507125E3.6010505@astro.uio.no> On 10/07/2012 08:48 AM, Dag Sverre Seljebotn wrote: > On 10/07/2012 08:41 AM, Jianbao Tao wrote: >> Hi, >> >> I am developing a Python wrapper of the NASA CDF C library in Cython. I >> have got a working version of it now, but it is slower than the >> counterpart in IDL. For loading the same file, mine takes about 400 ms, >> whereas the IDL version takes about 290 ms. >> >> The main overhead in my code is caused by a for-loop of >> element-by-element copying. Here is the relevant code in cython: >> #-------------------------------- code >> ----------------------------------------------------- >> #-- double >> realData = numpy.zeros(lenData, np_dtype) >> >> dblEntry = malloc(lenData * sizeof(double)) >> status = CDFlib( >> SELECT_, zVAR_RECCOUNT_, numRecs, >> NULL_) >> status = CDFlib( >> GET_, zVAR_HYPERDATA_, dblEntry, >> NULL_) >> for ii in range(lenData): >> realData[ii] = dblEntry[ii] >> realData.shape = np_shape >> free(dblEntry) > > You don't say what np_dtype is here (or the Cython variable declaration > for it). > > Assuming it is np.double and "cdef np.ndarray[double] realData", what > you should do is simple pass the buffer of realData to the CDFlib function: > > status = CDFlib(GET_, ..., &realData[0], NULL) > > Then there's no need for copying. > > This is really what you should do anyway, then if the dtype is different > leave it to the "astype" function (but then comparisons with IDL should > take into account the dtype conversion). To really answer your question (though in this case you should use a different approach), what you should use to copy data efficiently is the C memcpy function. Dag Sverre From jniehof at lanl.gov Sun Oct 7 09:57:35 2012 From: jniehof at lanl.gov (Jonathan T. Niehof) Date: Sun, 07 Oct 2012 07:57:35 -0600 Subject: [Numpy-discussion] How to copy data from a C array to a numpy array efficiently? In-Reply-To: References: Message-ID: <50718A4F.8070703@lanl.gov> On 10/07/2012 12:41 AM, Jianbao Tao wrote: > Hi, > > I am developing a Python wrapper of the NASA CDF C library in Cython. I > have got a working version of it now, but it is slower than the > counterpart in IDL. For loading the same file, mine takes about 400 ms, > whereas the IDL version takes about 290 ms. Does spacepy.pycdf not work well for you? (Not Cython, of course...) As Dag pointed out, passing a pointer to the numpy array data in to the CDF library works very nicely, although you may need some contortions to handle column-major zVars. (Making the numpy array Fortran order doesn't quite do it since the record dimension doesn't move around.) -- Jonathan Niehof ISR-3 Space Data Systems Los Alamos National Laboratory MS-D466 Los Alamos, NM 87545 Phone: 505-667-9595 email: jniehof at lanl.gov Correspondence / Technical data or Software Publicly Available From thouis at gmail.com Sun Oct 7 10:15:40 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Sun, 7 Oct 2012 10:15:40 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: I plan to import all the Trac issues to github by the end of this week. I want to get an up-to-date snapshot of the Trac DB, and run another test import with it (just to make sure there's nothing in recent bugs that isn't handled). Previous test imports here: https://github.com/thouis/numpy-trac-migration/issues Ray From jianbao.tao at gmail.com Sun Oct 7 15:36:42 2012 From: jianbao.tao at gmail.com (Jianbao Tao) Date: Sun, 7 Oct 2012 12:36:42 -0700 Subject: [Numpy-discussion] How to copy data from a C array to a numpy array efficiently? Message-ID: Dag, Thanks for your tip. I didn't know that I could use a numpy array directly. I tried that out, and it works like a charm. :-) Jonathan, Glad that you mentioned spacepy. I actually tried spacepy before I decided to write my own wrapper. However, on the one hand, I can't install spacepy on my mac (10.8 OS X). On the other hand, spacepy has too much stuff that I don't need, and I don't think I can rip spacepy.pycdf off the whole package. So, I eventually decided to write my own wrapper. However, as a space physics scientist, I'd like to see spacepy become better and more generic. For the moment, it seems to me that spacepy is more tailored to suit LANL's needs, especially about radiation belt research, than being a generic tool for space physics. Cheers, Jianbao -------------- next part -------------- An HTML attachment was scrubbed... URL: From dg.gmane at thesamovar.net Sun Oct 7 18:36:54 2012 From: dg.gmane at thesamovar.net (Dan Goodman) Date: Mon, 08 Oct 2012 00:36:54 +0200 Subject: [Numpy-discussion] set_printoptions precision and single floats In-Reply-To: References: Message-ID: <50720406.4010907@thesamovar.net> On 06/10/2012 18:17, Ralf Gommers wrote: > Do you mean scalars or arrays? For me set_printoptions only affects > arrays and not scalars. Both float32 and float64 arrays work as advertised: Yep, I mean scalars (although as Warren noted, it also affects e.g. array(1.23456789)). Dan From josef.pktd at gmail.com Sun Oct 7 22:15:52 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 7 Oct 2012 22:15:52 -0400 Subject: [Numpy-discussion] set_printoptions precision and single floats In-Reply-To: <50720406.4010907@thesamovar.net> References: <50720406.4010907@thesamovar.net> Message-ID: On Sun, Oct 7, 2012 at 6:36 PM, Dan Goodman wrote: > On 06/10/2012 18:17, Ralf Gommers wrote: >> Do you mean scalars or arrays? For me set_printoptions only affects >> arrays and not scalars. Both float32 and float64 arrays work as advertised: > > Yep, I mean scalars (although as Warren noted, it also affects e.g. > array(1.23456789)). My impression from previous discussions on the mailing list is that this is by design: printing of scalars and scalar arrays is handed off to python which doesn't have a set_printoptions. printing of arrays is under the control of numpy I don't see a way to influence the __str__ or __repr__ of a python float. Josef > > Dan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From thomas.p.krauss at gmail.com Tue Oct 9 10:30:14 2012 From: thomas.p.krauss at gmail.com (Tom Krauss) Date: Tue, 9 Oct 2012 09:30:14 -0500 Subject: [Numpy-discussion] swig interface file (numpy.i) warning Message-ID: Hi, I've been happy to use numpy.i for generating SWIG interfaces to C++. For a while, I've noticed this warning while compiling: /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" and today tried to get rid of the warning. So, in numpy.i, I followed the warning's advice. I added the # def here: %{ #ifndef SWIG_FILE_WITH_INIT # define NO_IMPORT_ARRAY #endif #include "stdio.h" *#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION* #include %} SWIG was happy, but when compiling the C++ wrapper, there were many warnings followed by many errors. The warnings were for redefinition of NPY_MIN_BYTE and similar. The errors were for all kinds of stuff, excerpt here: native_wrap.cpp:3632: error: ?PyArray_NOTYPE? was not declared in this scope native_wrap.cpp:3633: error: cannot convert ?PyObject*? to ?const PyArrayObject*? for argument ?1? to ?int PyArray_TYPE(const PyArrayObject*)? native_wrap.cpp: At global scope: native_wrap.cpp:3877: error: ?intp? has not been declared native_wrap.cpp: In function ?int require_fortran(PyArrayObject*)?: native_wrap.cpp:3929: error: ?struct tagPyArrayObject? has no member named ?nd? native_wrap.cpp:3933: error: ?struct tagPyArrayObject? has no member named ?flags? native_wrap.cpp:3933: error: ?FARRAY? was not declared in this scope native_wrap.cpp:20411: error: ?struct tagPyArrayObject? has no member named ?data? It looks like there is a new C API for numpy, and the version of numpy.i that I have doesn't use it. Is there a new version of numpy.i available (or in development) that works with the new API? Short term it will just get rid of a warning but I am interested in a good long term solution in case I need to upgrade numpy. Thanks, Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Oct 9 10:50:56 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 9 Oct 2012 08:50:56 -0600 Subject: [Numpy-discussion] swig interface file (numpy.i) warning In-Reply-To: References: Message-ID: Hi Tom, On Tue, Oct 9, 2012 at 8:30 AM, Tom Krauss wrote: > Hi, > > I've been happy to use numpy.i for generating SWIG interfaces to C++. > > For a while, I've noticed this warning while compiling: > /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2: > warning: #warning "Using deprecated NumPy API, disable it by #defining > NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" > > and today tried to get rid of the warning. > > So, in numpy.i, I followed the warning's advice. I added the # def here: > > %{ > #ifndef SWIG_FILE_WITH_INIT > # define NO_IMPORT_ARRAY > #endif > #include "stdio.h" > *#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION* > #include > %} > > SWIG was happy, but when compiling the C++ wrapper, there were many > warnings followed by many errors. The warnings were for redefinition > of NPY_MIN_BYTE and similar. The errors were for all kinds of stuff, > excerpt here: > native_wrap.cpp:3632: error: ?PyArray_NOTYPE? was not declared in this > scope > native_wrap.cpp:3633: error: cannot convert ?PyObject*? to ?const > PyArrayObject*? for argument ?1? to ?int PyArray_TYPE(const PyArrayObject*)? > native_wrap.cpp: At global scope: > native_wrap.cpp:3877: error: ?intp? has not been declared > native_wrap.cpp: In function ?int require_fortran(PyArrayObject*)?: > native_wrap.cpp:3929: error: ?struct tagPyArrayObject? has no member named > ?nd? > native_wrap.cpp:3933: error: ?struct tagPyArrayObject? has no member named > ?flags? > native_wrap.cpp:3933: error: ?FARRAY? was not declared in this scope > native_wrap.cpp:20411: error: ?struct tagPyArrayObject? has no member > named ?data? > > It looks like there is a new C API for numpy, and the version of numpy.i > that I have doesn't use it. > > Is there a new version of numpy.i available (or in development) that works > with the new API? Short term it will just get rid of a warning but I am > interested in a good long term solution in case I need to upgrade numpy. > > In the long term we would like to hide the ndarray internals, essentially making them private. There are still some incomplete areas, f2py and, apparently, numpy.i. Your feedback here is quite helpful and if you have some time we can try to get this straightened out. Could you attach the code you are trying to interface? If you have a github account you could also set up a branch where we could work on this. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wfspotz at sandia.gov Tue Oct 9 11:52:15 2012 From: wfspotz at sandia.gov (Bill Spotz) Date: Tue, 9 Oct 2012 09:52:15 -0600 Subject: [Numpy-discussion] [EXTERNAL] swig interface file (numpy.i) warning In-Reply-To: References: Message-ID: <7F19AD02-EE44-4BF2-AC13-B4EF8A04117E@sandia.gov> Tom, Charles, If you discuss this further, be sure to CC me. -Bill Spotz On Oct 9, 2012, at 8:50 AM, Charles R Harris wrote: > Hi Tom, > > On Tue, Oct 9, 2012 at 8:30 AM, Tom Krauss wrote: > Hi, > > I've been happy to use numpy.i for generating SWIG interfaces to C++. > > For a while, I've noticed this warning while compiling: > /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" > > and today tried to get rid of the warning. > > So, in numpy.i, I followed the warning's advice. I added the # def here: > > %{ > #ifndef SWIG_FILE_WITH_INIT > # define NO_IMPORT_ARRAY > #endif > #include "stdio.h" > #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION > #include > %} > > SWIG was happy, but when compiling the C++ wrapper, there were many warnings followed by many errors. The warnings were for redefinition of NPY_MIN_BYTE and similar. The errors were for all kinds of stuff, excerpt here: > native_wrap.cpp:3632: error: ?PyArray_NOTYPE? was not declared in this scope > native_wrap.cpp:3633: error: cannot convert ?PyObject*? to ?const PyArrayObject*? for argument ?1? to ?int PyArray_TYPE(const PyArrayObject*)? > native_wrap.cpp: At global scope: > native_wrap.cpp:3877: error: ?intp? has not been declared > native_wrap.cpp: In function ?int require_fortran(PyArrayObject*)?: > native_wrap.cpp:3929: error: ?struct tagPyArrayObject? has no member named ?nd? > native_wrap.cpp:3933: error: ?struct tagPyArrayObject? has no member named ?flags? > native_wrap.cpp:3933: error: ?FARRAY? was not declared in this scope > native_wrap.cpp:20411: error: ?struct tagPyArrayObject? has no member named ?data? > > It looks like there is a new C API for numpy, and the version of numpy.i that I have doesn't use it. > > Is there a new version of numpy.i available (or in development) that works with the new API? Short term it will just get rid of a warning but I am interested in a good long term solution in case I need to upgrade numpy. > > > In the long term we would like to hide the ndarray internals, essentially making them private. There are still some incomplete areas, f2py and, apparently, numpy.i. Your feedback here is quite helpful and if you have some time we can try to get this straightened out. Could you attach the code you are trying to interface? If you have a github account you could also set up a branch where we could work on this. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From thomas.p.krauss at gmail.com Tue Oct 9 14:47:57 2012 From: thomas.p.krauss at gmail.com (Tom Krauss) Date: Tue, 9 Oct 2012 13:47:57 -0500 Subject: [Numpy-discussion] [EXTERNAL] swig interface file (numpy.i) warning In-Reply-To: <7F19AD02-EE44-4BF2-AC13-B4EF8A04117E@sandia.gov> References: <7F19AD02-EE44-4BF2-AC13-B4EF8A04117E@sandia.gov> Message-ID: I can't attach the exact code but would be happy to provide something simple that has the same issue. I'll post something here when I can get to it. - Tom On Tue, Oct 9, 2012 at 10:52 AM, Bill Spotz wrote: > Tom, Charles, > > If you discuss this further, be sure to CC me. > > -Bill Spotz > > On Oct 9, 2012, at 8:50 AM, Charles R Harris wrote: > > > Hi Tom, > > > > On Tue, Oct 9, 2012 at 8:30 AM, Tom Krauss > wrote: > > Hi, > > > > I've been happy to use numpy.i for generating SWIG interfaces to C++. > > > > For a while, I've noticed this warning while compiling: > > > /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2: > warning: #warning "Using deprecated NumPy API, disable it by #defining > NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" > > > > and today tried to get rid of the warning. > > > > So, in numpy.i, I followed the warning's advice. I added the # def here: > > > > %{ > > #ifndef SWIG_FILE_WITH_INIT > > # define NO_IMPORT_ARRAY > > #endif > > #include "stdio.h" > > #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION > > #include > > %} > > > > SWIG was happy, but when compiling the C++ wrapper, there were many > warnings followed by many errors. The warnings were for redefinition of > NPY_MIN_BYTE and similar. The errors were for all kinds of stuff, excerpt > here: > > native_wrap.cpp:3632: error: ?PyArray_NOTYPE? was not declared in this > scope > > native_wrap.cpp:3633: error: cannot convert ?PyObject*? to ?const > PyArrayObject*? for argument ?1? to ?int PyArray_TYPE(const PyArrayObject*)? > > native_wrap.cpp: At global scope: > > native_wrap.cpp:3877: error: ?intp? has not been declared > > native_wrap.cpp: In function ?int require_fortran(PyArrayObject*)?: > > native_wrap.cpp:3929: error: ?struct tagPyArrayObject? has no member > named ?nd? > > native_wrap.cpp:3933: error: ?struct tagPyArrayObject? has no member > named ?flags? > > native_wrap.cpp:3933: error: ?FARRAY? was not declared in this scope > > native_wrap.cpp:20411: error: ?struct tagPyArrayObject? has no member > named ?data? > > > > It looks like there is a new C API for numpy, and the version of numpy.i > that I have doesn't use it. > > > > Is there a new version of numpy.i available (or in development) that > works with the new API? Short term it will just get rid of a warning but I > am interested in a good long term solution in case I need to upgrade numpy. > > > > > > In the long term we would like to hide the ndarray internals, > essentially making them private. There are still some incomplete areas, > f2py and, apparently, numpy.i. Your feedback here is quite helpful and if > you have some time we can try to get this straightened out. Could you > attach the code you are trying to interface? If you have a github account > you could also set up a branch where we could work on this. > > > > Chuck > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ** Bill Spotz ** > ** Sandia National Laboratories Voice: (505)845-0170 ** > ** P.O. Box 5800 Fax: (505)284-0154 ** > ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.p.krauss at gmail.com Tue Oct 9 23:18:58 2012 From: thomas.p.krauss at gmail.com (Tom Krauss) Date: Tue, 9 Oct 2012 22:18:58 -0500 Subject: [Numpy-discussion] [EXTERNAL] swig interface file (numpy.i) warning In-Reply-To: References: <7F19AD02-EE44-4BF2-AC13-B4EF8A04117E@sandia.gov> Message-ID: This code reproduces the error - I think it is small enough for email. (large) numpy.i not included, let me know if you want that too. Makefile will need to be tailored to your environment. If it's more convenient, or you have trouble reproducing, I can create a branch on github - let me know. On Tue, Oct 9, 2012 at 1:47 PM, Tom Krauss wrote: > I can't attach the exact code but would be happy to provide something > simple that has the same issue. I'll post something here when I can get to > it. > - Tom > > > On Tue, Oct 9, 2012 at 10:52 AM, Bill Spotz wrote: > >> Tom, Charles, >> >> If you discuss this further, be sure to CC me. >> >> -Bill Spotz >> >> On Oct 9, 2012, at 8:50 AM, Charles R Harris wrote: >> >> > Hi Tom, >> > >> > On Tue, Oct 9, 2012 at 8:30 AM, Tom Krauss >> wrote: >> > Hi, >> > >> > I've been happy to use numpy.i for generating SWIG interfaces to C++. >> > >> > For a while, I've noticed this warning while compiling: >> > >> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2: >> warning: #warning "Using deprecated NumPy API, disable it by #defining >> NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" >> > >> > and today tried to get rid of the warning. >> > >> > So, in numpy.i, I followed the warning's advice. I added the # def >> here: >> > >> > %{ >> > #ifndef SWIG_FILE_WITH_INIT >> > # define NO_IMPORT_ARRAY >> > #endif >> > #include "stdio.h" >> > #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION >> > #include >> > %} >> > >> > SWIG was happy, but when compiling the C++ wrapper, there were many >> warnings followed by many errors. The warnings were for redefinition of >> NPY_MIN_BYTE and similar. The errors were for all kinds of stuff, excerpt >> here: >> > native_wrap.cpp:3632: error: ?PyArray_NOTYPE? was not declared in this >> scope >> > native_wrap.cpp:3633: error: cannot convert ?PyObject*? to ?const >> PyArrayObject*? for argument ?1? to ?int PyArray_TYPE(const PyArrayObject*)? >> > native_wrap.cpp: At global scope: >> > native_wrap.cpp:3877: error: ?intp? has not been declared >> > native_wrap.cpp: In function ?int require_fortran(PyArrayObject*)?: >> > native_wrap.cpp:3929: error: ?struct tagPyArrayObject? has no member >> named ?nd? >> > native_wrap.cpp:3933: error: ?struct tagPyArrayObject? has no member >> named ?flags? >> > native_wrap.cpp:3933: error: ?FARRAY? was not declared in this scope >> > native_wrap.cpp:20411: error: ?struct tagPyArrayObject? has no member >> named ?data? >> > >> > It looks like there is a new C API for numpy, and the version of >> numpy.i that I have doesn't use it. >> > >> > Is there a new version of numpy.i available (or in development) that >> works with the new API? Short term it will just get rid of a warning but I >> am interested in a good long term solution in case I need to upgrade numpy. >> > >> > >> > In the long term we would like to hide the ndarray internals, >> essentially making them private. There are still some incomplete areas, >> f2py and, apparently, numpy.i. Your feedback here is quite helpful and if >> you have some time we can try to get this straightened out. Could you >> attach the code you are trying to interface? If you have a github account >> you could also set up a branch where we could work on this. >> > >> > Chuck >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> ** Bill Spotz ** >> ** Sandia National Laboratories Voice: (505)845-0170 ** >> ** P.O. Box 5800 Fax: (505)284-0154 ** >> ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: example.tar.gz Type: application/x-gzip Size: 1173 bytes Desc: not available URL: From njs at pobox.com Wed Oct 10 09:42:13 2012 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 10 Oct 2012 14:42:13 +0100 Subject: [Numpy-discussion] Fwd: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: Message-ID: This PR submitted a few months ago adds a substantial new API to numpy, so it'd be great to get more review. No-one's replied yet, though... Any thoughts, anyone? Is it useful, could it be better...? -n On 9 Jun 2012 22:47, "Nathaniel Smith" wrote: > [Manual PR notification] > > ---------- Forwarded message ---------- > From: timcera > Date: Sat, Jun 9, 2012 at 10:13 PM > Subject: [numpy] ENH: Initial implementation of a 'neighbor' calculation > (#303) > To: njsmith > > > Each element is assigned the result of a function based on it's neighbors. > Neighbors are selected based on a weight array. > > It uses the new pad routines to pad arrays if neighboring values are > required that would be off the edge of the input array. > > Will be great to have the masked array settled because right now you > can only sort of exclude from the neighborhood using a zero in the > weight array. Zero or np.IGNORE don't affect np.sum, but functions > like np.mean and np.std would give different answers. Because of this > my early implementations of neighbor included an optional mask array > along with the weight array, but I decided would be best to wait for > the new masked arrays. > > This in some ways could be considered a generalization of a > convolution, and comparison with existing numpy/scipy convolution > results are included in the tests. The advantage to neighbor is that > any function that accepts a 1-d array, and returns a single result, > can be used instead of convolution only using summation. The > convolution functions require the weight array to be flipped to get > the same answer as neighbor. > > You can merge this Pull Request by running: > > git pull https://github.com/timcera/numpy neighbor > > Or you can view, comment on it, or merge it online at: > > https://github.com/numpy/numpy/pull/303 > > -- Commit Summary -- > > * ENH: Initial implementation of a 'neighbor' calculation where the each > > -- File Changes -- > > M numpy/lib/__init__.py (2) > A numpy/lib/neighbor.py (305) > A numpy/lib/tests/test_neighbor.py (278) > > -- Patch Links -- > > https://github.com/numpy/numpy/pull/303.patch > https://github.com/numpy/numpy/pull/303.diff > > --- > Reply to this email directly or view it on GitHub: > https://github.com/numpy/numpy/pull/303 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim at cerazone.net Wed Oct 10 12:55:57 2012 From: tim at cerazone.net (Cera, Tim) Date: Wed, 10 Oct 2012 12:55:57 -0400 Subject: [Numpy-discussion] Fwd: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: Message-ID: On Wed, Oct 10, 2012 at 1:58 AM, Travis E. Oliphant < notifications at github.com> wrote: > I'm not sure what to make of no comments on this PR. This seems like a > useful addition. @timcera are you still > interested in having this PR merged? > Yes. I mentioned in PR comments that the lack of discussion is because my code engenders speechless awe in anyone who looks at it. :-) Of course speechless awe can come from two different reasons! Hopefully it is because my code is so awesome. Seriously, I really wanted some input, especially after I found #31 . On Wed, Oct 10, 2012 at 7:24 AM, Eric Moore wrote: > This seems to be trying to solve a very similar problem to #31 > Internally I implemented something like rolling window, but I don't return the windows. Instead the contents of the windows are used for calculation of each windows 'central' cell in the results array. After seeing the rolling window function I thought it might be nice to bring that out into a callable function, so that similar functionality would be available. That particular function isn't useful to me directly, but perhaps others? Kindest regards, Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Thu Oct 11 05:50:53 2012 From: Nicolas.Rougier at inria.fr (Nicolas Rougier) Date: Thu, 11 Oct 2012 11:50:53 +0200 Subject: [Numpy-discussion] [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: Message-ID: <8A55AC08-E589-4777-AA2A-C23DDC6B1064@inria.fr> I missed the original post but I personally find this addition especially useful for my work in computational neuroscience. I did something vaguely similar in a small framework (http://dana.loria.fr/, you can look more specifically at http://dana.loria.fr/doc/connection.html for details). Examples are available from: http://dana.loria.fr/examples.html The actual computation can be made in several ways depending on the properties of the kernel but the idea is to compute an array "K" such that given an array "A" and a kernel "k", A*K holds the expected result. This also work with sparse array for example when the kernel is very small. I suspect the PR will be quite efficient compared to what I did. Nicolas On Oct 10, 2012, at 18:55 , Cera, Tim wrote: > On Wed, Oct 10, 2012 at 1:58 AM, Travis E. Oliphant wrote: > I'm not sure what to make of no comments on this PR. This seems like a useful addition. @timcera are you still interested in having this PR merged? > > > Yes. > > I mentioned in PR comments that the lack of discussion is because my code engenders speechless awe in anyone who looks at it. :-) Of course speechless awe can come from two different reasons! Hopefully it is because my code is so awesome. > > Seriously, I really wanted some input, especially after I found #31. > > > On Wed, Oct 10, 2012 at 7:24 AM, Eric Moore wrote: > This seems to be trying to solve a very similar problem to #31 > > Internally I implemented something like rolling window, but I don't return the windows. Instead the contents of the windows are used for calculation of each windows 'central' cell in the results array. > > After seeing the rolling window function I thought it might be nice to bring that out into a callable function, so that similar functionality would be available. That particular function isn't useful to me directly, but perhaps others? > > Kindest regards, > Tim > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From sebastian at sipsolutions.net Thu Oct 11 08:40:16 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 11 Oct 2012 14:40:16 +0200 Subject: [Numpy-discussion] Fwd: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: Message-ID: <1349959216.3183.74.camel@sebastian-laptop> On Wed, 2012-10-10 at 12:55 -0400, Cera, Tim wrote: > On Wed, Oct 10, 2012 at 1:58 AM, Travis E. > Oliphant wrote: > I'm not sure what to make of no comments on this PR. This > seems like a useful addition. @timcera are you still > interested in having this PR merged? > > > Internally I implemented something like rolling window, but I don't > return the windows. Instead the contents of the windows are used for > calculation of each windows 'central' cell in the results array. > Rolling window can help with everything but the I guess typical case of neighbor(..., mode='same', pad=None), where the idea must fail since not all neighborhoods are the same size. It is probably quite a bit faster to replace the shapeintersect with it in those cases, but not sure if it is worth it. What I personally do not quite like is that for the case of the function being a ufunc, it not being (..., mode='same', pad=None) and weights=np.ones(...), the rolling window approach will be much faster as it can be fully vectorized with the new numpy versions. But thats just me and the documentation would have a "see also", so I guess users should be able to figure that out if they need the speed. Plus I don't see an elegant way to find the neighborhoods for (mode='same', pad=None) so having a function to help with it should be nice. On a general note, maybe it would make sense to allow a binary mask instead of the np.isfinite check and would it be better if the returned arrays are not raveled unless there is this mask? Also there is a ** missing for the kwargs in the function call. > > After seeing the rolling window function I thought it might be nice to > bring that out into a callable function, so that similar functionality > would be available. That particular function isn't useful to me > directly, but perhaps others? > > > Kindest regards, > Tim > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From njs at pobox.com Thu Oct 11 09:48:53 2012 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 11 Oct 2012 14:48:53 +0100 Subject: [Numpy-discussion] [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: <8A55AC08-E589-4777-AA2A-C23DDC6B1064@inria.fr> References: <8A55AC08-E589-4777-AA2A-C23DDC6B1064@inria.fr> Message-ID: On Thu, Oct 11, 2012 at 10:50 AM, Nicolas Rougier wrote: > I missed the original post but I personally find this addition especially useful for my work in computational neuroscience. > > I did something vaguely similar in a small framework (http://dana.loria.fr/, you can look more specifically at http://dana.loria.fr/doc/connection.html for details). Examples are available from: http://dana.loria.fr/examples.html > > The actual computation can be made in several ways depending on the properties of the kernel but the idea is to compute an array "K" such that given an array "A" and a kernel "k", A*K holds the expected result. This also work with sparse array for example when the kernel is very small. I suspect the PR will be quite efficient compared to what I did. Would the current PR be useful to you if merged as-is? A common pitfall with these sorts of contributions is that we realize only after merging it that there is some tiny detail of the API that makes it not-quite-usable for some people with related problems, so it'd be awesome if you could take a closer look. -n From njs at pobox.com Thu Oct 11 15:08:53 2012 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 11 Oct 2012 20:08:53 +0100 Subject: [Numpy-discussion] Fwd: [numpy] When accessing MaskedArray rows, always return an mvoid object (#483) In-Reply-To: References: Message-ID: This PR changes the behaviour of masked record arrays. The rationale makes sense to me, but I don't actually use this functionality. Any masked array users want to chime in? -n ---------- Forwarded message ---------- From: "Thomas Robitaille" Date: 11 Oct 2012 19:35 Subject: [numpy] When accessing MaskedArray rows, always return an mvoid object (#483) To: "numpy/numpy" The current behavior of MaskedArray when accessing a row object is that if all masks are False, the row is an np.void object, and if any are set to True, it is a np.ma.core.mvoid object. The issue with this is that users can access/modify masks for rows that already have at least one mask set to True, but not for rows that have no masks set. For example: In [1]: a = ma.array(zip([1,2,3]), mask=[0,1,0], dtype=[('x', int)]) In [2]: a[0].mask --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in () ----> 1 a[0].mask AttributeError: 'numpy.void' object has no attribute 'mask' In [3]: a[1].mask Out[3]: (True,) There is no reason why a row should behave differently whether any values are masked or not, so this PR defaults to always returning an mvoid object, otherwise mask values cannot be set for rows that start off with no mask values set. (Of course, the present implementation in Numpy also has a performance impact - for arrays with many fields, each call to a row has to flatten the mask of the whole record just to check what type to return, which is inefficient. With this PR, row access should be faster for masked arrays. But this is secondary.) ------------------------------ You can merge this Pull Request by running: git pull https://github.com/astrofrog/numpy fix-masked-getitem Or view, comment on, or merge it at: https://github.com/numpy/numpy/pull/483 Commit Summary - When accessing MaskedArray rows, always return an mvoid object File Changes - *M* numpy/ma/core.py (10) Patch Links - https://github.com/numpy/numpy/pull/483.patch - https://github.com/numpy/numpy/pull/483.diff ? Reply to this email directly or view it on GitHub. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Fri Oct 12 08:33:47 2012 From: sturla at molden.no (Sturla Molden) Date: Fri, 12 Oct 2012 14:33:47 +0200 Subject: [Numpy-discussion] Fwd: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: Message-ID: <50780E2B.4040008@molden.no> On 10.10.2012 15:42, Nathaniel Smith wrote: > This PR submitted a few months ago adds a substantial new API to numpy, > so it'd be great to get more review. No-one's replied yet, though... > > Any thoughts, anyone? Is it useful, could it be better...? Fast neighbor search is what scipy.spatial.cKDTree is designed for. There is an brand new version in SciPy master. Sturla From tim at cerazone.net Fri Oct 12 09:29:40 2012 From: tim at cerazone.net (Cera, Tim) Date: Fri, 12 Oct 2012 09:29:40 -0400 Subject: [Numpy-discussion] Fwd: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: <50780E2B.4040008@molden.no> References: <50780E2B.4040008@molden.no> Message-ID: For the neighbor module, the neighborhood is input specified by the 'weight' array. All values in the neighborhood are processed by a function. In the geosciences, ArcGIS is a very important tool and the neighbor module is very loosely modeled after http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=An%20overview%20of%20the%20Neighborhood%20tools If the language is confusing, now is the time to change the names. Kindest regards, Tim On Fri, Oct 12, 2012 at 8:33 AM, Sturla Molden wrote: > On 10.10.2012 15:42, Nathaniel Smith wrote: > > This PR submitted a few months ago adds a substantial new API to numpy, > > so it'd be great to get more review. No-one's replied yet, though... > > > > Any thoughts, anyone? Is it useful, could it be better...? > > Fast neighbor search is what scipy.spatial.cKDTree is designed for. > There is an brand new version in SciPy master. > > Sturla > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim at cerazone.net Fri Oct 12 09:38:46 2012 From: tim at cerazone.net (Cera, Tim) Date: Fri, 12 Oct 2012 09:38:46 -0400 Subject: [Numpy-discussion] Fwd: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: <50780E2B.4040008@molden.no> Message-ID: If you followed the link http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=An%20overview%20of%20the%20Neighborhood%20tools note that the current neighborhood implementation that we are talking about implements the ArcGIS 'Focal*' functionality, not the 'Block*' ones. Note also that ArcGIS is limited to 2-d, and a 3x3 neighborhood. Kindest regards, Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwang at continuum.io Fri Oct 12 11:50:49 2012 From: pwang at continuum.io (Peter Wang) Date: Fri, 12 Oct 2012 10:50:49 -0500 Subject: [Numpy-discussion] Reminder: Last day of early registration for PyData NYC conference! Message-ID: Hi everyone, Just a friendly reminder that today is the final day of early registration for the PyData NYC conference later this month! We have a fantastic lineup of talks and workshops on a variety of topics related to Python for data analysis, including topics that are hard to find at other conferences (e.g. practical perspectives on Python and Hadoop, using Python and R, etc.). http://nyc2012.pydata.org/ Use the discount code "numpy" for a 20% discount off of registration! We are also looking for sponsors. We are proud to feature D. E. Shaw, JP Morgan, and Appnexus as gold sponsors. If your company or organization would like some visibility in front of a few hundred Python data hackers, please visit our sponsor information page: http://nyc2012.pydata.org/sponsors/becoming/ Thanks, Peter From sturla at molden.no Fri Oct 12 14:04:21 2012 From: sturla at molden.no (Sturla Molden) Date: Fri, 12 Oct 2012 20:04:21 +0200 Subject: [Numpy-discussion] Fwd: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: <50780E2B.4040008@molden.no> Message-ID: I'm still rather sure GIS functionality belongs in scipy.spatial instead of numpy. Sturla Sendt fra min iPad Den 12. okt. 2012 kl. 15:29 skrev "Cera, Tim" : > For the neighbor module, the neighborhood is input specified by the 'weight' array. All values in the neighborhood are processed by a function. > > In the geosciences, ArcGIS is a very important tool and the neighbor module is very loosely modeled after http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=An%20overview%20of%20the%20Neighborhood%20tools > > If the language is confusing, now is the time to change the names. > > Kindest regards, > Tim > > On Fri, Oct 12, 2012 at 8:33 AM, Sturla Molden wrote: >> On 10.10.2012 15:42, Nathaniel Smith wrote: >> > This PR submitted a few months ago adds a substantial new API to numpy, >> > so it'd be great to get more review. No-one's replied yet, though... >> > >> > Any thoughts, anyone? Is it useful, could it be better...? >> >> Fast neighbor search is what scipy.spatial.cKDTree is designed for. >> There is an brand new version in SciPy master. >> >> Sturla >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Fri Oct 12 12:02:29 2012 From: Nicolas.Rougier at inria.fr (Nicolas Rougier) Date: Fri, 12 Oct 2012 18:02:29 +0200 Subject: [Numpy-discussion] [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: <8A55AC08-E589-4777-AA2A-C23DDC6B1064@inria.fr> Message-ID: <9DF50AD4-DF54-4F31-9FC1-1BCFB6E96E85@inria.fr> Sorry, I'm away from the lab and did not have a chance to test is yet. I will do next week. Nicolas On Oct 11, 2012, at 15:48 , Nathaniel Smith wrote: > On Thu, Oct 11, 2012 at 10:50 AM, Nicolas Rougier > wrote: >> I missed the original post but I personally find this addition especially useful for my work in computational neuroscience. >> >> I did something vaguely similar in a small framework (http://dana.loria.fr/, you can look more specifically at http://dana.loria.fr/doc/connection.html for details). Examples are available from: http://dana.loria.fr/examples.html >> >> The actual computation can be made in several ways depending on the properties of the kernel but the idea is to compute an array "K" such that given an array "A" and a kernel "k", A*K holds the expected result. This also work with sparse array for example when the kernel is very small. I suspect the PR will be quite efficient compared to what I did. > > Would the current PR be useful to you if merged as-is? A common > pitfall with these sorts of contributions is that we realize only > after merging it that there is some tiny detail of the API that makes > it not-quite-usable for some people with related problems, so it'd be > awesome if you could take a closer look. > > -n > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From paul.anton.letnes at gmail.com Sat Oct 13 02:52:06 2012 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sat, 13 Oct 2012 08:52:06 +0200 Subject: [Numpy-discussion] ANN: WinPython v2.7.3.0 In-Reply-To: References: Message-ID: <6697FA16-B843-4EA7-9A45-4C09BABAA867@gmail.com> Hi Pierre, first I'd like to congratulate you on a product which seems to finally solve all the problems I have at work regarding python scripting. A portable, fully featured python distribution is indeed very useful. One question though: Is there a way to run winpython's IPython under cygwin, without a separate window popping up? Just setting the PATH doesn't quite cut it, unfortunately. I'm interested because cygwin doesn't come with many science oriented python packages at all. Cheers Paul On 24. sep. 2012, at 21:22, Pierre Raybaut wrote: > Hi all, > > I'm pleased to introduce my new contribution to the Python community: WinPython. > > WinPython v2.7.3.0 has been released and is available for 32-bit and > 64-bit Windows platforms: > http://code.google.com/p/winpython/ > > WinPython is a free open-source portable distribution of Python for > Windows, designed for scientists. > > It is a full-featured (see > http://code.google.com/p/winpython/wiki/PackageIndex) Python-based > scientific environment: > * Designed for scientists (thanks to the integrated libraries NumPy, > SciPy, Matplotlib, guiqwt, etc.: > * Regular *scientific users*: interactive data processing and > visualization using Python with Spyder > * *Advanced scientific users and software developers*: Python > applications development with Spyder, version control with Mercurial > and other development tools (like gettext) > * *Portable*: preconfigured, it should run out of the box on any > machine under Windows (without any installation requirements) and the > folder containing WinPython can be moved to any location (local, > network or removable drive) > * *Flexible*: one can install (or should I write "use" as it's > portable) as many WinPython versions as necessary (like isolated and > self-consistent environments), even if those versions are running > different versions of Python (2.7, 3.x in the near future) or > different architectures (32bit or 64bit) on the same machine > * *Customizable*: using the integrated package manager (wppm, as > WinPython Package Manager), it's possible to install, uninstall or > upgrade Python packages (see > http://code.google.com/p/winpython/wiki/WPPM for more details on > supported package formats). > > *WinPython is not an attempt to replace Python(x,y)*, this is just > something different (see > http://code.google.com/p/winpython/wiki/Roadmap): more flexible, > easier to maintain, movable and less invasive for the OS, but > certainly less user-friendly, with less packages/contents and without > any integration to Windows explorer [*]. > > [*] Actually there is an optional integration into Windows explorer, > providing the same features as the official Python installer regarding > file associations and context menu entry (this option may be activated > through the WinPython Control Panel). > > Enjoy! > -Pierre > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From tjhnson at gmail.com Sat Oct 13 14:25:29 2012 From: tjhnson at gmail.com (T J) Date: Sat, 13 Oct 2012 13:25:29 -0500 Subject: [Numpy-discussion] Fwd: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: <50780E2B.4040008@molden.no> Message-ID: On Fri, Oct 12, 2012 at 1:04 PM, Sturla Molden wrote: > I'm still rather sure GIS functionality belongs in scipy.spatial instead > of numpy. > > >From the link: """ FocalMax Finds the highest value for each cell location on an input grid within a specified neighborhood and sends it to the corresponding cell location on the output grid. """ Isn't this much more generic than GIS? Maybe the criteria should be: "Would you expect to find extensive discussion on this functionality within a standard text on spatial data structures?" Sliding windows and nice boundary handling seems less "spatial" to me, and much further from B-trees, quadtress, kd-trees, persistent data structures, etc. Can you elaborate your position a bit more? Sorry if I'm being dense. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Oct 14 13:53:11 2012 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 14 Oct 2012 17:53:11 +0000 (UTC) Subject: [Numpy-discussion] distutils & C++ & Fortran Message-ID: Hi, I'd like to link both C++ and Fortran code into a single Python extension, using numpy.distutils (for scipy.special). But it seems the distutils-based tools actually cannot do this? Does someone know if there's a way to work around this? -- Pauli Virtanen From jaredoconnell at gmail.com Sun Oct 14 14:04:11 2012 From: jaredoconnell at gmail.com (Jared O'Connell) Date: Sun, 14 Oct 2012 19:04:11 +0100 Subject: [Numpy-discussion] Welcome to the "NumPy-Discussion" mailing list (Digest mode) In-Reply-To: References: Message-ID: Hello! I am trying to build numpy 1.6.2 with python 2.7 in Ubuntu 12.04 and wish to link against serial ATLAS rather than the threaded version. I have modified site.cfg accordingly by uncommenting out the following lines: [blas_opt] libraries = f77blas, cblas, atlas [lapack_opt] libraries = lapack, f77blas, cblas, atlas However when I run python setup.py build it still links to the threaded versions eg. python setup.py build Running from numpy source directory.F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib Setting PTATLAS=ATLAS customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize LaheyFCompiler Could not locate executable lf95 customize PGroupFCompiler Could not locate executable pgfortran customize AbsoftFCompiler Could not locate executable f90 customize NAGFCompiler Found executable /usr/bin/f95 customize VastFCompiler customize CompaqFCompiler Could not locate executable fort customize IntelItaniumFCompiler Could not locate executable efort Could not locate executable efc customize IntelEM64TFCompiler customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/usr/lib/atlas-base -lptf77blas -lptcblas -latlas -o _configtest ATLAS version 3.8.4 built by buildd on Sat Sep 10 23:12:12 UTC 2011: UNAME : Linux crested 2.6.24-29-server #1 SMP Wed Aug 10 15:58:57 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_HAMMER -DATL_CPUMHZ=1993 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 393216 F77 : gfortran, version GNU Fortran (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 F77FLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 SMCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 SKCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 success! removing: _configtest.c _configtest.o _configtest Setting PTATLAS=ATLAS FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack not found in /usr/lib/atlas-base numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] ... Any ideas? best, Jared -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaredoconnell at gmail.com Sun Oct 14 14:04:39 2012 From: jaredoconnell at gmail.com (Jared O'Connell) Date: Sun, 14 Oct 2012 19:04:39 +0100 Subject: [Numpy-discussion] Compiling against serial ATLAS Message-ID: Hello! I am trying to build numpy 1.6.2 with python 2.7 in Ubuntu 12.04 and wish to link against serial ATLAS rather than the threaded version. I have modified site.cfg accordingly by uncommenting out the following lines: [blas_opt] libraries = f77blas, cblas, atlas [lapack_opt] libraries = lapack, f77blas, cblas, atlas However when I run python setup.py build it still links to the threaded versions eg. python setup.py build Running from numpy source directory.F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib Setting PTATLAS=ATLAS customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize LaheyFCompiler Could not locate executable lf95 customize PGroupFCompiler Could not locate executable pgfortran customize AbsoftFCompiler Could not locate executable f90 customize NAGFCompiler Found executable /usr/bin/f95 customize VastFCompiler customize CompaqFCompiler Could not locate executable fort customize IntelItaniumFCompiler Could not locate executable efort Could not locate executable efc customize IntelEM64TFCompiler customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/usr/lib/atlas-base -lptf77blas -lptcblas -latlas -o _configtest ATLAS version 3.8.4 built by buildd on Sat Sep 10 23:12:12 UTC 2011: UNAME : Linux crested 2.6.24-29-server #1 SMP Wed Aug 10 15:58:57 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_HAMMER -DATL_CPUMHZ=1993 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 393216 F77 : gfortran, version GNU Fortran (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 F77FLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 SMCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 SKCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 success! removing: _configtest.c _configtest.o _configtest Setting PTATLAS=ATLAS FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack not found in /usr/lib/atlas-base numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] ... Any ideas? best, Jared -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Sun Oct 14 20:24:19 2012 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sun, 14 Oct 2012 20:24:19 -0400 Subject: [Numpy-discussion] [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: <50780E2B.4040008@molden.no> References: <50780E2B.4040008@molden.no> Message-ID: <7B8972B1-C3D1-43E0-A92E-A5E99AF4B842@yale.edu> > On 10.10.2012 15:42, Nathaniel Smith wrote: >> This PR submitted a few months ago adds a substantial new API to numpy, >> so it'd be great to get more review. No-one's replied yet, though... >> >> Any thoughts, anyone? Is it useful, could it be better...? > > Fast neighbor search is what scipy.spatial.cKDTree is designed for. > There is an brand new version in SciPy master. It's not neighbor *search*, it's applying a function over an (arbitrary chosen and weighted) moving neighborhood in an nd array. It would be useful for the author of the PR to post a detailed comparison of this functionality with scipy.ndimage.generic_filter, which appears to have very similar functionality. Zach From millman at berkeley.edu Mon Oct 15 00:53:08 2012 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 14 Oct 2012 21:53:08 -0700 Subject: [Numpy-discussion] [ANN] CFP: SciPy India 2012 -- Dec 27-29 -- IIT Bombay Message-ID: Hello, The CFP for SciPy India 2012, to be held in IIT Bombay from December 27-29 is open. Please spread the word! Scipy.in is a conference providing opportunities to spread the use of the Python programming language in the Scientific Computing community in India. It provides a unique opportunity to interact with the "Who's who" of the Python for Scientific Computing fraternity and learn, understand, participate, and contribute to Scientific Computing using Python. Attendees of the conference and participants of the sprints planned will be able to access and review the tools available. They will also be able to learn domain-specific applications and how the tools apply to a plethora of application problems. One of the goals of the conference is to combine education, engineering, and science with computing through the medium of Python. This conference also aims to spread the use of Python for Scientific Computing in various fields and among different communities. Call for Papers ================ We look forward to your submissions on the use of Python for Scientific Computing and Education. This includes pedagogy, exploration, modeling and analysis from both applied and developmental perspectives. We welcome contributions from academia as well as industry. Submission of Papers ===================== If you wish to present your paper using this platform, please submit an abstract of 300 to 700 words describing the topic, including its relevance to scientific computing. Based on the number and quality of the submissions, the conference organizers will allot 10 - 30 minutes for each accepted talk. In addition to these talks, there will be an open session of lightning talks, during which any attendee who wishes to talk on a pertinent topic is invited to do a presentation not exceeding five minutes in duration. If you wish to present a talk at the conference, please follow the guidelines below. Submission Guidelines ====================== - Submit your proposals at scipy at fossee.in - Submissions whose main purpose is to promote a commercial product or service will be refused. - All accepted proposals must be presented at the SciPy conference by at least one author. Important Dates ================ - Call for proposals start: 27th September 2012, Thursday - Call for proposals end: 1st November 2012, Thursday - List of accepted proposals will be published: 19th November 2012, Monday - Submission of first presentation: 10th December 2012, Monday - Submission of final presentation(with final changes): 20th December 2012, Thursday From tim at cerazone.net Mon Oct 15 09:36:16 2012 From: tim at cerazone.net (Cera, Tim) Date: Mon, 15 Oct 2012 09:36:16 -0400 Subject: [Numpy-discussion] [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: <7B8972B1-C3D1-43E0-A92E-A5E99AF4B842@yale.edu> References: <50780E2B.4040008@molden.no> <7B8972B1-C3D1-43E0-A92E-A5E99AF4B842@yale.edu> Message-ID: On Sun, Oct 14, 2012 at 8:24 PM, Zachary Pincus wrote: > It would be useful for the author of the PR to post a detailed comparison > of this functionality with scipy.ndimage.generic_filter, which appears to > have very similar functionality. > I'll be durned. I created neighbor because I didn't find what I wanted, and to find now that I just didn't look in the right place is well ... Let's just say that I went for a long run last night. Searching for ndimage, I found that is has been around a long, long time. First in numarray, then moved to scipy. Really I could only nitpick about minor differences - kinda like a primary political campaign. On the face of it though, generic_filter looks better. First off it is written in C so likely will be faster and more efficient memory use. I didn't look at optimizing neighbor at all and at least my part of it is pure Python. Of course for all of the small differences, I like my choices better. :-) I would like to make a mild suggestion. Emphasis on mild. Maybe ndimage, in all or in part, should be brought into (back into?) Numpy and renamed. About the PR. Given that the neighbor functionality exists already, I will close the PR later today. Move along, nothing to see here... Side note: I wrote arraypad with the future idea that it would become easyish to include that functionality in other places, for example in neighbor. A Don't Repeat Yourself idea. Previously I had only seen Fortran pad capabilities in some of the Fast Fourier Transform functions. The generic_filter function includes several padding functions - written in C. This means that if arraypad needs be more efficient we have C code to base a better arraypad. Another side node: The measurements functions in ndimage are called zonal functions in the GIS field. Kindest regards, Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Mon Oct 15 14:26:43 2012 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 15 Oct 2012 14:26:43 -0400 Subject: [Numpy-discussion] [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) In-Reply-To: References: <50780E2B.4040008@molden.no> <7B8972B1-C3D1-43E0-A92E-A5E99AF4B842@yale.edu> Message-ID: <51DEB028-E25D-4AB6-BD7F-3322DC62269E@yale.edu> Hi Tim, It's tricky to find functionality sometimes because as you've seen numpy and especially scipy are spread over very diverse domains with very diverse terminology! Usually someone on one or the other of the lists can help folks find some functionality, if not by name than by description... In any case, though, I hope you'll keep your code around and accessible to people in standalone form. Despite being a bit slower than the ndimage stuff, it seems like you've got a better set of boundary conditions, and some other niceties that may appeal to others too. Zach > On Sun, Oct 14, 2012 at 8:24 PM, Zachary Pincus wrote: >> It would be useful for the author of the PR to post a detailed comparison of this functionality with scipy.ndimage.generic_filter, which appears to have very similar functionality. >> > I'll be durned. I created neighbor because I didn't find what I wanted, and to find now that I just didn't look in the right place is well ... Let's just say that I went for a long run last night. > > Searching for ndimage, I found that is has been around a long, long time. First in numarray, then moved to scipy. > > Really I could only nitpick about minor differences - kinda like a primary political campaign. On the face of it though, generic_filter looks better. First off it is written in C so likely will be faster and more efficient memory use. I didn't look at optimizing neighbor at all and at least my part of it is pure Python. Of course for all of the small differences, I like my choices better. :-) > > I would like to make a mild suggestion. Emphasis on mild. Maybe ndimage, in all or in part, should be brought into (back into?) Numpy and renamed. > > About the PR. Given that the neighbor functionality exists already, I will close the PR later today. Move along, nothing to see here... > > Side note: I wrote arraypad with the future idea that it would become easyish to include that functionality in other places, for example in neighbor. A Don't Repeat Yourself idea. Previously I had only seen Fortran pad capabilities in some of the Fast Fourier Transform functions. The generic_filter function includes several padding functions - written in C. This means that if arraypad needs be more efficient we have C code to base a better arraypad. > > Another side node: The measurements functions in ndimage are called zonal functions in the GIS field. > > Kindest regards, > Tim > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From byronblay at googlemail.com Tue Oct 16 11:36:51 2012 From: byronblay at googlemail.com (Byron Blay) Date: Tue, 16 Oct 2012 16:36:51 +0100 Subject: [Numpy-discussion] array flags not copied/deepcopied Message-ID: Copying / deepcopying an array does not copy the writeable flag: >>> import numpy >>> from copy import deepcopy >>> a = numpy.array([1,2,3,4,5]) >>> a.flags.writeable = False >>> b = deepcopy(a) >>> b.flags.writeable True Is this a bug? Should I raise an issue on github? Many thanks, Byron -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Oct 16 11:46:28 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 16 Oct 2012 09:46:28 -0600 Subject: [Numpy-discussion] distutils & C++ & Fortran In-Reply-To: References: Message-ID: On Sun, Oct 14, 2012 at 11:53 AM, Pauli Virtanen wrote: > Hi, > > I'd like to link both C++ and Fortran code into a single > Python extension, using numpy.distutils (for scipy.special). > > But it seems the distutils-based tools actually cannot do this? > Does someone know if there's a way to work around this? > > Looks like the answer is "no" ;) Is this a linking problem? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Oct 16 14:22:30 2012 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 16 Oct 2012 19:22:30 +0100 Subject: [Numpy-discussion] array flags not copied/deepcopied In-Reply-To: References: Message-ID: On Tue, Oct 16, 2012 at 4:36 PM, Byron Blay wrote: > Copying / deepcopying an array does not copy the writeable flag: > >>>> import numpy >>>> from copy import deepcopy >>>> a = numpy.array([1,2,3,4,5]) >>>> a.flags.writeable = False >>>> b = deepcopy(a) >>>> b.flags.writeable > True > > Is this a bug? > Should I raise an issue on github? I think this is intended? numpy doesn't really think of writeable and non-writeable arrays as being different sorts of objects; flags.writeable just records a fact about the underlying memory that the array refers to. If you make a new copy, then (by definition!) you get a nice new chunk of freshly allocated memory, and the memory is writeable. That said, this might not be the most useful approach. Can you elaborate on your situation where this is causing a problem? -n From thouis at gmail.com Tue Oct 16 16:06:02 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Tue, 16 Oct 2012 16:06:02 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Sun, Oct 7, 2012 at 10:15 AM, Thouis (Ray) Jones wrote: > I plan to import all the Trac issues to github by the end of this > week. I want to get an up-to-date snapshot of the Trac DB, and run > another test import with it (just to make sure there's nothing in > recent bugs that isn't handled). > > Previous test imports here: > https://github.com/thouis/numpy-trac-migration/issues I successfully imported all the issues from a more recent snapshot to the test repository as @numpy-gitbot, rather than @thouis (to save myself from getting pulled into every bug's discussion). If no one sees any problems with the latest imports (basically, the last 2000 or so by github issue #) , I think it's ready for the real transfer to the numpy github repository. Ray From pav at iki.fi Tue Oct 16 16:56:51 2012 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 16 Oct 2012 20:56:51 +0000 (UTC) Subject: [Numpy-discussion] distutils & C++ & Fortran References: Message-ID: Charles R Harris gmail.com> writes: > On Sun, Oct 14, 2012 at 11:53 AM, Pauli Virtanen iki.fi> wrote: > Hi, > I'd like to link both C++ and Fortran code into a single > Python extension, using ?numpy.distutils (for scipy.special). > But it seems the distutils-based tools actually cannot do this? > Does someone know if there's a way to work around this? > > Looks like the answer is "no" ;) Is this a linking problem? Pretty much. Knowing the Fortran or C++ runtime names / link flags on different compilers would be enough, but apparently this information is not written down anywhere in the build system. I guess I'll stick with the workaround of putting them in separate modules. Pauli From robert.kern at gmail.com Tue Oct 16 17:45:12 2012 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 16 Oct 2012 22:45:12 +0100 Subject: [Numpy-discussion] distutils & C++ & Fortran In-Reply-To: References: Message-ID: On Tue, Oct 16, 2012 at 9:56 PM, Pauli Virtanen wrote: > Charles R Harris gmail.com> writes: >> On Sun, Oct 14, 2012 at 11:53 AM, Pauli Virtanen iki.fi> wrote: >> Hi, >> I'd like to link both C++ and Fortran code into a single >> Python extension, using numpy.distutils (for scipy.special). >> But it seems the distutils-based tools actually cannot do this? >> Does someone know if there's a way to work around this? >> >> Looks like the answer is "no" ;) Is this a linking problem? > > Pretty much. Knowing the Fortran or C++ runtime names / link flags > on different compilers would be enough, but apparently this > information is not written down anywhere in the build system. With some platforms/compilers, we use the compiler itself as the linker, because it knows the right runtime library and link flags for the language, often because they are more variable with respect to specific compiler versions. -- Robert Kern From njs at pobox.com Tue Oct 16 17:54:00 2012 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 16 Oct 2012 22:54:00 +0100 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Tue, Oct 16, 2012 at 9:06 PM, Thouis (Ray) Jones wrote: > On Sun, Oct 7, 2012 at 10:15 AM, Thouis (Ray) Jones wrote: >> I plan to import all the Trac issues to github by the end of this >> week. I want to get an up-to-date snapshot of the Trac DB, and run >> another test import with it (just to make sure there's nothing in >> recent bugs that isn't handled). >> >> Previous test imports here: >> https://github.com/thouis/numpy-trac-migration/issues > > I successfully imported all the issues from a more recent snapshot to > the test repository as @numpy-gitbot, rather than @thouis (to save > myself from getting pulled into every bug's discussion). > > If no one sees any problems with the latest imports (basically, the > last 2000 or so by github issue #) , I think it's ready for the real > transfer to the numpy github repository. This is really fabulous; thanks for all the effort you've put in! But... I am quite concerned that we're still linking to attachments in the trac, e.g.: https://github.com/thouis/numpy-trac-migration/issues/6002#issuecomment-9492223 This means that we can't ever take down the trac without breaking all our github issues; we're committing to keeping the trac running indefinitely. Doesn't that kind of defeat a lot of the point...? -n From thouis at gmail.com Tue Oct 16 19:18:39 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Tue, 16 Oct 2012 19:18:39 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Tue, Oct 16, 2012 at 5:54 PM, Nathaniel Smith wrote: > On Tue, Oct 16, 2012 at 9:06 PM, Thouis (Ray) Jones wrote: >> On Sun, Oct 7, 2012 at 10:15 AM, Thouis (Ray) Jones wrote: >>> I plan to import all the Trac issues to github by the end of this >>> week. I want to get an up-to-date snapshot of the Trac DB, and run >>> another test import with it (just to make sure there's nothing in >>> recent bugs that isn't handled). >>> >>> Previous test imports here: >>> https://github.com/thouis/numpy-trac-migration/issues >> >> I successfully imported all the issues from a more recent snapshot to >> the test repository as @numpy-gitbot, rather than @thouis (to save >> myself from getting pulled into every bug's discussion). >> >> If no one sees any problems with the latest imports (basically, the >> last 2000 or so by github issue #) , I think it's ready for the real >> transfer to the numpy github repository. > > This is really fabulous; thanks for all the effort you've put in! > > But... I am quite concerned that we're still linking to attachments in > the trac, e.g.: > https://github.com/thouis/numpy-trac-migration/issues/6002#issuecomment-9492223 > > This means that we can't ever take down the trac without breaking all > our github issues; we're committing to keeping the trac running > indefinitely. Doesn't that kind of defeat a lot of the point...? Not actually. I have a snapshot of the attachments, as well, and once there's a place for them live (another repository, possibly), we can use the github API to edit the issues in place to point to the new URLs. Ray From sebastian.walter at gmail.com Wed Oct 17 05:38:14 2012 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Wed, 17 Oct 2012 11:38:14 +0200 Subject: [Numpy-discussion] how is y += x computed when y.strides = (0, 8) and x.strides=(16, 8) ? In-Reply-To: References: <1346892081.1210.17.camel@sebastian-laptop> Message-ID: I'd like to have a look at the implementation of iadd in numpy, but I'm having a real hard time to find the corresponding code. I'm basically stuck at https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/number.c#L487 Could someone give me a pointer where to find it? Respectively, could someone point me to some documentation where the (folder/file) structure of the numpy sources is explained? Sebastian On Thu, Sep 6, 2012 at 2:58 PM, Nathaniel Smith wrote: > On Thu, Sep 6, 2012 at 1:41 AM, Sebastian Berg > wrote: >> Hey, >> >> No idea if this is simply not support or just a bug, though I am >> guessing that such usage simply is not planned. > > I think that's right... currently numpy simply makes no guarantees > about what order ufunc loops will be performed in, or even if they > will be performed in any strictly sequential order. In ordinary cases > this lets it make various optimizations, but it means that you can't > count on any specific behaviour for the unusual case where different > locations in the output array are stored in overlapping memory. > > Fixing this would require two things: > (a) Some code to detect when an array may have internal overlaps (sort > of like np.may_share_memory for axes). Not entirely trivial. > (b) A "fallback mode" for ufuncs where if the code in (a) detects that > we are (probably) dealing with one of these arrays, it processes the > operations in some predictable order without buffering. > > I suppose if someone wanted to come up with these two pieces, and it > didn't look like it would cause slowdowns in common cases, the code in > (b) avoided creating duplicate code paths that increased maintenance > burden, etc., then probably no-one would object to making these arrays > act in a better defined way? I don't think most people are that > worried about this though. Your original code would be much clearer if > it just used np.sum... > > -n > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From teoliphant at gmail.com Wed Oct 17 11:22:28 2012 From: teoliphant at gmail.com (Travis Oliphant) Date: Wed, 17 Oct 2012 10:22:28 -0500 Subject: [Numpy-discussion] A change with minor compatibility questions Message-ID: <0B6DD8C1-BCB4-496B-8584-FB3C71CA8234@gmail.com> Hey all, https://github.com/numpy/numpy/pull/482 is a pull request that changes the hash function for numpy void scalars. These are the objects returned from fully indexing a structured array: array[i] if array is a 1-d structured array. Currently their hash function just hashes the pointer to the underlying data. This means that void scalars can be used as keys in a dictionary but the behavior is non-intuitive because another void scalar with the same data but pointing to a different region of memory will hash differently. The pull request makes it so that two void scalars with the same data will hash to the same value (using the same algorithm as a tuple hash). This pull request also only allows read-only scalars to be hashed. There is a small chance this will break someone's code if they relied on this behavior. I don't believe anyone is currently relying on this behavior -- but I've been proven wrong before. What do people on this list think? Should we raise a warning in the next release when a hash function on a void scalar is called or just make the change, put it in the release notes and make a few people change their code if needed. The problem was identified by a couple of users of NumPy currently which is why I think that people who have tried using numpy void scalars as keys aren't doing it right now but are instead converting them to tuples first. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Wed Oct 17 11:27:40 2012 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 17 Oct 2012 17:27:40 +0200 Subject: [Numpy-discussion] A change with minor compatibility questions In-Reply-To: <0B6DD8C1-BCB4-496B-8584-FB3C71CA8234@gmail.com> References: <0B6DD8C1-BCB4-496B-8584-FB3C71CA8234@gmail.com> Message-ID: <20121017152740.GT1622@phare.normalesup.org> On Wed, Oct 17, 2012 at 10:22:28AM -0500, Travis Oliphant wrote: > There is a small chance this will break someone's code if they relied on this > behavior. I don't believe anyone is currently relying on this behavior -- but > I've been proven wrong before. What do people on this list think? I think that the change is acceptable. Thanks for asking! > Should we raise a warning in the next release when a hash function on a void > scalar is called or just make the change, put it in the release notes and make > a few people change their code if needed. I think that there should be a section in the release notes that list such changes in a very visible way. I don't think that a warning is necessary here, but I guess that others might have a different point of view. G From d.s.seljebotn at astro.uio.no Wed Oct 17 12:56:36 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 17 Oct 2012 18:56:36 +0200 Subject: [Numpy-discussion] A change with minor compatibility questions In-Reply-To: <0B6DD8C1-BCB4-496B-8584-FB3C71CA8234@gmail.com> References: <0B6DD8C1-BCB4-496B-8584-FB3C71CA8234@gmail.com> Message-ID: <507EE344.4020505@astro.uio.no> On 10/17/2012 05:22 PM, Travis Oliphant wrote: > Hey all, > > https://github.com/numpy/numpy/pull/482 > > is a pull request that changes the hash function for numpy void > scalars. These are the objects returned from fully indexing a > structured array: array[i] if array is a 1-d structured array. > > Currently their hash function just hashes the pointer to the underlying > data. This means that void scalars can be used as keys in a > dictionary but the behavior is non-intuitive because another void scalar > with the same data but pointing to a different region of memory will > hash differently. > > The pull request makes it so that two void scalars with the same data > will hash to the same value (using the same algorithm as a tuple hash). > This pull request also only allows read-only scalars to be hashed. > > There is a small chance this will break someone's code if they relied on > this behavior. I don't believe anyone is currently relying on this > behavior -- but I've been proven wrong before. What do people on this > list think? I support working on fixing this, but if I understand your fix correctly this change just breaks things in a different way. Specifically, in this example: arr = np.ones(4, dtype=[('a', np.int64)]) x = arr[0] d = { x : 'value' } arr[0]['a'] = 4 print d[x] Does the last line raise a KeyError? If I understand correctly it does. (Of course, the current situation just breaks lookups in another situation.) I propose to BOTH make "x" unhashable (thus being a good Python citizen and following Python rules) AND provide "x.askey()" or "x.immutable()" which returns something immutable you can use as a key. The places where that breaks things is probably buggy code that must be fixed (either one way or the other) anyway. Perhaps a warning period is in order then (one would raise a warning in __hash__, telling people to use the "askey()" method). (I would really prefer to always have "x" be immutable, but that probably breaks working code.) Dag Sverre From d.s.seljebotn at astro.uio.no Wed Oct 17 13:48:51 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 17 Oct 2012 19:48:51 +0200 Subject: [Numpy-discussion] A change with minor compatibility questions In-Reply-To: <507EE344.4020505@astro.uio.no> References: <0B6DD8C1-BCB4-496B-8584-FB3C71CA8234@gmail.com> <507EE344.4020505@astro.uio.no> Message-ID: <507EEF83.2020603@astro.uio.no> On 10/17/2012 06:56 PM, Dag Sverre Seljebotn wrote: > On 10/17/2012 05:22 PM, Travis Oliphant wrote: >> Hey all, >> >> https://github.com/numpy/numpy/pull/482 >> >> is a pull request that changes the hash function for numpy void >> scalars. These are the objects returned from fully indexing a >> structured array: array[i] if array is a 1-d structured array. >> >> Currently their hash function just hashes the pointer to the underlying >> data. This means that void scalars can be used as keys in a >> dictionary but the behavior is non-intuitive because another void scalar >> with the same data but pointing to a different region of memory will >> hash differently. >> >> The pull request makes it so that two void scalars with the same data >> will hash to the same value (using the same algorithm as a tuple hash). >> This pull request also only allows read-only scalars to be hashed. >> >> There is a small chance this will break someone's code if they relied on >> this behavior. I don't believe anyone is currently relying on this >> behavior -- but I've been proven wrong before. What do people on this >> list think? > > I support working on fixing this, but if I understand your fix correctly > this change just breaks things in a different way. > > Specifically, in this example: > > arr = np.ones(4, dtype=[('a', np.int64)]) > x = arr[0] > d = { x : 'value' } > arr[0]['a'] = 4 > print d[x] > > Does the last line raise a KeyError? If I understand correctly it does. Argh. I overlooked both Travis' second commit, and the explicit mention of read-only above. Isn't it possible to produce a read-only array from a writeable one though, and so get a read-only scalar whose underlying value can still change? Anyway, sorry about being so quick to post. Dag Sverre From travis at continuum.io Wed Oct 17 15:08:13 2012 From: travis at continuum.io (Travis Oliphant) Date: Wed, 17 Oct 2012 14:08:13 -0500 Subject: [Numpy-discussion] A change with minor compatibility questions In-Reply-To: <507EEF83.2020603@astro.uio.no> References: <0B6DD8C1-BCB4-496B-8584-FB3C71CA8234@gmail.com> <507EE344.4020505@astro.uio.no> <507EEF83.2020603@astro.uio.no> Message-ID: On Oct 17, 2012, at 12:48 PM, Dag Sverre Seljebotn wrote: > On 10/17/2012 06:56 PM, Dag Sverre Seljebotn wrote: >> On 10/17/2012 05:22 PM, Travis Oliphant wrote: >>> Hey all, >>> >>> https://github.com/numpy/numpy/pull/482 >>> >>> is a pull request that changes the hash function for numpy void >>> scalars. These are the objects returned from fully indexing a >>> structured array: array[i] if array is a 1-d structured array. >>> >>> Currently their hash function just hashes the pointer to the underlying >>> data. This means that void scalars can be used as keys in a >>> dictionary but the behavior is non-intuitive because another void scalar >>> with the same data but pointing to a different region of memory will >>> hash differently. >>> >>> The pull request makes it so that two void scalars with the same data >>> will hash to the same value (using the same algorithm as a tuple hash). >>> This pull request also only allows read-only scalars to be hashed. >>> >>> There is a small chance this will break someone's code if they relied on >>> this behavior. I don't believe anyone is currently relying on this >>> behavior -- but I've been proven wrong before. What do people on this >>> list think? >> >> I support working on fixing this, but if I understand your fix correctly >> this change just breaks things in a different way. >> >> Specifically, in this example: >> >> arr = np.ones(4, dtype=[('a', np.int64)]) >> x = arr[0] >> d = { x : 'value' } >> arr[0]['a'] = 4 >> print d[x] >> >> Does the last line raise a KeyError? If I understand correctly it does. > > Argh. I overlooked both Travis' second commit, and the explicit mention > of read-only above. > > Isn't it possible to produce a read-only array from a writeable one > though, and so get a read-only scalar whose underlying value can still > change? Yes, it is possible to do that (just like it is currently possible to change a tuple with a C-extension or even Cython or a string with NumPy). We won't be able to prevent people from writing code that will have odd behavior, but we can communicate correctly about what one should do. -Travis > > Anyway, sorry about being so quick to post. > > Dag Sverre > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ralf.gommers at gmail.com Wed Oct 17 16:44:23 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 17 Oct 2012 22:44:23 +0200 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Tue, Oct 16, 2012 at 11:54 PM, Nathaniel Smith wrote: > On Tue, Oct 16, 2012 at 9:06 PM, Thouis (Ray) Jones > wrote: > > On Sun, Oct 7, 2012 at 10:15 AM, Thouis (Ray) Jones > wrote: > >> I plan to import all the Trac issues to github by the end of this > >> week. I want to get an up-to-date snapshot of the Trac DB, and run > >> another test import with it (just to make sure there's nothing in > >> recent bugs that isn't handled). > >> > >> Previous test imports here: > >> https://github.com/thouis/numpy-trac-migration/issues > > > > I successfully imported all the issues from a more recent snapshot to > > the test repository as @numpy-gitbot, rather than @thouis (to save > > myself from getting pulled into every bug's discussion). > > > > If no one sees any problems with the latest imports (basically, the > > last 2000 or so by github issue #) , I think it's ready for the real > > transfer to the numpy github repository. > > This is really fabulous; thanks for all the effort you've put in! > Looks great to me too, don't see anything missing. Thanks a lot Ray! Ralf > But... I am quite concerned that we're still linking to attachments in > the trac, e.g.: > > https://github.com/thouis/numpy-trac-migration/issues/6002#issuecomment-9492223 > > This means that we can't ever take down the trac without breaking all > our github issues; we're committing to keeping the trac running > indefinitely. Doesn't that kind of defeat a lot of the point...? > > -n > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Oct 17 20:58:02 2012 From: cournape at gmail.com (David Cournapeau) Date: Thu, 18 Oct 2012 02:58:02 +0200 Subject: [Numpy-discussion] how is y += x computed when y.strides = (0, 8) and x.strides=(16, 8) ? In-Reply-To: References: <1346892081.1210.17.camel@sebastian-laptop> Message-ID: On Wed, Oct 17, 2012 at 11:38 AM, Sebastian Walter wrote: > I'd like to have a look at the implementation of iadd in numpy, > but I'm having a real hard time to find the corresponding code. > > I'm basically stuck at > https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/number.c#L487 n_ops is essentially a map of function pointers set in the umath module (see PyArray_SetNumericOps, used in umath module). IOW, n_ops.add is a pointer to umath.add, itself set up through a generated file __umath_generated.c: f = PyUFunc_FromFuncAndData(add_functions, ...); PyDict_SetItemString(dictionary, "add", f); At that point, you need to look into the ufunc machinery: for double all along, the add type resolver should be pretty simple and in the end call DOUBLE_add. > > Could someone give me a pointer where to find it? > Respectively, could someone point me to some documentation where the > (folder/file) structure of the numpy sources is explained? There is sadly not much explanation on the code structure for the C part. src/multiarray contains the code for the multiarray extension (array, dtype, broadcasting, iteration) and src/umath contains the code for the umath extension (ufunc machinery + core loops implementation). David From simon.lieschke at orionhealth.com Wed Oct 17 21:34:56 2012 From: simon.lieschke at orionhealth.com (Simon Lieschke) Date: Thu, 18 Oct 2012 14:34:56 +1300 (NZDT) Subject: [Numpy-discussion] numpy.arrange not returning expected results (bug?) Message-ID: Hi, I've discovered calling numpy.arange(1.1, 17.1) and numpy(1.1, 16.1) both return the same results. Could this be a numpy bug, or is there some behaviour I'm possibly not aware of here? I've pasted in the results of an interactive Python session comparing and contrasting these with some other similar calls. I'm using NumPy 1.6.2. Thanks in advance, Simon 14:21:46.77 @ C:\Users\simon >python Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.arange(1.1, 17.1) array([ 1.1, 2.1, 3.1, 4.1, 5.1, 6.1, 7.1, 8.1, 9.1, 10.1, 11.1, 12.1, 13.1, 14.1, 15.1, 16.1]) >>> np.arange(1.1, 16.1) array([ 1.1, 2.1, 3.1, 4.1, 5.1, 6.1, 7.1, 8.1, 9.1, 10.1, 11.1, 12.1, 13.1, 14.1, 15.1, 16.1]) >>> np.arange(1.1, 15.1) array([ 1.1, 2.1, 3.1, 4.1, 5.1, 6.1, 7.1, 8.1, 9.1, 10.1, 11.1, 12.1, 13.1, 14.1]) >>> np.arange(1, 17) array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]) >>> np.arange(1, 16) array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]) >>> np.arange(1, 15) array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]) From scott.sinclair.za at gmail.com Thu Oct 18 03:55:35 2012 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 18 Oct 2012 09:55:35 +0200 Subject: [Numpy-discussion] numpy.arrange not returning expected results (bug?) In-Reply-To: References: Message-ID: On 18 October 2012 03:34, Simon Lieschke wrote: > I've discovered calling numpy.arange(1.1, 17.1) and numpy(1.1, 16.1) both > return the same results. Could this be a numpy bug, or is there some > behaviour I'm possibly not aware of here? Not a bug, it's because you're using floating point arguments. The docstring (http://docs.scipy.org/doc/numpy/reference/generated/numpy.arange.html) tells you that "For floating point arguments, the length of the result is ceil((stop - start)/step)". If you try In [1]: (17.1-1.1)/1.0 Out[1]: 16.0 In [2]: (16.1-1.1)/1.0 Out[2]: 15.000000000000002 In [3]: np.ceil((17.1-1.1)/1.0) Out[3]: 16.0 In [4]: np.ceil((16.1-1.1)/1.0) Out[4]: 16.0 you see that the length of the output array ends up being the same due to floating point round-off effects. You can achieve what you want using np.linspace(1.1, 17.1, num=17) etc.. Cheers, Scott From thouis at gmail.com Thu Oct 18 09:17:10 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Thu, 18 Oct 2012 09:17:10 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Wed, Oct 17, 2012 at 4:44 PM, Ralf Gommers wrote: > > > On Tue, Oct 16, 2012 at 11:54 PM, Nathaniel Smith wrote: >> >> On Tue, Oct 16, 2012 at 9:06 PM, Thouis (Ray) Jones >> wrote: >> > On Sun, Oct 7, 2012 at 10:15 AM, Thouis (Ray) Jones >> > wrote: >> >> I plan to import all the Trac issues to github by the end of this >> >> week. I want to get an up-to-date snapshot of the Trac DB, and run >> >> another test import with it (just to make sure there's nothing in >> >> recent bugs that isn't handled). >> >> >> >> Previous test imports here: >> >> https://github.com/thouis/numpy-trac-migration/issues >> > >> > I successfully imported all the issues from a more recent snapshot to >> > the test repository as @numpy-gitbot, rather than @thouis (to save >> > myself from getting pulled into every bug's discussion). >> > >> > If no one sees any problems with the latest imports (basically, the >> > last 2000 or so by github issue #) , I think it's ready for the real >> > transfer to the numpy github repository. >> >> This is really fabulous; thanks for all the effort you've put in! > > > Looks great to me too, don't see anything missing. Thanks a lot Ray! > > Ralf > > >> >> But... I am quite concerned that we're still linking to attachments in >> the trac, e.g.: >> >> https://github.com/thouis/numpy-trac-migration/issues/6002#issuecomment-9492223 >> >> This means that we can't ever take down the trac without breaking all >> our github issues; we're committing to keeping the trac running >> indefinitely. Doesn't that kind of defeat a lot of the point...? >> >> -n >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion I plan to run the migration tomorrow (I promised the git users that will be @-mentioned a day's warning, which I forgot to send until now). Ray From thouis at gmail.com Fri Oct 19 11:20:01 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Fri, 19 Oct 2012 11:20:01 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: I started the import with the oldest 75 and newest 125 Trac issues, and will wait a few hours to do the rest to allow feedback, just in case something is broken that I haven't noticed. I did make one change to better emulate Trac behavior. Some Trac usernames are also email addresses, which Trac anonymizes in its display. I decided it was safer to do the same. Ray Jones From travis at continuum.io Fri Oct 19 11:56:17 2012 From: travis at continuum.io (Travis Oliphant) Date: Fri, 19 Oct 2012 10:56:17 -0500 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: Kudos! Ray. Very impressive and useful work. -Travis On Oct 19, 2012, at 10:20 AM, Thouis (Ray) Jones wrote: > I started the import with the oldest 75 and newest 125 Trac issues, > and will wait a few hours to do the rest to allow feedback, just in > case something is broken that I haven't noticed. > > I did make one change to better emulate Trac behavior. Some Trac > usernames are also email addresses, which Trac anonymizes in its > display. I decided it was safer to do the same. > > Ray Jones > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From travis at continuum.io Fri Oct 19 12:10:44 2012 From: travis at continuum.io (Travis Oliphant) Date: Fri, 19 Oct 2012 11:10:44 -0500 Subject: [Numpy-discussion] Announcing Anaconda version 1.1 Message-ID: I just wanted to let everyone know about our new release of Anaconda which now has Spyder and Matplotlib working for Mac OS X and Windows. Right now, it's the best way to get the pre-requisites for Numba --- though I recommend getting the latest Numba from github as Numba is still under active development. * Anaconda 1.1 Announcement Continuum Analytics, Inc. is pleased to announce the release of Anaconda Pro 1.1, which extends Anaconda?s programming capabilities to the desktop. Anaconda Pro now includes an IDE (Spyder ) and plotting capabilities (Matplotlib ), as well as optimized versions of Numba Pro and IOPro . With these enhancements, AnacondaPro is a complete solution for server-side computation or client-side development. It is equally well-suited for supercomputers or for training in a classroom. Available for Windows, Mac OS X, and Linux, Anaconda is a Python distribution for scientific computing, engineering simulation, and business intelligence & data management. It includes the most popular numerical and scientific libraries used by scientists, engineers, and data analysts, with a single integrated and flexible installer. Continuum Analytics offers Enterprise-level support for Anaconda, covering both its open source libraries as well as the included commercial libraries from Continuum. For more information, to download a trial version of Anaconda Pro, or download the completely free Anaconda CE, click here . * * * * * Best regards, -Travis * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From thouis at gmail.com Fri Oct 19 16:46:54 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Fri, 19 Oct 2012 16:46:54 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Fri, Oct 19, 2012 at 11:20 AM, Thouis (Ray) Jones wrote: > I started the import with the oldest 75 and newest 125 Trac issues, > and will wait a few hours to do the rest to allow feedback, just in > case something is broken that I haven't noticed. > > I did make one change to better emulate Trac behavior. Some Trac > usernames are also email addresses, which Trac anonymizes in its > display. I decided it was safer to do the same. The import is running again, though I've been having some failures in a few comments and general hangs (these might be network related). I'm keeping track of which issues might have had difficulties. @endolith noticed that I didn't correctly relink #XXX trac id numbers to github id numbers (both trac and github create links automatically), so that will have to be handled by a postprocessing script (which it probably would have, anyway, since the github # isn't known before import). Ray From thouis at gmail.com Fri Oct 19 21:34:18 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Fri, 19 Oct 2012 21:34:18 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Fri, Oct 19, 2012 at 4:46 PM, Thouis (Ray) Jones wrote: > On Fri, Oct 19, 2012 at 11:20 AM, Thouis (Ray) Jones wrote: >> I started the import with the oldest 75 and newest 125 Trac issues, >> and will wait a few hours to do the rest to allow feedback, just in >> case something is broken that I haven't noticed. >> >> I did make one change to better emulate Trac behavior. Some Trac >> usernames are also email addresses, which Trac anonymizes in its >> display. I decided it was safer to do the same. > > The import is running again, though I've been having some failures in > a few comments and general hangs (these might be network related). > I'm keeping track of which issues might have had difficulties. > > @endolith noticed that I didn't correctly relink #XXX trac id numbers > to github id numbers (both trac and github create links > automatically), so that will have to be handled by a postprocessing > script (which it probably would have, anyway, since the github # isn't > known before import). Import has finished. The following trac #s had issues in creating the comments (I think due to network problems): 182, 297, 619, 621, 902, 904, 909 913, 914, 915, 1044, 1526. I'll review them and see if I can pull in anything missing I'll also work on a script for updating the trac crossrefs to github crossrefs. In the "no good deed goes unpunished" category, I accidentally logged in as myself (rather than numpy-gitbot) and pushed about 500 issues, so now I receive updates whenever one of them gets changed. At least most of them were closed, already... Ray From thouis at gmail.com Fri Oct 19 21:50:34 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Fri, 19 Oct 2012 21:50:34 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: Also, it looks like Trac issues 2228 and up weren't in the snapshot of the DB I had. Those should be imported after Trac is disabled for new bugs. Ray From kmichael.aye at gmail.com Fri Oct 19 23:49:16 2012 From: kmichael.aye at gmail.com (Michael Aye) Date: Fri, 19 Oct 2012 20:49:16 -0700 Subject: [Numpy-discussion] fromfile with sys.stdin? Message-ID: Is it possible to use sys.stdin as input for numpy.fromfile? I can't make it work. This simple example: ---- import sys import numpy as np # I know the stream format, so I skip parsing the header keys = ['lot','lon','tb','qual'] dt = [(key,'f8') for key in keys] # skip the header, it has 'end' in the last line while 'end' not in sys.stdin.readline(): pass # pass the stdin filepointer to fromfile arr = np.fromfile(sys.stdin, dtype = dt) print arr ---- produces this error: ---- (develop+)[maye at luna1 ~/src/diviner]$ pread des=~/data/data.des < ~/data/mydata.txt | python test_fromfile.py Traceback (most recent call last): File "test_fromfile.py", line 10, in arr = np.fromfile(sys.stdin, dtype = dt) IOError: could not seek in file ---- I also tried to catch the binary output stream into a binary file and read that with fromfile without any problem with the exact same method like here, skipping over the header and so on, so it should not be any problem with the rest of the code. Any hints welcome! From fperez.net at gmail.com Sat Oct 20 01:43:52 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 19 Oct 2012 22:43:52 -0700 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Fri, Oct 19, 2012 at 8:56 AM, Travis Oliphant wrote: > Kudos! Ray. > > Very impressive and useful work. Indeed, many thanks, Ray!! This has been a ton of work, and somewhat thankless b/c it's for a one-off thing. What I did for our lanunchpad bug migration was a far more hackish/dirty job precisely for that reason, so I applaud you for your patience to do this right. Cheers, f From ndbecker2 at gmail.com Sat Oct 20 07:11:34 2012 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 20 Oct 2012 07:11:34 -0400 Subject: [Numpy-discussion] print array in a form that can then be input Message-ID: I find it annoying that in casual use, if I print an array, that form can't be directly used as subsequent input (or can it?). What do others do about this? When I say casual, what I mean is, I write some long-running task and at the end, print some small array. Now I decide I'd like to cut/paste that into a new program. From davidmenhur at gmail.com Sat Oct 20 07:40:48 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Sat, 20 Oct 2012 13:40:48 +0200 Subject: [Numpy-discussion] print array in a form that can then be input In-Reply-To: References: Message-ID: For the case of a small array, you can use repr(). This will work as long as the array is not clipped (it is small enough). >>> a=np.arange(10) >>> print a [0 1 2 3 4 5 6 7 8 9] >>> repr(a) 'array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])' >>> a=np.arange(10000) >>> a.resize((100,100)) >>> print a [[ 0 1 2 ..., 97 98 99] [ 100 101 102 ..., 197 198 199] [ 200 201 202 ..., 297 298 299] ..., [9700 9701 9702 ..., 9797 9798 9799] [9800 9801 9802 ..., 9897 9898 9899] [9900 9901 9902 ..., 9997 9998 9999]] >>> repr(a) 'array([[ 0, 1, 2, ..., 97, 98, 99],\n [ 100, 101, 102, ..., 197, 198, 199],\n [ 200, 201, 202, ..., 297, 298, 299],\n ..., \n [9700, 9701, 9702, ..., 9797, 9798, 9799],\n [9800, 9801, 9802, ..., 9897, 9898, 9899],\n [9900, 9901, 9902, ..., 9997, 9998, 9999]])' David. On Sat, Oct 20, 2012 at 1:11 PM, Neal Becker wrote: > I find it annoying that in casual use, if I print an array, that form can't be > directly used as subsequent input (or can it?). > > What do others do about this? When I say casual, what I mean is, I write some > long-running task and at the end, print some small array. Now I decide I'd like > to cut/paste that into a new program. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From fperez.net at gmail.com Sat Oct 20 16:05:57 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 20 Oct 2012 13:05:57 -0700 Subject: [Numpy-discussion] print array in a form that can then be input In-Reply-To: References: Message-ID: On Sat, Oct 20, 2012 at 4:40 AM, Da?id wrote: >>>> a=np.arange(10) >>>> print a > [0 1 2 3 4 5 6 7 8 9] >>>> repr(a) > 'array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])' > Note that you don't need to explicitly call repr() at the interactive prompt: by default, Python prints the repr of an object when you type its name: In [24]: a=np.arange(10) In [25]: repr(a) Out[25]: 'array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])' In [26]: a Out[26]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) (this is ipython, but the behavior is the same in plain python). Cheers, f From davidmenhur at gmail.com Sat Oct 20 16:18:30 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Sat, 20 Oct 2012 22:18:30 +0200 Subject: [Numpy-discussion] print array in a form that can then be input In-Reply-To: References: Message-ID: On Sat, Oct 20, 2012 at 10:05 PM, Fernando Perez wrote: > Note that you don't need to explicitly call repr() at the interactive > prompt: by default, Python prints the repr of an object when you type > its name: True, and good point to notice. Nevertheless print executes str(), if present, and it is the output of any non-interactive script. Just to complete your clarification. :) From rahulgarg44 at gmail.com Sun Oct 21 10:27:43 2012 From: rahulgarg44 at gmail.com (Rahul Garg) Date: Sun, 21 Oct 2012 10:27:43 -0400 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests Message-ID: Hi. I am a PhD student at McGill University and I am developing a compiler for Python for CPUs and GPUs. For CPUs, I build upon LLVM. For GPUs, I generate OpenCL and I have also implemented some library functions on the GPU myself. The restriction that it is only for numerical code and intended for NumPy users. The compiler is aware of simple things in NumPy like matrix multiplication, slicing operators, strided layouts, some library functions (though limited at this time) and the negative indexing semantics etc. However, the compiler is not limited to vector code. Scalar code or manually written loops also work. However, only numerical datatypes are supported with no support for lists, dicts, classes etc. First class functions are not currently supported but are on the roadmap. You will have to add some type annotations to your functions. If you have a compatible GPU, you can also use the GPU by indicating which parts to run on the GPU. Otherwise you can just use it to run your code on the CPU. As an example, simple scalar code like fibonacci function works fine. Simple loops like those used in stencil-type computations are also working. Parallel-for loops are also provided and working. Simple vector oriented code is also working fine on both CPU and GPU. The system is being tested on Ubuntu 12.04 and tested with Python 2.7 (though I think should work with other Python 2.x variants). For GPUs, I am ensuring that the system works with AMD and Nvidia GPUs. The compiler is in early stages and I am looking for test cases. The project will be open-sourced in November under Apache 2 and thereafter will be developed in an open repo. If you have some simple code that I can use as a benchmark that I can use to test and evaluate the compiler, that will be very helpful. Some annotations will be required, which I can help you write. I will be VERY grateful to anyone who can provide test cases. In turn, it will help improve the compiler and everyone will benefit. Some of you may be wondering how it compares to Numba. Well it is essentially very similar in the idea. So why build a new compiler then? Actually the project I am building is not specific to Python. I am building a far more general compiler infrastructure for array languages, and Python frontend is just one small part of the project. For example, I am also working on a MATLAB frontend. (Some of you may remember me from an earlier compiler project which unfortunately went nowhere. This is a different project and this time I am determined to convert it into a usable system. I realize the proof is in the pudding, so I hope to convince people by releasing code soon.) thanks, Rahul From aron at ahmadia.net Sun Oct 21 12:01:56 2012 From: aron at ahmadia.net (Aron Ahmadia) Date: Sun, 21 Oct 2012 17:01:56 +0100 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: References: Message-ID: Hi Rahul, Very cool! I'm looking forward to seeing some performance results! Anders Logg posted a computational challenge to G+ about a month ago, and we got entries in Octave, Fortran, Python, and Julia (all implementing the same solution from Jed Brown). The challenge is here: https://plus.google.com/116518787475147930287/posts/jiULACjiGnW Here is my simple attempt at Cythonizing Jed's Octave code: https://gist.github.com/3893361 The best solution in Fortran took 38 microseconds. The best Python solution clocked in at around 445. The Julia solution implemented by Jed took around 224 microseconds, a good LLVM solution should come close to or beat that. Hope this helps. Aron On Sun, Oct 21, 2012 at 3:27 PM, Rahul Garg wrote: > Hi. > > I am a PhD student at McGill University and I am developing a compiler > for Python for CPUs and GPUs. For CPUs, I build upon LLVM. For GPUs, I > generate OpenCL and I have also implemented some library functions on > the GPU myself. The restriction that it is only for numerical code and > intended for NumPy users. The compiler is aware of simple things in > NumPy like matrix multiplication, slicing operators, strided layouts, > some library functions (though limited at this time) and the negative > indexing semantics etc. However, the compiler is not limited to vector > code. Scalar code or manually written loops also work. However, only > numerical datatypes are supported with no support for lists, dicts, > classes etc. First class functions are not currently supported but are > on the roadmap. You will have to add some type annotations to your > functions. If you have a compatible GPU, you can also use the GPU by > indicating which parts to run on the GPU. Otherwise you can just use > it to run your code on the CPU. > > As an example, simple scalar code like fibonacci function works fine. > Simple loops like those used in stencil-type computations are also > working. Parallel-for loops are also provided and working. Simple > vector oriented code is also working fine on both CPU and GPU. The > system is being tested on Ubuntu 12.04 and tested with Python 2.7 > (though I think should work with other Python 2.x variants). For GPUs, > I am ensuring that the system works with AMD and Nvidia GPUs. > > The compiler is in early stages and I am looking for test cases. The > project will be open-sourced in November under Apache 2 and thereafter > will be developed in an open repo. If you have some simple code that I > can use as a benchmark that I can use to test and evaluate the > compiler, that will be very helpful. Some annotations will be > required, which I can help you write. I will be VERY grateful to > anyone who can provide test cases. In turn, it will help improve the > compiler and everyone will benefit. > > Some of you may be wondering how it compares to Numba. Well it is > essentially very similar in the idea. So why build a new compiler > then? Actually the project I am building is not specific to Python. I > am building a far more general compiler infrastructure for array > languages, and Python frontend is just one small part of the project. > For example, I am also working on a MATLAB frontend. > > (Some of you may remember me from an earlier compiler project which > unfortunately went nowhere. This is a different project and this time > I am determined to convert it into a usable system. I realize the > proof is in the pudding, so I hope to convince people by releasing > code soon.) > > thanks, > Rahul > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldcroft at head.cfa.harvard.edu Sun Oct 21 12:39:56 2012 From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft) Date: Sun, 21 Oct 2012 12:39:56 -0400 Subject: [Numpy-discussion] Comparing rows in a structured masked array raises exception Message-ID: When comparing rows of a structured masked array I'm getting an exception. A similar operation on an structured ndarray gives the expected True/False result. Note that this exception only occurs if one or more of the mask values are True, since otherwise both row objects are np.void and the ndarray behavior is obtained. (See PR #483, which seems like a good idea to me, compatibility issues notwithstanding). >>> from numpy import ma >>> marr = ma.array([(1,2), (3,4)], dtype=[('x', int), ('y', int)]) >>> marr.mask[0][0] = True >>> marr.mask[1][0] = True >>> marr[0] == marr[1] Traceback (most recent call last): File "", line 1, in marr[0] == marr[1] File "/data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/ma/core.py", line 3573, in __eq__ check = ndarray.__eq__(self.filled(0), odata).view(type(self)) TypeError: descriptor '__eq__' requires a 'numpy.ndarray' object but received a 'numpy.void' >>> arr = np.array([(1,2), (3,4)], dtype=[('x', int), ('y', int)]) >>> arr[0] == arr[1] False >>> np.__version__ '1.6.2' Thanks, Tom From rahulgarg44 at gmail.com Sun Oct 21 14:57:04 2012 From: rahulgarg44 at gmail.com (Rahul Garg) Date: Sun, 21 Oct 2012 14:57:04 -0400 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: References: Message-ID: Thanks! I need to add support for eig and inv (will do this week, at least for CPU) but other than that, I should definitely be able to handle those kinds of benchmarks. rahul On Sun, Oct 21, 2012 at 12:01 PM, Aron Ahmadia wrote: > Hi Rahul, > > Very cool! I'm looking forward to seeing some performance results! Anders > Logg posted a computational challenge to G+ about a month ago, and we got > entries in Octave, Fortran, Python, and Julia (all implementing the same > solution from Jed Brown). The challenge is here: > > https://plus.google.com/116518787475147930287/posts/jiULACjiGnW > > Here is my simple attempt at Cythonizing Jed's Octave code: > > https://gist.github.com/3893361 > > The best solution in Fortran took 38 microseconds. The best Python solution > clocked in at around 445. The Julia solution implemented by Jed took around > 224 microseconds, a good LLVM solution should come close to or beat that. > > Hope this helps. > > Aron > > On Sun, Oct 21, 2012 at 3:27 PM, Rahul Garg wrote: >> >> Hi. >> >> I am a PhD student at McGill University and I am developing a compiler >> for Python for CPUs and GPUs. For CPUs, I build upon LLVM. For GPUs, I >> generate OpenCL and I have also implemented some library functions on >> the GPU myself. The restriction that it is only for numerical code and >> intended for NumPy users. The compiler is aware of simple things in >> NumPy like matrix multiplication, slicing operators, strided layouts, >> some library functions (though limited at this time) and the negative >> indexing semantics etc. However, the compiler is not limited to vector >> code. Scalar code or manually written loops also work. However, only >> numerical datatypes are supported with no support for lists, dicts, >> classes etc. First class functions are not currently supported but are >> on the roadmap. You will have to add some type annotations to your >> functions. If you have a compatible GPU, you can also use the GPU by >> indicating which parts to run on the GPU. Otherwise you can just use >> it to run your code on the CPU. >> >> As an example, simple scalar code like fibonacci function works fine. >> Simple loops like those used in stencil-type computations are also >> working. Parallel-for loops are also provided and working. Simple >> vector oriented code is also working fine on both CPU and GPU. The >> system is being tested on Ubuntu 12.04 and tested with Python 2.7 >> (though I think should work with other Python 2.x variants). For GPUs, >> I am ensuring that the system works with AMD and Nvidia GPUs. >> >> The compiler is in early stages and I am looking for test cases. The >> project will be open-sourced in November under Apache 2 and thereafter >> will be developed in an open repo. If you have some simple code that I >> can use as a benchmark that I can use to test and evaluate the >> compiler, that will be very helpful. Some annotations will be >> required, which I can help you write. I will be VERY grateful to >> anyone who can provide test cases. In turn, it will help improve the >> compiler and everyone will benefit. >> >> Some of you may be wondering how it compares to Numba. Well it is >> essentially very similar in the idea. So why build a new compiler >> then? Actually the project I am building is not specific to Python. I >> am building a far more general compiler infrastructure for array >> languages, and Python frontend is just one small part of the project. >> For example, I am also working on a MATLAB frontend. >> >> (Some of you may remember me from an earlier compiler project which >> unfortunately went nowhere. This is a different project and this time >> I am determined to convert it into a usable system. I realize the >> proof is in the pudding, so I hope to convince people by releasing >> code soon.) >> >> thanks, >> Rahul >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jason-sage at creativetrax.com Mon Oct 22 11:44:25 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Mon, 22 Oct 2012 10:44:25 -0500 Subject: [Numpy-discussion] matrix norm Message-ID: <508569D9.7010308@creativetrax.com> I'm curious why scipy/numpy defaults to calculating the Frobenius norm for matrices [1], when Matlab, Octave, and Mathematica all default to calculating the induced 2-norm [2]. Is it solely because the Frobenius norm is easier to calculate, or is there some other good mathematical reason for doing things differently? Thanks, Jason [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html [2] * Matlab (http://www.mathworks.com/help/matlab/ref/norm.html). * Octave (http://www.network-theory.co.uk/docs/octave3/octave_198.html). * Mathematica (http://reference.wolfram.com/mathematica/ref/Norm.html) From charlesr.harris at gmail.com Mon Oct 22 11:56:48 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 22 Oct 2012 09:56:48 -0600 Subject: [Numpy-discussion] matrix norm In-Reply-To: <508569D9.7010308@creativetrax.com> References: <508569D9.7010308@creativetrax.com> Message-ID: On Mon, Oct 22, 2012 at 9:44 AM, Jason Grout wrote: > I'm curious why scipy/numpy defaults to calculating the Frobenius norm > for matrices [1], when Matlab, Octave, and Mathematica all default to > calculating the induced 2-norm [2]. Is it solely because the Frobenius > norm is easier to calculate, or is there some other good mathematical > reason for doing things differently? > > Looks to me like Matlab, Octave, and Mathematica all default to the Frobenius norm . Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason-sage at creativetrax.com Mon Oct 22 12:00:22 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Mon, 22 Oct 2012 11:00:22 -0500 Subject: [Numpy-discussion] matrix norm In-Reply-To: References: <508569D9.7010308@creativetrax.com> Message-ID: <50856D96.5000608@creativetrax.com> On 10/22/12 10:56 AM, Charles R Harris wrote: > > > On Mon, Oct 22, 2012 at 9:44 AM, Jason Grout > > wrote: > > I'm curious why scipy/numpy defaults to calculating the Frobenius norm > for matrices [1], when Matlab, Octave, and Mathematica all default to > calculating the induced 2-norm [2]. Is it solely because the Frobenius > norm is easier to calculate, or is there some other good mathematical > reason for doing things differently? > > > Looks to me like Matlab, Octave, and Mathematica all default to the > Frobenius norm . > Am I not reading their docs correctly? * Matlab (http://www.mathworks.com/help/matlab/ref/norm.html). "n = norm(X) is the same as n = norm(X,2)." (and "n = norm(X,2) returns the 2-norm of X.") * Octave (http://www.network-theory.co.uk/docs/octave3/octave_198.html). "Compute the p-norm of the matrix a. If the second argument is missing, p = 2 is assumed." * Mathematica (http://reference.wolfram.com/mathematica/ref/Norm.html) "For matrices, Norm[m] gives the maximum singular value of m." Thanks, Jason From charlesr.harris at gmail.com Mon Oct 22 12:08:46 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 22 Oct 2012 10:08:46 -0600 Subject: [Numpy-discussion] matrix norm In-Reply-To: <50856D96.5000608@creativetrax.com> References: <508569D9.7010308@creativetrax.com> <50856D96.5000608@creativetrax.com> Message-ID: On Mon, Oct 22, 2012 at 10:00 AM, Jason Grout wrote: > On 10/22/12 10:56 AM, Charles R Harris wrote: > > > > > > On Mon, Oct 22, 2012 at 9:44 AM, Jason Grout > > > > wrote: > > > > I'm curious why scipy/numpy defaults to calculating the Frobenius > norm > > for matrices [1], when Matlab, Octave, and Mathematica all default to > > calculating the induced 2-norm [2]. Is it solely because the > Frobenius > > norm is easier to calculate, or is there some other good mathematical > > reason for doing things differently? > > > > > > Looks to me like Matlab, Octave, and Mathematica all default to the > > Frobenius norm . > > > > Am I not reading their docs correctly? > > * Matlab (http://www.mathworks.com/help/matlab/ref/norm.html). > > "n = norm(X) is the same as n = norm(X,2)." (and "n = norm(X,2) returns > the 2-norm of X.") > > * Octave (http://www.network-theory.co.uk/docs/octave3/octave_198.html). > > The 2-norm and the Frobenius norm are the same thing. > "Compute the p-norm of the matrix a. If the second argument is missing, > p = 2 is assumed." > > * Mathematica (http://reference.wolfram.com/mathematica/ref/Norm.html) > > "For matrices, Norm[m] gives the maximum singular value of m." > > OK, looks like Mathematica does return the induced (operator) norm. I didn't see that bit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Mon Oct 22 12:09:24 2012 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 22 Oct 2012 11:09:24 -0500 Subject: [Numpy-discussion] matrix norm In-Reply-To: References: <508569D9.7010308@creativetrax.com> Message-ID: On Mon, Oct 22, 2012 at 10:56 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Mon, Oct 22, 2012 at 9:44 AM, Jason Grout wrote: > >> I'm curious why scipy/numpy defaults to calculating the Frobenius norm >> for matrices [1], when Matlab, Octave, and Mathematica all default to >> calculating the induced 2-norm [2]. Is it solely because the Frobenius >> norm is easier to calculate, or is there some other good mathematical >> reason for doing things differently? >> >> > Looks to me like Matlab, Octave, and Mathematica all default to the > Frobenius norm . > Not octave: octave-3.4.0:26> a = [1 2; 3 4] a = 1 2 3 4 The default norm (for a 2x2 matrix) is the spectral norm: octave-3.4.0:27> norm(a) ans = 5.4650 octave-3.4.0:28> norm(a, 2) ans = 5.4650 octave-3.4.0:29> svd(a)(1) ans = 5.4650 not the Frobenius norm: octave-3.4.0:30> norm(a, 'fro') ans = 5.4772 octave-3.4.0:31> sqrt(sum(sum(a.**2))) ans = 5.4772 Warren > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason-sage at creativetrax.com Mon Oct 22 12:14:53 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Mon, 22 Oct 2012 11:14:53 -0500 Subject: [Numpy-discussion] matrix norm In-Reply-To: References: <508569D9.7010308@creativetrax.com> <50856D96.5000608@creativetrax.com> Message-ID: <508570FD.9060007@creativetrax.com> On 10/22/12 11:08 AM, Charles R Harris wrote: > > The 2-norm and the Frobenius norm are the same thing. For vectors, but I was talking about matrices and induced p-norms (sorry for not being clear). Warren pointed out that the spectral norm (the induced 2-norm) is used in Octave as the default. Does someone have matlab to test their implementation? The fact that matlab has a separate command for the Frobenius norm indicates that they also may be using the spectral norm for the default matrix norm. Thanks, Jason From souheil.inati at nih.gov Mon Oct 22 12:20:11 2012 From: souheil.inati at nih.gov (Inati, Souheil (NIH/NIMH) [E]) Date: Mon, 22 Oct 2012 12:20:11 -0400 Subject: [Numpy-discussion] matrix norm In-Reply-To: <508570FD.9060007@creativetrax.com> References: <508569D9.7010308@creativetrax.com> <50856D96.5000608@creativetrax.com> <508570FD.9060007@creativetrax.com> Message-ID: <3E9FC674-87F7-4F55-9E98-988D267CB058@nih.gov> matlab default is the 2-norm. norm(X) is equivalent to norm(X,2) norm(X,'fro') is the frobenius norm. There are others and some special cases for vectors. http://www.mathworks.com/help/matlab/ref/norm.html On Oct 22, 2012, at 12:14 PM, Jason Grout wrote: > On 10/22/12 11:08 AM, Charles R Harris wrote: > >> >> The 2-norm and the Frobenius norm are the same thing. > > For vectors, but I was talking about matrices and induced p-norms (sorry > for not being clear). Warren pointed out that the spectral norm (the > induced 2-norm) is used in Octave as the default. Does someone have > matlab to test their implementation? The fact that matlab has a > separate command for the Frobenius norm indicates that they also may be > using the spectral norm for the default matrix norm. > > Thanks, > > Jason > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pmhobson at gmail.com Mon Oct 22 12:50:27 2012 From: pmhobson at gmail.com (Paul Hobson) Date: Mon, 22 Oct 2012 09:50:27 -0700 Subject: [Numpy-discussion] Announcing Anaconda version 1.1 In-Reply-To: References: Message-ID: On Fri, Oct 19, 2012 at 9:10 AM, Travis Oliphant wrote: > I just wanted to let everyone know about our new release of Anaconda which > now has Spyder and Matplotlib working for Mac OS X and Windows. > > [snip] > For more information, to download a trial version of Anaconda Pro, or > download the completely free Anaconda CE, click here. Travis, Just downloaded the linux version of this to try out on a server. When I try to execute the bash script, I get the following message: -bash-3.2$ ./AnacondaCE-1.1-linux.sh ERROR: size of AnacondaCE-1.1-linux.sh should be 248021729 bytes And the file actually is... -bash-3.2$ ls -l total 242452 -rwx--x--x 1 pnrfield pnrfield 248017374 Oct 19 16:39 AnacondaCE-1.1-linux.sh Thanks, -paul From pmhobson at gmail.com Mon Oct 22 13:19:31 2012 From: pmhobson at gmail.com (Paul Hobson) Date: Mon, 22 Oct 2012 10:19:31 -0700 Subject: [Numpy-discussion] Announcing Anaconda version 1.1 In-Reply-To: References: Message-ID: Sorry for the noise. Using curl to directly grab the file (instead of copying from another machine) does the trick. -paul On Mon, Oct 22, 2012 at 9:50 AM, Paul Hobson wrote: > On Fri, Oct 19, 2012 at 9:10 AM, Travis Oliphant wrote: >> I just wanted to let everyone know about our new release of Anaconda which >> now has Spyder and Matplotlib working for Mac OS X and Windows. >> >> [snip] >> For more information, to download a trial version of Anaconda Pro, or >> download the completely free Anaconda CE, click here. > > Travis, > > Just downloaded the linux version of this to try out on a server. When > I try to execute the bash script, I get the following message: > > -bash-3.2$ ./AnacondaCE-1.1-linux.sh > ERROR: size of AnacondaCE-1.1-linux.sh should be 248021729 bytes > > And the file actually is... > -bash-3.2$ ls -l > total 242452 > -rwx--x--x 1 pnrfield pnrfield 248017374 Oct 19 16:39 AnacondaCE-1.1-linux.sh > > Thanks, > -paul From jason-sage at creativetrax.com Mon Oct 22 15:04:44 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Mon, 22 Oct 2012 14:04:44 -0500 Subject: [Numpy-discussion] matrix norm In-Reply-To: <508569D9.7010308@creativetrax.com> References: <508569D9.7010308@creativetrax.com> Message-ID: <508598CC.2010307@creativetrax.com> On 10/22/12 10:44 AM, Jason Grout wrote: > I'm curious why scipy/numpy defaults to calculating the Frobenius norm > for matrices [1], when Matlab, Octave, and Mathematica all default to > calculating the induced 2-norm [2]. Is it solely because the Frobenius > norm is easier to calculate, or is there some other good mathematical > reason for doing things differently? I think we've established that the other software mentioned does indeed use the spectral norm by default. I'm still curious: what was the reason for breaking with the norm (pun intended :)? Any chance that in a (probably far distant) future release, the norm default could be changed to conform with matlab/octave/mathematica's view of the world? It's not a huge deal to me now that I know to watch out for it, but it did just bite me and a student a few times. Thanks, Jason From pav at iki.fi Mon Oct 22 16:03:56 2012 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 22 Oct 2012 20:03:56 +0000 (UTC) Subject: [Numpy-discussion] matrix norm References: <508569D9.7010308@creativetrax.com> <508598CC.2010307@creativetrax.com> Message-ID: Jason Grout creativetrax.com> writes: [clip] > I think we've established that the other software mentioned does indeed > use the spectral norm by default. I'm still curious: what was the > reason for breaking with the norm (pun intended :)? Any chance that in > a (probably far distant) future release, the norm default could be > changed to conform with matlab/octave/mathematica's view of the world? > It's not a huge deal to me now that I know to watch out for it, but it > did just bite me and a student a few times. The trail leads to here: http://projects.scipy.org/numpy/attachment/ticket/36/numpy-6-norm-change- default.diff Seems like the chances of learning the reason why this change was done are pretty slim. -- Pauli Virtanen From robert.kern at gmail.com Mon Oct 22 16:08:52 2012 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 22 Oct 2012 21:08:52 +0100 Subject: [Numpy-discussion] matrix norm In-Reply-To: References: <508569D9.7010308@creativetrax.com> <508598CC.2010307@creativetrax.com> Message-ID: On Mon, Oct 22, 2012 at 9:03 PM, Pauli Virtanen wrote: > Jason Grout creativetrax.com> writes: > [clip] >> I think we've established that the other software mentioned does indeed >> use the spectral norm by default. I'm still curious: what was the >> reason for breaking with the norm (pun intended :)? Any chance that in >> a (probably far distant) future release, the norm default could be >> changed to conform with matlab/octave/mathematica's view of the world? >> It's not a huge deal to me now that I know to watch out for it, but it >> did just bite me and a student a few times. > > The trail leads to here: > http://projects.scipy.org/numpy/attachment/ticket/36/numpy-6-norm-change- > default.diff > > Seems like the chances of learning the reason why this change was done > are pretty slim. http://mail.scipy.org/pipermail/numpy-discussion/2006-March/019194.html -- Robert Kern From jason-sage at creativetrax.com Mon Oct 22 16:09:39 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Mon, 22 Oct 2012 15:09:39 -0500 Subject: [Numpy-discussion] matrix norm In-Reply-To: References: <508569D9.7010308@creativetrax.com> <508598CC.2010307@creativetrax.com> Message-ID: <5085A803.3090207@creativetrax.com> On 10/22/12 3:03 PM, Pauli Virtanen wrote: > Jason Grout creativetrax.com> writes: > [clip] >> I think we've established that the other software mentioned does indeed >> use the spectral norm by default. I'm still curious: what was the >> reason for breaking with the norm (pun intended :)? Any chance that in >> a (probably far distant) future release, the norm default could be >> changed to conform with matlab/octave/mathematica's view of the world? >> It's not a huge deal to me now that I know to watch out for it, but it >> did just bite me and a student a few times. > > The trail leads to here: > http://projects.scipy.org/numpy/attachment/ticket/36/numpy-6-norm-change- > default.diff > > Seems like the chances of learning the reason why this change was done > are pretty slim. Thanks for tracking this down. It looks like the default used to be the spectral norm. I guess there still is a question of if it is possible to change it back at this point. FYI, in Sage, we were following the numpy convention (default = Frobenius norm) for numerical matrices, but we are now switching the default to be the spectral norm [1]. Thanks, Jason [1] http://trac.sagemath.org/sage_trac/ticket/13643 From jason-sage at creativetrax.com Mon Oct 22 16:16:30 2012 From: jason-sage at creativetrax.com (Jason Grout) Date: Mon, 22 Oct 2012 15:16:30 -0500 Subject: [Numpy-discussion] matrix norm In-Reply-To: References: <508569D9.7010308@creativetrax.com> <508598CC.2010307@creativetrax.com> Message-ID: <5085A99E.8020509@creativetrax.com> On 10/22/12 3:08 PM, Robert Kern wrote: > > http://mail.scipy.org/pipermail/numpy-discussion/2006-March/019194.html > Ah, so it was basically speed that was the issue. I won't push this further. I'll just note that I was confused for a bit, and I probably won't be the last person confused about it, given the conventions in the other software packages. Jason From davidmenhur at gmail.com Mon Oct 22 16:23:51 2012 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Mon, 22 Oct 2012 22:23:51 +0200 Subject: [Numpy-discussion] matrix norm In-Reply-To: <5085A99E.8020509@creativetrax.com> References: <508569D9.7010308@creativetrax.com> <508598CC.2010307@creativetrax.com> <5085A99E.8020509@creativetrax.com> Message-ID: In this case, I would say a note in the documentation would be in order, remarking the fact that this default is not what other packages take as default. On Mon, Oct 22, 2012 at 10:16 PM, Jason Grout wrote: > On 10/22/12 3:08 PM, Robert Kern wrote: >> >> http://mail.scipy.org/pipermail/numpy-discussion/2006-March/019194.html >> > > Ah, so it was basically speed that was the issue. > > I won't push this further. I'll just note that I was confused for a > bit, and I probably won't be the last person confused about it, given > the conventions in the other software packages. > > Jason > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From thouis at gmail.com Mon Oct 22 23:05:24 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Mon, 22 Oct 2012 23:05:24 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Fri, Oct 19, 2012 at 9:34 PM, Thouis (Ray) Jones wrote: > On Fri, Oct 19, 2012 at 4:46 PM, Thouis (Ray) Jones wrote: >> On Fri, Oct 19, 2012 at 11:20 AM, Thouis (Ray) Jones wrote: >>> I started the import with the oldest 75 and newest 125 Trac issues, >>> and will wait a few hours to do the rest to allow feedback, just in >>> case something is broken that I haven't noticed. >>> >>> I did make one change to better emulate Trac behavior. Some Trac >>> usernames are also email addresses, which Trac anonymizes in its >>> display. I decided it was safer to do the same. >> >> The import is running again, though I've been having some failures in >> a few comments and general hangs (these might be network related). >> I'm keeping track of which issues might have had difficulties. >> >> @endolith noticed that I didn't correctly relink #XXX trac id numbers >> to github id numbers (both trac and github create links >> automatically), so that will have to be handled by a postprocessing >> script (which it probably would have, anyway, since the github # isn't >> known before import). > > Import has finished. > > The following trac #s had issues in creating the comments (I think due > to network problems): 182, 297, 619, 621, 902, 904, 909 913, 914, 915, > 1044, 1526. I'll review them and see if I can pull in anything > missing > > I'll also work on a script for updating the trac crossrefs to github crossrefs. > > In the "no good deed goes unpunished" category, I accidentally logged > in as myself (rather than numpy-gitbot) and pushed about 500 issues, > so now I receive updates whenever one of them gets changed. At least > most of them were closed, already... I just updated the cross-issue-references to use github rather than Trac id numbers. Stupidly, I may have accidentally removed comments that were added in the last few days to issues moved from trac to github. Hopefully not, or at least not many. It's probably a good idea to turn off Trac, soon, to keep too many new bugs from needing to be ported, and old bugs being commented on. The latter is more of a pain to deal with. Ray From m.schultz at fz-juelich.de Tue Oct 23 02:35:42 2012 From: m.schultz at fz-juelich.de (maschu) Date: Mon, 22 Oct 2012 23:35:42 -0700 (PDT) Subject: [Numpy-discussion] re carray from list of lists Message-ID: <34590361.post@talk.nabble.com> Dear all, I am new to this list and still consider myself a python newby even though I have worked with it for almost two years now. My question may be a bit academic (because there is numpy.genfromtext and matplotlib.mlab.csv2rec), yet it caused me quite a bit of searching before I found out how to convert a list of lists into a recarray. Here is an example code: ----- start of code ---------------------------- import numpy as np import csv reader=csv.reader(open("csvfile.csv","rb"),delimiter=';') x=list(reader) # array of string arrays h=x.pop(0) # first line has column headers # convert data to recarray (must make list of tuples!) xx=[tuple(elem) for elem in x] z=np.dtype(('S16, f8, f4, f4')) res=np.array(xx,dtype=z) res.dtype.names=h # set header names ----- end of code ---------------------------- Doesn't this consume too much memory? This first creates the list of lists as x, then converts this into a list of tuples xx, and then forms a recarray res from xx. Why is there the limitation that a recarray must be given a list of tuples instead of a list of lists (which comes out of the csv.reader, for example)? Thanks for any hints to make this more efficient or explanations why these things are done the way they are done. Martin -- View this message in context: http://old.nabble.com/recarray-from-list-of-lists-tp34590361p34590361.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From cournape at gmail.com Tue Oct 23 10:58:42 2012 From: cournape at gmail.com (David Cournapeau) Date: Tue, 23 Oct 2012 16:58:42 +0200 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Tue, Oct 23, 2012 at 5:05 AM, Thouis (Ray) Jones wrote: > On Fri, Oct 19, 2012 at 9:34 PM, Thouis (Ray) Jones wrote: >> On Fri, Oct 19, 2012 at 4:46 PM, Thouis (Ray) Jones wrote: >>> On Fri, Oct 19, 2012 at 11:20 AM, Thouis (Ray) Jones wrote: >>>> I started the import with the oldest 75 and newest 125 Trac issues, >>>> and will wait a few hours to do the rest to allow feedback, just in >>>> case something is broken that I haven't noticed. >>>> >>>> I did make one change to better emulate Trac behavior. Some Trac >>>> usernames are also email addresses, which Trac anonymizes in its >>>> display. I decided it was safer to do the same. >>> >>> The import is running again, though I've been having some failures in >>> a few comments and general hangs (these might be network related). >>> I'm keeping track of which issues might have had difficulties. >>> >>> @endolith noticed that I didn't correctly relink #XXX trac id numbers >>> to github id numbers (both trac and github create links >>> automatically), so that will have to be handled by a postprocessing >>> script (which it probably would have, anyway, since the github # isn't >>> known before import). >> >> Import has finished. >> >> The following trac #s had issues in creating the comments (I think due >> to network problems): 182, 297, 619, 621, 902, 904, 909 913, 914, 915, >> 1044, 1526. I'll review them and see if I can pull in anything >> missing >> >> I'll also work on a script for updating the trac crossrefs to github crossrefs. >> >> In the "no good deed goes unpunished" category, I accidentally logged >> in as myself (rather than numpy-gitbot) and pushed about 500 issues, >> so now I receive updates whenever one of them gets changed. At least >> most of them were closed, already... > > I just updated the cross-issue-references to use github rather than > Trac id numbers. Stupidly, I may have accidentally removed comments > that were added in the last few days to issues moved from trac to > github. Hopefully not, or at least not many. > > It's probably a good idea to turn off Trac, soon, to keep too many new > bugs from needing to be ported, and old bugs being commented on. The > latter is more of a pain to deal with. I will look into making the NumPy trac read-only. It should not be too complicated to extend Pauli's code to redirect the tickets part to github issues. Have we decided what to do with the wiki content ? David From travis at continuum.io Tue Oct 23 11:13:34 2012 From: travis at continuum.io (Travis Oliphant) Date: Tue, 23 Oct 2012 10:13:34 -0500 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: <5E3DAC7D-5B5C-4713-BA96-1FC663D3C8A4@continuum.io> On Oct 23, 2012, at 9:58 AM, David Cournapeau wrote: > On Tue, Oct 23, 2012 at 5:05 AM, Thouis (Ray) Jones wrote: >> On Fri, Oct 19, 2012 at 9:34 PM, Thouis (Ray) Jones wrote: >>> On Fri, Oct 19, 2012 at 4:46 PM, Thouis (Ray) Jones wrote: >>>> On Fri, Oct 19, 2012 at 11:20 AM, Thouis (Ray) Jones wrote: >>>>> I started the import with the oldest 75 and newest 125 Trac issues, >>>>> and will wait a few hours to do the rest to allow feedback, just in >>>>> case something is broken that I haven't noticed. >>>>> >>>>> I did make one change to better emulate Trac behavior. Some Trac >>>>> usernames are also email addresses, which Trac anonymizes in its >>>>> display. I decided it was safer to do the same. >>>> >>>> The import is running again, though I've been having some failures in >>>> a few comments and general hangs (these might be network related). >>>> I'm keeping track of which issues might have had difficulties. >>>> >>>> @endolith noticed that I didn't correctly relink #XXX trac id numbers >>>> to github id numbers (both trac and github create links >>>> automatically), so that will have to be handled by a postprocessing >>>> script (which it probably would have, anyway, since the github # isn't >>>> known before import). >>> >>> Import has finished. >>> >>> The following trac #s had issues in creating the comments (I think due >>> to network problems): 182, 297, 619, 621, 902, 904, 909 913, 914, 915, >>> 1044, 1526. I'll review them and see if I can pull in anything >>> missing >>> >>> I'll also work on a script for updating the trac crossrefs to github crossrefs. >>> >>> In the "no good deed goes unpunished" category, I accidentally logged >>> in as myself (rather than numpy-gitbot) and pushed about 500 issues, >>> so now I receive updates whenever one of them gets changed. At least >>> most of them were closed, already... >> >> I just updated the cross-issue-references to use github rather than >> Trac id numbers. Stupidly, I may have accidentally removed comments >> that were added in the last few days to issues moved from trac to >> github. Hopefully not, or at least not many. >> >> It's probably a good idea to turn off Trac, soon, to keep too many new >> bugs from needing to be ported, and old bugs being commented on. The >> latter is more of a pain to deal with. > > I will look into making the NumPy trac read-only. It should not be too > complicated to extend Pauli's code to redirect the tickets part to > github issues. > > Have we decided what to do with the wiki content ? > I believe there is a wiki dump command in trac wiki. We should put that content linked off the numpy pages at github. Thanks for helping with this. -Travis > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From nouiz at nouiz.org Tue Oct 23 11:41:11 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Tue, 23 Oct 2012 11:41:11 -0400 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: References: Message-ID: Did you saw the gpu nd array project? We try to do something similar but only for the GPU. https://github.com/inducer/compyte/wiki Fred On Sun, Oct 21, 2012 at 2:57 PM, Rahul Garg wrote: > Thanks! I need to add support for eig and inv (will do this week, at > least for CPU) but other than that, I should definitely be able to > handle those kinds of benchmarks. > > rahul > > On Sun, Oct 21, 2012 at 12:01 PM, Aron Ahmadia wrote: >> Hi Rahul, >> >> Very cool! I'm looking forward to seeing some performance results! Anders >> Logg posted a computational challenge to G+ about a month ago, and we got >> entries in Octave, Fortran, Python, and Julia (all implementing the same >> solution from Jed Brown). The challenge is here: >> >> https://plus.google.com/116518787475147930287/posts/jiULACjiGnW >> >> Here is my simple attempt at Cythonizing Jed's Octave code: >> >> https://gist.github.com/3893361 >> >> The best solution in Fortran took 38 microseconds. The best Python solution >> clocked in at around 445. The Julia solution implemented by Jed took around >> 224 microseconds, a good LLVM solution should come close to or beat that. >> >> Hope this helps. >> >> Aron >> >> On Sun, Oct 21, 2012 at 3:27 PM, Rahul Garg wrote: >>> >>> Hi. >>> >>> I am a PhD student at McGill University and I am developing a compiler >>> for Python for CPUs and GPUs. For CPUs, I build upon LLVM. For GPUs, I >>> generate OpenCL and I have also implemented some library functions on >>> the GPU myself. The restriction that it is only for numerical code and >>> intended for NumPy users. The compiler is aware of simple things in >>> NumPy like matrix multiplication, slicing operators, strided layouts, >>> some library functions (though limited at this time) and the negative >>> indexing semantics etc. However, the compiler is not limited to vector >>> code. Scalar code or manually written loops also work. However, only >>> numerical datatypes are supported with no support for lists, dicts, >>> classes etc. First class functions are not currently supported but are >>> on the roadmap. You will have to add some type annotations to your >>> functions. If you have a compatible GPU, you can also use the GPU by >>> indicating which parts to run on the GPU. Otherwise you can just use >>> it to run your code on the CPU. >>> >>> As an example, simple scalar code like fibonacci function works fine. >>> Simple loops like those used in stencil-type computations are also >>> working. Parallel-for loops are also provided and working. Simple >>> vector oriented code is also working fine on both CPU and GPU. The >>> system is being tested on Ubuntu 12.04 and tested with Python 2.7 >>> (though I think should work with other Python 2.x variants). For GPUs, >>> I am ensuring that the system works with AMD and Nvidia GPUs. >>> >>> The compiler is in early stages and I am looking for test cases. The >>> project will be open-sourced in November under Apache 2 and thereafter >>> will be developed in an open repo. If you have some simple code that I >>> can use as a benchmark that I can use to test and evaluate the >>> compiler, that will be very helpful. Some annotations will be >>> required, which I can help you write. I will be VERY grateful to >>> anyone who can provide test cases. In turn, it will help improve the >>> compiler and everyone will benefit. >>> >>> Some of you may be wondering how it compares to Numba. Well it is >>> essentially very similar in the idea. So why build a new compiler >>> then? Actually the project I am building is not specific to Python. I >>> am building a far more general compiler infrastructure for array >>> languages, and Python frontend is just one small part of the project. >>> For example, I am also working on a MATLAB frontend. >>> >>> (Some of you may remember me from an earlier compiler project which >>> unfortunately went nowhere. This is a different project and this time >>> I am determined to convert it into a usable system. I realize the >>> proof is in the pudding, so I hope to convince people by releasing >>> code soon.) >>> >>> thanks, >>> Rahul >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From heng at cantab.net Tue Oct 23 11:48:14 2012 From: heng at cantab.net (Henry Gomersall) Date: Tue, 23 Oct 2012 16:48:14 +0100 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: References: Message-ID: <1351007294.3585.5.camel@farnsworth> On Tue, 2012-10-23 at 11:41 -0400, Fr?d?ric Bastien wrote: > Did you saw the gpu nd array project? We try to do something similar > but only for the GPU. > Out of interest, is there a reason why the backend for Numpy could not be written entirely in OpenCL? Assuming of course all the relevant backends are up to scratch. Is there a fundamental reason why targetting a CPU through OpenCL is worse than doing it exclusively through C or C++? Henry From rahulgarg44 at gmail.com Tue Oct 23 11:59:00 2012 From: rahulgarg44 at gmail.com (Rahul Garg) Date: Tue, 23 Oct 2012 11:59:00 -0400 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: References: Message-ID: Thanks! I did not know about the project. Looking into it. rahul On Tue, Oct 23, 2012 at 11:41 AM, Fr?d?ric Bastien wrote: > Did you saw the gpu nd array project? We try to do something similar > but only for the GPU. > > https://github.com/inducer/compyte/wiki > > Fred > > On Sun, Oct 21, 2012 at 2:57 PM, Rahul Garg wrote: >> Thanks! I need to add support for eig and inv (will do this week, at >> least for CPU) but other than that, I should definitely be able to >> handle those kinds of benchmarks. >> >> rahul >> >> On Sun, Oct 21, 2012 at 12:01 PM, Aron Ahmadia wrote: >>> Hi Rahul, >>> >>> Very cool! I'm looking forward to seeing some performance results! Anders >>> Logg posted a computational challenge to G+ about a month ago, and we got >>> entries in Octave, Fortran, Python, and Julia (all implementing the same >>> solution from Jed Brown). The challenge is here: >>> >>> https://plus.google.com/116518787475147930287/posts/jiULACjiGnW >>> >>> Here is my simple attempt at Cythonizing Jed's Octave code: >>> >>> https://gist.github.com/3893361 >>> >>> The best solution in Fortran took 38 microseconds. The best Python solution >>> clocked in at around 445. The Julia solution implemented by Jed took around >>> 224 microseconds, a good LLVM solution should come close to or beat that. >>> >>> Hope this helps. >>> >>> Aron >>> >>> On Sun, Oct 21, 2012 at 3:27 PM, Rahul Garg wrote: >>>> >>>> Hi. >>>> >>>> I am a PhD student at McGill University and I am developing a compiler >>>> for Python for CPUs and GPUs. For CPUs, I build upon LLVM. For GPUs, I >>>> generate OpenCL and I have also implemented some library functions on >>>> the GPU myself. The restriction that it is only for numerical code and >>>> intended for NumPy users. The compiler is aware of simple things in >>>> NumPy like matrix multiplication, slicing operators, strided layouts, >>>> some library functions (though limited at this time) and the negative >>>> indexing semantics etc. However, the compiler is not limited to vector >>>> code. Scalar code or manually written loops also work. However, only >>>> numerical datatypes are supported with no support for lists, dicts, >>>> classes etc. First class functions are not currently supported but are >>>> on the roadmap. You will have to add some type annotations to your >>>> functions. If you have a compatible GPU, you can also use the GPU by >>>> indicating which parts to run on the GPU. Otherwise you can just use >>>> it to run your code on the CPU. >>>> >>>> As an example, simple scalar code like fibonacci function works fine. >>>> Simple loops like those used in stencil-type computations are also >>>> working. Parallel-for loops are also provided and working. Simple >>>> vector oriented code is also working fine on both CPU and GPU. The >>>> system is being tested on Ubuntu 12.04 and tested with Python 2.7 >>>> (though I think should work with other Python 2.x variants). For GPUs, >>>> I am ensuring that the system works with AMD and Nvidia GPUs. >>>> >>>> The compiler is in early stages and I am looking for test cases. The >>>> project will be open-sourced in November under Apache 2 and thereafter >>>> will be developed in an open repo. If you have some simple code that I >>>> can use as a benchmark that I can use to test and evaluate the >>>> compiler, that will be very helpful. Some annotations will be >>>> required, which I can help you write. I will be VERY grateful to >>>> anyone who can provide test cases. In turn, it will help improve the >>>> compiler and everyone will benefit. >>>> >>>> Some of you may be wondering how it compares to Numba. Well it is >>>> essentially very similar in the idea. So why build a new compiler >>>> then? Actually the project I am building is not specific to Python. I >>>> am building a far more general compiler infrastructure for array >>>> languages, and Python frontend is just one small part of the project. >>>> For example, I am also working on a MATLAB frontend. >>>> >>>> (Some of you may remember me from an earlier compiler project which >>>> unfortunately went nowhere. This is a different project and this time >>>> I am determined to convert it into a usable system. I realize the >>>> proof is in the pudding, so I hope to convince people by releasing >>>> code soon.) >>>> >>>> thanks, >>>> Rahul >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ralf.gommers at gmail.com Tue Oct 23 12:23:29 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 23 Oct 2012 18:23:29 +0200 Subject: [Numpy-discussion] Issue tracking In-Reply-To: <5E3DAC7D-5B5C-4713-BA96-1FC663D3C8A4@continuum.io> References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> <5E3DAC7D-5B5C-4713-BA96-1FC663D3C8A4@continuum.io> Message-ID: On Tue, Oct 23, 2012 at 5:13 PM, Travis Oliphant wrote: > > On Oct 23, 2012, at 9:58 AM, David Cournapeau wrote: > > > On Tue, Oct 23, 2012 at 5:05 AM, Thouis (Ray) Jones > wrote: > >> On Fri, Oct 19, 2012 at 9:34 PM, Thouis (Ray) Jones > wrote: > >>> On Fri, Oct 19, 2012 at 4:46 PM, Thouis (Ray) Jones > wrote: > >>>> On Fri, Oct 19, 2012 at 11:20 AM, Thouis (Ray) Jones < > thouis at gmail.com> wrote: > >>>>> I started the import with the oldest 75 and newest 125 Trac issues, > >>>>> and will wait a few hours to do the rest to allow feedback, just in > >>>>> case something is broken that I haven't noticed. > >>>>> > >>>>> I did make one change to better emulate Trac behavior. Some Trac > >>>>> usernames are also email addresses, which Trac anonymizes in its > >>>>> display. I decided it was safer to do the same. > >>>> > >>>> The import is running again, though I've been having some failures in > >>>> a few comments and general hangs (these might be network related). > >>>> I'm keeping track of which issues might have had difficulties. > >>>> > >>>> @endolith noticed that I didn't correctly relink #XXX trac id numbers > >>>> to github id numbers (both trac and github create links > >>>> automatically), so that will have to be handled by a postprocessing > >>>> script (which it probably would have, anyway, since the github # isn't > >>>> known before import). > >>> > >>> Import has finished. > >>> > >>> The following trac #s had issues in creating the comments (I think due > >>> to network problems): 182, 297, 619, 621, 902, 904, 909 913, 914, 915, > >>> 1044, 1526. I'll review them and see if I can pull in anything > >>> missing > >>> > >>> I'll also work on a script for updating the trac crossrefs to github > crossrefs. > >>> > >>> In the "no good deed goes unpunished" category, I accidentally logged > >>> in as myself (rather than numpy-gitbot) and pushed about 500 issues, > >>> so now I receive updates whenever one of them gets changed. At least > >>> most of them were closed, already... > >> > >> I just updated the cross-issue-references to use github rather than > >> Trac id numbers. Stupidly, I may have accidentally removed comments > >> that were added in the last few days to issues moved from trac to > >> github. Hopefully not, or at least not many. > >> > >> It's probably a good idea to turn off Trac, soon, to keep too many new > >> bugs from needing to be ported, and old bugs being commented on. The > >> latter is more of a pain to deal with. > > > > I will look into making the NumPy trac read-only. It should not be too > > complicated to extend Pauli's code to redirect the tickets part to > > github issues. > > > > Have we decided what to do with the wiki content ? > > > > I believe there is a wiki dump command in trac wiki. We should put that > content linked off the numpy pages at github. > Please don't do that. Most of the content is either links to what was already moved into the numpy git repo, or very outdated stuff. I see at least two better options: - just leave the wiki pages as is, make them read only and add a clear warning "outdated" at the top. - move the rest of the useful content into the git repo, remove the rest Ralf > > Thanks for helping with this. > > -Travis > > > > > > David > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim at cerazone.net Tue Oct 23 13:11:44 2012 From: tim at cerazone.net (Cera, Tim) Date: Tue, 23 Oct 2012 13:11:44 -0400 Subject: [Numpy-discussion] Is there a way to reset an accumulate function? Message-ID: I have an array that is peppered throughout in random spots with 'nan'. I would like to use 'cumsum', but I want it to reset the accumulation to 0 whenever a 'nan' is encountered. Is there a way to do this? Aside from a loop - which is what I am going to setup here in a moment. Kindest regards, Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Oct 23 13:24:28 2012 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 23 Oct 2012 18:24:28 +0100 Subject: [Numpy-discussion] Is there a way to reset an accumulate function? In-Reply-To: References: Message-ID: There's nothing like that built into numpy, no. I guess you could use reduceat to calculate the total for each span of non-nans and then replace each nan with the negative of that value so that plain cumsum would work, but a loop is going to be easier (and a loop in Cython will be faster). -n On 23 Oct 2012 18:11, "Cera, Tim" wrote: > I have an array that is peppered throughout in random spots with 'nan'. I > would like to use 'cumsum', but I want it to reset the accumulation to 0 > whenever a 'nan' is encountered. Is there a way to do this? Aside from a > loop - which is what I am going to setup here in a moment. > > Kindest regards, > Tim > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Tue Oct 23 13:19:27 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Tue, 23 Oct 2012 13:19:27 -0400 Subject: [Numpy-discussion] Is there a way to reset an accumulate function? In-Reply-To: References: Message-ID: Hi, Why not start conting from the end of the vector until you find a nan? Your problem do not need to check the full vector. Fred On Tue, Oct 23, 2012 at 1:11 PM, Cera, Tim wrote: > I have an array that is peppered throughout in random spots with 'nan'. I > would like to use 'cumsum', but I want it to reset the accumulation to 0 > whenever a 'nan' is encountered. Is there a way to do this? Aside from a > loop - which is what I am going to setup here in a moment. > > Kindest regards, > Tim > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From thouis at gmail.com Tue Oct 23 14:57:34 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Tue, 23 Oct 2012 14:57:34 -0400 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Tue, Oct 23, 2012 at 10:58 AM, David Cournapeau wrote: > I will look into making the NumPy trac read-only. It should not be too > complicated to extend Pauli's code to redirect the tickets part to > github issues. If you need the map of trac IDs to github IDs, I have code to grab that. Ray From amcmorl at gmail.com Tue Oct 23 15:12:26 2012 From: amcmorl at gmail.com (Angus McMorland) Date: Tue, 23 Oct 2012 15:12:26 -0400 Subject: [Numpy-discussion] Is there a way to reset an accumulate function? In-Reply-To: References: Message-ID: On 23 October 2012 13:11, Cera, Tim wrote: > I have an array that is peppered throughout in random spots with 'nan'. I > would like to use 'cumsum', but I want it to reset the accumulation to 0 > whenever a 'nan' is encountered. Is there a way to do this? Aside from a > loop - which is what I am going to setup here in a moment. How about this hackish solution, for a quick non-looping fix? In [39]: a = np.array([1,2,3,4,np.nan,1,2,3,np.nan,3]) idx = np.flatnonzero(np.isnan(a)) a_ = a.copy() a_[idx] = 0 np.add.reduceat(a_, np.hstack((0,idx))) Out[39]: array([ 10., 6., 3.]) Note that if the last element of a is nan, you get a final 0 in the result. Angus -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From cournape at gmail.com Tue Oct 23 15:57:11 2012 From: cournape at gmail.com (David Cournapeau) Date: Tue, 23 Oct 2012 21:57:11 +0200 Subject: [Numpy-discussion] Issue tracking In-Reply-To: References: <3C64F2BE-50E5-403C-9022-71233A6E3449@continuum.io> Message-ID: On Tue, Oct 23, 2012 at 8:57 PM, Thouis (Ray) Jones wrote: > On Tue, Oct 23, 2012 at 10:58 AM, David Cournapeau wrote: >> I will look into making the NumPy trac read-only. It should not be too >> complicated to extend Pauli's code to redirect the tickets part to >> github issues. > > If you need the map of trac IDs to github IDs, I have code to grab that. Oh, I meant something much simpler: just redirect the view issues link into GH :) David From tim at cerazone.net Tue Oct 23 20:04:07 2012 From: tim at cerazone.net (Cera, Tim) Date: Tue, 23 Oct 2012 20:04:07 -0400 Subject: [Numpy-discussion] Is there a way to reset an accumulate function? In-Reply-To: References: Message-ID: > How about this hackish solution, for a quick non-looping fix? > > In [39]: a = np.array([1,2,3,4,np.nan,1,2,3,np.nan,3]) > idx = np.flatnonzero(np.isnan(a)) > a_ = a.copy() > a_[idx] = 0 > np.add.reduceat(a_, np.hstack((0,idx))) > Out[39]: array([ 10., 6., 3.]) > Close, but not exactly what I need. I want the 'cumsum', so given the 'a' in your example: array([1,2,6,10,0,1,3,6,0,3]) I just made a loop, testing for 'nan'. Not elegant, but it works so I am not complaining. Kindest regards, Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Oct 24 04:47:51 2012 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Oct 2012 09:47:51 +0100 Subject: [Numpy-discussion] Is there a way to reset an accumulate function? In-Reply-To: References: Message-ID: On Tue, Oct 23, 2012 at 6:11 PM, Cera, Tim wrote: > I have an array that is peppered throughout in random spots with 'nan'. I > would like to use 'cumsum', but I want it to reset the accumulation to 0 > whenever a 'nan' is encountered. Is there a way to do this? Aside from a > loop - which is what I am going to setup here in a moment. How about this? def nancumsum(x): nans = np.isnan(x) x = np.array(x) x[nans] = 0 reset_idx = np.zeros(len(x), dtype=int) reset_idx[nans] = np.arange(len(x))[nans] reset_idx = np.maximum.accumulate(reset_idx) cumsum = np.cumsum(x) cumsum = cumsum - cumsum[reset_idx] return cumsum -- Robert Kern From gnurser at gmail.com Wed Oct 24 07:18:17 2012 From: gnurser at gmail.com (George Nurser) Date: Wed, 24 Oct 2012 12:18:17 +0100 Subject: [Numpy-discussion] einsum slow vs (tensor)dot Message-ID: Hi, I was just looking at the einsum function. To me, it's a really elegant and clear way of doing array operations, which is the core of what numpy is about. It removes the need to remember a range of functions, some of which I find tricky (e.g. tile). Unfortunately the present implementation seems ~ 4-6x slower than dot or tensordot for decent size arrays. I suspect it is because the implementation does not use blas/lapack calls. cheers, George Nurser. E.g. (in ipython on Mac OS X 10.6, python 2.7.3, numpy 1.6.2 from macports) a = np.arange(600000.).reshape(1500,400) b = np.arange(240000.).reshape(400,600) c = np.arange(600) d = np.arange(400) %timeit np.einsum('ij,jk', a, b) 10 loops, best of 3: 156 ms per loop %timeit np.dot(a,b) 10 loops, best of 3: 27.4 ms per loop %timeit np.einsum('i,ij,j',d,b,c) 1000 loops, best of 3: 709 us per loop %timeit np.dot(d,np.dot(b,c)) 10000 loops, best of 3: 121 us per loop or abig = np.arange(4800.).reshape(6,8,100) bbig = np.arange(1920.).reshape(8,6,40) %timeit np.einsum('ijk,jil->kl', abig, bbig) 1000 loops, best of 3: 425 us per loop %timeit np.tensordot(abig,bbig, axes=([1,0],[0,1])) 10000 loops, best of 3: 105 us per loop -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis-bz-gg at t-online.de Wed Oct 24 13:33:18 2012 From: denis-bz-gg at t-online.de (denis) Date: Wed, 24 Oct 2012 19:33:18 +0200 Subject: [Numpy-discussion] np.linalg.lstsq with several columns all 0 => huge x ? Message-ID: Folks, np.linalg.lstsq of a random-uniform A 50 x 32 with 3 columns all 0 returns x[:3] 0 as expected, but 4 columns all 0 => huge x: lstsq (50, 32) with 4 columns all 0: [ -3.7e+09 -3.6e+13 -1.9e+13 -2.9e+12 7.3e-01 ... This may be a roundoff problem, or even a Mac Altivec lapack bug, not worth looking into. linalg.svd is ok though, odd. Summary: if you run linalg.lstsq on big arrays, either check max |x| or regularize, do lstsq( vstack( A, weight * eye(dim) ), hstack( b, zeros(dim) )) cheers -- denis From josef.pktd at gmail.com Wed Oct 24 13:59:37 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 24 Oct 2012 13:59:37 -0400 Subject: [Numpy-discussion] np.linalg.lstsq with several columns all 0 => huge x ? In-Reply-To: References: Message-ID: On Wed, Oct 24, 2012 at 1:33 PM, denis wrote: > Folks, > np.linalg.lstsq of a random-uniform A 50 x 32 with 3 columns all 0 > returns x[:3] 0 as expected, > but 4 columns all 0 => huge x: > lstsq (50, 32) with 4 columns all 0: > [ -3.7e+09 -3.6e+13 -1.9e+13 -2.9e+12 7.3e-01 ... > > This may be a roundoff problem, or even a Mac Altivec lapack bug, > not worth looking into. linalg.svd is ok though, odd. > > Summary: if you run linalg.lstsq on big arrays, > either check max |x| > or regularize, do lstsq( vstack( A, weight * eye(dim) ), > hstack( b, zeros(dim) )) lstsq has rcond argument to do (I think) essentially the same. might need to be increased in your example. Josef > > cheers > -- denis > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From kmichael.aye at gmail.com Wed Oct 24 15:00:27 2012 From: kmichael.aye at gmail.com (Michael Aye) Date: Wed, 24 Oct 2012 12:00:27 -0700 Subject: [Numpy-discussion] how to pipe into numpy arrays? Message-ID: As numpy.fromfile seems to require full file object functionalities like seek, I can not use it with the sys.stdin pipe. So how could I stream a binary pipe directly into numpy? I can imagine storing the data in a string and use StringIO but the files are 3.6 GB large, just the binary, and that will most likely be much more as a string object. Reading binary files on disk is NOT the problem, I would like to avoid the temporary file if possible. From ben.root at ou.edu Wed Oct 24 21:19:03 2012 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 24 Oct 2012 21:19:03 -0400 Subject: [Numpy-discussion] how to pipe into numpy arrays? In-Reply-To: References: Message-ID: On Wed, Oct 24, 2012 at 3:00 PM, Michael Aye wrote: > As numpy.fromfile seems to require full file object functionalities > like seek, I can not use it with the sys.stdin pipe. > So how could I stream a binary pipe directly into numpy? > I can imagine storing the data in a string and use StringIO but the > files are 3.6 GB large, just the binary, and that will most likely be > much more as a string object. > Reading binary files on disk is NOT the problem, I would like to avoid > the temporary file if possible. > > I haven't tried this myself, but there is a numpy.frombuffer() function as well. Maybe that could be used here? Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim at cerazone.net Wed Oct 24 21:24:03 2012 From: tim at cerazone.net (Cera, Tim) Date: Wed, 24 Oct 2012 21:24:03 -0400 Subject: [Numpy-discussion] Is there a way to reset an accumulate function? In-Reply-To: References: Message-ID: On Wed, Oct 24, 2012 at 4:47 AM, Robert Kern wrote: > How about this? > > > def nancumsum(x): > nans = np.isnan(x) > x = np.array(x) > x[nans] = 0 > reset_idx = np.zeros(len(x), dtype=int) > reset_idx[nans] = np.arange(len(x))[nans] > reset_idx = np.maximum.accumulate(reset_idx) > cumsum = np.cumsum(x) > cumsum = cumsum - cumsum[reset_idx] > return cumsum > Thank you for putting in the time to look at this. It doesn't work for the first group of numbers if x[0] is non-zero. Could perhaps concatenate a np.nan at the beginning to force a reset and adjust the returned array to not include the dummy value... def nancumsum(x): x = np.concatenate(([np.nan], x)) nans = np.isnan(x) x = np.array(x) x[nans] = 0 reset_idx = np.zeros(len(x), dtype=int) reset_idx[nans] = np.arange(len(x))[nans] reset_idx = np.maximum.accumulate(reset_idx) cumsum = np.cumsum(x) cumsum = cumsum - cumsum[reset_idx] return cumsum[1:] >>> a array([ 4., 1., 2., 0., 18., 5., 6., 0., 8., 9.], dtype=float32) If no np.nan, then 'nancumsum' and 'np.cumsum' should be the same... >>> np.cumsum(a) array([ 4., 5., 7., 7., 25., 30., 36., 36., 44., 53.], dtype=float32) >>> nancumsum(a) array([ 4., 5., 7., 7., 25., 30., 36., 36., 44., 53.]) >>> a[3] = np.nan >>> np.cumsum(a) array([ 4., 5., 7., nan, nan, nan, nan, nan, nan, nan], dtype=float32) >>> nancumsum(a) array([ 4., 5., 7., 0., 18., 23., 29., 29., 37., 46.]) Excellent! Kindest regards, Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From dtustudy68 at hotmail.com Thu Oct 25 00:01:47 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Wed, 24 Oct 2012 22:01:47 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. Message-ID: Hi, All, I am trying to install numpy from http://www.scipy.org/Download . by git clone git://github.com/numpy/numpy.git numpy But, when I ran python setup.py install I got: SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel Where to get python-dev ? I tried: $ easy_install python-develSearching for python-develReading http://pypi.python.org/simple/python-devel/Couldn't find index page for 'python-devel' (maybe misspelled?)Scanning index of all packages (this may take a while)Reading http://pypi.python.org/simple/No local packages or download links found for python-develerror: Could not find suitable distribution for Requirement.parse('python-devel') and $ easy_install python-devSearching for python-devReading http://pypi.python.org/simple/python-dev/Couldn't find index page for 'python-dev' (maybe misspelled?)Scanning index of all packages (this may take a while)Reading http://pypi.python.org/simple/No local packages or download links found for python-deverror: Could not find suitable distribution for Requirement.parse('python-dev') -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Oct 25 02:05:47 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 25 Oct 2012 08:05:47 +0200 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: Message-ID: On Thu, Oct 25, 2012 at 6:01 AM, Jack Bryan wrote: > Hi, All, > > I am trying to install numpy from http://www.scipy.org/Download . > > by > > git clone git://github.com/numpy/numpy.git numpy > > > But, when I ran > > python setup.py install > > I got: > > SystemError: Cannot compile 'Python.h'. Perhaps you need to install > python-dev|python-devel > > Where to get python-dev ? > If you're on OS X you have to install XCode, preferably with the optional SDKs for previous platforms. On Linux you can install python-dev or similar with your package manager. Ralf > I tried: > > $ easy_install python-devel > Searching for python-devel > Reading http://pypi.python.org/simple/python-devel/ > Couldn't find index page for 'python-devel' (maybe misspelled?) > Scanning index of all packages (this may take a while) > Reading http://pypi.python.org/simple/ > No local packages or download links found for python-devel > error: Could not find suitable distribution for > Requirement.parse('python-devel') > > and > > $ easy_install python-dev > Searching for python-dev > Reading http://pypi.python.org/simple/python-dev/ > Couldn't find index page for 'python-dev' (maybe misspelled?) > Scanning index of all packages (this may take a while) > Reading http://pypi.python.org/simple/ > No local packages or download links found for python-dev > error: Could not find suitable distribution for > Requirement.parse('python-dev') > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Thu Oct 25 02:17:42 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Thu, 25 Oct 2012 08:17:42 +0200 Subject: [Numpy-discussion] how to pipe into numpy arrays? In-Reply-To: References: Message-ID: <5088D986.20703@astro.uio.no> On 10/24/2012 09:00 PM, Michael Aye wrote: > As numpy.fromfile seems to require full file object functionalities > like seek, I can not use it with the sys.stdin pipe. > So how could I stream a binary pipe directly into numpy? > I can imagine storing the data in a string and use StringIO but the > files are 3.6 GB large, just the binary, and that will most likely be > much more as a string object. A Python 2 string is just a bytes object and would take 3.6 GB as well (or did you mean in text encoding?) > Reading binary files on disk is NOT the problem, I would like to avoid > the temporary file if possible. Read in chunks? Something like 1) Create array arr 2) arr_bytes = arr.view(np.uint8).reshape(np.prod(arr.shape)) # check that modifying arr_bytes modifies arr, # if not, work with reshape arguments 3) while not done: arr_bytes[i:i + chunk_size] = f.read(chunk_size) ... Alternatively, one could write some C or Cython code to read directly into the NumPy array buffer, which avoids an extra copy over the memory bus of the data. (Since unfortunately it doesn't look like "fromfile" has an out argument.) Dag Sverre From d.s.seljebotn at astro.uio.no Thu Oct 25 02:19:30 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Thu, 25 Oct 2012 08:19:30 +0200 Subject: [Numpy-discussion] how to pipe into numpy arrays? In-Reply-To: <5088D986.20703@astro.uio.no> References: <5088D986.20703@astro.uio.no> Message-ID: <5088D9F2.3080401@astro.uio.no> On 10/25/2012 08:17 AM, Dag Sverre Seljebotn wrote: > On 10/24/2012 09:00 PM, Michael Aye wrote: >> As numpy.fromfile seems to require full file object functionalities >> like seek, I can not use it with the sys.stdin pipe. >> So how could I stream a binary pipe directly into numpy? >> I can imagine storing the data in a string and use StringIO but the >> files are 3.6 GB large, just the binary, and that will most likely be >> much more as a string object. > > A Python 2 string is just a bytes object and would take 3.6 GB as well > (or did you mean in text encoding?) > >> Reading binary files on disk is NOT the problem, I would like to avoid >> the temporary file if possible. > > Read in chunks? Something like > > 1) Create array arr > > 2) > > arr_bytes = arr.view(np.uint8).reshape(np.prod(arr.shape)) > # check that modifying arr_bytes modifies arr, > # if not, work with reshape arguments > > 3) > > while not done: > arr_bytes[i:i + chunk_size] = f.read(chunk_size) > ... > > Alternatively, one could write some C or Cython code to read directly > into the NumPy array buffer, which avoids an extra copy over the memory > bus of the data. (Since unfortunately it doesn't look like "fromfile" > has an out argument.) Actually, as long as you make sure chunk_size is on the order of 1 MB or so, the Python overhead may not matter and the chunks fit in cache so an extra copy is avoided, so a C solution may be overkill. Dag Sverre From denis-bz-gg at t-online.de Thu Oct 25 03:49:40 2012 From: denis-bz-gg at t-online.de (denis) Date: Thu, 25 Oct 2012 09:49:40 +0200 Subject: [Numpy-discussion] np.linalg.lstsq with several columns all 0 => huge x ? In-Reply-To: References: Message-ID: On 24/10/2012 19:59, josef.pktd at gmail.com wrote: > On Wed, Oct 24, 2012 at 1:33 PM, denis wrote: >> Folks, >> np.linalg.lstsq of a random-uniform A 50 x 32 with 3 columns all 0 >> returns x[:3] 0 as expected, >> but 4 columns all 0 => huge x: > lstsq has rcond argument to do (I think) essentially the same. > might need to be increased in your example. > > Josef rcond 1e-3 works, 1e-6 does not, for these sizes; the returned singular values don't change much from 3 to 4 columns of 0s, so why the big jump ? cheers -- denis From dtustudy68 at hotmail.com Thu Oct 25 08:32:05 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Thu, 25 Oct 2012 06:32:05 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: , Message-ID: Thanks for your invitation. I cannot use rpm command because I am not root user and cannot get sys-adm authorization. Any help will be appreciated. Thanks Date: Thu, 25 Oct 2012 08:05:47 +0200 From: ralf.gommers at gmail.com To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. On Thu, Oct 25, 2012 at 6:01 AM, Jack Bryan wrote: Hi, All, I am trying to install numpy from http://www.scipy.org/Download . by git clone git://github.com/numpy/numpy.git numpy But, when I ran python setup.py install I got: SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel Where to get python-dev ? If you're on OS X you have to install XCode, preferably with the optional SDKs for previous platforms. On Linux you can install python-dev or similar with your package manager. Ralf I tried: $ easy_install python-develSearching for python-develReading http://pypi.python.org/simple/python-devel/ Couldn't find index page for 'python-devel' (maybe misspelled?)Scanning index of all packages (this may take a while)Reading http://pypi.python.org/simple/ No local packages or download links found for python-develerror: Could not find suitable distribution for Requirement.parse('python-devel') and $ easy_install python-devSearching for python-devReading http://pypi.python.org/simple/python-dev/Couldn't find index page for 'python-dev' (maybe misspelled?) Scanning index of all packages (this may take a while)Reading http://pypi.python.org/simple/No local packages or download links found for python-dev error: Could not find suitable distribution for Requirement.parse('python-dev') _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From dtustudy68 at hotmail.com Thu Oct 25 10:23:08 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Thu, 25 Oct 2012 08:23:08 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: , , , Message-ID: Hi, I have installed python2.6 in my local dir. But, when I used python setup.py install --user to install numpy , I got: $ python setup.py install --user'import site' failed; use -v for tracebackTraceback (most recent call last): File "setup.py", line 18, in import osImportError: No module named os in python2.6 I got: >>> import osTraceback (most recent call last): File "", line 1, in ImportError: No module named os Any help will be appreciated. thanks From: dtustudy68 at hotmail.com To: numpy-discussion at scipy.org Date: Thu, 25 Oct 2012 06:32:05 -0600 Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. Thanks for your invitation. I cannot use rpm command because I am not root user and cannot get sys-adm authorization. Any help will be appreciated. Thanks Date: Thu, 25 Oct 2012 08:05:47 +0200 From: ralf.gommers at gmail.com To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. On Thu, Oct 25, 2012 at 6:01 AM, Jack Bryan wrote: Hi, All, I am trying to install numpy from http://www.scipy.org/Download . by git clone git://github.com/numpy/numpy.git numpy But, when I ran python setup.py install I got: SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel Where to get python-dev ? If you're on OS X you have to install XCode, preferably with the optional SDKs for previous platforms. On Linux you can install python-dev or similar with your package manager. Ralf I tried: $ easy_install python-develSearching for python-develReading http://pypi.python.org/simple/python-devel/ Couldn't find index page for 'python-devel' (maybe misspelled?)Scanning index of all packages (this may take a while)Reading http://pypi.python.org/simple/ No local packages or download links found for python-develerror: Could not find suitable distribution for Requirement.parse('python-devel') and $ easy_install python-devSearching for python-devReading http://pypi.python.org/simple/python-dev/Couldn't find index page for 'python-dev' (maybe misspelled?) Scanning index of all packages (this may take a while)Reading http://pypi.python.org/simple/No local packages or download links found for python-dev error: Could not find suitable distribution for Requirement.parse('python-dev') _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron at ahmadia.net Thu Oct 25 11:00:07 2012 From: aron at ahmadia.net (Aron Ahmadia) Date: Thu, 25 Oct 2012 16:00:07 +0100 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: Message-ID: Hi Jack, >>>> import os > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named os You've got a broken local python install. Did you try following the instructions here: http://stackoverflow.com/a/622810/122022? It is probably easiest to ask your system administrator to install python-devel for you (and python-numpy while they are at it). A From dtustudy68 at hotmail.com Thu Oct 25 11:39:24 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Thu, 25 Oct 2012 09:39:24 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: , , , , Message-ID: Hi, Aron, I have to install in my local dir. The sys-adm cannot help me. Any help will be appreciated. > Date: Thu, 25 Oct 2012 16:00:07 +0100 > From: aron at ahmadia.net > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. > > Hi Jack, > > >>>> import os > > Traceback (most recent call last): > > File "", line 1, in > > ImportError: No module named os > > You've got a broken local python install. Did you try following the > instructions here: http://stackoverflow.com/a/622810/122022? > > It is probably easiest to ask your system administrator to install > python-devel for you (and python-numpy while they are at it). > > A > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron at ahmadia.net Thu Oct 25 12:07:06 2012 From: aron at ahmadia.net (Aron Ahmadia) Date: Thu, 25 Oct 2012 17:07:06 +0100 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: Message-ID: I'm repeating myself, but did you try following the instructions here: http://stackoverflow.com/a/622810/122022? -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmhobson at gmail.com Thu Oct 25 14:28:31 2012 From: pmhobson at gmail.com (Paul Hobson) Date: Thu, 25 Oct 2012 11:28:31 -0700 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: Message-ID: On Wed, Oct 24, 2012 at 9:01 PM, Jack Bryan wrote: > Hi, All, > > I am trying to install numpy from http://www.scipy.org/Download . > > by > > git clone git://github.com/numpy/numpy.git numpy > > > But, when I ran > > python setup.py install > > I got: > > SystemError: Cannot compile 'Python.h'. Perhaps you need to install > python-dev|python-devel > > Where to get python-dev ? > > I tried: > > $ easy_install python-devel > Searching for python-devel > Reading http://pypi.python.org/simple/python-devel/ > Couldn't find index page for 'python-devel' (maybe misspelled?) > Scanning index of all packages (this may take a while) > Reading http://pypi.python.org/simple/ > No local packages or download links found for python-devel > error: Could not find suitable distribution for > Requirement.parse('python-devel') > > and > > $ easy_install python-dev > Searching for python-dev > Reading http://pypi.python.org/simple/python-dev/ > Couldn't find index page for 'python-dev' (maybe misspelled?) > Scanning index of all packages (this may take a while) > Reading http://pypi.python.org/simple/ > No local packages or download links found for python-dev > error: Could not find suitable distribution for > Requirement.parse('python-dev') Jack, Since you don't have admin privileges, I'm going to emphatically recommend that you use AnacondaCE. Sine you're on RedHat, it's as simple as: $ curl http://09c8d0b2229f813c1b93-c95ac804525aac4b6dba79b00b39d1d3.r79.cf1.rackcdn.com/AnacondaCE-1.1-linux.sh > anaconda.sh $ bash anaconda.sh Follow the prompts and you're done with a lot more than just Numpy. -paul From wardefar at iro.umontreal.ca Thu Oct 25 17:48:26 2012 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Thu, 25 Oct 2012 17:48:26 -0400 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure Message-ID: I submitted a pull request and one of the Travis builds is failing: https://travis-ci.org/#!/numpy/numpy/jobs/2933551 Given my changes, https://github.com/dwf/numpy/commit/4c88fdafc003397d6879f81bf59f68adeeb59f2b I don't see how the masked array module (responsible for the failing test) could possibly be affected in this manner (RuntimeWarning raised by encountering an invalid value in power()). Moreover, I cannot reproduce it after setting NPY_SEPARATE_COMPILATION on my own machine (whatever that does -- I'm not quite clear). Does anyone have any insight into what's going on here? Can anyone else reproduce it outside of Travis? Thanks, David From wardefar at iro.umontreal.ca Thu Oct 25 17:54:37 2012 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Thu, 25 Oct 2012 17:54:37 -0400 Subject: [Numpy-discussion] einsum slow vs (tensor)dot In-Reply-To: References: Message-ID: On Wed, Oct 24, 2012 at 7:18 AM, George Nurser wrote: > Hi, > > I was just looking at the einsum function. > To me, it's a really elegant and clear way of doing array operations, which > is the core of what numpy is about. > It removes the need to remember a range of functions, some of which I find > tricky (e.g. tile). > > Unfortunately the present implementation seems ~ 4-6x slower than dot or > tensordot for decent size arrays. > I suspect it is because the implementation does not use blas/lapack calls. > > cheers, George Nurser. Hi George, IIRC (and I haven't dug into it heavily; not a physicist so I don't encounter this notation often), einsum implements a superset of what dot or tensordot (and the corresponding BLAS calls) can do. So, I think that logic is needed to carve out the special cases in which an einsum can be performed quickly with BLAS. Pull requests in this vein would certainly be welcome, but requires the attention of someone who really understands how einsum works/can work. David From sebastian at sipsolutions.net Thu Oct 25 18:15:04 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 26 Oct 2012 00:15:04 +0200 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: References: Message-ID: <1351203304.2532.9.camel@sebastian-laptop> On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote: > I submitted a pull request and one of the Travis builds is failing: > > https://travis-ci.org/#!/numpy/numpy/jobs/2933551 > Don't worry about that failure on Travis... It happens randomly on at the moment and its unrelated to anything you are doing. I am not sure though you can change behavior like that since you also change the default behavior of the `.copy()` method and someone might rely on that? Maybe making it specific to the copy model would make it unlikely that anyone relies on the default, it would seem sensible that copy.copy(array) does basically the same as np.copy(array) and not as the method .copy, though ideally maybe the differences could be removed in the long run I guess. Regards, Sebastian > Given my changes, > > https://github.com/dwf/numpy/commit/4c88fdafc003397d6879f81bf59f68adeeb59f2b > > I don't see how the masked array module (responsible for the failing > test) could possibly be affected in this manner (RuntimeWarning raised > by encountering an invalid value in power()). > > Moreover, I cannot reproduce it after setting NPY_SEPARATE_COMPILATION > on my own machine (whatever that does -- I'm not quite clear). > > Does anyone have any insight into what's going on here? Can anyone > else reproduce it outside of Travis? > > Thanks, > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From wardefar at iro.umontreal.ca Thu Oct 25 18:58:19 2012 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Thu, 25 Oct 2012 18:58:19 -0400 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: <1351203304.2532.9.camel@sebastian-laptop> References: <1351203304.2532.9.camel@sebastian-laptop> Message-ID: On Thu, Oct 25, 2012 at 6:15 PM, Sebastian Berg wrote: > On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote: > Don't worry about that failure on Travis... It happens randomly on at > the moment and its unrelated to anything you are doing. Ah, okay. I figured it was something like that. > I am not sure though you can change behavior like that since you also > change the default behavior of the `.copy()` method and someone might > rely on that? Oops, you're right. I assumed I was changing __copy__ only. Pull request updated. Given that behaviour is documented it really ought to be tested. I'll add one. > Maybe making it specific to the copy model would make it > unlikely that anyone relies on the default, it would seem sensible that > copy.copy(array) does basically the same as np.copy(array) and not as > the method .copy, though ideally maybe the differences could be removed > in the long run I guess. Agreed, but for now the .copy() method's default shouldn't change. I think the scikit-learn usecase I described is a good reason why the copy protocol methods should maintain data ordering, though. David From josef.pktd at gmail.com Thu Oct 25 20:39:20 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 25 Oct 2012 20:39:20 -0400 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: References: <1351203304.2532.9.camel@sebastian-laptop> Message-ID: On Thu, Oct 25, 2012 at 6:58 PM, David Warde-Farley wrote: > On Thu, Oct 25, 2012 at 6:15 PM, Sebastian Berg > wrote: >> On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote: > >> Don't worry about that failure on Travis... It happens randomly on at >> the moment and its unrelated to anything you are doing. > > Ah, okay. I figured it was something like that. > >> I am not sure though you can change behavior like that since you also >> change the default behavior of the `.copy()` method and someone might >> rely on that? > > Oops, you're right. I assumed I was changing __copy__ only. Pull > request updated. > > Given that behaviour is documented it really ought to be tested. I'll add one. > >> Maybe making it specific to the copy model would make it >> unlikely that anyone relies on the default, it would seem sensible that >> copy.copy(array) does basically the same as np.copy(array) and not as >> the method .copy, though ideally maybe the differences could be removed >> in the long run I guess. > > Agreed, but for now the .copy() method's default shouldn't change. I > think the scikit-learn usecase I described is a good reason why the > copy protocol methods should maintain data ordering, though. I think this might be something that could wait for a big numpy version change. At least I always assumed that a new array created by copying is the default numpy C order (unless otherwise requested). I never rely on it except sometimes when I think about speed of operation in fortran versus c order. np.copy says: This is equivalent to >>> np.array(a, copy=True) numpy.ma.copy has the order keyword with "c" as default changing the default from "C" to "A" doesn't look like a minor API change. (I'm using numpy 1.5 help file, so maybe I'm outdated.) Josef > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Thu Oct 25 21:27:24 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 25 Oct 2012 19:27:24 -0600 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: <1351203304.2532.9.camel@sebastian-laptop> References: <1351203304.2532.9.camel@sebastian-laptop> Message-ID: Hi Sebastian, On Thu, Oct 25, 2012 at 4:15 PM, Sebastian Berg wrote: > On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote: > > I submitted a pull request and one of the Travis builds is failing: > > > > https://travis-ci.org/#!/numpy/numpy/jobs/2933551 > > > Don't worry about that failure on Travis... It happens randomly on at > the moment and its unrelated to anything you are doing. > I am not sure though you can change behavior like that since you also > change the default behavior of the `.copy()` method and someone might > rely on that? Maybe making it specific to the copy model would make it > unlikely that anyone relies on the default, it would seem sensible that > copy.copy(array) does basically the same as np.copy(array) and not as > the method .copy, though ideally maybe the differences could be removed > in the long run I guess. > > You seem to becoming more involved in the code maintenance. Would you be interested in gaining commit rights at some point? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dtustudy68 at hotmail.com Thu Oct 25 21:40:17 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Thu, 25 Oct 2012 19:40:17 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: , , , , , , Message-ID: Hi, Aron, I have followed the instructions on the url, after installing python 2.6, I tried to install numpy and got : [~/numpy/numpy] $ /home/user/myName/.local/bin/python setup.py installTraceback (most recent call last): File "setup.py", line 22, in import subprocess File "/home/user/myName/.local/lib/python2.6/subprocess.py", line 401, in import selectImportError: No module named select Any help will be appreciated. Date: Thu, 25 Oct 2012 17:07:06 +0100 From: aron at ahmadia.net To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. I'm repeating myself, but did you try following the instructions here: http://stackoverflow.com/a/622810/122022? _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From wardefar at iro.umontreal.ca Thu Oct 25 23:48:28 2012 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Thu, 25 Oct 2012 23:48:28 -0400 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: References: <1351203304.2532.9.camel@sebastian-laptop> Message-ID: On Thu, Oct 25, 2012 at 8:39 PM, wrote: > On Thu, Oct 25, 2012 at 6:58 PM, David Warde-Farley > wrote: >> On Thu, Oct 25, 2012 at 6:15 PM, Sebastian Berg >> wrote: >>> On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote: >> >>> Don't worry about that failure on Travis... It happens randomly on at >>> the moment and its unrelated to anything you are doing. >> >> Ah, okay. I figured it was something like that. >> >>> I am not sure though you can change behavior like that since you also >>> change the default behavior of the `.copy()` method and someone might >>> rely on that? >> >> Oops, you're right. I assumed I was changing __copy__ only. Pull >> request updated. >> >> Given that behaviour is documented it really ought to be tested. I'll add one. >> >>> Maybe making it specific to the copy model would make it >>> unlikely that anyone relies on the default, it would seem sensible that >>> copy.copy(array) does basically the same as np.copy(array) and not as >>> the method .copy, though ideally maybe the differences could be removed >>> in the long run I guess. >> >> Agreed, but for now the .copy() method's default shouldn't change. I >> think the scikit-learn usecase I described is a good reason why the >> copy protocol methods should maintain data ordering, though. > > I think this might be something that could wait for a big numpy version change. > > At least I always assumed that a new array created by copying is the default > numpy C order (unless otherwise requested). > I never rely on it except sometimes when I think about speed of operation in > fortran versus c order. > > np.copy says: > This is equivalent to >>>> np.array(a, copy=True) > > numpy.ma.copy has the order keyword with "c" as default > > changing the default from "C" to "A" doesn't look like a minor API change. Hi Josef, Just to be clear: this pull request ( https://github.com/numpy/numpy/pull/2699 ) affects only the behaviour when arrays are copied with the standard library "copy" module. The use of the .copy() method of ndarrays, or the numpy.copy library function, remains the same, with all defaults intact. An oversight on my part had briefly changed it, but Sebastian pointed this out and I've updated the PR. The main application is, as I said, with objects that have ndarray members and interface with external code that relies on Fortran ordering of those ndarray members (of which there are several in scikit-learn, as one example). An object once deepcopy'd should ideally "just work" in any situation where the original worked, but with the current implementation of the copy protocol methods this wasn't possible. Frequent users of .copy()/np.copy() who are familiar with *NumPy's* API behaviour should be largely unaffected. David From josef.pktd at gmail.com Fri Oct 26 01:04:11 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 26 Oct 2012 01:04:11 -0400 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: References: <1351203304.2532.9.camel@sebastian-laptop> Message-ID: On Thu, Oct 25, 2012 at 11:48 PM, David Warde-Farley wrote: > On Thu, Oct 25, 2012 at 8:39 PM, wrote: >> On Thu, Oct 25, 2012 at 6:58 PM, David Warde-Farley >> wrote: >>> On Thu, Oct 25, 2012 at 6:15 PM, Sebastian Berg >>> wrote: >>>> On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote: >>> >>>> Don't worry about that failure on Travis... It happens randomly on at >>>> the moment and its unrelated to anything you are doing. >>> >>> Ah, okay. I figured it was something like that. >>> >>>> I am not sure though you can change behavior like that since you also >>>> change the default behavior of the `.copy()` method and someone might >>>> rely on that? >>> >>> Oops, you're right. I assumed I was changing __copy__ only. Pull >>> request updated. >>> >>> Given that behaviour is documented it really ought to be tested. I'll add one. >>> >>>> Maybe making it specific to the copy model would make it >>>> unlikely that anyone relies on the default, it would seem sensible that >>>> copy.copy(array) does basically the same as np.copy(array) and not as >>>> the method .copy, though ideally maybe the differences could be removed >>>> in the long run I guess. >>> >>> Agreed, but for now the .copy() method's default shouldn't change. I >>> think the scikit-learn usecase I described is a good reason why the >>> copy protocol methods should maintain data ordering, though. >> >> I think this might be something that could wait for a big numpy version change. >> >> At least I always assumed that a new array created by copying is the default >> numpy C order (unless otherwise requested). >> I never rely on it except sometimes when I think about speed of operation in >> fortran versus c order. >> >> np.copy says: >> This is equivalent to >>>>> np.array(a, copy=True) >> >> numpy.ma.copy has the order keyword with "c" as default >> >> changing the default from "C" to "A" doesn't look like a minor API change. > > Hi Josef, > > Just to be clear: this pull request ( > https://github.com/numpy/numpy/pull/2699 ) affects only the behaviour > when arrays are copied with the standard library "copy" module. The > use of the .copy() method of ndarrays, or the numpy.copy library > function, remains the same, with all defaults intact. An oversight on > my part had briefly changed it, but Sebastian pointed this out and > I've updated the PR. Fine, I didn't understand that part correctly. I have no opinion in that case. (In statsmodels we only use copy the array method and through np.array().) Thanks for the clarification Josef > > The main application is, as I said, with objects that have ndarray > members and interface with external code that relies on Fortran > ordering of those ndarray members (of which there are several in > scikit-learn, as one example). An object once deepcopy'd should > ideally "just work" in any situation where the original worked, but > with the current implementation of the copy protocol methods this > wasn't possible. > > Frequent users of .copy()/np.copy() who are familiar with *NumPy's* > API behaviour should be largely unaffected. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From d.warde.farley at gmail.com Fri Oct 26 01:27:03 2012 From: d.warde.farley at gmail.com (David Warde-Farley) Date: Fri, 26 Oct 2012 01:27:03 -0400 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: References: <1351203304.2532.9.camel@sebastian-laptop> Message-ID: On Fri, Oct 26, 2012 at 1:04 AM, wrote: > Fine, I didn't understand that part correctly. > > I have no opinion in that case. > (In statsmodels we only use copy the array method and through np.array().) Do you implement __copy__ or __deepcopy__ on your objects? If not, client code using statsmodels might try to use the copy module and run into the same problem. David From gnurser at gmail.com Fri Oct 26 05:51:23 2012 From: gnurser at gmail.com (George Nurser) Date: Fri, 26 Oct 2012 10:51:23 +0100 Subject: [Numpy-discussion] einsum slow vs (tensor)dot In-Reply-To: References: Message-ID: On 25 October 2012 22:54, David Warde-Farley wrote: > On Wed, Oct 24, 2012 at 7:18 AM, George Nurser wrote: > > Hi, > > > > I was just looking at the einsum function. > > To me, it's a really elegant and clear way of doing array operations, > which > > is the core of what numpy is about. > > It removes the need to remember a range of functions, some of which I > find > > tricky (e.g. tile). > > > > Unfortunately the present implementation seems ~ 4-6x slower than dot or > > tensordot for decent size arrays. > > I suspect it is because the implementation does not use blas/lapack > calls. > > > > cheers, George Nurser. > > Hi George, > > IIRC (and I haven't dug into it heavily; not a physicist so I don't > encounter this notation often), einsum implements a superset of what > dot or tensordot (and the corresponding BLAS calls) can do. So, I > think that logic is needed to carve out the special cases in which an > einsum can be performed quickly with BLAS. > Hi David, Yes, that's my reading of the situation as well. > Pull requests in this vein would certainly be welcome, but requires > the attention of someone who really understands how einsum works/can > work. > ...and I guess how to interface w BLAS/LAPACK. cheers, George. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Oct 26 09:03:41 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 26 Oct 2012 13:03:41 +0000 (UTC) Subject: [Numpy-discussion] error of install numpy on linux redhat. References: , , , , , , Message-ID: Jack Bryan hotmail.com> writes: [clip] > [~/numpy/numpy] $ /home/user/myName/.local/bin/python setup.py install > Traceback (most recent call last): > ? File "setup.py", line 22, in > ? ? import subprocess > ? File "/home/user/myName/.local/lib/python2.6/subprocess.py", line 401, in > ? ? import select > ImportError: No module named select This means your Python 2.6 installation is still broken, so it is not related to Numpy. Start trying to install Numpy only after you get >>> import subprocess to work properly. Googling reveals that also others have had problems with missing select module when compiling Python. Apparently the machine you are using is not a standard Redhat installation, but something on it is strange. -- Pauli Virtanen From dtustudy68 at hotmail.com Fri Oct 26 09:48:51 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Fri, 26 Oct 2012 07:48:51 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: , , , , , , , , , , , , , , Message-ID: Hi, Th python installed by sys-adm can import subprocess and select w/o any problems. But, it does not support python-dev. So, I need to install a python that can support python-dev and install other python modules such as numpy and scipy. Any help will be appreciated. > To: numpy-discussion at scipy.org > From: pav at iki.fi > Date: Fri, 26 Oct 2012 13:03:41 +0000 > Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. > > Jack Bryan hotmail.com> writes: > [clip] > > [~/numpy/numpy] $ /home/user/myName/.local/bin/python setup.py install > > Traceback (most recent call last): > > File "setup.py", line 22, in > > import subprocess > > File "/home/user/myName/.local/lib/python2.6/subprocess.py", line 401, in > > > import select > > ImportError: No module named select > > This means your Python 2.6 installation is still broken, > so it is not related to Numpy. > > Start trying to install Numpy only after you get > > >>> import subprocess > > to work properly. > > Googling reveals that also others have had problems with missing select > module when compiling Python. > > Apparently the machine you are using is not a standard Redhat installation, > but something on it is strange. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Oct 26 09:52:30 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 26 Oct 2012 09:52:30 -0400 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: References: <1351203304.2532.9.camel@sebastian-laptop> Message-ID: On Fri, Oct 26, 2012 at 1:27 AM, David Warde-Farley wrote: > On Fri, Oct 26, 2012 at 1:04 AM, wrote: > >> Fine, I didn't understand that part correctly. >> >> I have no opinion in that case. >> (In statsmodels we only use copy the array method and through np.array().) > > Do you implement __copy__ or __deepcopy__ on your objects? If not, > client code using statsmodels might try to use the copy module and run > into the same problem. We don't implement either. I haven't heard yet of a use case for copying a model or results instance in statsmodels (that doesn't have a more memory efficient alternative). I make a note about this if we need it in future. (asides: We had to work around pickling problems. We try to use views instead of copies. Nothing depends on fortran versus c order of arrays except speed, and I think linalg will make copies if users provide data arrays with the wrong properties. We never systematically investigated differences in speed depending on array order or views, and some differences will have changed with recent numpy versions. ) Josef > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From dtustudy68 at hotmail.com Fri Oct 26 11:33:51 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Fri, 26 Oct 2012 09:33:51 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: , , , , , , , , , , , , , , Message-ID: During the make: I got : Failed to find the necessary bits to build these modules: _tkinter bsddb185 dl imageop sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. running build_scripts During the make install : I got : running install_egg_info Removing /home/user/myname/usr/lib/python2.6/site-packages/Python-2.6.8-py2.6.egg-info Writing /home/user/myname/usr/lib/python2.6/site-packages/Python-2.6.8-py2.6.egg-info if test -f /home/user/myname/.local/bin/python -o -h /home/user/jding/.local/bin/python; \ then rm -f /home/user/myname/.local/bin/python; \ else true; \ fi (cd /home/user/myname/.local/bin; ln python2.6 python) rm -f /home/user/myname/.local/bin/python-config (cd /home/user/myname/.local/bin; ln -s python2.6-config python-config) /usr/bin/install -c -m 644 ./Misc/python.man \ /home/user/myname/.local/share/man/man1/python.1 When I launched python form command line,I got: [~/python_268/Python-2.6.8] $ ~/.local/bin/python Python 2.6.8 (unknown, Oct 26 2012, 11:18:54) [GCC 4.3.2 20081105 (Red Hat 4.3.2-20)] on linux2 When I make test I got: 327 tests OK. 1 test failed: test_site 38 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses test_dl test_gl test_imageop test_imgfile test_kqueue test_linuxaudiodev test_macos test_macostools test_normalization test_ossaudiodev test_pep277 test_py3kwarn test_scriptpackages test_smtpnet test_socketserver test_startfile test_sunaudiodev test_tcl test_timeout test_unicode_file test_urllib2net test_urllibnet test_winreg test_winsound test_zipfile64 1 skip unexpected on linux2: test_tcl make: *** [test] Error 1 Is this a problem ? Any help will be appreciated. > To: numpy-discussion at scipy.org > From: pav at iki.fi > Date: Fri, 26 Oct 2012 13:03:41 +0000 > Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. > > Jack Bryan hotmail.com> writes: > [clip] > > [~/numpy/numpy] $ /home/user/myName/.local/bin/python setup.py install > > Traceback (most recent call last): > > File "setup.py", line 22, in > > import subprocess > > File "/home/user/myName/.local/lib/python2.6/subprocess.py", line 401, in > > > import select > > ImportError: No module named select > > This means your Python 2.6 installation is still broken, > so it is not related to Numpy. > > Start trying to install Numpy only after you get > > >>> import subprocess > > to work properly. > > Googling reveals that also others have had problems with missing select > module when compiling Python. > > Apparently the machine you are using is not a standard Redhat installation, > but something on it is strange. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.robitaille at gmail.com Fri Oct 26 14:54:46 2012 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 26 Oct 2012 20:54:46 +0200 Subject: [Numpy-discussion] MaskedArray, view, and fill_value Message-ID: Hi everyone, Could someone explain the following behavior? >>> from numpy import ma >>> x = ma.array([1,2,3]) >>> x.fill_value = 1 >>> x.view(ma.MaskedArray) masked_array(data = [1 2 3], mask = False, fill_value = 999999) i.e. the fill_value gets reset. It looks from ma/core.py like it is deliberate: # Make sure to reset the _fill_value if needed if getattr(output, '_fill_value', None) is not None: output._fill_value = None but this doesn't make any sense to me... Any ideas? I'm thinking maybe it should be: # Make sure to reset the _fill_value if needed if getattr(output, '_fill_value', None) is not None and dtype is not None: output._fill_value = None because then it would make sense to reset the fill_value if the dtype has changed. Any thoughts? Thanks, Tom From sebastian at sipsolutions.net Fri Oct 26 17:10:03 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 26 Oct 2012 23:10:03 +0200 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: References: <1351203304.2532.9.camel@sebastian-laptop> Message-ID: <1351285803.4141.12.camel@sebastian-laptop> Hey On Thu, 2012-10-25 at 19:27 -0600, Charles R Harris wrote: > Hi Sebastian, > > > You seem to becoming more involved in the code maintenance. Would you > be interested in gaining commit rights at some point? > Maybe, but honestly I am not sure if I will keep following numpy very closely in the future. Regards, Sebastian > Chuck > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pav at iki.fi Fri Oct 26 17:53:42 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 26 Oct 2012 21:53:42 +0000 (UTC) Subject: [Numpy-discussion] error of install numpy on linux redhat. References: , , , , , , , , , , , , , , Message-ID: Jack Bryan hotmail.com> writes: [clip] > Is this a problem ? You need to ensure that >>> import subprocess works. I don't know how to fix that. Rather than trying to compile Python yourself and hitting errors that you are unable to resolve, it would be more productive to grab either: http://continuum.io/downloads.html http://www.enthought.com/products/epd.php as suggested earlier in this thread. -- Pauli Virtanen From dtustudy68 at hotmail.com Fri Oct 26 18:02:04 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Fri, 26 Oct 2012 16:02:04 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: , , , , , , , ,,, , , , , , , , , , , , , , , Message-ID: Hi, I cannot afford the cost of buying these products. Why does not python's website provide some FAQ help about this kind of common problems ? (such as modules missing) ? I have to work on the open source ones . Any help will be appreciated. > To: numpy-discussion at scipy.org > From: pav at iki.fi > Date: Fri, 26 Oct 2012 21:53:42 +0000 > Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. > > Jack Bryan hotmail.com> writes: > [clip] > > Is this a problem ? > > You need to ensure that > > >>> import subprocess > > works. I don't know how to fix that. > > Rather than trying to compile Python yourself and hitting errors that > you are unable to resolve, it would be more productive to grab either: > > http://continuum.io/downloads.html > > http://www.enthought.com/products/epd.php > > as suggested earlier in this thread. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Oct 26 18:26:15 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 26 Oct 2012 16:26:15 -0600 Subject: [Numpy-discussion] copy/deepcopy pull request, Travis build failure In-Reply-To: <1351285803.4141.12.camel@sebastian-laptop> References: <1351203304.2532.9.camel@sebastian-laptop> <1351285803.4141.12.camel@sebastian-laptop> Message-ID: On Fri, Oct 26, 2012 at 3:10 PM, Sebastian Berg wrote: > Hey > > On Thu, 2012-10-25 at 19:27 -0600, Charles R Harris wrote: > > Hi Sebastian, > > > > > > > You seem to becoming more involved in the code maintenance. Would you > > be interested in gaining commit rights at some point? > > > Maybe, but honestly I am not sure if I will keep following numpy very > closely in the future. > > Fair enough. We need to get new people to get involved, so if you find yourself tending to more involvement it is a definite option. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Oct 26 19:36:56 2012 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 26 Oct 2012 23:36:56 +0000 (UTC) Subject: [Numpy-discussion] error of install numpy on linux redhat. References: , , , , , , , , , , , , , , , , , , , , , , , , Message-ID: Jack Bryan hotmail.com> writes: > Hi, I cannot afford the cost of buying these products. The Anaconda community edition is free. EPD is free for academic use. The issue you are facing is a rare one --- I have not had similar issues myself on different platforms, including HPC clusters. -- Pauli Virtanen From dtustudy68 at hotmail.com Fri Oct 26 22:06:51 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Fri, 26 Oct 2012 20:06:51 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: , , , , , , , , , , ,,, , , , , , , , , , , , , , , , , , , , , , , , Message-ID: Thanks for your help. Why do the Anaconda and EPD can help me fix the problem ? Any help will be appreciated. > To: numpy-discussion at scipy.org > From: pav at iki.fi > Date: Fri, 26 Oct 2012 23:36:56 +0000 > Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. > > Jack Bryan hotmail.com> writes: > > Hi, I cannot afford the cost of buying these products. > > The Anaconda community edition is free. > > EPD is free for academic use. > > The issue you are facing is a rare one --- I have not had similar > issues myself on different platforms, including HPC clusters. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Oct 27 04:10:28 2012 From: cournape at gmail.com (David Cournapeau) Date: Sat, 27 Oct 2012 10:10:28 +0200 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: Message-ID: On Sat, Oct 27, 2012 at 4:06 AM, Jack Bryan wrote: > Thanks for your help. > > Why do the Anaconda and EPD can help me fix the problem ? Because they are an all-included distribution of python with numpy/scipy and many other packages. You don't need to compile anything. David From dtustudy68 at hotmail.com Sat Oct 27 09:27:01 2012 From: dtustudy68 at hotmail.com (Jack Bryan) Date: Sat, 27 Oct 2012 07:27:01 -0600 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: , , , , , , , , , , , , , , Message-ID: Thanks for your reply. Do they support other python modules ? such as metplotlib, IPython ? Do their products expire after some time so that I have to install it from scratch ? Any help will be appreciated. > Date: Sat, 27 Oct 2012 10:10:28 +0200 > From: cournape at gmail.com > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. > > On Sat, Oct 27, 2012 at 4:06 AM, Jack Bryan wrote: > > Thanks for your help. > > > > Why do the Anaconda and EPD can help me fix the problem ? > > Because they are an all-included distribution of python with > numpy/scipy and many other packages. You don't need to compile > anything. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron at ahmadia.net Sat Oct 27 09:51:50 2012 From: aron at ahmadia.net (Aron Ahmadia) Date: Sat, 27 Oct 2012 14:51:50 +0100 Subject: [Numpy-discussion] error of install numpy on linux redhat. In-Reply-To: References: Message-ID: Dear Jack, You are starting to ask questions that should be easily answered by 5 minutes of your own time using a search engine. Speaking for myself, I'm a volunteer here, and I want to help people learn and use Python, but I also expect that you will put some of your own effort into tracking down advice that people have given you. There have been two suggestions here for you to try (Anaconda CE and EPD Free), both of them contain the libraries you are asking about, and advertise this broadly on their web pages. Install instructions have also been provided to you and are on the web pages. If you are having trouble understanding or accessing the sites, let us know, but for now I can only assume that you are not doing your own homework. -Aron On Sat, Oct 27, 2012 at 2:27 PM, Jack Bryan wrote: > Thanks for your reply. > > Do they support other python modules ? such as metplotlib, IPython ? > > Do their products expire after some time so that I have to install it from > scratch ? > > Any help will be appreciated. > > > > Date: Sat, 27 Oct 2012 10:10:28 +0200 > > From: cournape at gmail.com > > To: numpy-discussion at scipy.org > > > Subject: Re: [Numpy-discussion] error of install numpy on linux redhat. > > > > On Sat, Oct 27, 2012 at 4:06 AM, Jack Bryan > wrote: > > > Thanks for your help. > > > > > > Why do the Anaconda and EPD can help me fix the problem ? > > > > Because they are an all-included distribution of python with > > numpy/scipy and many other packages. You don't need to compile > > anything. > > > > David > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From damon.mcdougall at gmail.com Sat Oct 27 16:58:15 2012 From: damon.mcdougall at gmail.com (Damon McDougall) Date: Sat, 27 Oct 2012 21:58:15 +0100 Subject: [Numpy-discussion] Blaze Message-ID: I just saw this on a news site I read pretty often and thought this community would be interested. It certainly looks useful! https://speakerdeck.com/sdiehl/blaze-next-generation-numpy Sorry if it's been posted already. Best wishes, Damon -- Damon McDougall http://www.damon-is-a-geek.com B2.39 Mathematics Institute University of Warwick Coventry West Midlands CV4 7AL United Kingdom From charlesr.harris at gmail.com Sat Oct 27 19:14:32 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 27 Oct 2012 17:14:32 -0600 Subject: [Numpy-discussion] Blaze In-Reply-To: References: Message-ID: On Sat, Oct 27, 2012 at 2:58 PM, Damon McDougall wrote: > I just saw this on a news site I read pretty often and thought this > community would be interested. It certainly looks useful! > > https://speakerdeck.com/sdiehl/blaze-next-generation-numpy > > Sorry if it's been posted already. > Best wishes, > Damon > > Thanks. I've been trying to keep an eye on this but it is difficult to find the details. I think the beta was scheduled for sometime in December. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From radek.machulka at gmail.com Mon Oct 29 06:29:10 2012 From: radek.machulka at gmail.com (Radek Machulka) Date: Mon, 29 Oct 2012 11:29:10 +0100 Subject: [Numpy-discussion] numpy.savez(_compressed) in loop Message-ID: <1896437.YHDg1QmCBS@edge> Hi, is there a way how to save more arrays into single npy (npz if possible) file in loop? Something like: fw = open('foo.bar', 'wb') while foo: arr = np.array(bar) np.savez_compressed(fw, arr) fw.close() Or some workaround maybe? I go through hundreds of thousands arrays and can not keep them in memory. Yes, I can save each array into single file, but I would better have them all in single one. Thanks Radek From larry.paltrow at gmail.com Mon Oct 29 04:40:28 2012 From: larry.paltrow at gmail.com (Larry Paltrow) Date: Mon, 29 Oct 2012 01:40:28 -0700 Subject: [Numpy-discussion] nan result from np.linalg.lstsq() Message-ID: I'm having some trouble using the linalg.lstsq() function with certain data and trying to understand why. It's always returning nans when I feed it this numpy array of float64 data: data = df.close.values #coming from a pandas dataframe type(data) >>> numpy.ndarray data.dtype >>> dtype('float64') data >>> array([ 1.31570348, 1.31565421, 1.3157375 , ..., 1.32175 , 1.32180441, 1.321775 ]) xi = np.arange(0,len(data)) A = np.vstack([xi, np.ones(len(xi))]).T A >>> array([[ 0.00000000e+00, 1.00000000e+00], [ 1.00000000e+00, 1.00000000e+00], [ 2.00000000e+00, 1.00000000e+00], ..., [ 2.87800000e+03, 1.00000000e+00], [ 2.87900000e+03, 1.00000000e+00], [ 2.88000000e+03, 1.00000000e+00]]) m, c = np.linalg.lstsq(A, data)[0] m, c >>> (nan, nan) oy, do not understand. Does anyone else know why? -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Oct 29 04:49:20 2012 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Oct 2012 08:49:20 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?numpy=2Esavez=28=5Fcompressed=29_in_?= =?utf-8?q?loop?= References: <1896437.YHDg1QmCBS@edge> Message-ID: Radek Machulka gmail.com> writes: > is there a way how to save more arrays into single npy (npz if possible) file > in loop? Something like: > > fw = open('foo.bar', 'wb') > while foo: > arr = np.array(bar) > np.savez_compressed(fw, arr) > fw.close() > > Or some workaround maybe? I go through hundreds of thousands arrays and can > not keep them in memory. Yes, I can save each array into single file, but I > would better have them all in single one. [clip] You can take a look at the implementation of savez_compressed. The npz files are nothing but zip files with npy format files inside them, so you should be able to build them up by stuffing files saved by np.save() into a zipfile.ZipFile. However, a better option could be to use a another data format such as hdf5. -- Pauli Virtanen From pav at iki.fi Mon Oct 29 04:50:30 2012 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Oct 2012 08:50:30 +0000 (UTC) Subject: [Numpy-discussion] nan result from np.linalg.lstsq() References: Message-ID: Larry Paltrow gmail.com> writes: > I'm having some trouble using the linalg.lstsq() function > with certain data and trying to understand why. It's > always returning nans when I feed it this numpy array of float64 data: > > data = df.close.values #coming from a pandas dataframe > > type(data) > >>>?numpy.ndarray Maybe you have NaNs in the array? (i.e. np.isnan(data) is True) -- Pauli Virtanen From larry.paltrow at gmail.com Mon Oct 29 05:01:53 2012 From: larry.paltrow at gmail.com (Larry Paltrow) Date: Mon, 29 Oct 2012 02:01:53 -0700 Subject: [Numpy-discussion] nan result from np.linalg.lstsq() In-Reply-To: References: Message-ID: np.isnan(data) is True >>> False On Mon, Oct 29, 2012 at 1:50 AM, Pauli Virtanen wrote: > Larry Paltrow gmail.com> writes: > > I'm having some trouble using the linalg.lstsq() function > > with certain data and trying to understand why. It's > > always returning nans when I feed it this numpy array of float64 data: > > > > data = df.close.values #coming from a pandas dataframe > > > > type(data) > > >>> numpy.ndarray > > Maybe you have NaNs in the array? (i.e. np.isnan(data) is True) > > -- > Pauli Virtanen > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Mon Oct 29 05:12:35 2012 From: e.antero.tammi at gmail.com (eat) Date: Mon, 29 Oct 2012 11:12:35 +0200 Subject: [Numpy-discussion] nan result from np.linalg.lstsq() In-Reply-To: References: Message-ID: Hi, On Mon, Oct 29, 2012 at 11:01 AM, Larry Paltrow wrote: > np.isnan(data) is True > >>> False > Check with: np.all(~np.isnan(x)) My 2 cents, -eat > > > On Mon, Oct 29, 2012 at 1:50 AM, Pauli Virtanen wrote: > >> Larry Paltrow gmail.com> writes: >> > I'm having some trouble using the linalg.lstsq() function >> > with certain data and trying to understand why. It's >> > always returning nans when I feed it this numpy array of float64 data: >> > >> > data = df.close.values #coming from a pandas dataframe >> > >> > type(data) >> > >>> numpy.ndarray >> >> Maybe you have NaNs in the array? (i.e. np.isnan(data) is True) >> >> -- >> Pauli Virtanen >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry.paltrow at gmail.com Mon Oct 29 05:21:12 2012 From: larry.paltrow at gmail.com (Larry Paltrow) Date: Mon, 29 Oct 2012 02:21:12 -0700 Subject: [Numpy-discussion] nan result from np.linalg.lstsq() In-Reply-To: References: Message-ID: Ok thanks, I figured np.isnan(data) is True is what we want but wasn't certain. I think it should describe the same thing. np.all(~np.isnan(data)) >>> False Seems to be all non-nan On Mon, Oct 29, 2012 at 2:12 AM, eat wrote: > Hi, > > On Mon, Oct 29, 2012 at 11:01 AM, Larry Paltrow wrote: > >> np.isnan(data) is True >> >>> False >> > Check with: > np.all(~np.isnan(x)) > > My 2 cents, > -eat > >> >> >> On Mon, Oct 29, 2012 at 1:50 AM, Pauli Virtanen wrote: >> >>> Larry Paltrow gmail.com> writes: >>> > I'm having some trouble using the linalg.lstsq() function >>> > with certain data and trying to understand why. It's >>> > always returning nans when I feed it this numpy array of float64 data: >>> > >>> > data = df.close.values #coming from a pandas dataframe >>> > >>> > type(data) >>> > >>> numpy.ndarray >>> >>> Maybe you have NaNs in the array? (i.e. np.isnan(data) is True) >>> >>> -- >>> Pauli Virtanen >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Oct 29 05:36:16 2012 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Oct 2012 09:36:16 +0000 (UTC) Subject: [Numpy-discussion] nan result from np.linalg.lstsq() References: Message-ID: Larry Paltrow gmail.com> writes: [clip] > np.all(~np.isnan(data)) > >>> False > > Seems to be all non-nan No, it means you have NaNs in your data. -- Pauli Virtanen From ben.root at ou.edu Mon Oct 29 09:54:38 2012 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 29 Oct 2012 09:54:38 -0400 Subject: [Numpy-discussion] Regression in mpl: AttributeError: incompatible shape for a non-contiguous array Message-ID: This error started showing up in the test suite for mpl when using numpy master. AttributeError: incompatible shape for a non-contiguous array The tracebacks all point back to various code points where we are trying to set the shape of an array, e.g., offsets.shape = (-1, 2) Those lines haven't changed in a couple of years, and was intended to be done this way to raise an error when reshaping would result in a copy (since we needed to use the original in those places). I don't know how these arrays have become non-contiguous, so I am wondering if there was some sort of attribute that got screwed up somewhere (maybe with views?) Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Mon Oct 29 10:33:53 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 29 Oct 2012 15:33:53 +0100 Subject: [Numpy-discussion] Regression in mpl: AttributeError: incompatible shape for a non-contiguous array In-Reply-To: References: Message-ID: <1351521233.2669.17.camel@sebastian-laptop> Hey, On Mon, 2012-10-29 at 09:54 -0400, Benjamin Root wrote: > This error started showing up in the test suite for mpl when using > numpy master. > > AttributeError: incompatible shape for a non-contiguous array > > The tracebacks all point back to various code points where we are > trying to set the shape of an array, e.g., > > offsets.shape = (-1, 2) > Could you give a hint what these arrays history (how it was created) and maybe .shape/.strides is? Sounds like the array is not contiguous when it is expected to be, or the attribute setting itself fails in some corner cases on master? Regards, Sebastian > Those lines haven't changed in a couple of years, and was intended to > be done this way to raise an error when reshaping would result in a > copy (since we needed to use the original in those places). I don't > know how these arrays have become non-contiguous, so I am wondering if > there was some sort of attribute that got screwed up somewhere (maybe > with views?) > > Cheers! > Ben Root > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ben.root at ou.edu Mon Oct 29 10:43:15 2012 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 29 Oct 2012 10:43:15 -0400 Subject: [Numpy-discussion] Regression in mpl: AttributeError: incompatible shape for a non-contiguous array In-Reply-To: <1351521233.2669.17.camel@sebastian-laptop> References: <1351521233.2669.17.camel@sebastian-laptop> Message-ID: On Mon, Oct 29, 2012 at 10:33 AM, Sebastian Berg wrote: > Hey, > > On Mon, 2012-10-29 at 09:54 -0400, Benjamin Root wrote: > > This error started showing up in the test suite for mpl when using > > numpy master. > > > > AttributeError: incompatible shape for a non-contiguous array > > > > The tracebacks all point back to various code points where we are > > trying to set the shape of an array, e.g., > > > > offsets.shape = (-1, 2) > > > Could you give a hint what these arrays history (how it was created) and > maybe .shape/.strides is? Sounds like the array is not contiguous when > it is expected to be, or the attribute setting itself fails in some > corner cases on master? > > Regards, > > Sebastian > > The original reporter of the bug dug into the commit list and suspects it was this one: https://github.com/numpy/numpy/commit/02ebf8b3e7674a6b8a06636feaa6c761fcdf4e2d However, it might be earlier than that (he is currently doing a clean rebuild to make sure). As for the history: offsets = np.asanyarray(offsets) offsets.shape = (-1, 2) # Make it Nx2 Where "offsets" comes in from (possibly) user-supplied data. Nothing really all that special. I will see if I can get stride information. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrickmarshwx at gmail.com Mon Oct 29 11:04:08 2012 From: patrickmarshwx at gmail.com (Patrick Marsh) Date: Mon, 29 Oct 2012 10:04:08 -0500 Subject: [Numpy-discussion] Regression in mpl: AttributeError: incompatible shape for a non-contiguous array In-Reply-To: References: <1351521233.2669.17.camel@sebastian-laptop> Message-ID: Turns out it isn't the commit I thought it was. I'm currently going through a git bisect to track down the actual commit that introduced this bug. I'll post back when I've found it. PTM --- Patrick Marsh Ph.D. Candidate / Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com On Mon, Oct 29, 2012 at 9:43 AM, Benjamin Root wrote: > > > On Mon, Oct 29, 2012 at 10:33 AM, Sebastian Berg < > sebastian at sipsolutions.net> wrote: > >> Hey, >> >> On Mon, 2012-10-29 at 09:54 -0400, Benjamin Root wrote: >> > This error started showing up in the test suite for mpl when using >> > numpy master. >> > >> > AttributeError: incompatible shape for a non-contiguous array >> > >> > The tracebacks all point back to various code points where we are >> > trying to set the shape of an array, e.g., >> > >> > offsets.shape = (-1, 2) >> > >> Could you give a hint what these arrays history (how it was created) and >> maybe .shape/.strides is? Sounds like the array is not contiguous when >> it is expected to be, or the attribute setting itself fails in some >> corner cases on master? >> >> Regards, >> >> Sebastian >> >> > The original reporter of the bug dug into the commit list and suspects it > was this one: > > > https://github.com/numpy/numpy/commit/02ebf8b3e7674a6b8a06636feaa6c761fcdf4e2d > > However, it might be earlier than that (he is currently doing a clean > rebuild to make sure). > > As for the history: > > offsets = np.asanyarray(offsets) > offsets.shape = (-1, 2) # Make it Nx2 > > Where "offsets" comes in from (possibly) user-supplied data. Nothing > really all that special. I will see if I can get stride information. > > Ben Root > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Mon Oct 29 11:11:43 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Mon, 29 Oct 2012 11:11:43 -0400 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: <1351007294.3585.5.camel@farnsworth> References: <1351007294.3585.5.camel@farnsworth> Message-ID: On Tue, Oct 23, 2012 at 11:48 AM, Henry Gomersall wrote: > On Tue, 2012-10-23 at 11:41 -0400, Fr?d?ric Bastien wrote: >> Did you saw the gpu nd array project? We try to do something similar >> but only for the GPU. >> > Out of interest, is there a reason why the backend for Numpy could not > be written entirely in OpenCL? > > Assuming of course all the relevant backends are up to scratch. > > Is there a fundamental reason why targetting a CPU through OpenCL is > worse than doing it exclusively through C or C++? First, opencl do not allow us to do pointor arythmetique. So when taking a slice of an ndarray, we can't just more the pointor. So we need to change the object structure. I didn't do any speed anylysis of this, but I think that by using OpenCL, it would have a bigger overhead. So it is only useful for "big" ndarray. I don't have any size in mind too. I don't know, but if we could access the opencl data directly from C/C++, we could bypass this for small array if we want. But maybe this is not possible! Fred From patrickmarshwx at gmail.com Mon Oct 29 11:15:33 2012 From: patrickmarshwx at gmail.com (Patrick Marsh) Date: Mon, 29 Oct 2012 10:15:33 -0500 Subject: [Numpy-discussion] Regression in mpl: AttributeError: incompatible shape for a non-contiguous array In-Reply-To: References: <1351521233.2669.17.camel@sebastian-laptop> Message-ID: I've tracked down the problem to this commit: https://github.com/numpy/numpy/commit/c48156dfdc408f0a1e59ef54ac490cccbd6b8d73 Patrick.Marsh at buxton numpy> git bisect good c48156dfdc408f0a1e59ef54ac490cccbd6b8d73 is the first bad commit commit c48156dfdc408f0a1e59ef54ac490cccbd6b8d73 Author: Sebastian Berg Date: Sun Oct 21 18:50:28 2012 +0200 API: Change Flags Updateing to allow C-/F-contiguous arrays This changes UpdateFlags to ignore 1-dimensional axis when setting C-/F-contiguous flags. Updates both flags always now. --- Patrick Marsh Ph.D. Candidate / Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com On Mon, Oct 29, 2012 at 10:04 AM, Patrick Marsh wrote: > Turns out it isn't the commit I thought it was. I'm currently going > through a git bisect to track down the actual commit that introduced this > bug. I'll post back when I've found it. > > > PTM > --- > Patrick Marsh > Ph.D. Candidate / Liaison to the HWT > School of Meteorology / University of Oklahoma > Cooperative Institute for Mesoscale Meteorological Studies > National Severe Storms Laboratory > http://www.patricktmarsh.com > > > > On Mon, Oct 29, 2012 at 9:43 AM, Benjamin Root wrote: > >> >> >> On Mon, Oct 29, 2012 at 10:33 AM, Sebastian Berg < >> sebastian at sipsolutions.net> wrote: >> >>> Hey, >>> >>> On Mon, 2012-10-29 at 09:54 -0400, Benjamin Root wrote: >>> > This error started showing up in the test suite for mpl when using >>> > numpy master. >>> > >>> > AttributeError: incompatible shape for a non-contiguous array >>> > >>> > The tracebacks all point back to various code points where we are >>> > trying to set the shape of an array, e.g., >>> > >>> > offsets.shape = (-1, 2) >>> > >>> Could you give a hint what these arrays history (how it was created) and >>> maybe .shape/.strides is? Sounds like the array is not contiguous when >>> it is expected to be, or the attribute setting itself fails in some >>> corner cases on master? >>> >>> Regards, >>> >>> Sebastian >>> >>> >> The original reporter of the bug dug into the commit list and suspects it >> was this one: >> >> >> https://github.com/numpy/numpy/commit/02ebf8b3e7674a6b8a06636feaa6c761fcdf4e2d >> >> However, it might be earlier than that (he is currently doing a clean >> rebuild to make sure). >> >> As for the history: >> >> offsets = np.asanyarray(offsets) >> offsets.shape = (-1, 2) # Make it Nx2 >> >> Where "offsets" comes in from (possibly) user-supplied data. Nothing >> really all that special. I will see if I can get stride information. >> >> Ben Root >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Mon Oct 29 11:18:36 2012 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 29 Oct 2012 11:18:36 -0400 Subject: [Numpy-discussion] Regression in mpl: AttributeError: incompatible shape for a non-contiguous array In-Reply-To: References: <1351521233.2669.17.camel@sebastian-laptop> Message-ID: On Mon, Oct 29, 2012 at 11:04 AM, Patrick Marsh wrote: > Turns out it isn't the commit I thought it was. I'm currently going > through a git bisect to track down the actual commit that introduced this > bug. I'll post back when I've found it. > > > PTM > --- > Patrick Marsh > Ph.D. Candidate / Liaison to the HWT > School of Meteorology / University of Oklahoma > Cooperative Institute for Mesoscale Meteorological Studies > National Severe Storms Laboratory > http://www.patricktmarsh.com > > > > On Mon, Oct 29, 2012 at 9:43 AM, Benjamin Root wrote: > >> >> >> On Mon, Oct 29, 2012 at 10:33 AM, Sebastian Berg < >> sebastian at sipsolutions.net> wrote: >> >>> Hey, >>> >>> On Mon, 2012-10-29 at 09:54 -0400, Benjamin Root wrote: >>> > This error started showing up in the test suite for mpl when using >>> > numpy master. >>> > >>> > AttributeError: incompatible shape for a non-contiguous array >>> > >>> > The tracebacks all point back to various code points where we are >>> > trying to set the shape of an array, e.g., >>> > >>> > offsets.shape = (-1, 2) >>> > >>> Could you give a hint what these arrays history (how it was created) and >>> maybe .shape/.strides is? Sounds like the array is not contiguous when >>> it is expected to be, or the attribute setting itself fails in some >>> corner cases on master? >>> >>> Regards, >>> >>> Sebastian >>> >>> >> The original reporter of the bug dug into the commit list and suspects it >> was this one: >> >> >> https://github.com/numpy/numpy/commit/02ebf8b3e7674a6b8a06636feaa6c761fcdf4e2d >> >> However, it might be earlier than that (he is currently doing a clean >> rebuild to make sure). >> >> As for the history: >> >> offsets = np.asanyarray(offsets) >> offsets.shape = (-1, 2) # Make it Nx2 >> >> Where "offsets" comes in from (possibly) user-supplied data. Nothing >> really all that special. I will see if I can get stride information. >> >> Ben Root >> >> Further digging reveals that the code fails when the array is originally 1-D. I had an array with shape (2,) and stride (8,). The reshaping should result in a shape of (1, 2). Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrickmarshwx at gmail.com Mon Oct 29 11:23:12 2012 From: patrickmarshwx at gmail.com (Patrick Marsh) Date: Mon, 29 Oct 2012 10:23:12 -0500 Subject: [Numpy-discussion] Regression in mpl: AttributeError: incompatible shape for a non-contiguous array In-Reply-To: References: <1351521233.2669.17.camel@sebastian-laptop> Message-ID: I went ahead and filed a ticket (https://github.com/numpy/numpy/issues/2700) for historical purposes?and so those on the MPL list can follow up. PTM --- Patrick Marsh Ph.D. Candidate / Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com On Mon, Oct 29, 2012 at 10:18 AM, Benjamin Root wrote: > > > On Mon, Oct 29, 2012 at 11:04 AM, Patrick Marsh wrote: > >> Turns out it isn't the commit I thought it was. I'm currently going >> through a git bisect to track down the actual commit that introduced this >> bug. I'll post back when I've found it. >> >> >> PTM >> --- >> Patrick Marsh >> Ph.D. Candidate / Liaison to the HWT >> School of Meteorology / University of Oklahoma >> Cooperative Institute for Mesoscale Meteorological Studies >> National Severe Storms Laboratory >> http://www.patricktmarsh.com >> >> >> >> On Mon, Oct 29, 2012 at 9:43 AM, Benjamin Root wrote: >> >>> >>> >>> On Mon, Oct 29, 2012 at 10:33 AM, Sebastian Berg < >>> sebastian at sipsolutions.net> wrote: >>> >>>> Hey, >>>> >>>> On Mon, 2012-10-29 at 09:54 -0400, Benjamin Root wrote: >>>> > This error started showing up in the test suite for mpl when using >>>> > numpy master. >>>> > >>>> > AttributeError: incompatible shape for a non-contiguous array >>>> > >>>> > The tracebacks all point back to various code points where we are >>>> > trying to set the shape of an array, e.g., >>>> > >>>> > offsets.shape = (-1, 2) >>>> > >>>> Could you give a hint what these arrays history (how it was created) and >>>> maybe .shape/.strides is? Sounds like the array is not contiguous when >>>> it is expected to be, or the attribute setting itself fails in some >>>> corner cases on master? >>>> >>>> Regards, >>>> >>>> Sebastian >>>> >>>> >>> The original reporter of the bug dug into the commit list and suspects >>> it was this one: >>> >>> >>> https://github.com/numpy/numpy/commit/02ebf8b3e7674a6b8a06636feaa6c761fcdf4e2d >>> >>> However, it might be earlier than that (he is currently doing a clean >>> rebuild to make sure). >>> >>> As for the history: >>> >>> offsets = np.asanyarray(offsets) >>> offsets.shape = (-1, 2) # Make it Nx2 >>> >>> Where "offsets" comes in from (possibly) user-supplied data. Nothing >>> really all that special. I will see if I can get stride information. >>> >>> Ben Root >>> >>> > Further digging reveals that the code fails when the array is originally > 1-D. I had an array with shape (2,) and stride (8,). The reshaping should > result in a shape of (1, 2). > > Ben Root > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Mon Oct 29 11:24:54 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 29 Oct 2012 16:24:54 +0100 Subject: [Numpy-discussion] Regression in mpl: AttributeError: incompatible shape for a non-contiguous array In-Reply-To: References: <1351521233.2669.17.camel@sebastian-laptop> Message-ID: <1351524294.2669.20.camel@sebastian-laptop> Thanks a lot! I feared it would have something to do with those changes, turns out I tried with the wrong numpy version. But at least its not in 1.7. for sure so nothing to worry about :). I will have a look at it. On Mon, 2012-10-29 at 10:15 -0500, Patrick Marsh wrote: > I've tracked down the problem to this > commit: https://github.com/numpy/numpy/commit/c48156dfdc408f0a1e59ef54ac490cccbd6b8d73 > > > > > > > Patrick.Marsh at buxton numpy> git bisect good > c48156dfdc408f0a1e59ef54ac490cccbd6b8d73 is the first bad commit > commit c48156dfdc408f0a1e59ef54ac490cccbd6b8d73 > Author: Sebastian Berg > Date: Sun Oct 21 18:50:28 2012 +0200 > > > API: Change Flags Updateing to allow C-/F-contiguous arrays > > This changes UpdateFlags to ignore 1-dimensional axis when > setting C-/F-contiguous flags. Updates both flags always now. > > > > > > > > > > > --- > Patrick Marsh > Ph.D. Candidate / Liaison to the HWT > School of Meteorology / University of Oklahoma > Cooperative Institute for Mesoscale Meteorological Studies > National Severe Storms Laboratory > http://www.patricktmarsh.com > > > > On Mon, Oct 29, 2012 at 10:04 AM, Patrick Marsh > wrote: > Turns out it isn't the commit I thought it was. I'm currently > going through a git bisect to track down the actual commit > that introduced this bug. I'll post back when I've found it. > > > > > PTM > --- > Patrick Marsh > Ph.D. Candidate / Liaison to the HWT > School of Meteorology / University of Oklahoma > Cooperative Institute for Mesoscale Meteorological Studies > National Severe Storms Laboratory > http://www.patricktmarsh.com > > > > On Mon, Oct 29, 2012 at 9:43 AM, Benjamin Root > wrote: > > > > On Mon, Oct 29, 2012 at 10:33 AM, Sebastian Berg > wrote: > Hey, > > On Mon, 2012-10-29 at 09:54 -0400, Benjamin > Root wrote: > > This error started showing up in the test > suite for mpl when using > > numpy master. > > > > AttributeError: incompatible shape for a > non-contiguous array > > > > The tracebacks all point back to various > code points where we are > > trying to set the shape of an array, e.g., > > > > offsets.shape = (-1, 2) > > > > Could you give a hint what these arrays > history (how it was created) and > maybe .shape/.strides is? Sounds like the > array is not contiguous when > it is expected to be, or the attribute setting > itself fails in some > corner cases on master? > > Regards, > > Sebastian > > > > The original reporter of the bug dug into the commit > list and suspects it was this one: > > https://github.com/numpy/numpy/commit/02ebf8b3e7674a6b8a06636feaa6c761fcdf4e2d > > However, it might be earlier than that (he is > currently doing a clean rebuild to make sure). > > As for the history: > > offsets = np.asanyarray(offsets) > offsets.shape = (-1, 2) # Make > it Nx2 > > Where "offsets" comes in from (possibly) user-supplied > data. Nothing really all that special. I will see if > I can get stride information. > > Ben Root > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From heng at cantab.net Mon Oct 29 11:26:07 2012 From: heng at cantab.net (Henry Gomersall) Date: Mon, 29 Oct 2012 15:26:07 +0000 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: References: <1351007294.3585.5.camel@farnsworth> Message-ID: <1351524367.13677.13.camel@farnsworth> On Mon, 2012-10-29 at 11:11 -0400, Fr?d?ric Bastien wrote: > > Assuming of course all the relevant backends are up to scratch. > > > > Is there a fundamental reason why targetting a CPU through OpenCL is > > worse than doing it exclusively through C or C++? > > First, opencl do not allow us to do pointor arythmetique. So when > taking a slice of an ndarray, we can't just more the pointor. So we > need to change the object structure. > > I didn't do any speed anylysis of this, but I think that by using > OpenCL, it would have a bigger overhead. So it is only useful for > "big" ndarray. I don't have any size in mind too. I don't know, but if > we could access the opencl data directly from C/C++, we could bypass > this for small array if we want. But maybe this is not possible! My understanding is that when running OpenCL on CPU, one can simply map memory from a host pointer using CL_MEM_USE_HOST_PTR during buffer creation. On a CPU, this will result in no copies being made. The overhead is clearly an issue, and was the subject of my question. I wouldn't be surprised to find that the speedup associated with the free multithreading that comes with OpenCL on CPU, along with the vector data types mapping nicely to SSE etc, would make OpenCL on CPU faster on any reasonably sized array. It strikes me that if there is a neat way in which numpy objects can be represented by coherent versions in both main memory and device memory, then OpenCL could be used when it makes sense (either on CPU or GPU), and the CPU natively when _it_ makes sense. Henry From nouiz at nouiz.org Mon Oct 29 11:49:31 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Mon, 29 Oct 2012 11:49:31 -0400 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: <1351524367.13677.13.camel@farnsworth> References: <1351007294.3585.5.camel@farnsworth> <1351524367.13677.13.camel@farnsworth> Message-ID: That is possible. The gpu nd array project I talked above work on the CPU and the GPU in OpenCL and with CUDA. But there is much stuff that is in numpy that we didn't ported. We started integrating it into Theano. So this mean the GPU code from Theano will be ported to this project, so there will be more code available later. If some people are willing to help/collaborate don't hesitate. I think we should collaborate much more on GPU/OpenCL code then we do now. That was part of the goal of gpu nd array. There is the PyCUDA and PyOpenCL authors that also collaborate with us. Fred On Mon, Oct 29, 2012 at 11:26 AM, Henry Gomersall wrote: > On Mon, 2012-10-29 at 11:11 -0400, Fr?d?ric Bastien wrote: >> > Assuming of course all the relevant backends are up to scratch. >> > >> > Is there a fundamental reason why targetting a CPU through OpenCL is >> > worse than doing it exclusively through C or C++? >> >> First, opencl do not allow us to do pointor arythmetique. So when >> taking a slice of an ndarray, we can't just more the pointor. So we >> need to change the object structure. >> >> I didn't do any speed anylysis of this, but I think that by using >> OpenCL, it would have a bigger overhead. So it is only useful for >> "big" ndarray. I don't have any size in mind too. I don't know, but if >> we could access the opencl data directly from C/C++, we could bypass >> this for small array if we want. But maybe this is not possible! > > My understanding is that when running OpenCL on CPU, one can simply map > memory from a host pointer using CL_MEM_USE_HOST_PTR during buffer > creation. On a CPU, this will result in no copies being made. > > The overhead is clearly an issue, and was the subject of my question. I > wouldn't be surprised to find that the speedup associated with the free > multithreading that comes with OpenCL on CPU, along with the vector data > types mapping nicely to SSE etc, would make OpenCL on CPU faster on any > reasonably sized array. > > It strikes me that if there is a neat way in which numpy objects can be > represented by coherent versions in both main memory and device memory, > then OpenCL could be used when it makes sense (either on CPU or GPU), > and the CPU natively when _it_ makes sense. > > Henry > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From heng at cantab.net Mon Oct 29 11:53:50 2012 From: heng at cantab.net (Henry Gomersall) Date: Mon, 29 Oct 2012 15:53:50 +0000 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: References: <1351007294.3585.5.camel@farnsworth> <1351524367.13677.13.camel@farnsworth> Message-ID: <1351526030.13677.15.camel@farnsworth> On Mon, 2012-10-29 at 11:49 -0400, Fr?d?ric Bastien wrote: > That is possible. > Great! > The gpu nd array project I talked above work on the CPU and the GPU in > OpenCL and with CUDA. But there is much stuff that is in numpy that we > didn't ported. This is: https://github.com/inducer/compyte/wiki right? > > We started integrating it into Theano. So this mean the GPU code from > Theano will be ported to this project, so there will be more code > available later. > > If some people are willing to help/collaborate don't hesitate. I think > we should collaborate much more on GPU/OpenCL code then we do now. > That was part of the goal of gpu nd array. There is the PyCUDA and > PyOpenCL authors that also collaborate with us. I'm interested, but somewhat an opencl rooky with limited time! I'll have a play when I have a moment. Cheers, Henry From nouiz at nouiz.org Mon Oct 29 12:01:51 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Mon, 29 Oct 2012 12:01:51 -0400 Subject: [Numpy-discussion] NumPy to CPU+GPU compiler, looking for tests In-Reply-To: <1351526030.13677.15.camel@farnsworth> References: <1351007294.3585.5.camel@farnsworth> <1351524367.13677.13.camel@farnsworth> <1351526030.13677.15.camel@farnsworth> Message-ID: On Mon, Oct 29, 2012 at 11:53 AM, Henry Gomersall wrote: > On Mon, 2012-10-29 at 11:49 -0400, Fr?d?ric Bastien wrote: >> That is possible. >> > Great! Just to be clear, I mean this is possible to make it work. We do not do that for now. Also, sharing the memory on the CPU and GPU is not trivial if we want that to be efficient. So I would be surprised that we make it work except by copy when needed. Most of the time, if the data is on the GPU, it will be better to keep it there, as the transfer overhead will probably be too high to gain efficiency by using the GPU. >> The gpu nd array project I talked above work on the CPU and the GPU in >> OpenCL and with CUDA. But there is much stuff that is in numpy that we >> didn't ported. > > This is: https://github.com/inducer/compyte/wiki right? yes >> >> We started integrating it into Theano. So this mean the GPU code from >> Theano will be ported to this project, so there will be more code >> available later. >> >> If some people are willing to help/collaborate don't hesitate. I think >> we should collaborate much more on GPU/OpenCL code then we do now. >> That was part of the goal of gpu nd array. There is the PyCUDA and >> PyOpenCL authors that also collaborate with us. > > I'm interested, but somewhat an opencl rooky with limited time! I'll > have a play when I have a moment. Great. I think that people work better when they work on stuff that motivate them. If you have any algo event if simple that you want to work on do not hesitate. Fred From larry.paltrow at gmail.com Mon Oct 29 12:05:49 2012 From: larry.paltrow at gmail.com (Larry Paltrow) Date: Mon, 29 Oct 2012 09:05:49 -0700 Subject: [Numpy-discussion] nan result from np.linalg.lstsq() In-Reply-To: References: Message-ID: doh! thanks On Mon, Oct 29, 2012 at 2:36 AM, Pauli Virtanen wrote: > Larry Paltrow gmail.com> writes: > [clip] > > np.all(~np.isnan(data)) > > >>> False > > > > Seems to be all non-nan > > No, it means you have NaNs in your data. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.warde.farley at gmail.com Mon Oct 29 16:37:49 2012 From: d.warde.farley at gmail.com (David Warde-Farley) Date: Mon, 29 Oct 2012 16:37:49 -0400 Subject: [Numpy-discussion] numpy.savez(_compressed) in loop In-Reply-To: <1896437.YHDg1QmCBS@edge> References: <1896437.YHDg1QmCBS@edge> Message-ID: On Mon, Oct 29, 2012 at 6:29 AM, Radek Machulka wrote: > Hi, > > is there a way how to save more arrays into single npy (npz if possible) file > in loop? Something like: > > fw = open('foo.bar', 'wb') > while foo: > arr = np.array(bar) > np.savez_compressed(fw, arr) > fw.close() > > Or some workaround maybe? I go through hundreds of thousands arrays and can > not keep them in memory. Yes, I can save each array into single file, but I > would better have them all in single one. Note that you can save several npy arrays into a single file descriptor. The NPY format is smart enough that given the header, numpy.load() knows how many items to load. Moreover, if passed an open file object, numpy.load() leaves its cursor position intact, such that something like this works: In [1]: import numpy as np In [2]: f = open('x.dat', 'wb') In [3]: np.save(f, np.arange(5)) In [4]: np.save(f, np.arange(10)) In [5]: f.close() In [6]: with open('x.dat', 'rb') as f: ...: while True: ...: try: ...: print np.load(f) ...: except IOError: ...: break ...: [0 1 2 3 4] [0 1 2 3 4 5 6 7 8 9] This is something of a hack. You could do better than catching the IOError, e.g. saving a 0d array containing the number of forthcoming arrays, or a sentinel array containing "None" at the end to indicate there are no more real arrays to serialize. Like Pauli said, it's probably worthwhile to consider HDF5. David From paul.anton.letnes at gmail.com Tue Oct 30 13:25:32 2012 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Tue, 30 Oct 2012 18:25:32 +0100 Subject: [Numpy-discussion] numpy.savez(_compressed) in loop In-Reply-To: <1896437.YHDg1QmCBS@edge> References: <1896437.YHDg1QmCBS@edge> Message-ID: On 29. okt. 2012, at 11:29, Radek Machulka wrote: > Hi, > > is there a way how to save more arrays into single npy (npz if possible) file > in loop? Something like: > > fw = open('foo.bar', 'wb') > while foo: > arr = np.array(bar) > np.savez_compressed(fw, arr) > fw.close() > > Or some workaround maybe? I go through hundreds of thousands arrays and can > not keep them in memory. Yes, I can save each array into single file, but I > would better have them all in single one. As Pauli said, hdf5 is nicer for such things. I foresee: import h5py fw = h5py.File('foo.hdf5', 'w') while foo: arr = np.array(bar) fw['arr'] = arr fw.close() Good luck Paul From ralf.gommers at gmail.com Tue Oct 30 18:08:36 2012 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 30 Oct 2012 23:08:36 +0100 Subject: [Numpy-discussion] ticket 2228: Scientific package seeing ABI change in 1.6.x Message-ID: Hi, Ticket 2228 says ABI was broken in 1.6.x. Specifically, NPY_CHAR in the NPY_TYPES enum seems to be have been moved. Can anyone comment on why the 3 datetime related values were inserted instead of appended? Ralf P.S. that ticket has escaped the Github move. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Oct 30 21:46:06 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 30 Oct 2012 19:46:06 -0600 Subject: [Numpy-discussion] ticket 2228: Scientific package seeing ABI change in 1.6.x In-Reply-To: References: Message-ID: On Tue, Oct 30, 2012 at 4:08 PM, Ralf Gommers wrote: > Hi, > > Ticket 2228 says ABI was broken in 1.6.x. Specifically, NPY_CHAR in the > NPY_TYPES enum seems to be have been moved. Can anyone comment on why the 3 > datetime related values were inserted instead of appended? > I don't know, although having NPY_CHAR after NPY_NTYPES goes back to 1.0.3 NPY_NTYPES, NPY_NOTYPE, NPY_CHAR, /* special flag */ And I expect it was desired to keep it there on the expectation that there was a reason for it. The decision not to append was in 1.4.0 NPY_DATETIME, NPY_TIMEDELTA, NPY_NTYPES, NPY_NOTYPE, NPY_CHAR, /* special flag */ And probably due to Robert Kern or Travis, IIRC who worked on getting it in. I don't see a good way to get around the ABI break, I think the question going forward needs to be whether we leave it after NPY_NTYPES or make it part of the unchanging ABI, and I suspect we need to know what the 'special flag' comment means before we can make that decision. My suspicion is that it wasn't considered a real numeric type, but rather a flag marking a special string type, in which case it probably doesn't really belong among the types, which I think is also indicated by NPY_NOTYPE. Moving NPY_CHAR could have implications we would want to check, but I'd generally favor moving it all else being equal. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Oct 30 23:26:50 2012 From: travis at continuum.io (Travis Oliphant) Date: Tue, 30 Oct 2012 22:26:50 -0500 Subject: [Numpy-discussion] ticket 2228: Scientific package seeing ABI change in 1.6.x In-Reply-To: References: Message-ID: The NPY_CHAR is not a "real type". There are no type-coercion functions attached to it nor ufuncs nor a full dtype object. However, it is used to mimic old Numeric character arrays (especially for copying a string). It should have been deprecated before changing the ABI. I don't think it was realized that it was part of the ABI (mostly for older codes that depended on Numeric). I think it was just another oversight that inserting type-codes changes this part of the ABI. The positive side is that It's a small part of the ABI and not many codes should depend on it. At this point, I'm not sure what can be done, except to document that NPY_CHAR has been deprecated in 1.7.0 and remove it in 1.8.0 to avoid future ABI difficulties. The short answer, is that codes that use NPY_CHAR must be recompiled to be compatible with 1.6.0. -Travis On Oct 30, 2012, at 8:46 PM, Charles R Harris wrote: > > > On Tue, Oct 30, 2012 at 4:08 PM, Ralf Gommers wrote: > Hi, > > Ticket 2228 says ABI was broken in 1.6.x. Specifically, NPY_CHAR in the NPY_TYPES enum seems to be have been moved. Can anyone comment on why the 3 datetime related values were inserted instead of appended? > > I don't know, although having NPY_CHAR after NPY_NTYPES goes back to 1.0.3 > > > NPY_NTYPES, > NPY_NOTYPE, > > NPY_CHAR, /* special flag */ > > And I expect it was desired to keep it there on the expectation that there was a reason for it. The decision not to append was in 1.4.0 > > NPY_DATETIME, NPY_TIMEDELTA, > NPY_NTYPES, > NPY_NOTYPE, > NPY_CHAR, /* special flag */ > > And probably due to Robert Kern or Travis, IIRC who worked on getting it in. > > I don't see a good way to get around the ABI break, I think the question going forward needs to be whether we leave it after NPY_NTYPES or make it part of the unchanging ABI, and I suspect we need to know what the 'special flag' comment means before we can make that decision. My suspicion is that it wasn't considered a real numeric type, but rather a flag marking a special string type, in which case it probably doesn't really belong among the types, which I think is also indicated by NPY_NOTYPE. Moving NPY_CHAR could have implications we would want to check, but I'd generally favor moving it all else being equal. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Oct 31 08:05:37 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 31 Oct 2012 06:05:37 -0600 Subject: [Numpy-discussion] ticket 2228: Scientific package seeing ABI change in 1.6.x In-Reply-To: References: Message-ID: On Tue, Oct 30, 2012 at 9:26 PM, Travis Oliphant wrote: > The NPY_CHAR is not a "real type". There are no type-coercion functions > attached to it nor ufuncs nor a full dtype object. However, it is used > to mimic old Numeric character arrays (especially for copying a string). > > It should have been deprecated before changing the ABI. I don't think it > was realized that it was part of the ABI (mostly for older codes that > depended on Numeric). I think it was just another oversight that > inserting type-codes changes this part of the ABI. > > The positive side is that It's a small part of the ABI and not many codes > should depend on it. At this point, I'm not sure what can be done, except > to document that NPY_CHAR has been deprecated in 1.7.0 and remove it in > 1.8.0 to avoid future ABI difficulties. > > The short answer, is that codes that use NPY_CHAR must be recompiled to be > compatible with 1.6.0. > > IIRC, it was proposed to remove it at one point, but the STScI folks wanted to keep it because their software depended on it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From thouis at gmail.com Wed Oct 31 11:13:04 2012 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Wed, 31 Oct 2012 11:13:04 -0400 Subject: [Numpy-discussion] ticket 2228: Scientific package seeing ABI change in 1.6.x In-Reply-To: References: Message-ID: On Tue, Oct 30, 2012 at 6:08 PM, Ralf Gommers wrote: > [...] > P.S. that ticket has escaped the Github move. The github move only included up to 2225 or so. Anything after that will have to be imported when Trac is redirected to github. I believe David Cournapeau is going to do that at some point (hopefully soon). Ray From klonuo at gmail.com Wed Oct 31 16:04:29 2012 From: klonuo at gmail.com (klo uo) Date: Wed, 31 Oct 2012 21:04:29 +0100 Subject: [Numpy-discussion] Simple question about scatter plot graph Message-ID: I guess this is simple for many on this list I have 20 samples with ~420 points each, and scatter plot (lag_plot) for all samples is attached 16x16 grid pattern is easily visible, but I can't make the meaning Thanks [image: Inline image 1] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 129862 bytes Desc: not available URL: From Catherine.M.Moroney at jpl.nasa.gov Wed Oct 31 19:23:33 2012 From: Catherine.M.Moroney at jpl.nasa.gov (Moroney, Catherine M (388D)) Date: Wed, 31 Oct 2012 23:23:33 +0000 Subject: [Numpy-discussion] using numpy.argmax to index into another array Message-ID: <7B9A5D9B4E8E10489D50EE851ABCA73E35A0311F@ap-embx-sp10.RES.AD.JPL> Hello Everybody, I have the following problem that I would be interested in finding an easy/elegant solution to. I've got it working, but my solution is exceedingly clunky and I'm sure that there must be a better way. Here is the (boiled-down) problem: I have 2 different 3-d array, shaped (4,4,4), "a" and "b" I can find the max values and location of those max values in "a" in the 0-th dimension using max and argmax resulting in a 4x4 matrix. So far, very easy. I then want to find the values in "b" that correspond to the maximum values in a. This is where I got stuck. Below find the sample code I used (pretty clumsy stuff ...) Can somebody show a better (less clumsy) way of finding bmax in the code excerpt below? >>> a = numpy.random.randint(0, high=15, size=(4,4,4)) >>> b = numpy.random.randint(0, high=15, size=(4,4,4)) >>> amax = a.argmax(axis=0) >>> idx2 = [0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3] >>> idx1 = [0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3] >>> idx_all = (amax.flatten(), numpy.array(idx1), numpy.array(idx2)) >>> bmax = b[idx_all].reshape(4,4) >>> a array([[[ 7, 1, 7, 10], [11, 5, 9, 0], [13, 1, 13, 14], [ 4, 13, 4, 7]], [[ 2, 11, 4, 5], [ 6, 12, 14, 9], [ 0, 4, 5, 14], [13, 5, 3, 4]], [[ 1, 4, 5, 4], [ 7, 13, 1, 11], [ 9, 2, 10, 4], [ 3, 4, 10, 13]], [[14, 6, 7, 1], [ 8, 0, 8, 2], [ 6, 8, 7, 12], [ 4, 9, 0, 12]]]) >>> idx array([[3, 1, 0, 0], [0, 2, 1, 2], [0, 3, 0, 0], [1, 0, 2, 2]]) >>> b array([[[ 4, 2, 3, 2], [ 9, 6, 1, 4], [ 1, 10, 13, 11], [10, 8, 1, 10]], [[ 3, 13, 7, 7], [ 4, 7, 9, 7], [ 3, 13, 10, 10], [13, 8, 14, 6]], [[14, 10, 13, 6], [13, 2, 12, 12], [ 9, 6, 3, 8], [ 0, 7, 8, 11]], [[12, 10, 3, 0], [11, 7, 0, 4], [ 9, 7, 11, 12], [ 7, 12, 1, 1]]]) >>> bmax = b[idx_all].reshape(4,4) >>> bmax array([[12, 13, 3, 2], [ 9, 2, 9, 12], [ 1, 7, 13, 11], [13, 8, 8, 11]]) Catherine From chris.barker at noaa.gov Wed Oct 31 20:08:12 2012 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 31 Oct 2012 17:08:12 -0700 Subject: [Numpy-discussion] numpy.savez(_compressed) in loop In-Reply-To: References: <1896437.YHDg1QmCBS@edge> Message-ID: On Mon, Oct 29, 2012 at 1:37 PM, David Warde-Farley wrote: > This is something of a hack. but a cool one... > Like Pauli said, it's probably worthwhile to consider HDF5. But HDF5 is a big dependency -- it can be a pain to build. IT's great for what it does well, and great for interchanging data with other systems that use it, but if all you need to do is save and reload arrays, it's pretty heavy weight -- that's why save and savez were written. It looks like the python zipfile module supports adding to existing zip files, to It should be pretty easy to add this functionality to savez (and savez_compressed) pretty cleanly. I'll bet a patch would be accepted! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From josef.pktd at gmail.com Wed Oct 31 20:17:39 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 31 Oct 2012 20:17:39 -0400 Subject: [Numpy-discussion] Simple question about scatter plot graph In-Reply-To: References: Message-ID: On Wed, Oct 31, 2012 at 4:04 PM, klo uo wrote: > I guess this is simple for many on this list > I have 20 samples with ~420 points each, and scatter plot (lag_plot) for > all samples is attached > 16x16 grid pattern is easily visible, but I can't make the meaning > I don't have much of an idea what we are supposed to see, except that there might not be much autocorrelation. Is this grided data and some scatter points might actually be many points on top of each other so we don't see all points and not the frequencey distribution? Is y on a continuous, metric scale or are all grid points different categories, observations. Josef > > Thanks > [image: Inline image 1] > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 129862 bytes Desc: not available URL: From wardefar at iro.umontreal.ca Wed Oct 31 20:22:25 2012 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Wed, 31 Oct 2012 20:22:25 -0400 Subject: [Numpy-discussion] using numpy.argmax to index into another array In-Reply-To: <7B9A5D9B4E8E10489D50EE851ABCA73E35A0311F@ap-embx-sp10.RES.AD.JPL> References: <7B9A5D9B4E8E10489D50EE851ABCA73E35A0311F@ap-embx-sp10.RES.AD.JPL> Message-ID: On Wed, Oct 31, 2012 at 7:23 PM, Moroney, Catherine M (388D) wrote: > Hello Everybody, > > I have the following problem that I would be interested in finding an easy/elegant solution to. > I've got it working, but my solution is exceedingly clunky and I'm sure that there must be a > better way. > > Here is the (boiled-down) problem: > > I have 2 different 3-d array, shaped (4,4,4), "a" and "b" > > I can find the max values and location of those max values in "a" in the 0-th dimension > using max and argmax resulting in a 4x4 matrix. So far, very easy. > > I then want to find the values in "b" that correspond to the maximum values in a. > This is where I got stuck. > > Below find the sample code I used (pretty clumsy stuff ...) > Can somebody show a better (less clumsy) way of finding bmax > in the code excerpt below? Hi Catherine, It's not a huge improvement, but one simple thing is that you can generate those index arrays easily and avoid the pesky reshaping at the end like so: In [27]: idx2, idx3 = numpy.mgrid[0:4, 0:4] In [28]: b[amax, idx2, idx3] Out[28]: array([[12, 1, 13, 3], [ 8, 11, 9, 10], [11, 11, 1, 10], [ 5, 7, 9, 5]]) In [29]: b[amax, idx2, idx3] == bmax Out[29]: array([[ True, True, True, True], [ True, True, True, True], [ True, True, True, True], [ True, True, True, True]], dtype=bool) numpy.ogrid would work here too, and will use a bit less memory. mgrid and ogrid are special objects from the numpy.lib.index_tricks module that generate, respectively, a "mesh grid" and an "open mesh grid" (where the unchanging dimensions are of length 1) when indexed like so. In the latter case, broadcasting takes effect when you index with them. Cheers, David From sebastian at sipsolutions.net Wed Oct 31 20:30:53 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 01 Nov 2012 01:30:53 +0100 Subject: [Numpy-discussion] using numpy.argmax to index into another array In-Reply-To: References: <7B9A5D9B4E8E10489D50EE851ABCA73E35A0311F@ap-embx-sp10.RES.AD.JPL> Message-ID: <1351729853.2570.6.camel@sebastian-laptop> Hey, On Wed, 2012-10-31 at 20:22 -0400, David Warde-Farley wrote: > On Wed, Oct 31, 2012 at 7:23 PM, Moroney, Catherine M (388D) > wrote: > > Hello Everybody, > > > > I have the following problem that I would be interested in finding an easy/elegant solution to. > > I've got it working, but my solution is exceedingly clunky and I'm sure that there must be a > > better way. > > > > Here is the (boiled-down) problem: > > > > I have 2 different 3-d array, shaped (4,4,4), "a" and "b" > > > > I can find the max values and location of those max values in "a" in the 0-th dimension > > using max and argmax resulting in a 4x4 matrix. So far, very easy. > > > > I then want to find the values in "b" that correspond to the maximum values in a. > > This is where I got stuck. > > > > Below find the sample code I used (pretty clumsy stuff ...) > > Can somebody show a better (less clumsy) way of finding bmax > > in the code excerpt below? > Its a bit off-label usage (as the documentation says), so maybe its slower for larger arrays, but as long as you choose along axis 0, you could also use: np.choose(amax, b) > Hi Catherine, > > It's not a huge improvement, but one simple thing is that you can > generate those index arrays easily and avoid the pesky reshaping at > the end like so: > > In [27]: idx2, idx3 = numpy.mgrid[0:4, 0:4] > > In [28]: b[amax, idx2, idx3] > Out[28]: > array([[12, 1, 13, 3], > [ 8, 11, 9, 10], > [11, 11, 1, 10], > [ 5, 7, 9, 5]]) > > In [29]: b[amax, idx2, idx3] == bmax > Out[29]: > array([[ True, True, True, True], > [ True, True, True, True], > [ True, True, True, True], > [ True, True, True, True]], dtype=bool) > > numpy.ogrid would work here too, and will use a bit less memory. mgrid > and ogrid are special objects from the numpy.lib.index_tricks module > that generate, respectively, a "mesh grid" and an "open mesh grid" > (where the unchanging dimensions are of length 1) when indexed like > so. In the latter case, broadcasting takes effect when you index with > them. > > Cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sebastian at sipsolutions.net Wed Oct 31 20:42:27 2012 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 01 Nov 2012 01:42:27 +0100 Subject: [Numpy-discussion] using numpy.argmax to index into another array In-Reply-To: <1351729853.2570.6.camel@sebastian-laptop> References: <7B9A5D9B4E8E10489D50EE851ABCA73E35A0311F@ap-embx-sp10.RES.AD.JPL> <1351729853.2570.6.camel@sebastian-laptop> Message-ID: <1351730547.2570.8.camel@sebastian-laptop> On Thu, 2012-11-01 at 01:30 +0100, Sebastian Berg wrote: > Hey, > > On Wed, 2012-10-31 at 20:22 -0400, David Warde-Farley wrote: > > On Wed, Oct 31, 2012 at 7:23 PM, Moroney, Catherine M (388D) > > wrote: > > > Hello Everybody, > > > > > > I have the following problem that I would be interested in finding an easy/elegant solution to. > > > I've got it working, but my solution is exceedingly clunky and I'm sure that there must be a > > > better way. > > > > > > Here is the (boiled-down) problem: > > > > > > I have 2 different 3-d array, shaped (4,4,4), "a" and "b" > > > > > > I can find the max values and location of those max values in "a" in the 0-th dimension > > > using max and argmax resulting in a 4x4 matrix. So far, very easy. > > > > > > I then want to find the values in "b" that correspond to the maximum values in a. > > > This is where I got stuck. > > > > > > Below find the sample code I used (pretty clumsy stuff ...) > > > Can somebody show a better (less clumsy) way of finding bmax > > > in the code excerpt below? > > > > Its a bit off-label usage (as the documentation says), so maybe its > slower for larger arrays, but as long as you choose along axis 0, you > could also use: > > np.choose(amax, b) > Sorry, nevermind that. choose just is not made for this, it would only works for very small arrays... > > Hi Catherine, > > > > It's not a huge improvement, but one simple thing is that you can > > generate those index arrays easily and avoid the pesky reshaping at > > the end like so: > > > > In [27]: idx2, idx3 = numpy.mgrid[0:4, 0:4] > > > > In [28]: b[amax, idx2, idx3] > > Out[28]: > > array([[12, 1, 13, 3], > > [ 8, 11, 9, 10], > > [11, 11, 1, 10], > > [ 5, 7, 9, 5]]) > > > > In [29]: b[amax, idx2, idx3] == bmax > > Out[29]: > > array([[ True, True, True, True], > > [ True, True, True, True], > > [ True, True, True, True], > > [ True, True, True, True]], dtype=bool) > > > > numpy.ogrid would work here too, and will use a bit less memory. mgrid > > and ogrid are special objects from the numpy.lib.index_tricks module > > that generate, respectively, a "mesh grid" and an "open mesh grid" > > (where the unchanging dimensions are of length 1) when indexed like > > so. In the latter case, broadcasting takes effect when you index with > > them. > > > > Cheers, > > > > David > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From klonuo at gmail.com Wed Oct 31 20:59:07 2012 From: klonuo at gmail.com (klo uo) Date: Thu, 1 Nov 2012 01:59:07 +0100 Subject: [Numpy-discussion] Simple question about scatter plot graph In-Reply-To: References: Message-ID: Thanks for your reply I suppose, variable length signals are split on equal parts and dominant harmonic is extracted. Then scatter plot shows this pattern, which has some low correlation, but I can't abstract what could be concluded from grid pattern, as I lack statistical knowledge. Maybe it's saying that data is quantized, which can't be easily seen from single sample bar chart, but perhaps scatter plot suggests that? That's only my wild guess On Thu, Nov 1, 2012 at 1:17 AM, wrote: > I don't have much of an idea what we are supposed to see, except that > there might not be much autocorrelation. > > Is this grided data and some scatter points might actually be many points > on top of each other so we don't see all points and not the frequencey > distribution? > Is y on a continuous, metric scale or are all grid points different > categories, observations. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Oct 31 21:12:19 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 31 Oct 2012 21:12:19 -0400 Subject: [Numpy-discussion] Simple question about scatter plot graph In-Reply-To: References: Message-ID: On Wed, Oct 31, 2012 at 8:59 PM, klo uo wrote: > Thanks for your reply > > I suppose, variable length signals are split on equal parts and dominant > harmonic is extracted. Then scatter plot shows this pattern, which has some > low correlation, but I can't abstract what could be concluded from grid > pattern, as I lack statistical knowledge. > Maybe it's saying that data is quantized, which can't be easily seen from > single sample bar chart, but perhaps scatter plot suggests that? That's only > my wild guess http://pandasplotting.blogspot.ca/2012/06/lag-plot.html In general you would see a lag autocorrelation structure in the plot. My guess is that even if there is a pattern in your data we might not see it because we don't see plots that are plotted on top of each other. We only see the support of the y_t, y_{t+1} transition (points that are at least once in the sample), but not the frequencies (or conditional distribution). If that's the case, then reduce alpha level so many points on top of each other are darker, or colorcode the histogram for each y_t: bincount for each y_t and normalize, or use np.histogram directly for each y_t, then assign to each point a colorscale depending on it's frequency. Did you calculate the correlation? (But maybe linear correlation won't show much.) Josef > > > > On Thu, Nov 1, 2012 at 1:17 AM, wrote: >> >> I don't have much of an idea what we are supposed to see, except that >> there might not be much autocorrelation. >> >> Is this grided data and some scatter points might actually be many points >> on top of each other so we don't see all points and not the frequencey >> distribution? >> Is y on a continuous, metric scale or are all grid points different >> categories, observations. > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ben.root at ou.edu Wed Oct 31 21:24:35 2012 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 31 Oct 2012 21:24:35 -0400 Subject: [Numpy-discussion] Simple question about scatter plot graph In-Reply-To: References: Message-ID: On Wednesday, October 31, 2012, wrote: > On Wed, Oct 31, 2012 at 8:59 PM, klo uo > > wrote: > > Thanks for your reply > > > > I suppose, variable length signals are split on equal parts and dominant > > harmonic is extracted. Then scatter plot shows this pattern, which has > some > > low correlation, but I can't abstract what could be concluded from grid > > pattern, as I lack statistical knowledge. > > Maybe it's saying that data is quantized, which can't be easily seen from > > single sample bar chart, but perhaps scatter plot suggests that? That's > only > > my wild guess > > http://pandasplotting.blogspot.ca/2012/06/lag-plot.html > In general you would see a lag autocorrelation structure in the plot. > > My guess is that even if there is a pattern in your data we might not > see it because we don't see plots that are plotted on top of each > other. We only see the support of the y_t, y_{t+1} transition (points > that are at least once in the sample), but not the frequencies (or > conditional distribution). > > If that's the case, then > reduce alpha level so many points on top of each other are darker, or > colorcode the histogram for each y_t: bincount for each y_t and > normalize, or use np.histogram directly for each y_t, then assign to > each point a colorscale depending on it's frequency. > > Did you calculate the correlation? (But maybe linear correlation won't > show much.) > > Josef The answer is hexbin() in matplotlib when you have many points laying on or near each other. Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From klonuo at gmail.com Wed Oct 31 21:43:41 2012 From: klonuo at gmail.com (klo uo) Date: Thu, 1 Nov 2012 02:43:41 +0100 Subject: [Numpy-discussion] Simple question about scatter plot graph In-Reply-To: References: Message-ID: OK, thanks guys for your suggestions, which I'll try tomorrow I did correlation first, but no significant values Then I did linear regression, one sample to rest and while there I spotted this grid pattern I was using pandas lag_plot, but it's same plot when I do MPL scatter one sample on others -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkhilmer at gmail.com Wed Oct 31 21:57:07 2012 From: jkhilmer at gmail.com (Jonathan Hilmer) Date: Wed, 31 Oct 2012 19:57:07 -0600 Subject: [Numpy-discussion] Simple question about scatter plot graph In-Reply-To: References: Message-ID: Obviously there are some real patterns there, but when interpreting low-resolution plots visually, be careful of Moire effects: view the following image at multiple zoom levels as an example. http://upload.wikimedia.org/wikipedia/commons/4/42/Divers_-_Illustrated_London_News_Feb_6_1873-2.PNG My own data is extremely deceptive when viewed on a computer monitor at typical resolutions, and physical printouts or zoomed-in variants show dramatically different patterns. Jonathan On Wed, Oct 31, 2012 at 7:43 PM, klo uo wrote: > OK, thanks guys for your suggestions, which I'll try tomorrow > > I did correlation first, but no significant values > Then I did linear regression, one sample to rest and while there I spotted > this grid pattern > I was using pandas lag_plot, but it's same plot when I do MPL scatter one > sample on others > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pwang at continuum.io Wed Oct 31 23:20:03 2012 From: pwang at continuum.io (Peter Wang) Date: Wed, 31 Oct 2012 23:20:03 -0400 Subject: [Numpy-discussion] Blaze In-Reply-To: References: Message-ID: On Sat, Oct 27, 2012 at 7:14 PM, Charles R Harris wrote: > Thanks. I've been trying to keep an eye on this but it is difficult to > find the details. I think the beta was scheduled for sometime in December. > Hey guys, We're hoping to get a first release out by the end of the November, per Stephen's slides. It's taken a while to get some of the fundamentals hashed out to a point that we were happy with, and that could form a solid core to iterate on. Cheers, Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: