From fperez.net at gmail.com Wed Jul 1 03:40:39 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 1 Jul 2009 00:40:39 -0700 Subject: [Numpy-discussion] Tutorial topics for SciPy'09 Conference In-Reply-To: References: Message-ID: Hi, On Mon, Jun 1, 2009 at 10:20 PM, Fernando Perez wrote: > The time for the Scipy'09 conference is rapidly approaching, and we > would like to both announce the plan for tutorials and solicit > feedback from everyone on topics of interest. rather than rehash much here, where it's not easy to paste a table, I've posted a note with the poll results here: http://fdoperez.blogspot.com/2009/06/scipy-advanced-tutorials-results.html The short and plain-text-friendly version is the final topic ranking: 1 Advanced topics in matplotlib use 2 Advanced numpy 3 Designing scientific interfaces with Traits 4 Mayavi/TVTK 5 Cython 6 Symbolic computing with sympy 7 Statistics with Scipy 8 Using GPUs with PyCUDA 9 Testing strategies for scientific codes 10 Parallel computing in Python and mpi4py 11 Sparse Linear Algebra with Scipy 12 Structured and record arrays in numpy 13 Design patterns for efficient iterator-based scientific codes 14 Sage 15 The TimeSeries scikit 16 Hermes: high order Finite Element Methods 17 Graph theory with NetworkX We're currently contacting speakers, and we'll let you know once a final list is made with confirmed speakers. Cheers, f From emmanuelle.gouillart at normalesup.org Wed Jul 1 04:17:51 2009 From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart) Date: Wed, 1 Jul 2009 10:17:51 +0200 Subject: [Numpy-discussion] np.memmap and memory usage Message-ID: <20090701081751.GA6814@phare.normalesup.org> Hello, I'm using numpy.memmap to open big 3-D arrays of Xray tomography data. After I have created a new array using memmap, I modify the contrast of every Z-slice (along the first dimension) inside a for loop, for a better visualization of the data. Although I call memmap.flush after each modification of a Z-slice, the memory used by Ipython keeps increasing at every new iteration. At the end of the loop, the memory used by Ipython is of the order of magnitude of the size of the data file (1.8Go !). I would have expected that the maximum amount of memory used would corresponde to only one Z-slice of the 3D array. See the code snapshots below for more details. Is this an expected behaviour? How can I reduce the amount of memory used by Ipython and still process my data? import numpy as np imin, imax = -2, 2 data = np.memmap(filename = 'data.vol', mode='r+', dtype=float32, shape=(512,981, 981)) for sl in data: sl[slimax] = imax data.flush() # tried with and without this line Thanks in advance, Emmanuelle From david at ar.media.kyoto-u.ac.jp Wed Jul 1 03:59:10 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 01 Jul 2009 16:59:10 +0900 Subject: [Numpy-discussion] Numpy complex types, packing and C99 Message-ID: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> Hi, I would like to add an explicit configuration test to check that our complex type is compatible with the C99 complex type (when available). Is this ok ? As currently defined, complex c types (npy_cfloat, etc...) are not defined in a way such as they are binary compatible with the C99 complex type. Strictly speaking, packing the complex type is an ABI break, but we already make the assumption in ufunc, so we would have wrong results/crashes currently if it were not packed, so I believe the check is harmless by itself. The rationale is that I would like to add support for complex math in npy_math (cabs, cexp, etc...). As I would like to use the C99 complex type (when available), this requires that numpy complex type is ABI compatible with C99 type. cheers, David From nwagner at iam.uni-stuttgart.de Wed Jul 1 04:35:34 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Wed, 01 Jul 2009 10:35:34 +0200 Subject: [Numpy-discussion] permutation symbol In-Reply-To: References: Message-ID: On Tue, 30 Jun 2009 13:51:15 -0600 Charles R Harris wrote: > On Tue, Jun 30, 2009 at 12:26 PM, Nils Wagner > wrote: > >> On Tue, 30 Jun 2009 11:10:39 -0600 >> Charles R Harris wrote: >> > On Tue, Jun 30, 2009 at 10:56 AM, Charles R Harris < >> > charlesr.harris at gmail.com> wrote: >> > >> >> >> >> >> >> On Tue, Jun 30, 2009 at 10:40 AM, Nils Wagner < >> >> nwagner at iam.uni-stuttgart.de> wrote: >> >> >> >>> On Tue, 30 Jun 2009 10:27:05 -0600 >> >>> Charles R Harris wrote: >> >>> > On Tue, Jun 30, 2009 at 5:11 AM, Nils Wagner >> >>> > wrote: >> >>> > >> >>> >> On Tue, 30 Jun 2009 11:22:34 +0200 >> >>> >> "Nils Wagner" >>wrote: >> >>> >> >> >>> >>> Hi all, >> >>> >>> >> >>> >>> How can I build the following product with numpy >> >>> >>> >> >>> >>> q_i = \varepsilon_{ijk} q_{kj} >> >>> >>> >> >>> >>> where \varepsilon_{ijk} denotes the permutation >> >>>symbol. >> >>> >>> >> >>> >>> Nils >> >>> >>> >> >>> >> Sorry for replying to myself. >> >>> >> The permutation symbol is also known as the >> >>>Levi-Civita >> >>> >>symbol. >> >>> >> I found an explicit expression at >> >>> >> http://en.wikipedia.org/wiki/Levi-Civita_symbol >> >>> >> >> >>> >> How do I build the product of the Levi-Civita >>symbol >> >>> >>\varepsilon_{ijk} and >> >>> >> the two dimensional array >> >>> >> q_{kj}, i,j,k = 1,2,3 ? >> >>> >> >> >>> > >> >>> > Write it out explicitly. It essentially >> >>>antisymmetrizes >> >>> >q and the three off >> >>> > diagonal elements can then be treated as a vector. >> >>> >Depending on how q is >> >>> > formed and the resulting vector is used there may >>be >> >>> >other things you can do >> >>> > when you use it in a more general expression. If >>this >> >>>is >> >>> >part of a general >> >>> > calculation there might be other ways of >>expressing >> >>>it. >> >>> > >> >>> > Chuck >> >>> >> >>> Hi Chuck, >> >>> >> >>> Thank you for your response. >> >>> The problem at hand is described in a paper by >>Angeles >> >>> namely equation (17c) in >> >>> "Automatic computation of the screw parameters of >> >>> rigid-body motions. >> >>> Part I: Finitely-separated positions" >> >>> Journal of Dynamic systems, Measurement and Control, >> >>>Vol. >> >>> 108 (1986) pp. 32-38 >> >>> >> >> >> >> You can solve this problem using quaternions also, in >> >>which case it reduces >> >> to an eigenvalue problem. You will note that such >>things >> >>as PCA are used in >> >> the papers that reference the cited work so you can't >> >>really get around that >> >> bit of inefficiency. >> >> >> > >> > Here's a reference to the quaternion approach: >> > >>http://people.csail.mit.edu/bkph/papers/Absolute_Orientation.pdf. >> >You can >> > get the translation part from the motion of the >> >centroid. >> > >> > If you are into abstractions you will note that the >> >problem reduces to >> > minimising a quadratic form in the quaternion >> >components. The rest is just >> > algebra ;) >> > >> > Chuck >> >> It turns out that the product is simply an invariant of >>a >> 3 \times 3 matrix. >> >> from numpy import array, zeros, identity >> from numpy.linalg import norm >> >> >> def vect(A): >> """ linear invariant of a 3 x 3 matrix """ >> tmp = zeros(3,float) >> tmp[0] = 0.5*(A[2,1]-A[1,2]) >> tmp[1] = 0.5*(A[0,2]-A[2,0]) >> tmp[2] = 0.5*(A[1,0]-A[0,1]) >> >> return tmp > > > Out of curiosity, where did the .5 come from? It is not >normally part of the > Levi-Civita symbol. > > Chuck Hi Chuck, It's my fault. The components of the invariant q are given by q_i = 0.5 \varepsilon_{ijk} q_{kj} Nils From pav at iki.fi Wed Jul 1 05:14:39 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 1 Jul 2009 09:14:39 +0000 (UTC) Subject: [Numpy-discussion] np.memmap and memory usage References: <20090701081751.GA6814@phare.normalesup.org> Message-ID: Wed, 01 Jul 2009 10:17:51 +0200, Emmanuelle Gouillart kirjoitti: > I'm using numpy.memmap to open big 3-D arrays of Xray tomography > data. After I have created a new array using memmap, I modify the > contrast of every Z-slice (along the first dimension) inside a for loop, > for a better visualization of the data. Although I call memmap.flush > after each modification of a Z-slice, the memory used by Ipython keeps > increasing at every new iteration. At the end of the loop, the memory > used by Ipython is of the order of magnitude of the size of the data > file (1.8Go !). I would have expected that the maximum amount of memory > used would corresponde to only one Z-slice of the 3D array. See the code > snapshots below for more details. > > Is this an expected behaviour? How can I reduce the amount of > memory used by Ipython and still process my data? How do you measure the memory used? Note that on Linux, "top" includes the size of OS caches for the memmap in the RSS size of a process. You can try to monitor "free" instead: $ free total used free shared buffers cached Mem: 12300488 11485664 814824 0 642928 7960736 -/+ buffers/cache: 2882000 9418488 Swap: 7847712 2428 7845284 If the memory is used by OS caches, the "used" number on the "-/+ buffers/ cache" line should stay constant while the program runs. In this case, what is most likely actually taking up memory is the OS buffering the data in memory, before writing it to disk. Linux has at least some system-wide parameters available that tune the aggressiveness of data cachine. I suppose there may also be some file-specific settings, but I have no idea what they are. -- Pauli Virtanen From magawake at gmail.com Wed Jul 1 07:57:15 2009 From: magawake at gmail.com (Mag Gam) Date: Wed, 1 Jul 2009 07:57:15 -0400 Subject: [Numpy-discussion] Multi thread loading data Message-ID: <1cbd6f830907010457la959372n120d9be0b145629a@mail.gmail.com> Is it possible to use loadtxt in a mult thread way? Basically, I want to process a very large CSV file (100+ million records) and instead of loading thousand elements into a buffer process and then load another 1 thousand elements and process and so on... I was wondering if there is a technique where I can use multiple processors to do this faster. TIA From faltet at pytables.org Wed Jul 1 09:04:08 2009 From: faltet at pytables.org (Francesc Alted) Date: Wed, 1 Jul 2009 15:04:08 +0200 Subject: [Numpy-discussion] np.memmap and memory usage In-Reply-To: <20090701081751.GA6814@phare.normalesup.org> References: <20090701081751.GA6814@phare.normalesup.org> Message-ID: <200907011504.11140.faltet@pytables.org> A Wednesday 01 July 2009 10:17:51 Emmanuelle Gouillart escrigu?: > Hello, > > I'm using numpy.memmap to open big 3-D arrays of Xray tomography > data. After I have created a new array using memmap, I modify the > contrast of every Z-slice (along the first dimension) inside a for loop, > for a better visualization of the data. Although I call memmap.flush > after each modification of a Z-slice, the memory used by Ipython keeps > increasing at every new iteration. At the end of the loop, the memory > used by Ipython is of the order of magnitude of the size of the data file > (1.8Go !). I would have expected that the maximum amount of memory used > would corresponde to only one Z-slice of the 3D array. See the code > snapshots below for more details. > > Is this an expected behaviour? I think so, yes. As all the file (1.8 GB) has to be processed, the OS will need to load everything in RAM at least once. The memory containing already processed data is not 'freed' by the OS even if you try to flush() the slices (this just force memory data to be saved on-disk), but rather when the OS needs it again (but still, they will be swapped out to disk, effectively consuming resources). I'm afraid that the only way to actually 'free' the memory is to close() the *complete* dataset. > How can I reduce the amount of > memory used by Ipython and still process my data? There are a number of interfaces that can deal with binary data on-disk so that they can read&write slices of data easily. By using this approach, you only have to load the appropriate slice, operate and write it again. That's similar to how numpy.memmap works, but it uses far less memory (just the loaded slice). For example, by using the PyTables interface [1], you could do: f = tb.openFile(filename+".h5", "r+") data = f.root.data for nrow, sl in enumerate(data): sl[slimax] = imax data[nrow] = sl f.close() which is similar to your memmap-based approach: data = np.memmap(filename+".bin", mode='r+', dtype='f4', shape=shape) for sl in data: sl[slimax] = imax just that it takes far less memory (47 MB vs 1.9 GB) and besides, it has no other limitation than your available space on disk (compare this with the limit of your virtual memory when using memmap). The speeds are similar too: Using numpy.memmap Time creating data file: 8.534 Time processing data file: 32.742 Using tables Time creating data file: 2.88 Time processing data file: 32.615 However, you can still speed-up out-of-core computations by using the recently introduced tables.Expr class (PyTables 2.2b1, see [2]), which uses a combination of the Numexpr [3] and PyTables advanced computing capabilities: f = tb.openFile(filename+".h5", "r+") data = f.root.data expr = tb.Expr("where(dataimax, imax, data)") expr.setOutput(data) expr.eval() f.close() and the timings for this venue are: Using tables.Expr Time creating data file: 2.393 Time processing data file: 18.25 which is around a 75% faster than a pure memmap/PyTables approach. Further, if your data is compressible, you can probably achieve additional speed-ups by using a fast compressor (like LZO, which is supported by PyTables right out-of-the-box). I'm attaching the script I've used for producing the above timings. You may find it useful for trying this out against your own data. [1] http://www.pytables.org [2] http://www.pytables.org/download/preliminary/ [3] http://code.google.com/p/numexpr/ HTH, -- Francesc Alted -------------- next part -------------- A non-text attachment was scrubbed... Name: memmap-tables-Expr.py Type: text/x-python Size: 1924 bytes Desc: not available URL: From emmanuelle.gouillart at normalesup.org Wed Jul 1 09:11:36 2009 From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart) Date: Wed, 1 Jul 2009 15:11:36 +0200 Subject: [Numpy-discussion] np.memmap and memory usage In-Reply-To: References: <20090701081751.GA6814@phare.normalesup.org> Message-ID: <20090701131136.GB6814@phare.normalesup.org> Hi Pauli, thank you for your answer! I was indeed measuring the memory used with top, which is not the best tool for understanding what really happens. I monitored "free" during the execution of my program and indeed, the used numbers on the "+/-buffers/cache" line stays roughly constant (fluctuations are smaller than say 5Mo). When I close my program this number gets smaller by a number of the order of magnitude of one Z-slice of my array (35 Mo), so it is possible that the buffer is always full. However, as I call memmap.flush, the buffer should be emptied at each iteration, right? I observed other strange things, for example if I create a first array with np.memmap, I get a MemoryError if I try to create a second array (without doing anything with the first array but creating it). The sum of the sizes of the two files is 3.8 Go, much more than my RAM size, but as I didn't load anything explicitely into the memory, I don't understand why it is not possible to have both arrays defined... Anyway, I need to read a bit of documentation about data caching in Linux, as you suggested. Thanks again, Emmanuelle > > Is this an expected behaviour? How can I reduce the amount of > > memory used by Ipython and still process my data? > How do you measure the memory used? Note that on Linux, "top" includes > the size of OS caches for the memmap in the RSS size of a process. > You can try to monitor "free" instead: > $ free > total used free shared buffers cached > Mem: 12300488 11485664 814824 0 642928 7960736 > -/+ buffers/cache: 2882000 9418488 > Swap: 7847712 2428 7845284 > If the memory is used by OS caches, the "used" number on the "-/+ buffers/ > cache" line should stay constant while the program runs. > In this case, what is most likely actually taking up memory is the OS > buffering the data in memory, before writing it to disk. Linux has at > least some system-wide parameters available that tune the aggressiveness > of data cachine. I suppose there may also be some file-specific settings, > but I have no idea what they are. From emmanuelle.gouillart at normalesup.org Wed Jul 1 09:53:20 2009 From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart) Date: Wed, 1 Jul 2009 15:53:20 +0200 Subject: [Numpy-discussion] np.memmap and memory usage In-Reply-To: <200907011504.11140.faltet@pytables.org> References: <20090701081751.GA6814@phare.normalesup.org> <200907011504.11140.faltet@pytables.org> Message-ID: <20090701135320.GA14913@phare.normalesup.org> Hi Francesc, many thanks for this very detailed and informative answer! This list is really great :D. I'm going to install pytables at once and I will try your scripts with my data. As your computations were made with files of the same size as mine, hopefully it should run also quite well on my computer. I'll let you know! Thanks again, Emmanuelle On Wed, Jul 01, 2009 at 03:04:08PM +0200, Francesc Alted wrote: > A Wednesday 01 July 2009 10:17:51 Emmanuelle Gouillart escrigu?: > > Hello, > > I'm using numpy.memmap to open big 3-D arrays of Xray tomography > > data. After I have created a new array using memmap, I modify the > > contrast of every Z-slice (along the first dimension) inside a for loop, > > for a better visualization of the data. Although I call memmap.flush > > after each modification of a Z-slice, the memory used by Ipython keeps > > increasing at every new iteration. At the end of the loop, the memory > > used by Ipython is of the order of magnitude of the size of the data file > > (1.8Go !). I would have expected that the maximum amount of memory used > > would corresponde to only one Z-slice of the 3D array. See the code > > snapshots below for more details. > > Is this an expected behaviour? > I think so, yes. As all the file (1.8 GB) has to be processed, the OS will > need to load everything in RAM at least once. The memory containing already > processed data is not 'freed' by the OS even if you try to flush() the slices > (this just force memory data to be saved on-disk), but rather when the OS > needs it again (but still, they will be swapped out to disk, effectively > consuming resources). I'm afraid that the only way to actually 'free' the > memory is to close() the *complete* dataset. > > How can I reduce the amount of > > memory used by Ipython and still process my data? > There are a number of interfaces that can deal with binary data on-disk so > that they can read&write slices of data easily. By using this approach, you > only have to load the appropriate slice, operate and write it again. That's > similar to how numpy.memmap works, but it uses far less memory (just the > loaded slice). > For example, by using the PyTables interface [1], you could do: > f = tb.openFile(filename+".h5", "r+") > data = f.root.data > for nrow, sl in enumerate(data): > sl[sl sl[sl>imax] = imax > data[nrow] = sl > f.close() > which is similar to your memmap-based approach: > data = np.memmap(filename+".bin", mode='r+', dtype='f4', shape=shape) > for sl in data: > sl[sl sl[sl>imax] = imax > just that it takes far less memory (47 MB vs 1.9 GB) and besides, it has no > other limitation than your available space on disk (compare this with the > limit of your virtual memory when using memmap). The speeds are similar too: > Using numpy.memmap > Time creating data file: 8.534 > Time processing data file: 32.742 > Using tables > Time creating data file: 2.88 > Time processing data file: 32.615 > However, you can still speed-up out-of-core computations by using the recently > introduced tables.Expr class (PyTables 2.2b1, see [2]), which uses a > combination of the Numexpr [3] and PyTables advanced computing capabilities: > f = tb.openFile(filename+".h5", "r+") > data = f.root.data > expr = tb.Expr("where(data expr.setOutput(data) > expr.eval() > expr = tb.Expr("where(data>imax, imax, data)") > expr.setOutput(data) > expr.eval() > f.close() > and the timings for this venue are: > Using tables.Expr > Time creating data file: 2.393 > Time processing data file: 18.25 > which is around a 75% faster than a pure memmap/PyTables approach. > Further, if your data is compressible, you can probably achieve additional > speed-ups by using a fast compressor (like LZO, which is supported by PyTables > right out-of-the-box). > I'm attaching the script I've used for producing the above timings. You may > find it useful for trying this out against your own data. > [1] http://www.pytables.org > [2] http://www.pytables.org/download/preliminary/ > [3] http://code.google.com/p/numexpr/ > HTH, From cournape at gmail.com Wed Jul 1 10:25:55 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 1 Jul 2009 23:25:55 +0900 Subject: [Numpy-discussion] np.memmap and memory usage In-Reply-To: References: <20090701081751.GA6814@phare.normalesup.org> Message-ID: <5b8d13220907010725r4b4c2d58g8b49910335bfb8b7@mail.gmail.com> On Wed, Jul 1, 2009 at 6:14 PM, Pauli Virtanen wrote: > Wed, 01 Jul 2009 10:17:51 +0200, Emmanuelle Gouillart kirjoitti: >> ? ? ? I'm using numpy.memmap to open big 3-D arrays of Xray tomography >> data. After I have created a new array using memmap, I modify the >> contrast of every Z-slice (along the first dimension) inside a for loop, >> for a better visualization of the data. Although I call memmap.flush >> after each modification of a Z-slice, the memory used by Ipython keeps >> increasing at every new iteration. At the end of the loop, the memory >> used by Ipython is of the order of magnitude of the size of the data >> file (1.8Go !). I would have expected that the maximum amount of memory >> used would corresponde to only one Z-slice of the 3D array. See the code >> snapshots below for more details. >> >> ? ? ? Is this an expected behaviour? How can I reduce the amount of >> memory used by Ipython and still process my data? > > How do you measure the memory used? Note that on Linux, "top" includes > the size of OS caches for the memmap in the RSS size of a process. > You can try to monitor "free" instead: A good tool on linux is exmap: http://ktown.kde.org/~seli/memory/analysis.html Unfortunately, it is static, but more accurate than free or top. David From faltet at pytables.org Wed Jul 1 11:22:50 2009 From: faltet at pytables.org (Francesc Alted) Date: Wed, 1 Jul 2009 17:22:50 +0200 Subject: [Numpy-discussion] np.memmap and memory usage In-Reply-To: <200907011504.11140.faltet@pytables.org> References: <20090701081751.GA6814@phare.normalesup.org> <200907011504.11140.faltet@pytables.org> Message-ID: <200907011722.50944.faltet@pytables.org> A Wednesday 01 July 2009 15:04:08 Francesc Alted escrigu?: > However, you can still speed-up out-of-core computations by using the > recently introduced tables.Expr class (PyTables 2.2b1, see [2]), which uses > a combination of the Numexpr [3] and PyTables advanced computing > capabilities: > > f = tb.openFile(filename+".h5", "r+") > data = f.root.data > expr = tb.Expr("where(data expr.setOutput(data) > expr.eval() > expr = tb.Expr("where(data>imax, imax, data)") > expr.setOutput(data) > expr.eval() > f.close() > > and the timings for this venue are: > > Using tables.Expr > Time creating data file: 2.393 > Time processing data file: 18.25 > > which is around a 75% faster than a pure memmap/PyTables approach. Ops, I suddenly realized that the above can be further accelerated by combining both expressions into a nested one. Something like: f = tb.openFile(filename+".h5", "r+") data = f.root.data # Complex expression that spawns several lines follows expr = tb.Expr(""" where( where(dataimax, imax, data) """) expr.setOutput(data) expr.eval() f.close() With this change, the computation time is now: Using tables.Expr Time creating data file: 2.18 Time processing data file: 10.992 which represents another 65% of improvement over the version using two expressions (and 3x faster than the numpy.memmap version). > Further, if your data is compressible, you can probably achieve additional > speed-ups by using a fast compressor (like LZO, which is supported by > PyTables right out-of-the-box). As I was curious, I've tried activating the LZO compressor. Here are the results: Using tables.Expr Time creating data file: 3.123 Time processing data file: 12.533 Mmh, contrarily to my expectations, this hasn't accelerated the computations. My guess is that data being very simple and synthetic, the compression ratio is very high (200x), forcing the compressor/uncompressor to do a lot of work here. However, with real-life data the speed could effectively improve. OTOH, using a faster compressor could be very advantageous here too :) Cheers, -- Francesc Alted -------------- next part -------------- A non-text attachment was scrubbed... Name: memmap-tables-Expr2.py Type: text/x-python Size: 2243 bytes Desc: not available URL: From nwagner at iam.uni-stuttgart.de Wed Jul 1 14:37:22 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Wed, 01 Jul 2009 20:37:22 +0200 Subject: [Numpy-discussion] make html Message-ID: Hi all, I am using >>> numpy.__version__ '1.4.0.dev7094' make html yields /home/nwagner/svn/numpy/doc/source/reference/generated/numpy.trunc.rst:: WARNING: document isn't included in any toctree done preparing documents... done Exception occurred: 2%] reference/generalized_ufuncs ures File "/home/nwagner/local/lib64/python2.6/site-packages/Sphinx-0.6.1-py2.6.egg/sphinx/environment.py", line 921, in get_toc_for toc = self.tocs[docname].deepcopy() KeyError: 'reference/generalized_ufuncs' The full traceback has been saved in /tmp/sphinx-err-wlA8vA.log, if you want to report the issue to the author. Please also report this if it was a user error, so that a better error message can be provided next time. Send reports to sphinx-dev at googlegroups.com. Thanks! make: *** [html] Fehler 1 Nils From pav at iki.fi Wed Jul 1 14:59:39 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 1 Jul 2009 18:59:39 +0000 (UTC) Subject: [Numpy-discussion] make html References: Message-ID: On 2009-07-01, Nils Wagner wrote: > I am using > >>>> numpy.__version__ > '1.4.0.dev7094' > > make html yields > > /home/nwagner/svn/numpy/doc/source/reference/generated/numpy.trunc.rst:: > WARNING: document isn't included in any toctree > done > preparing documents... done > Exception occurred: 2%] reference/generalized_ufuncs > ures > File > "/home/nwagner/local/lib64/python2.6/site-packages/Sphinx-0.6.1-py2.6.egg/sphinx/environment.py", > line 921, in get_toc_for > toc = self.tocs[docname].deepcopy() > KeyError: 'reference/generalized_ufuncs' Do "make clean" first. -- Pauli Virtanen From charlesr.harris at gmail.com Wed Jul 1 15:30:22 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Jul 2009 13:30:22 -0600 Subject: [Numpy-discussion] Numpy complex types, packing and C99 In-Reply-To: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> Message-ID: On Wed, Jul 1, 2009 at 1:59 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi, > > I would like to add an explicit configuration test to check that our > complex type is compatible with the C99 complex type (when available). > Is this ok ? > > As currently defined, complex c types (npy_cfloat, etc...) are not > defined in a way such as they are binary compatible with the C99 complex > type. Strictly speaking, packing the complex type is an ABI break, but > we already make the assumption in ufunc, so we would have wrong > results/crashes currently if it were not packed, so I believe the check > is harmless by itself. The rationale is that I would like to add support > for complex math in npy_math (cabs, cexp, etc...). As I would like to > use the C99 complex type (when available), this requires that numpy > complex type is ABI compatible with C99 type. > What do you mean by "packing" ? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Jul 1 15:57:05 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 1 Jul 2009 19:57:05 +0000 (UTC) Subject: [Numpy-discussion] Numpy complex types, packing and C99 References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> Message-ID: On 2009-07-01, Charles R Harris wrote: > On Wed, Jul 1, 2009 at 1:59 AM, David Cournapeau wrote: >> I would like to add an explicit configuration test to check that our >> complex type is compatible with the C99 complex type (when available). >> Is this ok? Seems OK to me. >> As currently defined, complex c types (npy_cfloat, etc...) are not >> defined in a way such as they are binary compatible with the C99 complex >> type. Strictly speaking, packing the complex type is an ABI break, but >> we already make the assumption in ufunc, so we would have wrong >> results/crashes currently if it were not packed, so I believe the check Is there a reason not to pack our complex number struct? I think if we bump the ABI version, changing this should be OK. >> is harmless by itself. The rationale is that I would like to add support >> for complex math in npy_math (cabs, cexp, etc...). As I would like to >> use the C99 complex type (when available), this requires that numpy >> complex type is ABI compatible with C99 type. Useful goal: this would allow using npy_math functions as a compatibility fallback when the C99 ones are not available. > What do you mean by "packing" ? C99 defines a complex type such that the real and imag parts are packed as in an array: double complex t; real = ((double*)&t)[0]; imag = ((double*)&t)[1]; Numpy defines this type as a struct, which may imply additional padding. -- Pauli Virtanen From charlesr.harris at gmail.com Wed Jul 1 16:19:31 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Jul 2009 14:19:31 -0600 Subject: [Numpy-discussion] Numpy complex types, packing and C99 In-Reply-To: References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> Message-ID: On Wed, Jul 1, 2009 at 1:57 PM, Pauli Virtanen wrote: > On 2009-07-01, Charles R Harris wrote: > > On Wed, Jul 1, 2009 at 1:59 AM, David Cournapeau < > david at ar.media.kyoto-u.ac.jp> wrote: > >> I would like to add an explicit configuration test to check that our > >> complex type is compatible with the C99 complex type (when available). > >> Is this ok? > > Seems OK to me. > > >> As currently defined, complex c types (npy_cfloat, etc...) are not > >> defined in a way such as they are binary compatible with the C99 complex > >> type. Strictly speaking, packing the complex type is an ABI break, but > >> we already make the assumption in ufunc, so we would have wrong > >> results/crashes currently if it were not packed, so I believe the check > > Is there a reason not to pack our complex number struct? I think > if we bump the ABI version, changing this should be OK. > > >> is harmless by itself. The rationale is that I would like to add support > >> for complex math in npy_math (cabs, cexp, etc...). As I would like to > >> use the C99 complex type (when available), this requires that numpy > >> complex type is ABI compatible with C99 type. > > Useful goal: this would allow using npy_math functions as a > compatibility fallback when the C99 ones are not available. > > > What do you mean by "packing" ? > > C99 defines a complex type such that the real and imag parts are > packed as in an array: > > double complex t; > real = ((double*)&t)[0]; > imag = ((double*)&t)[1]; > > Numpy defines this type as a struct, which may imply additional > padding. > OK. I don't see a problem then. As David says the ufuncs already work that way. Hmm... do we want to implement creal{l,,f} and cimag{l,,f} also? They are built in functions in gcc, can we detect that? Anyway, those functions would make rewriting the current complex functions pretty easy. Or do we want to rewrite the current functions to be ABI compatible with c99? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jul 1 16:25:25 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Jul 2009 14:25:25 -0600 Subject: [Numpy-discussion] Numpy complex types, packing and C99 In-Reply-To: References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> Message-ID: On Wed, Jul 1, 2009 at 1:57 PM, Pauli Virtanen wrote: > On 2009-07-01, Charles R Harris wrote: > > On Wed, Jul 1, 2009 at 1:59 AM, David Cournapeau < > david at ar.media.kyoto-u.ac.jp> wrote: > >> I would like to add an explicit configuration test to check that our > >> complex type is compatible with the C99 complex type (when available). > >> Is this ok? > > Seems OK to me. > > >> As currently defined, complex c types (npy_cfloat, etc...) are not > >> defined in a way such as they are binary compatible with the C99 complex > >> type. Strictly speaking, packing the complex type is an ABI break, but > >> we already make the assumption in ufunc, so we would have wrong > >> results/crashes currently if it were not packed, so I believe the check > > Is there a reason not to pack our complex number struct? I think > if we bump the ABI version, changing this should be OK. > This bothers me a bit. Since the two forms should normally be compatible maybe we can use a union or some other way to preserve the current ABI. It would be nice to have a deprecation warning too, but I can't figure out how to deprecate a struct declaration. A compiler warning perhaps? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Jul 1 16:34:00 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 1 Jul 2009 20:34:00 +0000 (UTC) Subject: [Numpy-discussion] Numpy complex types, packing and C99 References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> Message-ID: On 2009-07-01, Charles R Harris wrote: [clip] > OK. I don't see a problem then. As David says the ufuncs > already work that way. Hmm... do we want to > implement creal{l,,f} and cimag{l,,f} also? They might come useful. > They are built in functions in gcc, can we detect that? Don't know, probably we can? > Anyway, those functions would make rewriting the current > complex functions pretty easy. Or do we want to rewrite the > current functions to be ABI compatible with c99? I think that for scipy.special this would be useful. Of course, the operator semantics for complex numbers can't be used there, but still sticking closer to C99 could be useful. A separate question is whether we want to implement Numpy's complex ufuncs using the C99 ones when they are available. I had a branch for this: http://github.com/pv/numpy-work/tree/c99-complex-umath This has some risks: the system-provided complex-valued functions may be broken in different ways, or suffer from some subtle flaws. This is likely more common than having broken real-valued functions that have been around quite a while longer. (To give an example: casinh(1e-20) == 0 with GNU Libc 2.9.) -- Pauli Virtanen From cournape at gmail.com Wed Jul 1 19:47:13 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 2 Jul 2009 08:47:13 +0900 Subject: [Numpy-discussion] Numpy complex types, packing and C99 In-Reply-To: References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220907011647v4f2c157er24286197ced1bd90@mail.gmail.com> On Thu, Jul 2, 2009 at 5:25 AM, Charles R Harris wrote: > > > On Wed, Jul 1, 2009 at 1:57 PM, Pauli Virtanen wrote: >> >> On 2009-07-01, Charles R Harris wrote: >> > On Wed, Jul 1, 2009 at 1:59 AM, David Cournapeau >> > wrote: >> >> ? ?I would like to add an explicit configuration test to check that our >> >> complex type is compatible with the C99 complex type (when available). >> >> Is this ok? >> >> Seems OK to me. >> >> >> ? ?As currently defined, complex c types (npy_cfloat, etc...) are not >> >> defined in a way such as they are binary compatible with the C99 >> >> complex >> >> type. Strictly speaking, packing the complex type is an ABI break, but >> >> we already make the assumption in ufunc, so we would have wrong >> >> results/crashes currently if it were not packed, so I believe the check >> >> Is there a reason not to pack our complex number struct? I think >> if we bump the ABI version, changing this should be OK. > > This bothers me a bit. Since the two forms should normally be compatible > maybe we can use a union or some other way to preserve the current ABI. It > would be nice to have a deprecation warning too, but I can't figure out how > to deprecate a struct declaration. A compiler warning perhaps? I think it is a non issue, because we already rely on the assumption. I was just thinking about checking that sizeof(complex) == sizeof(double) * 2 at configuration check as a safeguard (and deal with it on platforms where it fails, using compiler-dependent pragma). Currently, if it is not packed, the complex ufunc are horribly wrong :) I don't see how using a union would help preserving anything: the only way to force packing would be to define complex as double[2] (as fftw does for fftw_complex), but this is not practical because you can't return an array in C. cheers, David From cournape at gmail.com Wed Jul 1 20:02:02 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 2 Jul 2009 09:02:02 +0900 Subject: [Numpy-discussion] Numpy complex types, packing and C99 In-Reply-To: References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220907011702q6b050a14g7b0f0be01bd19485@mail.gmail.com> On Thu, Jul 2, 2009 at 5:34 AM, Pauli Virtanen wrote: > Don't know, probably we can? yes, they are functions (the builtins are __real__ and __imag__) > > I think that for scipy.special this would be useful. Of course, > the operator semantics for complex numbers can't be used there, > but still sticking closer to C99 could be useful. that's one reason I was interested in getting this started :) The current state of scipy.special really saddens me. I also needs some of those functions at the C level for windows x64. > > A separate question is whether we want to implement Numpy's > complex ufuncs using the C99 ones when they are available. > I had a branch for this: > > ? ?http://github.com/pv/numpy-work/tree/c99-complex-umath I even went further: I used the C99 complex type (npy_cdouble is a typedef to complex if C99 complex is available). > This has some risks: the system-provided complex-valued functions > may be broken in different ways, or suffer from some subtle > flaws. This is likely more common than having broken real-valued > functions that have been around quite a while longer. (To give an > example: casinh(1e-20) == 0 with GNU Libc 2.9.) True, but we can deal with this once we have tests: we can force to use our own, fixed implementations on broken platforms. The glibc complex functions are indeed not great, I have noticed quite a few problems for special value handling (e.g. cabs(inf + I * NAN) is nan, but should be inf, etc....). I think you will agree that having tests for the edge cases is an acceptable solution :) David From elaine.angelino at gmail.com Wed Jul 1 21:02:53 2009 From: elaine.angelino at gmail.com (Elaine Angelino) Date: Wed, 1 Jul 2009 21:02:53 -0400 Subject: [Numpy-discussion] ndarray from column data Message-ID: <901520e20907011802y3c8c856anaea0b8de7a2b5726@mail.gmail.com> Hi there -- Is there a fast way to make a numpy ndarray from column data? For example, suppose I want to make an ndarray with 2 rows and 3 columns of different data types based on the following column data: C0 = [1,2] C1 = ['a','b'] C2 = [3.3,4.4] I could create an empty ndarray and fill the columns one by one: X = numpy.core.ndarray((2,), dtype=' From faltet at pytables.org Thu Jul 2 03:58:15 2009 From: faltet at pytables.org (Francesc Alted) Date: Thu, 2 Jul 2009 09:58:15 +0200 Subject: [Numpy-discussion] ndarray from column data In-Reply-To: <901520e20907011802y3c8c856anaea0b8de7a2b5726@mail.gmail.com> References: <901520e20907011802y3c8c856anaea0b8de7a2b5726@mail.gmail.com> Message-ID: <200907020958.15884.faltet@pytables.org> A Thursday 02 July 2009 03:02:53 Elaine Angelino escrigu?: > Hi there -- > > Is there a fast way to make a numpy ndarray from column data? > > For example, suppose I want to make an ndarray with 2 rows and 3 columns of > different data types based on the following column data: > > C0 = [1,2] > C1 = ['a','b'] > C2 = [3.3,4.4] > > I could create an empty ndarray and fill the columns one by one: > > X = numpy.core.ndarray((2,), dtype=' X['f0'] = C0 > X['f1'] = C1 > X['f2'] = C2 > > The result is the same as: > > X = numpy.array([(1,'a',3.3), (2,'b',4.4)], dtype=' > but I would like to make X directly from the column data. > > [ I know that numpy.core.records.fromarrays will produce a numpy recarray > from column data, but this of course is a recarray and not a ndarray! For > ex: > > X = numpy.numpy.core.records.fromarrays([C0,C1,C2]) ] What's wrong with recarrays? In any case, if you need a true ndarray object you can always do: ndarr = recarr.view(np.ndarray) and you are done. -- Francesc Alted From seb.binet at gmail.com Thu Jul 2 04:12:37 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Thu, 2 Jul 2009 10:12:37 +0200 Subject: [Numpy-discussion] pep-3118 extended struct format parser Message-ID: <200907021012.38069.binet@cern.ch> hi there, at last scipy'08 sprint, somebody (apologies for my brain fade) was working on being able to parse the extended struct format string so one could do: Nested structure :: struct { int ival; struct { unsigned short sval; unsigned char bval; unsigned char cval; } sub; } """i:ival: T{ H:sval: B:bval: B:cval: }:sub: """ Nested array :: struct { int ival; double data[16*4]; } """i:ival: (16,4)d:data: """ is this available somewhere ? (being it in pure python or fast-C implemented, I don't really care for the moment :) ) cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From cournape at gmail.com Thu Jul 2 07:56:11 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 2 Jul 2009 20:56:11 +0900 Subject: [Numpy-discussion] Numpy complex types, packing and C99 In-Reply-To: <5b8d13220907011702q6b050a14g7b0f0be01bd19485@mail.gmail.com> References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> <5b8d13220907011702q6b050a14g7b0f0be01bd19485@mail.gmail.com> Message-ID: <5b8d13220907020456y44b699c7p6f135f863e9db76@mail.gmail.com> On Thu, Jul 2, 2009 at 9:02 AM, David Cournapeau wrote: > > True, but we can deal ?with this once we have tests: we can force to > use our own, fixed implementations on broken platforms. The glibc > complex functions are indeed not great, I have noticed quite a few > problems for special value handling (e.g. cabs(inf + I * NAN) is nan, > but should be inf, etc....). Ok, here we are. There are two branches. - the first one, complex_umath_tests, is a branch with thorough covering of special values. I use the C99 standard for reference: http://github.com/cournape/numpy/tree/complex_umath_tests The main file is http://github.com/cournape/numpy/blob/c29e793c220f48e317adafb8c765acd4db13cb0d/numpy/core/tests/test_umath_complex.py Around 10 tests fail on Linux ATM. - the second branch is the interesting one: http://github.com/cournape/numpy/tree/C99_complex_math It implements clog, cexp, creal, cimag, carg, cabs, ccos, csin and csqrt. The list is dictated by my needs for windows 64, but the list can grow arbitrarily, of course. Most the implementations are taken from msun (the math library of FreeBSD). Unfortunately, they don't implement that many functions compared to the glibc. If I merge the two branches and use the npymath complex functions in umath, there is no unit test failures anymore :) I think I will merge the complex_umath_tests branch soon (platform-specific failures on build bot will be interesting), unless someone sees a problem with it. cheers, David From sccolbert at gmail.com Thu Jul 2 11:14:53 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 2 Jul 2009 11:14:53 -0400 Subject: [Numpy-discussion] Multi thread loading data In-Reply-To: <1cbd6f830907010457la959372n120d9be0b145629a@mail.gmail.com> References: <1cbd6f830907010457la959372n120d9be0b145629a@mail.gmail.com> Message-ID: <7f014ea60907020814g68274820l6e7b1bf5295a5383@mail.gmail.com> can you hold the entire file in memory as single array with room to spare? If so, you could use multiprocessing and load a bunch of smaller arrays, then join them all together. It wont be super fast, because serializing a numpy array is somewhat slow when using multiprocessing. That said, its still faster than disk transfers. I'm sure some numpy expert will come on here though and give you a much better idea. On Wed, Jul 1, 2009 at 7:57 AM, Mag Gam wrote: > Is it possible to use loadtxt in a mult thread way? Basically, I want > to process a very large CSV file (100+ million records) and instead of > loading thousand elements into a buffer process and then load another > 1 thousand elements and process and so on... > > I was wondering if there is a technique where I can use multiple > processors to do this faster. > > TIA > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From seb.haase at gmail.com Thu Jul 2 11:30:03 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 2 Jul 2009 17:30:03 +0200 Subject: [Numpy-discussion] Multi thread loading data In-Reply-To: <7f014ea60907020814g68274820l6e7b1bf5295a5383@mail.gmail.com> References: <1cbd6f830907010457la959372n120d9be0b145629a@mail.gmail.com> <7f014ea60907020814g68274820l6e7b1bf5295a5383@mail.gmail.com> Message-ID: On Thu, Jul 2, 2009 at 5:14 PM, Chris Colbert wrote: > can you hold the entire file in memory as single array with room to spare? > If so, you could use multiprocessing and load a bunch of smaller > arrays, then join them all together. > > It wont be super fast, because serializing a numpy array is somewhat > slow when using multiprocessing. That said, its still faster than disk > transfers. > > I'm ?sure some numpy expert will come on here though and give you a > much better idea. > > > > On Wed, Jul 1, 2009 at 7:57 AM, Mag Gam wrote: >> Is it possible to use loadtxt in a mult thread way? Basically, I want >> to process a very large CSV file (100+ million records) and instead of >> loading thousand elements into a buffer process and then load another >> 1 thousand elements and process and so on... >> >> I was wondering if there is a technique where I can use multiple >> processors to do this faster. >> >> TIA Do you know about the GIL (global interpreter lock) in Python ? It means that Python isn't doing "real" multithreading... Only if one thread is e.g. doing some slow or blocking io stuff, the other thread could keep work, e.g. doing CPU-heavy numpy stuff. But you would get 2-CPU numpy code - except for some C-implemented "long running" operations -- these should be programmed in a way that releases the GIL so that the other CPU could go on doing it's Python code. HTH, Sebastian Haase From sccolbert at gmail.com Thu Jul 2 11:38:54 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 2 Jul 2009 11:38:54 -0400 Subject: [Numpy-discussion] Multi thread loading data In-Reply-To: References: <1cbd6f830907010457la959372n120d9be0b145629a@mail.gmail.com> <7f014ea60907020814g68274820l6e7b1bf5295a5383@mail.gmail.com> Message-ID: <7f014ea60907020838t63a077dejf7e7cd010f042b0f@mail.gmail.com> Who are quoting Sebastian? Multiprocessing is a python package that spawns multiple python processes, effectively side-stepping the GIL, and provides easy mechanisms for IPC. Hence the need for serialization.... On Thu, Jul 2, 2009 at 11:30 AM, Sebastian Haase wrote: > On Thu, Jul 2, 2009 at 5:14 PM, Chris Colbert wrote: >> can you hold the entire file in memory as single array with room to spare? >> If so, you could use multiprocessing and load a bunch of smaller >> arrays, then join them all together. >> >> It wont be super fast, because serializing a numpy array is somewhat >> slow when using multiprocessing. That said, its still faster than disk >> transfers. >> >> I'm ?sure some numpy expert will come on here though and give you a >> much better idea. >> >> >> >> On Wed, Jul 1, 2009 at 7:57 AM, Mag Gam wrote: >>> Is it possible to use loadtxt in a mult thread way? Basically, I want >>> to process a very large CSV file (100+ million records) and instead of >>> loading thousand elements into a buffer process and then load another >>> 1 thousand elements and process and so on... >>> >>> I was wondering if there is a technique where I can use multiple >>> processors to do this faster. >>> >>> TIA > > Do you know about the GIL (global interpreter lock) in Python ? > It ?means that Python isn't doing "real" multithreading... > Only if one thread is e.g. doing some slow or blocking io stuff, the > other thread could keep work, e.g. doing CPU-heavy numpy stuff. > But you would get 2-CPU numpy code - except for some C-implemented > "long running" operations -- these should be programmed in a way that > releases the GIL so that the other CPU could go on doing it's Python > code. > > HTH, > Sebastian Haase > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From seb.haase at gmail.com Thu Jul 2 12:04:41 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 2 Jul 2009 18:04:41 +0200 Subject: [Numpy-discussion] Multi thread loading data In-Reply-To: <7f014ea60907020838t63a077dejf7e7cd010f042b0f@mail.gmail.com> References: <1cbd6f830907010457la959372n120d9be0b145629a@mail.gmail.com> <7f014ea60907020814g68274820l6e7b1bf5295a5383@mail.gmail.com> <7f014ea60907020838t63a077dejf7e7cd010f042b0f@mail.gmail.com> Message-ID: On Thu, Jul 2, 2009 at 5:38 PM, Chris Colbert wrote: > Who are quoting Sebastian? > > Multiprocessing is a python package that spawns multiple python > processes, effectively side-stepping the GIL, and provides easy > mechanisms for IPC. Hence the need for serialization.... > I was replying to the OP's email Regarding your comment: can separate processes not access the same memory space !? via shared memory ... I think there was a discussion about this not to long ago on this list. -S. > On Thu, Jul 2, 2009 at 11:30 AM, Sebastian Haase wrote: >> On Thu, Jul 2, 2009 at 5:14 PM, Chris Colbert wrote: >>> can you hold the entire file in memory as single array with room to spare? >>> If so, you could use multiprocessing and load a bunch of smaller >>> arrays, then join them all together. >>> >>> It wont be super fast, because serializing a numpy array is somewhat >>> slow when using multiprocessing. That said, its still faster than disk >>> transfers. >>> >>> I'm ?sure some numpy expert will come on here though and give you a >>> much better idea. >>> >>> >>> >>> On Wed, Jul 1, 2009 at 7:57 AM, Mag Gam wrote: >>>> Is it possible to use loadtxt in a mult thread way? Basically, I want >>>> to process a very large CSV file (100+ million records) and instead of >>>> loading thousand elements into a buffer process and then load another >>>> 1 thousand elements and process and so on... >>>> >>>> I was wondering if there is a technique where I can use multiple >>>> processors to do this faster. >>>> >>>> TIA >> >> Do you know about the GIL (global interpreter lock) in Python ? >> It ?means that Python isn't doing "real" multithreading... >> Only if one thread is e.g. doing some slow or blocking io stuff, the >> other thread could keep work, e.g. doing CPU-heavy numpy stuff. >> But you would get 2-CPU numpy code - except for some C-implemented >> "long running" operations -- these should be programmed in a way that >> releases the GIL so that the other CPU could go on doing it's Python >> code. >> >> HTH, >> Sebastian Haase >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sccolbert at gmail.com Thu Jul 2 13:08:53 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 2 Jul 2009 13:08:53 -0400 Subject: [Numpy-discussion] Multi thread loading data In-Reply-To: References: <1cbd6f830907010457la959372n120d9be0b145629a@mail.gmail.com> <7f014ea60907020814g68274820l6e7b1bf5295a5383@mail.gmail.com> <7f014ea60907020838t63a077dejf7e7cd010f042b0f@mail.gmail.com> Message-ID: <7f014ea60907021008i34df44bem80f3bf3967990726@mail.gmail.com> I'm relatively certain its possible, but then you have to deal with locks, semaphores, synchronization, etc... On Thu, Jul 2, 2009 at 12:04 PM, Sebastian Haase wrote: > On Thu, Jul 2, 2009 at 5:38 PM, Chris Colbert wrote: >> Who are quoting Sebastian? >> >> Multiprocessing is a python package that spawns multiple python >> processes, effectively side-stepping the GIL, and provides easy >> mechanisms for IPC. Hence the need for serialization.... >> > I was replying to the OP's email > > Regarding your comment: can separate processes not access the same > memory space !? via shared memory ... > I think there was a discussion about this not to long ago on this list. > > -S. > > > >> On Thu, Jul 2, 2009 at 11:30 AM, Sebastian Haase wrote: >>> On Thu, Jul 2, 2009 at 5:14 PM, Chris Colbert wrote: >>>> can you hold the entire file in memory as single array with room to spare? >>>> If so, you could use multiprocessing and load a bunch of smaller >>>> arrays, then join them all together. >>>> >>>> It wont be super fast, because serializing a numpy array is somewhat >>>> slow when using multiprocessing. That said, its still faster than disk >>>> transfers. >>>> >>>> I'm ?sure some numpy expert will come on here though and give you a >>>> much better idea. >>>> >>>> >>>> >>>> On Wed, Jul 1, 2009 at 7:57 AM, Mag Gam wrote: >>>>> Is it possible to use loadtxt in a mult thread way? Basically, I want >>>>> to process a very large CSV file (100+ million records) and instead of >>>>> loading thousand elements into a buffer process and then load another >>>>> 1 thousand elements and process and so on... >>>>> >>>>> I was wondering if there is a technique where I can use multiple >>>>> processors to do this faster. >>>>> >>>>> TIA >>> >>> Do you know about the GIL (global interpreter lock) in Python ? >>> It ?means that Python isn't doing "real" multithreading... >>> Only if one thread is e.g. doing some slow or blocking io stuff, the >>> other thread could keep work, e.g. doing CPU-heavy numpy stuff. >>> But you would get 2-CPU numpy code - except for some C-implemented >>> "long running" operations -- these should be programmed in a way that >>> releases the GIL so that the other CPU could go on doing it's Python >>> code. >>> >>> HTH, >>> Sebastian Haase >>> _______________________________________________ >>> Numpy-discussion mailing list >>> Numpy-discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dyamins at gmail.com Thu Jul 2 14:15:13 2009 From: dyamins at gmail.com (Dan Yamins) Date: Thu, 2 Jul 2009 14:15:13 -0400 Subject: [Numpy-discussion] ndarray from column data In-Reply-To: <200907020958.15884.faltet@pytables.org> References: <901520e20907011802y3c8c856anaea0b8de7a2b5726@mail.gmail.com> <200907020958.15884.faltet@pytables.org> Message-ID: <15e4667e0907021115i6d7c41a5o624d3dda93213b52@mail.gmail.com> > > What's wrong with recarrays? In any case, if you need a true ndarray > object > you can always do: > > ndarr = recarr.view(np.ndarray) > > and you are done. > I have a question about this though. The object "ndarr" will consist of "records", e.g.: In [96]: type(ndarr[0]) Out[96]: If ndarr had been defined directly via ndarray constructors, then you'd get: In [99]: type(ndarr[0]) Out[99]: I guess this might not be such a big deal. But what is the difference between the "record" class nad the "void" class? Does this matter for anything at all? Thanks Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kxroberto at googlemail.com Thu Jul 2 15:00:19 2009 From: kxroberto at googlemail.com (Robert) Date: Thu, 02 Jul 2009 21:00:19 +0200 Subject: [Numpy-discussion] ndarray from column data In-Reply-To: <901520e20907011802y3c8c856anaea0b8de7a2b5726@mail.gmail.com> References: <901520e20907011802y3c8c856anaea0b8de7a2b5726@mail.gmail.com> Message-ID: Elaine Angelino wrote: > Hi there -- > > Is there a fast way to make a numpy ndarray from column data? > > For example, suppose I want to make an ndarray with 2 rows and 3 columns > of different data types based on the following column data: > > C0 = [1,2] > C1 = ['a','b'] > C2 = [3.3,4.4] > > I could create an empty ndarray and fill the columns one by one: > > X = numpy.core.ndarray((2,), dtype=' X['f0'] = C0 > X['f1'] = C1 > X['f2'] = C2 > > The result is the same as: > > X = numpy.array([(1,'a',3.3), (2,'b',4.4)], dtype=' > but I would like to make X directly from the column data. > > [ I know that numpy.core.records.fromarrays will produce a numpy > recarray from column data, but this of course is a recarray and not a > ndarray! For ex: > > X = numpy.numpy.core.records.fromarrays([C0,C1,C2]) ] > > Thanks for any help, > > Elaine > >>> np.array(zip(C0,C1,C2), dtype=' 2D there is >>> np.column_stack((C0,C1,C2)) array([['1', 'a', '3.3'], ['2', 'b', '4.4']], dtype='|S8') Robert From pav at iki.fi Thu Jul 2 15:44:16 2009 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 2 Jul 2009 19:44:16 +0000 (UTC) Subject: [Numpy-discussion] Numpy complex types, packing and C99 References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> <5b8d13220907011702q6b050a14g7b0f0be01bd19485@mail.gmail.com> <5b8d13220907020456y44b699c7p6f135f863e9db76@mail.gmail.com> Message-ID: On 2009-07-02, David Cournapeau wrote: > I think I will merge the complex_umath_tests branch soon > (platform-specific failures on build bot will be interesting), unless > someone sees a problem with it. I think we tried this already (my c99-umath-funcs branch had TestC99 special case tests that were in Numpy trunk for a while). The outcome was that the current implementations of the complex functions don't have essentially any special-case behavior that's consistent across different platforms. And our buildbot coverage is not large enough, so users will report failing tests even if buildbots are OK... After repeating this cycle a couple of times, IIRC only some special cases of log survived :) Of course, if you meant to merge the tests first to the new implementations and that to trunk, this sounds better. -- Pauli Virtanen From paz117 at gmail.com Thu Jul 2 18:42:09 2009 From: paz117 at gmail.com (Peter Kelley) Date: Thu, 2 Jul 2009 18:42:09 -0400 Subject: [Numpy-discussion] Using loadtxt to read in mixed data types In-Reply-To: <3614653f0907021504x1becf8fbu7f98a364231b4c3b@mail.gmail.com> References: <3614653f0907021504x1becf8fbu7f98a364231b4c3b@mail.gmail.com> Message-ID: <3614653f0907021542g284c38e1t55091cf4301e57af@mail.gmail.com> Hey Everyone, I am reading in a file of columns with mixed data types, and the number of columns can vary and their format is inputted by the user. So I came up with this: dt=dtype({'names': [x for x in event_fields], 'formats': [b for b in event_format]}) eventArray = loadtxt(behEventFile,dt) where event_format is ['long', 'int', 'int', 'int', 'int', 'int', 'str', 'str', 'int', 'int', 'int', 'int', 'int', 'int', 'int', 'int'] I get TypeError: data type not understood, and I think it is because the event format is a list of strings not data types. Does anyone have know how to convert the list of strings into the data types for dtype. -Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Thu Jul 2 18:53:21 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 3 Jul 2009 00:53:21 +0200 Subject: [Numpy-discussion] Using loadtxt to read in mixed data types In-Reply-To: <3614653f0907021542g284c38e1t55091cf4301e57af@mail.gmail.com> References: <3614653f0907021504x1becf8fbu7f98a364231b4c3b@mail.gmail.com> <3614653f0907021542g284c38e1t55091cf4301e57af@mail.gmail.com> Message-ID: <9457e7c80907021553j770d8209y4de9366889671437@mail.gmail.com> Hi Peter 2009/7/3 Peter Kelley : > I get TypeError: data type not understood, and I think it is because the > event format is a list of strings not data types. Does anyone have know how > to convert the list of strings into the data types for dtype. In your example the problem actually comes in with the string 'long', which cannot be converted to a dtype using dtype('long') As for the strings, I think you'll have to pick the length of the longest string and use that as the dtype: np.dtype('S5') # where 5 is the longest Good luck. Regards St?fan From pgmdevlist at gmail.com Thu Jul 2 18:56:27 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 2 Jul 2009 18:56:27 -0400 Subject: [Numpy-discussion] Using loadtxt to read in mixed data types In-Reply-To: <3614653f0907021542g284c38e1t55091cf4301e57af@mail.gmail.com> References: <3614653f0907021504x1becf8fbu7f98a364231b4c3b@mail.gmail.com> <3614653f0907021542g284c38e1t55091cf4301e57af@mail.gmail.com> Message-ID: <84886DAB-7E51-45DE-914C-AE656071141B@gmail.com> On Jul 2, 2009, at 6:42 PM, Peter Kelley wrote: > > Hey Everyone, > > I am reading in a file of columns with mixed data types, and the > number of columns can vary and their format is inputted by the user. > So I came up with this: > > dt=dtype({'names': [x for x in event_fields], 'formats': [b for b in > event_format]}) > > eventArray = loadtxt(behEventFile,dt) > > where event_format is ['long', 'int', 'int', 'int', 'int', 'int', > 'str', 'str', 'int', 'int', 'int', 'int', 'int', 'int', 'int', 'int'] > > I get TypeError: data type not understood, and I think it is because > the event format is a list of strings not data types. Does anyone > have know how to convert the list of strings into the data types for > dtype. What about 'formats':[eval(b) for b in event_format] Should it fail, try something like: dtype([(x,eval(b)) for (x,b) in zip(event_fields, event_format)]) At least you force dtype to have the same nb of names & formats. > > -Peter > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From paz117 at gmail.com Thu Jul 2 19:30:42 2009 From: paz117 at gmail.com (Peter Kelley) Date: Thu, 2 Jul 2009 19:30:42 -0400 Subject: [Numpy-discussion] Using loadtxt to read in mixed data types In-Reply-To: <84886DAB-7E51-45DE-914C-AE656071141B@gmail.com> References: <3614653f0907021504x1becf8fbu7f98a364231b4c3b@mail.gmail.com> <3614653f0907021542g284c38e1t55091cf4301e57af@mail.gmail.com> <84886DAB-7E51-45DE-914C-AE656071141B@gmail.com> Message-ID: <3614653f0907021630y213554a1v6f9e24141094658d@mail.gmail.com> Thanks Pierre, For some reason 'formats':[eval(b) for b in event_format] didn't work, but as you said should it fail try dtype([(x,eval(b)) for (x,b) in zip(event_fields, event_format)]) which seems to be working. Interestingly before when I had typed this out by hand, both using tuples and a dictionary seemed to work as they should. Thanks again, -Peter From cournape at gmail.com Thu Jul 2 21:53:57 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 3 Jul 2009 10:53:57 +0900 Subject: [Numpy-discussion] Numpy complex types, packing and C99 In-Reply-To: References: <4A4B174E.60600@ar.media.kyoto-u.ac.jp> <5b8d13220907011702q6b050a14g7b0f0be01bd19485@mail.gmail.com> <5b8d13220907020456y44b699c7p6f135f863e9db76@mail.gmail.com> Message-ID: <5b8d13220907021853i2ed22317w9abc9b6b01dbbe6b@mail.gmail.com> On Fri, Jul 3, 2009 at 4:44 AM, Pauli Virtanen wrote: > I think we tried this already (my c99-umath-funcs branch had > TestC99 special case tests that were in Numpy trunk for a while). > > The outcome was that the current implementations of the complex > functions don't have essentially any special-case behavior that's > consistent across different platforms. And our buildbot coverage > is not large enough, so users will report failing tests even if > buildbots are OK... After repeating this cycle a couple of times, > IIRC only some special cases of log survived :) I think the situation is better now, because we have the infrastructure to deal with this: we have portable nan, inf, negative zero and corresponding macros (I have just added copysign recently to the trunk, which was the last missing for every case AFAIK). Preliminary tests on both windows (with MS compilers) and linux indicate that many special cases are already dealt with correctly. I don't have tests for branch cuts yet, though, only signed 0, nan and inf handling. > Of course, if you meant to merge the tests first to the new > implementations and that to trunk, this sounds better. The idea was to merge all the tests, but set them as knownfailure, and enable them everytime I merge a new function into the trunk. This way, one can easily check the differences between successive revisions ? But that's not a crucial point. I can keep everything in one branch if you think it is better. David From faltet at pytables.org Fri Jul 3 04:11:34 2009 From: faltet at pytables.org (Francesc Alted) Date: Fri, 3 Jul 2009 10:11:34 +0200 Subject: [Numpy-discussion] ndarray from column data In-Reply-To: <15e4667e0907021115i6d7c41a5o624d3dda93213b52@mail.gmail.com> References: <901520e20907011802y3c8c856anaea0b8de7a2b5726@mail.gmail.com> <200907020958.15884.faltet@pytables.org> <15e4667e0907021115i6d7c41a5o624d3dda93213b52@mail.gmail.com> Message-ID: <200907031011.34746.faltet@pytables.org> A Thursday 02 July 2009 20:15:13 Dan Yamins escrigu?: > > What's wrong with recarrays? In any case, if you need a true ndarray > > object > > you can always do: > > > > ndarr = recarr.view(np.ndarray) > > > > and you are done. > > I have a question about this though. The object "ndarr" will consist of > "records", e.g.: > > In [96]: type(ndarr[0]) > Out[96]: > > If ndarr had been defined directly via ndarray constructors, then you'd > get: > > In [99]: type(ndarr[0]) > Out[99]: > > I guess this might not be such a big deal. But what is the difference > between the "record" class nad the "void" class? Does this matter for > anything at all? The most apparent difference is that the record class implements accessors to fields as attributes: In [17]: b = np.rec.array([(1,2.,'abc')], 'i4,f8,S3') In [18]: b[0] Out[18]: (1, 2.0, 'abc') In [19]: b[0].f0 Out[19]: 1 In [20]: b[0].f2 Out[20]: 'abc' while void type does not: In [15]: a = np.array([(1,2.,'abc')], 'i4,f8,S3') In [21]: a[0].f0 --------------------------------------------------------------------------- AttributeError: 'numpy.void' object has no attribute 'f0' and you must make use of __getitem__() instead: In [22]: a[0]['f0'] Out[22]: 1 In [23]: a[0]['f2'] Out[23]: 'abc' Cheers, -- Francesc Alted From neilcrighton at gmail.com Fri Jul 3 04:30:59 2009 From: neilcrighton at gmail.com (Neil Crighton) Date: Fri, 3 Jul 2009 08:30:59 +0000 (UTC) Subject: [Numpy-discussion] Using loadtxt to read in mixed data types References: <3614653f0907021504x1becf8fbu7f98a364231b4c3b@mail.gmail.com> <3614653f0907021542g284c38e1t55091cf4301e57af@mail.gmail.com> <84886DAB-7E51-45DE-914C-AE656071141B@gmail.com> Message-ID: Pierre GM gmail.com> writes: > What about > 'formats':[eval(b) for b in event_format] > > Should it fail, try something like: > dtype([(x,eval(b)) for (x,b) in zip(event_fields, event_format)]) > > At least you force dtype to have the same nb of names & formats. > You could use data = np.genfromtxt(filename, names=listofname, dtype=None) Then you just need to specify the column names, and not the dtypes (they are inferred from the data). There are probably backwards compatibility issues, but it would be great if dtype=None was the default for genfromtxt. Neil From seb.haase at gmail.com Fri Jul 3 05:15:10 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Fri, 3 Jul 2009 11:15:10 +0200 Subject: [Numpy-discussion] argwhere does not accept py list Message-ID: Hi, should this not be accepted: >>> N.argwhere([4,0,2,1,3]) ? instead I get Traceback (most recent call last): File "", line 1, in File "./numpy/core/numeric.py", line 510, in argwhere AttributeError: 'list' object has no attribute 'nonzero' >>> N.argwhere(N.array([4,0,2,1,3])) [[0] [2] [3] [4]] >>> N.__version__ '1.3.0' >>> Just a note. -Sebastian Haase From silva at lma.cnrs-mrs.fr Fri Jul 3 05:48:45 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Fri, 03 Jul 2009 11:48:45 +0200 Subject: [Numpy-discussion] roots and high-order polynomial Message-ID: <1246614525.2871.30.camel@localhost.localdomain> Hello Has anyone looked at the behaviour of the (polynomial) roots function for high-order polynomials ? I have an application which internally searches for the roots of a polynomial. It works nicely for order less than 20, and then has an erratic behaviour for upper values... I looked into the source and I wondered that roots is based on the eigenvalues of the companion matrix. For high-order, this latter is rather sparse. Would it improve anything to compute the eigenvalues using sparse solvers? -- Fabrice Silva Laboratory of Mechanics and Acoustics - CNRS 31 chemin Joseph Aiguier, 13402 Marseille, France. From nwagner at iam.uni-stuttgart.de Fri Jul 3 05:52:05 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 03 Jul 2009 11:52:05 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: <1246614525.2871.30.camel@localhost.localdomain> References: <1246614525.2871.30.camel@localhost.localdomain> Message-ID: On Fri, 03 Jul 2009 11:48:45 +0200 Fabrice Silva wrote: > Hello > Has anyone looked at the behaviour of the (polynomial) >roots function > for high-order polynomials ? I have an application which >internally > searches for the roots of a polynomial. It works nicely >for order less > than 20, and then has an erratic behaviour for upper >values... > You will need multiprecision arithmetic in that case. It's an ill-conditioned problem. > I looked into the source and I wondered that roots is >based on the > eigenvalues of the companion matrix. For high-order, >this latter is > rather sparse. Would it improve anything to compute the >eigenvalues > using sparse solvers? No. Cheers, Nils From scott.sinclair.za at gmail.com Fri Jul 3 06:44:55 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Fri, 3 Jul 2009 12:44:55 +0200 Subject: [Numpy-discussion] argwhere does not accept py list In-Reply-To: References: Message-ID: <6a17e9ee0907030344o14d6a394q381952358b076d0e@mail.gmail.com> >2009/7/3 Sebastian Haase : > Hi, > should this not be accepted: >>>> N.argwhere([4,0,2,1,3]) > ? > instead I get > > Traceback (most recent call last): > ?File "", line 1, in > ?File "./numpy/core/numeric.py", line 510, in argwhere > AttributeError: 'list' object has no attribute 'nonzero' >>>> N.argwhere(N.array([4,0,2,1,3])) > [[0] > ?[2] > ?[3] > ?[4]] >>>> N.__version__ > '1.3.0' >>>> A fix could be a simple as applying the following diff (or similar + tests) Index: numpy/core/numeric.py =================================================================== --- numpy/core/numeric.py (revision 7095) +++ numpy/core/numeric.py (working copy) @@ -535,7 +535,7 @@ [1, 2]]) """ - return asarray(a.nonzero()).T + return transpose(asarray(a).nonzero()) >>> import numpy as np >>> a = [4,0,2,1,3] >>> np.argwhere(a) Traceback (most recent call last): File "", line 1, in File "/home/scott/.virtualenvs/numpy-dev/lib/python2.6/site-packages/numpy/core/numeric.py", line 538, in argwhere return asarray(a.nonzero()).T AttributeError: 'list' object has no attribute 'nonzero' >>> np.argwhere(np.asarray(a)) array([[0], [2], [3], [4]]) >>> np.transpose(np.asarray(a).nonzero()) array([[0], [2], [3], [4]]) >>> Cheers, Scott From silva at lma.cnrs-mrs.fr Fri Jul 3 08:26:39 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Fri, 03 Jul 2009 14:26:39 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: References: <1246614525.2871.30.camel@localhost.localdomain> Message-ID: <1246623999.2871.137.camel@localhost.localdomain> Le vendredi 03 juillet 2009 ? 11:52 +0200, Nils Wagner a ?crit : > You will need multiprecision arithmetic in that case. > It's an ill-conditioned problem. I may have said that the solution are of the same order of magnitude, so that the ratio between the lowest and the highest absolute values of the eigenvalues are rather close to one. Does it help ? Looking in the debian repository, I found python-gmpy, an interface GMP to Python. But how could it be used to compute the eigenvalues of the companion matrix ? I will look in the doc... -- Fabrice Silva LMA UPR CNRS 7051 From nwagner at iam.uni-stuttgart.de Fri Jul 3 08:43:50 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 03 Jul 2009 14:43:50 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: <1246623999.2871.137.camel@localhost.localdomain> References: <1246614525.2871.30.camel@localhost.localdomain> <1246623999.2871.137.camel@localhost.localdomain> Message-ID: On Fri, 03 Jul 2009 14:26:39 +0200 Fabrice Silva wrote: > Le vendredi 03 juillet 2009 ? 11:52 +0200, Nils Wagner a >?crit : >> You will need multiprecision arithmetic in that case. >> It's an ill-conditioned problem. > > I may have said that the solution are of the same order >of magnitude, so > that the ratio between the lowest and the highest >absolute values of the > eigenvalues are rather close to one. > Does it help ? > > Looking in the debian repository, I found python-gmpy, >an interface GMP > to Python. But how could it be used to compute the >eigenvalues of the > companion matrix ? I will look in the doc... > > -- >Fabrice Silva > LMA UPR CNRS 7051 > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Just curious - Can you provide us with the coefficients of your polynomial ? Nils From silva at lma.cnrs-mrs.fr Fri Jul 3 10:54:29 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Fri, 03 Jul 2009 16:54:29 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: References: <1246614525.2871.30.camel@localhost.localdomain> <1246623999.2871.137.camel@localhost.localdomain> Message-ID: <1246632869.2871.154.camel@localhost.localdomain> Le vendredi 03 juillet 2009 ? 14:43 +0200, Nils Wagner a ?crit : > Just curious - Can you provide us with the coefficients of > your polynomial ? Working case : Polynomial.c = [ -1.34100085e+57 +0.00000000e+00j -2.28806781e+55 +0.00000000e+00j -4.34808480e+54 -3.27208577e+36j -2.44499178e+52 -7.45966050e+34j -4.84319247e+51 -1.13783265e+34j 1.84249681e+47 +1.01074670e+33j -2.27761972e+48 +2.06997398e+31j 5.35043855e+45 +1.50239041e+30j -4.40346465e+44 +3.49745869e+27j 1.16055383e+42 -6.38037062e+25j -2.79370528e+40 -1.60550605e+23j 4.72851679e+37 +7.68429228e+20j -2.51340384e+35 +2.92105561e+17j] A non-working case : pretty large range of values! Polynomial.c = [ -1.34100085e+219 +0.00000000e+000j -3.87620512e+217 +0.00000000e+000j -1.59644076e+218 -4.96044532e+200j -4.36379116e+216 -1.78047353e+201j -8.66515071e+216 -7.57270655e+199j -2.22899258e+215 +1.20695742e+200j -2.84415205e+215 +2.04437917e+198j -6.84491536e+213 +4.11962630e+198j -6.31415522e+213 +8.60411394e+196j -1.41164416e+212 +6.40341637e+195j -1.00484766e+212 +1.05851522e+194j -2.06870108e+210 -6.90610706e+194j -1.18557253e+210 -7.04790806e+192j -2.22304125e+208 +8.69684716e+191j -1.05816505e+208 +4.10351383e+190j -1.78206407e+206 -3.01765476e+190j -7.22785804e+205 -3.75229199e+188j -1.07357199e+204 +1.60517046e+188j -3.79912531e+203 +5.72547583e+186j -4.85746209e+201 -6.62989846e+185j -1.53768394e+201 +6.29641862e+183j -1.63653463e+199 -3.94069765e+183j -4.77535548e+198 +2.70460428e+180j -4.02975379e+196 -8.22561006e+180j -1.12909884e+196 -3.28170925e+178j -7.00288278e+193 +1.87782740e+178j -2.00766909e+193 +5.47118767e+176j -7.99891319e+190 +5.56488189e+175j -2.63703112e+190 +1.09163655e+174j -4.91486942e+187 +6.37859777e+172j -2.49451367e+187 +1.07180124e+171j 1.17174241e+183 +4.42130463e+169j -1.63881690e+184 -1.39641308e+167j 2.63207649e+181 -5.52568751e+165j -7.08593327e+180 +8.81583002e+163j 1.88293454e+178 +2.44846832e+162j -1.85373357e+177 +1.80169585e+160j 5.67531810e+174 +8.34051533e+158j -2.54119737e+173 +6.83930275e+156j 6.95713759e+170 -3.36080695e+154j -1.36763548e+169 -4.25101484e+151j 2.31484033e+166 +2.65804116e+149j -1.19894847e+164 +1.50542119e+146j] -- Fabrice Silva LMA UPR CNRS 7051 From charlesr.harris at gmail.com Fri Jul 3 12:00:37 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 3 Jul 2009 10:00:37 -0600 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: <1246614525.2871.30.camel@localhost.localdomain> References: <1246614525.2871.30.camel@localhost.localdomain> Message-ID: On Fri, Jul 3, 2009 at 3:48 AM, Fabrice Silva wrote: > Hello > Has anyone looked at the behaviour of the (polynomial) roots function > for high-order polynomials ? I have an application which internally > searches for the roots of a polynomial. It works nicely for order less > than 20, and then has an erratic behaviour for upper values... > What do you mean by erratic? Are the computed roots different from known roots? The connection between polynomial coefficients and polynomial values becomes somewhat vague when the polynomial degree becomes large, it is numerically ill conditioned. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Fri Jul 3 13:30:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 03 Jul 2009 19:30:43 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: <1246623999.2871.137.camel@localhost.localdomain> References: <1246614525.2871.30.camel@localhost.localdomain> <1246623999.2871.137.camel@localhost.localdomain> Message-ID: <4A4E4043.2010709@student.matnat.uio.no> Fabrice Silva wrote: > Le vendredi 03 juillet 2009 ? 11:52 +0200, Nils Wagner a ?crit : >> You will need multiprecision arithmetic in that case. >> It's an ill-conditioned problem. > > I may have said that the solution are of the same order of magnitude, so > that the ratio between the lowest and the highest absolute values of the > eigenvalues are rather close to one. > Does it help ? > > Looking in the debian repository, I found python-gmpy, an interface GMP > to Python. But how could it be used to compute the eigenvalues of the > companion matrix ? I will look in the doc... > One of the easiest ways to calculate with GMP and similar, with Python, is with Sage: http://www.sagemath.org Finding the roots of a polynomial should be very natural within Sage. -- Dag Sverre From pav at iki.fi Fri Jul 3 15:08:44 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 3 Jul 2009 19:08:44 +0000 (UTC) Subject: [Numpy-discussion] roots and high-order polynomial References: <1246614525.2871.30.camel@localhost.localdomain> Message-ID: On 2009-07-03, Charles R Harris wrote: > roots? The connection between polynomial coefficients and polynomial values > becomes somewhat vague when the polynomial degree becomes large, it is > numerically ill conditioned. In addition to switching to higher precision than machine precision, another alternative could be choosing a stabler (eg. Chebyshev) polynomial basis than the {1,x,x^2,...} one. For the Chebyshev series f(x) = sum_{j=0}^N a_j T_j(x), the companion matrix is (check!) A = zeros((N, N)) A[0,1] = 1 j = arange(1, N-1) A[j, j+1] = 0.5 A[j, j-1] = 0.5 A[-1,:] = -0.5*a[:-1]/a[-1] A[-1,-2] += 0.5 and the zeros are x = linalg.eig(A)[0] See eg. http://dx.doi.org/10.1016/j.apnum.2005.09.007 for more. -- Pauli Virtanen From alan at ajackson.org Fri Jul 3 19:38:01 2009 From: alan at ajackson.org (Alan Jackson) Date: Fri, 3 Jul 2009 18:38:01 -0500 Subject: [Numpy-discussion] Bug in the F distribution? Message-ID: <20090703183801.7ec5fb56@ajackson.org> I either found a bug in the F distribution, or I'm really messed up. From a table I find dfnum dfden F(P<.01) 10 10 4.85 11 10 4.78 11 11 4.46 10 11 4.54 So let's calculate the same quantities using numpy... import scipy.stats as stats import numpy as np In [89]: stats.scoreatpercentile(np.random.f(10,10,1000000), 99) Out[89]: 4.8575912131878365 In [90]: stats.scoreatpercentile(np.random.f(11,10,1000000), 99) Out[90]: 5.2721528315236501 In [91]: stats.scoreatpercentile(np.random.f(11,11,1000000), 99) Out[91]: 4.4695161332631841 In [92]: stats.scoreatpercentile(np.random.f(10,11,1000000), 99) Out[92]: 4.1229323443042674 So at 10,10 and 11,11 it works (maybe), but all the other values are clearly off. I tried re-running the example I put into the documentation last summer, which worked, and I don't get the right answer any longer. -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From alan at ajackson.org Fri Jul 3 22:21:42 2009 From: alan at ajackson.org (Alan Jackson) Date: Fri, 3 Jul 2009 21:21:42 -0500 Subject: [Numpy-discussion] Bug in the F distribution? In-Reply-To: <20090703183801.7ec5fb56@ajackson.org> References: <20090703183801.7ec5fb56@ajackson.org> Message-ID: <20090703212142.6a38541f@ajackson.org> I've tried the same scheme using R and it seems to give the right answers > quantile( rf(10000000,10,10), .99) 99% 4.84548 > quantile( rf(10000000,11,10), .99) 99% 4.770002 > quantile( rf(10000000,11,11), .99) 99% 4.465655 > quantile( rf(10000000,10,11), .99) 99% 4.539423 >> I either found a bug in the F distribution, or I'm really messed up. >> >> From a table I find >> >> dfnum dfden F(P<.01) >> 10 10 4.85 >> 11 10 4.78 >> 11 11 4.46 >> 10 11 4.54 >> >> So let's calculate the same quantities using numpy... >> >> import scipy.stats as stats >> import numpy as np >> In [89]: stats.scoreatpercentile(np.random.f(10,10,1000000), 99) >> Out[89]: 4.8575912131878365 >> In [90]: stats.scoreatpercentile(np.random.f(11,10,1000000), 99) >> Out[90]: 5.2721528315236501 >> In [91]: stats.scoreatpercentile(np.random.f(11,11,1000000), 99) >> Out[91]: 4.4695161332631841 >> In [92]: stats.scoreatpercentile(np.random.f(10,11,1000000), 99) >> Out[92]: 4.1229323443042674 >> >> >> So at 10,10 and 11,11 it works (maybe), but all the other values are clearly >> off. I tried re-running the example I put into the documentation last summer, >> which worked, and I don't get the right answer any longer. -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From dyamins at gmail.com Fri Jul 3 22:40:19 2009 From: dyamins at gmail.com (Dan Yamins) Date: Fri, 3 Jul 2009 22:40:19 -0400 Subject: [Numpy-discussion] __eq__ method for recarray returns recarray Message-ID: <15e4667e0907031940l2a77b59eubb4d9791f6ed136@mail.gmail.com> If I have two recarrays with the same len and column headers, the __eq__ method returns the rich comparison, which is great. E.g. In [20]: x = np.rec.fromrecords([(1,2,'dd',.3),(33,2,'y',2.2),(2,3,'a',21.4),(3,4,'b',33.2)],names=['A','B','C','D']) In [21]: y = np.rec.fromrecords([(1,2,'dd',.3),(33,2,'y',2.2),(2,3,'a',21.4),(3,4,'b',33.2)],names=['A','B','C','D']) In [22]: x == y Out[22]: rec.array([ True, True, True, True], dtype=bool) But notice that the returned object is a recarray, not an array of bools. Why is this, and what is the purpose of having it this way? Similarly, if I subclass recarray, and say, in my subclass attach some attributes to x, then these attributes are attached to the result of the rich comparison. E.g. suppose I have a subclass of recarray called "NewRecarray" to which I attach an attribute .info. Then x = NewRecarray(data) y = NewRecarray(data) z = x == y Then z is a NewRecarray object and z.info = x.info. Is this the expected / proper behavior? Is there something wrong with the way I've subclassed recarray? Thanks, Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sat Jul 4 06:13:36 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 4 Jul 2009 12:13:36 +0200 Subject: [Numpy-discussion] argwhere does not accept py list In-Reply-To: <6a17e9ee0907030344o14d6a394q381952358b076d0e@mail.gmail.com> References: <6a17e9ee0907030344o14d6a394q381952358b076d0e@mail.gmail.com> Message-ID: <9457e7c80907040313y766990c0r969e1ea43821489f@mail.gmail.com> Thanks, Scott. This should now be fixed in SVN. 2009/7/3 Scott Sinclair : > Index: numpy/core/numeric.py > =================================================================== > --- numpy/core/numeric.py ? ? ? (revision 7095) > +++ numpy/core/numeric.py ? ? ? (working copy) > @@ -535,7 +535,7 @@ > ? ? ? ? ? ?[1, 2]]) > > ? ? """ > - ? ?return asarray(a.nonzero()).T > + ? ?return transpose(asarray(a).nonzero()) From pav+sp at iki.fi Sat Jul 4 08:59:42 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Sat, 4 Jul 2009 12:59:42 +0000 (UTC) Subject: [Numpy-discussion] Adding new functions to Numpy Message-ID: Hi, When you add new functions to Numpy, please include .. versionadded:: 1.4.0 in the Notes section of the function's docstring, and add the functions to an appropriate section of the reference guide. Like this: http://projects.scipy.org/numpy/changeset/7107 This way, users can keep track of which features are supported in which version. -- Pauli Virtanen From benparkla at gmail.com Sat Jul 4 13:53:27 2009 From: benparkla at gmail.com (Ben Park) Date: Sat, 4 Jul 2009 10:53:27 -0700 (PDT) Subject: [Numpy-discussion] assigning ma.masked. Strange behavior Message-ID: <24336615.post@talk.nabble.com> import numpy as np import numpy.ma as ma # There is no effect on the following assignment of ma.masked. a1 = ma.arange(10).reshape((2,5)) a1.ravel()[np.array([0,2,2])] = ma.masked # The following works: a1 = ma.arange(10) a1[np.array([0,2,2])] = ma.masked a1.shape = 2,5 -- View this message in context: http://www.nabble.com/assigning-ma.masked.-Strange-behavior-tp24336615p24336615.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From pav+sp at iki.fi Sat Jul 4 18:16:28 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Sat, 4 Jul 2009 22:16:28 +0000 (UTC) Subject: [Numpy-discussion] Problem with ML subscriptions? Message-ID: 2009-07-04 22:52 +0200, Fons Adriaensen wrote: [clip] > I subscribed to numpy-discussion almost two days ago. > I do receive messages from the list, but the ones I post > don't appear. I wrote to numpy-discussion-owner at scipy.org > but got no response so far. So it looks that either this > list doesn't want me or there is some other problem. > Anyway it seems impossible to reach the right people. I don't think I am able to help you here. Forwarding this to the ML, so the list admins know. -- Pauli Virtanen From stefan at sun.ac.za Sat Jul 4 18:22:37 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 5 Jul 2009 00:22:37 +0200 Subject: [Numpy-discussion] Problem with ML subscriptions? In-Reply-To: References: Message-ID: <9457e7c80907041522m1268eb64s4c4f71767feab80e@mail.gmail.com> 2009/7/5 Pauli Virtanen : > 2009-07-04 22:52 +0200, Fons Adriaensen wrote: > [clip] >> I subscribed to numpy-discussion almost two days ago. >> I do receive messages from the list, but the ones I post >> don't appear. I wrote to numpy-discussion-owner at scipy.org >> but got no response so far. So it looks that either this >> list doesn't want me or there is some other problem. >> Anyway it seems impossible to reach the right people. > > I don't think I am able to help you here. Forwarding this to the ML, so > the list admins know. I've CC'd Peter Wang, who is currently the most knowledgeable about that setup. Regards St?fan From afriedle at indiana.edu Sun Jul 5 11:45:50 2009 From: afriedle at indiana.edu (Andrew Friedley) Date: Sun, 05 Jul 2009 11:45:50 -0400 Subject: [Numpy-discussion] Add/multiply reduction confusion In-Reply-To: <4A48D2DB.10508@indiana.edu> References: <4A48D2DB.10508@indiana.edu> Message-ID: <4A50CAAE.5060400@indiana.edu> I figured this out in case anyone is interested. I found the check that does the type 'upcasting' in umath_ufunc_object.inc around line 3072 (NumPy 1.3.0). Turns out all I need to do is make sure my add and multiply ufuncs are actually named 'add' and 'multiply' and arrays will be upcasted appropriately. Maybe this is worth documenting somewhere, maybe in the UFunc C API? Or is it documented already, and I just missed it? Andrew Andrew Friedley wrote: > Hi, > > I'm trying to understand how integer types are upcast for add/multiply > operations for my GSoC project (Implementing Ufuncs using CorePy). > > The documentation says that for reduction with add/multiply operations, > integer types are 'upcast' to the int_ type (int64 on my system). What > exactly does this mean, internally? Where/when does the upcasting > occur? Is it a C-style cast, or a memory copy to a new temporary array? > > I'm a confused as to which low-level ufunc loop type is used (and why). > This is what I see: > > >>> a = numpy.arange(131072, dtype=numpy.int32) > >>> r = numpy.add.reduce(a) > >>> print type(r) > > >>> print hex(r) > 0x1ffff0000L > > Okay, fine. But I have my own ufunc, which defines only the following > types right now (I stripped it down for debugging): > > >>> print corefunc.add.types > ['ii->i', 'll->l'] > > NumPy has this, for comparison: > > >>> print numpy.add.types > ['??->?', 'bb->b', 'BB->B', 'hh->h', 'HH->H', 'ii->i', 'II->I', 'll->l', > 'LL->L', 'qq->q', 'QQ->Q', 'ff->f', 'dd->d', 'gg->g', 'FF->F', 'DD->D', > 'GG->G', 'OO->O'] > > Also just to verify I did this: > > >>> print numpy.typeDict['i'] > > >>> print numpy.typeDict['l'] > > > > Yet when I call my own ufunc, this happens: > > >>> a = numpy.arange(131072, dtype=numpy.int32) > >>> r = corefunc.add.reduce(a) > >>> print type(r) > > >>> print hex(r) > -0x10000 > > It looks like no upcasting is occurring here? My ii->i loop is being > used, not the ll->l loop.. why? I'm guessing this is something I am > doing wrong, any ideas what it is? > > Andrew > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From stefan at sun.ac.za Sun Jul 5 12:37:55 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 5 Jul 2009 18:37:55 +0200 Subject: [Numpy-discussion] Add/multiply reduction confusion In-Reply-To: <4A50CAAE.5060400@indiana.edu> References: <4A48D2DB.10508@indiana.edu> <4A50CAAE.5060400@indiana.edu> Message-ID: <9457e7c80907050937u305f3508o1829a00f28d9f8f5@mail.gmail.com> Hi Andrew 2009/7/5 Andrew Friedley : > I figured this out in case anyone is interested. > > I found the check that does the type 'upcasting' in > umath_ufunc_object.inc around line 3072 (NumPy 1.3.0). ?Turns out all I > need to do is make sure my add and multiply ufuncs are actually named > 'add' and 'multiply' and arrays will be upcasted appropriately. > > Maybe this is worth documenting somewhere, maybe in the UFunc C API? ?Or > is it documented already, and I just missed it? I'm sorry nobody came to your rescue earlier. There aren't that many people who are familiar with the details of the underlying numpy implementation. Would you please be so kind as to add your findings here: http://docs.scipy.org/numpy/docs/numpy-docs/reference/index.rst/#reference-index I haven't read through that document recently, so it may be in there already. Regards St?fan From afriedle at indiana.edu Sun Jul 5 14:47:18 2009 From: afriedle at indiana.edu (Andrew Friedley) Date: Sun, 05 Jul 2009 14:47:18 -0400 Subject: [Numpy-discussion] Add/multiply reduction confusion In-Reply-To: <9457e7c80907050937u305f3508o1829a00f28d9f8f5@mail.gmail.com> References: <4A48D2DB.10508@indiana.edu> <4A50CAAE.5060400@indiana.edu> <9457e7c80907050937u305f3508o1829a00f28d9f8f5@mail.gmail.com> Message-ID: <4A50F536.6040200@indiana.edu> St?fan van der Walt wrote: > 2009/7/5 Andrew Friedley : >> I found the check that does the type 'upcasting' in >> umath_ufunc_object.inc around line 3072 (NumPy 1.3.0). Turns out all I >> need to do is make sure my add and multiply ufuncs are actually named >> 'add' and 'multiply' and arrays will be upcasted appropriately. > Would you please be so kind as to add your findings here: > > http://docs.scipy.org/numpy/docs/numpy-docs/reference/index.rst/#reference-index > > I haven't read through that document recently, so it may be in there already. I created an account (afriedle) but looks like I don't have edit permissions. The user-side upcasting behavior for add/multiply is documented (though a little hidden) in the long paragraph right before the 'Available ufuncs' section. http://docs.scipy.org/numpy/docs/numpy-docs/reference/ufuncs.rst/#ufuncs I'm thinking something should be added on the C API side though, perhaps on the 'name' parameter of PyUFunc_FromFuncAndData(): http://docs.scipy.org/numpy/docs/numpy-docs/reference/c-api.ufunc.rst/#c-api-ufunc I would say something to the effect of (I borrowed from the user-side blurb :) : Specifying a name of 'add' or 'multiply' enables a special behavior for integer-typed reductions when no dtype is given. If the input type is an integer (or boolean) data type smaller than the size of the int_ data type, it will be internally upcast to the int_ (or uint) data type. Andrew From gael.varoquaux at normalesup.org Sun Jul 5 14:57:00 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 5 Jul 2009 20:57:00 +0200 Subject: [Numpy-discussion] Add/multiply reduction confusion In-Reply-To: <4A50F536.6040200@indiana.edu> References: <4A48D2DB.10508@indiana.edu> <4A50CAAE.5060400@indiana.edu> <9457e7c80907050937u305f3508o1829a00f28d9f8f5@mail.gmail.com> <4A50F536.6040200@indiana.edu> Message-ID: <20090705185700.GA8888@phare.normalesup.org> On Sun, Jul 05, 2009 at 02:47:18PM -0400, Andrew Friedley wrote: > St?fan van der Walt wrote: > > 2009/7/5 Andrew Friedley : > >> I found the check that does the type 'upcasting' in > >> umath_ufunc_object.inc around line 3072 (NumPy 1.3.0). Turns out all I > >> need to do is make sure my add and multiply ufuncs are actually named > >> 'add' and 'multiply' and arrays will be upcasted appropriately. > > Would you please be so kind as to add your findings here: > > http://docs.scipy.org/numpy/docs/numpy-docs/reference/index.rst/#reference-index > > I haven't read through that document recently, so it may be in there already. > I created an account (afriedle) but looks like I don't have edit > permissions. I have added you to the Editor list. Ga?l From geometrian at gmail.com Sun Jul 5 16:30:48 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 5 Jul 2009 13:30:48 -0700 Subject: [Numpy-discussion] Even Sphere Volume Message-ID: Hello, I'm trying to get a cloud particle system working. As part of determining the depth of every point (to get the light that can pass through), it makes sense that the volume should be of even density. The volume is a sphere. Currently, I'm using: vecs = numpy.random.standard_normal(size=(size,size,3)) magnitudes = numpy.sqrt((vecs*vecs).sum(axis=-1)) uvecs = vecs / magnitudes[...,numpy.newaxis] randlen = numpy.random.random((size,size)) randpoints = uvecs*randlen[...,numpy.newaxis] The particles' positions are set to randpoints. However, this creates only an even distribution of *directions *on the sphere. The particles are too closely centered around the center. What I want is a certain number of particles arranged evenly throughout the sphere. How do I do that? Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Sun Jul 5 16:44:45 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Sun, 5 Jul 2009 15:44:45 -0500 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: References: Message-ID: <49d6b3500907051344h45877b5chc41e3863f8249a47@mail.gmail.com> On Sun, Jul 5, 2009 at 3:30 PM, Ian Mallett wrote: > Hello, > > I'm trying to get a cloud particle system working. As part of determining > the depth of every point (to get the light that can pass through), it makes > sense that the volume should be of even density. The volume is a sphere. > Currently, I'm using: > > vecs = numpy.random.standard_normal(size=(size,size,3)) > magnitudes = numpy.sqrt((vecs*vecs).sum(axis=-1)) > uvecs = vecs / magnitudes[...,numpy.newaxis] > randlen = numpy.random.random((size,size)) > randpoints = uvecs*randlen[...,numpy.newaxis] > > The particles' positions are set to randpoints. However, this creates > only an even distribution of *directions *on the sphere. The particles > are too closely centered around the center. What I want is a certain number > of particles arranged evenly throughout the sphere. How do I do that? > > Thanks, > Ian > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Hey Ian, What are you using to visualize this cloud particle system, if any? I am interested in what you are working on, since I have a plan to create some cloud particle growth visualizations. To better understand the underlying physical concepts of the subject and demonstrate to others, as well as simulating the two different cloud condensation nuclei counter instruments. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Sun Jul 5 16:48:39 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 5 Jul 2009 13:48:39 -0700 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <49d6b3500907051344h45877b5chc41e3863f8249a47@mail.gmail.com> References: <49d6b3500907051344h45877b5chc41e3863f8249a47@mail.gmail.com> Message-ID: Presently, it's being rendered using point sprites/VBOs. It's supposed to be for a game I'm working on, but it's a good effect to have in the toolbox too :D -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sun Jul 5 16:50:40 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 5 Jul 2009 22:50:40 +0200 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: References: Message-ID: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> 2009/7/5 Ian Mallett : > I'm trying to get a cloud particle system working.? As part of determining > the depth of every point (to get the light that can pass through), it makes > sense that the volume should be of even density.? The volume is a sphere. > Currently, I'm using: > > vecs = numpy.random.standard_normal(size=(size,size,3)) > magnitudes = numpy.sqrt((vecs*vecs).sum(axis=-1)) > uvecs = vecs / magnitudes[...,numpy.newaxis] > randlen = numpy.random.random((size,size)) > randpoints = uvecs*randlen[...,numpy.newaxis] > > The particles' positions are set to randpoints.? However, this creates only > an even distribution of directions on the sphere.? The particles are too > closely centered around the center.? What I want is a certain number of > particles arranged evenly throughout the sphere.? How do I do that? I can think of two ways: Uniformly generate points in a cube, and throw away all those outside the sphere or, if you want to generate them directly, draw from a uniformly distribution and transform the coordinates appropriately. If you want more info on the latter approach, let me know. Regards St?fan From gokhansever at gmail.com Sun Jul 5 16:59:21 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Sun, 5 Jul 2009 15:59:21 -0500 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: References: <49d6b3500907051344h45877b5chc41e3863f8249a47@mail.gmail.com> Message-ID: <49d6b3500907051359t3e6f3a72o180de67a1c6bf91f@mail.gmail.com> On Sun, Jul 5, 2009 at 3:48 PM, Ian Mallett wrote: > Presently, it's being rendered using point sprites/VBOs. It's supposed to > be for a game I'm working on, but it's a good effect to have in the toolbox > too :D > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Is this supposed to be an answer to my response? If so, please explain what is point sprites and VBOs, and where is that toolbax in this context :) -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Sun Jul 5 17:48:20 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 5 Jul 2009 14:48:20 -0700 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> Message-ID: @St?fan: I thought of the first method. Let's hear the second approach. @G?khan: Yes. Toolbox is my metaphor for being able to do effects in OpenGL. Point sprites are images mapped onto vertices, VBOs are *v*ertex *b *uffer *o*bjects, that make stuff faster. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sun Jul 5 19:40:55 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 6 Jul 2009 01:40:55 +0200 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> Message-ID: <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> 2009/7/5 Ian Mallett : > @St?fan: I thought of the first method.? Let's hear the second approach. Please see the attached example. I start off by drawing random azimuth and elevation angles, as well as a radii: N = 1000 max_radius = 5 az = np.random.uniform(low=0, high=np.pi * 2, size=N) el = np.random.uniform(low=0, high=np.pi, size=N) r = np.random.uniform(size=N) You can imagine your volume consisting of a large number of concentric spherical surfaces (almost like those Russian nested dolls). We'd like to have all of those surfaces equally densely packed, but their surfaces increase in area by the third power with radius. To counter this effect we do r = r ** (1/3.) Now, imagine the elevation contours (like latitude on the earth) for one of those spherical surfaces. If we choose them equally spaced, we'll have a much higher concentration of points near the north and south poles. Instead, we choose them according to el = np.arccos(1 - 2*el) so that we have more contours close to the equator (where the contours are longer and need more points). >From a statistical point of view, the derivation is done using transformation of random variables: http://en.wikipedia.org/wiki/Probability_integral_transform http://en.wikipedia.org/wiki/Inverse_transform_sampling Regards St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: random_sphere.py Type: application/octet-stream Size: 436 bytes Desc: not available URL: From gokhansever at gmail.com Sun Jul 5 20:08:37 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Sun, 5 Jul 2009 19:08:37 -0500 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> Message-ID: <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> 2009/7/5 St?fan van der Walt > 2009/7/5 Ian Mallett : > > @St?fan: I thought of the first method. Let's hear the second approach. > > Please see the attached example. > > I start off by drawing random azimuth and elevation angles, as well as a > radii: > > N = 1000 > max_radius = 5 > > az = np.random.uniform(low=0, high=np.pi * 2, size=N) > el = np.random.uniform(low=0, high=np.pi, size=N) > r = np.random.uniform(size=N) > > You can imagine your volume consisting of a large number of concentric > spherical surfaces (almost like those Russian nested dolls). We'd > like to have all of those surfaces equally densely packed, but their > surfaces increase in area by the third power with radius. To counter > this effect we do > > r = r ** (1/3.) > > Now, imagine the elevation contours (like latitude on the earth) for > one of those spherical surfaces. If we choose them equally spaced, > we'll have a much higher concentration of points near the north and > south poles. Instead, we choose them according to > > el = np.arccos(1 - 2*el) > > so that we have more contours close to the equator (where the contours > are longer and need more points). > > >From a statistical point of view, the derivation is done using > transformation of random variables: > > http://en.wikipedia.org/wiki/Probability_integral_transform > http://en.wikipedia.org/wiki/Inverse_transform_sampling > > Regards > St?fan > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > St?fan, Do you have any idea, why spheres have rough edges on Mayavi? Secondly, how to add an efficient random movement visualization into your scene? Without considering collision, gravity, etc the details just letting all particles freely move into this given spherical border. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From d_l_goldsmith at yahoo.com Sun Jul 5 23:27:02 2009 From: d_l_goldsmith at yahoo.com (d_l_goldsmith at yahoo.com) Date: Sun, 5 Jul 2009 20:27:02 -0700 (PDT) Subject: [Numpy-discussion] The baffling behavior that just won't die Message-ID: <520986.66417.qm@web52112.mail.re2.yahoo.com> Sorry for the cross post, but almost immediately after hitting send (to scipy-user) I realized this is the more appropriate list for the below. :-( DG Hi, folks.? I'm having a problem using numpy.lookfor() that is very reminiscent of this thread: http://mail.scipy.org/pipermail/scipy-user/2008-June/017294.html The simplest reproduction of the problem is illustrated thusly: c:\Python25\Lib\site-packages\numpy>dir random.py Volume in drive C is SQ004816V03 Volume Serial Number is 8E67-90B7 Directory of c:\Python25\Lib\site-packages\numpy File Not Found c:\Python25\Lib\site-packages\numpy>python Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> 'lookfor' in dir(np) True >>> help(np.lookfor) cannot import name Random And that's all she wrote. I read the thread cited above and it led me to examine the goings-on using python -v; as warned, there was a waterfall of imports, but at the end of it all was: ? ? ? ? ? ? ? ???. ? ? ? ? ? ? ? ???: # C:\Python25\lib\repr.pyc matches C:\Python25\lib\repr.py import repr # precompiled from C:\Python25\lib\repr.pyc import collections # builtin # C:\Python25\lib\tempfile.pyc matches C:\Python25\lib\tempfile.py import tempfile # precompiled from C:\Python25\lib\tempfile.pyc import random # directory random # random\__init__.pyc matches random\__init__.py import random # precompiled from random\__init__.pyc # random\info.pyc matches random\info.py import random.info # precompiled from random\info.pyc import random.mtrand # dynamically loaded from random\mtrand.pyd cannot import name Random Please help.? Thanks! DG From geometrian at gmail.com Mon Jul 6 00:01:26 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sun, 5 Jul 2009 21:01:26 -0700 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> Message-ID: Thanks for the example, St?fan! I'm trying to work this into a position texture, and the arrays need interleaving. I tried a couple times. Here's what I have: az = numpy.random.uniform(0, numpy.pi * 2, size*size) el = numpy.random.uniform(0, numpy.pi, size*size) r = numpy.random.uniform(size*size) # Transform random variable r = r ** (1/3.) * 1 el = numpy.arccos(1 - 2*el) x = r * numpy.cos(az) * numpy.sin(el); y = r * numpy.sin(az) * numpy.sin(el); z = r * numpy.cos(el) randvecs = numpy.array([x,y,z]).reshape((size,size,3)) rgb = ((randvecs+1.0)/2.0)*255.0 surface = pygame.surfarray.make_surface(rgb) I think the arrays are being concatenated. Hence, visually, it doesn't look right... Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Jul 6 00:52:53 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 6 Jul 2009 06:52:53 +0200 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> Message-ID: <20090706045253.GB12897@phare.normalesup.org> On Sun, Jul 05, 2009 at 07:08:37PM -0500, G?khan SEVER wrote: > Do you have any idea, why spheres have rough edges on Mayavi? If you are using 'mlab.points3d', you want to increase the 'resolution' keyword argument. It is kept quite low so that you can plot gazzilions of points. Ga?l From gokhansever at gmail.com Mon Jul 6 01:18:32 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Mon, 6 Jul 2009 00:18:32 -0500 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <20090706045253.GB12897@phare.normalesup.org> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> <20090706045253.GB12897@phare.normalesup.org> Message-ID: <49d6b3500907052218q6f454c6k326647e555f44c8e@mail.gmail.com> On Sun, Jul 5, 2009 at 11:52 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Sun, Jul 05, 2009 at 07:08:37PM -0500, G?khan SEVER wrote: > > Do you have any idea, why spheres have rough edges on Mayavi? > > If you are using 'mlab.points3d', you want to increase the 'resolution' > keyword argument. It is kept quite low so that you can plot gazzilions of > points. > > Ga?l > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Thanks Ga?l, They look prettier with a higher resolution. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.sinclair.za at gmail.com Mon Jul 6 02:25:15 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Mon, 6 Jul 2009 08:25:15 +0200 Subject: [Numpy-discussion] assigning ma.masked. Strange behavior In-Reply-To: <24336615.post@talk.nabble.com> References: <24336615.post@talk.nabble.com> Message-ID: <6a17e9ee0907052325h13a0c9c2i4c7a6f7113f691c3@mail.gmail.com> > 2009/7/4 Ben Park : > > import numpy as np > import numpy.ma as ma > > # There is no effect on the following assignment of ma.masked. > a1 = ma.arange(10).reshape((2,5)) > a1.ravel()[np.array([0,2,2])] = ma.masked In some situations ravel has to return a copy of the data instead of a view. You're assigning ma.masked to elements of the copy, not tot elements of a1. >>> a1 = ma.arange(10).reshape((2,5)) >>> b = a1.ravel() >>> b[np.array([0,2,2])] = ma.masked >>> b masked_array(data = [-- 1 -- 3 4 5 6 7 8 9], mask = [ True False True False False False False False False False], fill_value = 999999) >>> a1 masked_array(data = [[0 1 2 3 4] [5 6 7 8 9]], mask = False, fill_value = 999999) >>> a1.ravel()[np.array([0,2,2])] = ma.masked >>> a1 masked_array(data = [[0 1 2 3 4] [5 6 7 8 9]], mask = False, fill_value = 999999) Cheers, Scott From pav at iki.fi Mon Jul 6 03:33:58 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 6 Jul 2009 07:33:58 +0000 (UTC) Subject: [Numpy-discussion] The baffling behavior that just won't die References: <520986.66417.qm@web52112.mail.re2.yahoo.com> Message-ID: Sun, 05 Jul 2009 20:27:02 -0700, d_l_goldsmith kirjoitti: [clip] > c:\Python25\Lib\site-packages\numpy>python Just don't run Python inside Numpy's package directory. This is not Numpy- specific: doing a thing like that just breaks relative imports. Also, Robert's answer in the old thread applies also here: instead of a file random.py, you have a package named 'random' in the working directory. -- Pauli Virtanen From d_l_goldsmith at yahoo.com Mon Jul 6 03:42:39 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Mon, 6 Jul 2009 00:42:39 -0700 (PDT) Subject: [Numpy-discussion] The baffling behavior that just won't die Message-ID: <495738.44154.qm@web52111.mail.re2.yahoo.com> Thanks, Pauli; learn something new everyday! ;-) DG --- On Mon, 7/6/09, Pauli Virtanen wrote: > From: Pauli Virtanen > Subject: Re: [Numpy-discussion] The baffling behavior that just won't die > To: numpy-discussion at scipy.org > Date: Monday, July 6, 2009, 12:33 AM > Sun, 05 Jul 2009 20:27:02 -0700, > d_l_goldsmith kirjoitti: > [clip] > > c:\Python25\Lib\site-packages\numpy>python > > Just don't run Python inside Numpy's package directory. > This is not Numpy- > specific: doing a thing like that just breaks relative > imports. > > Also, Robert's answer in the old thread applies also here: > instead of a > file random.py, you have a package named 'random' in the > working > directory. > > -- > Pauli Virtanen > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From david at ar.media.kyoto-u.ac.jp Mon Jul 6 03:44:48 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 06 Jul 2009 16:44:48 +0900 Subject: [Numpy-discussion] The baffling behavior that just won't die In-Reply-To: References: <520986.66417.qm@web52112.mail.re2.yahoo.com> Message-ID: <4A51AB70.3010406@ar.media.kyoto-u.ac.jp> Pauli Virtanen wrote: > Sun, 05 Jul 2009 20:27:02 -0700, d_l_goldsmith kirjoitti: > [clip] > >> c:\Python25\Lib\site-packages\numpy>python >> > > Just don't run Python inside Numpy's package directory. This is not Numpy- > specific: doing a thing like that just breaks relative imports. > I noticed that many windows users make this mistake. I wonder why, and if there is something (in the docs, etc...) that we could improve to avoid this. I can't understand why anyone would got into site-packages ? cheers, David From d_l_goldsmith at yahoo.com Mon Jul 6 05:05:09 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Mon, 6 Jul 2009 02:05:09 -0700 (PDT) Subject: [Numpy-discussion] The baffling behavior that just won't die Message-ID: <723712.30760.qm@web52102.mail.re2.yahoo.com> --- On Mon, 7/6/09, David Cournapeau wrote: > avoid this. I can't understand why anyone would got into > site-packages ? Why, to look at the source, of course - I did it all the time in Mac & Unix, too - must not have ever tried to run anything while in there I guess; maybe I knew this and forgot. Oh well, don't think I'll forget this time (just as I don't think I'll now forget what the qr factorization is all about). :-) DG > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From silva at lma.cnrs-mrs.fr Mon Jul 6 05:44:38 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Mon, 06 Jul 2009 11:44:38 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: References: <1246614525.2871.30.camel@localhost.localdomain> Message-ID: <1246873478.3573.25.camel@localhost.localdomain> Le vendredi 03 juillet 2009 ? 10:00 -0600, Charles R Harris a ?crit : > What do you mean by erratic? Are the computed roots different from > known roots? The connection between polynomial coefficients and > polynomial values becomes somewhat vague when the polynomial degree > becomes large, it is numerically ill conditioned. For an illustration of what I mean by 'erratic', see http://fsilva.perso.ec-marseille.fr/visible/script_als_nbmodes.png or http://fsilva.perso.ec-marseille.fr/visible/script_als_nbmodes.pdf with the real part of the roots on the left, and the imaginary part of the right: - for low orders (<26), a pair of complex conjugate roots appears when the order of polynomial is increased (with a step equal to 2). As expected in my physical application, their real parts are negative. - for higher order (>=26), there is no more 'hermitian symmetry' (complex conjugate solutions), and real parts become positive... The computation of the coefficients of the polynomial is out of topic, I already checked it and there is no errors. -- Fabrice Silva Laboratory of Mechanics and Acoustics - CNRS 31 chemin Joseph Aiguier, 13402 Marseille, France. From david at ar.media.kyoto-u.ac.jp Mon Jul 6 05:50:47 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 06 Jul 2009 18:50:47 +0900 Subject: [Numpy-discussion] The baffling behavior that just won't die In-Reply-To: <723712.30760.qm@web52102.mail.re2.yahoo.com> References: <723712.30760.qm@web52102.mail.re2.yahoo.com> Message-ID: <4A51C8F7.201@ar.media.kyoto-u.ac.jp> David Goldsmith wrote: > --- On Mon, 7/6/09, David Cournapeau wrote: > > >> avoid this. I can't understand why anyone would got into >> site-packages ? >> > > Why, to look at the source, of course - I did it all the time in Mac & Unix, too - must not have ever tried to run anything while in there I guess; maybe I knew this and forgot. Oh well, don't think I'll forget this time (just as I don't think I'll now forget what the qr factorization is all about). :-) > Ah, I get it now - since few people build from sources on windows, looking there is the only way to look at the sources. cheers, David From stefan at sun.ac.za Mon Jul 6 06:18:00 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 6 Jul 2009 12:18:00 +0200 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> Message-ID: <9457e7c80907060318p532caf2aib6fc4f317023959a@mail.gmail.com> 2009/7/6 Ian Mallett : > randvecs = numpy.array([x,y,z]).reshape((size,size,3)) Try numpy.array([x, y, z]).T.reshape((size, size, 3)) Regards St?fan From dagss at student.matnat.uio.no Mon Jul 6 09:30:32 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 06 Jul 2009 15:30:32 +0200 Subject: [Numpy-discussion] C API refcount question Message-ID: <4A51FC78.6030709@student.matnat.uio.no> I'm trying to fledge out Cython's numpy.pxd, and have a question: Is there documentation anywhere, or a simple-to-remember rule, on whether the return value of a NumPy C API is incref-ed or not? E.g. PyArray_BASE, PyArray_DESCR obviously does not incref; while I'm assuming that PyArray_New* must incref for successful operation. Furthermore PyArray_GETITEM must also incref. This kind of a priori guessing and source inspection gets tiresome for the hundreds of functions I must cover though... -- Dag Sverre From charlesr.harris at gmail.com Mon Jul 6 10:16:50 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 6 Jul 2009 08:16:50 -0600 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: <1246873478.3573.25.camel@localhost.localdomain> References: <1246614525.2871.30.camel@localhost.localdomain> <1246873478.3573.25.camel@localhost.localdomain> Message-ID: On Mon, Jul 6, 2009 at 3:44 AM, Fabrice Silva wrote: > Le vendredi 03 juillet 2009 ? 10:00 -0600, Charles R Harris a ?crit : > > > What do you mean by erratic? Are the computed roots different from > > known roots? The connection between polynomial coefficients and > > polynomial values becomes somewhat vague when the polynomial degree > > becomes large, it is numerically ill conditioned. > > For an illustration of what I mean by 'erratic', see > http://fsilva.perso.ec-marseille.fr/visible/script_als_nbmodes.png or > http://fsilva.perso.ec-marseille.fr/visible/script_als_nbmodes.pdf > with the real part of the roots on the left, and the imaginary part of > the right: > - for low orders (<26), a pair of complex conjugate roots appears when > the order of polynomial is increased (with a step equal to 2). As > expected in my physical application, their real parts are negative. > - for higher order (>=26), there is no more 'hermitian > symmetry' (complex conjugate solutions), and real parts become > positive... > Double precision breaks down at about degree 25 if things are well scaled, so that is suspicious in itself. Also, the companion matrix isn't Hermitean so that property of the roots isn't preserved by the algorithm. If it were Hermitean then eigh would be used instead of eig. That said, there are other ways of computing polynomial roots that might preserve the Hermitean property, but I don't know that that would solve your problem. > > The computation of the coefficients of the polynomial is out of topic, I > already checked it and there is no errors. > The problem is floating point round off error in representing the coefficients. Even if you know the coefficients exactly they can't generally be represented exactly in double precision. Any computational roundoff error just adds to that. If the coefficients were all integers I would have more confidence in the no error claim. Where do the coefficients come from? Perhaps there are options there. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jul 6 10:32:58 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 6 Jul 2009 08:32:58 -0600 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: References: <1246614525.2871.30.camel@localhost.localdomain> <1246873478.3573.25.camel@localhost.localdomain> Message-ID: On Mon, Jul 6, 2009 at 8:16 AM, Charles R Harris wrote: > > > On Mon, Jul 6, 2009 at 3:44 AM, Fabrice Silva wrote: > >> Le vendredi 03 juillet 2009 ? 10:00 -0600, Charles R Harris a ?crit : >> >> > What do you mean by erratic? Are the computed roots different from >> > known roots? The connection between polynomial coefficients and >> > polynomial values becomes somewhat vague when the polynomial degree >> > becomes large, it is numerically ill conditioned. >> >> For an illustration of what I mean by 'erratic', see >> http://fsilva.perso.ec-marseille.fr/visible/script_als_nbmodes.png or >> http://fsilva.perso.ec-marseille.fr/visible/script_als_nbmodes.pdf >> with the real part of the roots on the left, and the imaginary part of >> the right: >> - for low orders (<26), a pair of complex conjugate roots appears when >> the order of polynomial is increased (with a step equal to 2). As >> expected in my physical application, their real parts are negative. >> - for higher order (>=26), there is no more 'hermitian >> symmetry' (complex conjugate solutions), and real parts become >> positive... >> > > Double precision breaks down at about degree 25 if things are well scaled, > so that is suspicious in itself. Also, the companion matrix isn't Hermitean > so that property of the roots isn't preserved by the algorithm. If it were > Hermitean then eigh would be used instead of eig. That said, there are other > ways of computing polynomial roots that might preserve the Hermitean > property, but I don't know that that would solve your problem. > > I should mention that the computation of the eigenvalues of non-Hermitean matrices is numerically difficult in itself, much more so than for Hermitean matrices. The companion matrix approach is a convenient way to get the roots but I wouldn't trust it very far. Pauli's suggestion of using Chebyshev polynomials instead of power series might yield a better conditioned matrix. I think his suggestion worth pursuing if you can make the adjustment. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jul 6 10:35:32 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 6 Jul 2009 08:35:32 -0600 Subject: [Numpy-discussion] C API refcount question In-Reply-To: <4A51FC78.6030709@student.matnat.uio.no> References: <4A51FC78.6030709@student.matnat.uio.no> Message-ID: On Mon, Jul 6, 2009 at 7:30 AM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > I'm trying to fledge out Cython's numpy.pxd, and have a question: > > Is there documentation anywhere, or a simple-to-remember rule, on > whether the return value of a NumPy C API is incref-ed or not? > > E.g. PyArray_BASE, PyArray_DESCR obviously does not incref; while I'm > assuming that PyArray_New* must incref for successful operation. > Furthermore PyArray_GETITEM must also incref. > > This kind of a priori guessing and source inspection gets tiresome for > the hundreds of functions I must cover though... > Heh, I can't answer that question. I suppose we should figure it out and document it some day ;) Does Travis' book say anything? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From silva at lma.cnrs-mrs.fr Mon Jul 6 10:53:42 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Mon, 06 Jul 2009 16:53:42 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: References: <1246614525.2871.30.camel@localhost.localdomain> <1246873478.3573.25.camel@localhost.localdomain> Message-ID: <1246892022.3573.54.camel@localhost.localdomain> Le lundi 06 juillet 2009 ? 08:16 -0600, Charles R Harris a ?crit : > Double precision breaks down at about degree 25 if things are well > scaled, so that is suspicious in itself. Also, the companion matrix > isn't Hermitean so that property of the roots isn't preserved by the > algorithm. If it were Hermitian then eigh would be used instead of > eig. That said, there are other ways of computing polynomial roots > that might preserve the Hermitean property, but I don't know that that > would solve your problem. I think there is a misunderstanding: I was referring to the fact the solution had to be real or complex conjugate, and not that the companion matrix would be a hermitian matrix (which it isn't due to its construction). Something I forgot to tell is that the roots are complex eigenfrequencies of a physical system, the real and imaginary parts expressing the damping and the frequency, respectively. If a positive frequency is solution then the opposite negative is solution too but with the ?same-signed? damping. > The problem is floating point round off error in representing the > coefficients. Even if you know the coefficients exactly they can't > generally be represented exactly in double precision. Any > computational roundoff error just adds to that. If the coefficients > were all integers I would have more confidence in the no error claim. > > Where do the coefficients come from? Perhaps there are options there. Here is the construction: the coefficients are obtained from a modal analysis of a subsystem of a bigger system. A quantity called impedance depending of a variable X is the result of the combination of several terms: Z(X) = C1/(X-X1)+C1*/(X-X1*)+...+CN/(X-XN)+CN*/(X-XN*) where * denotes the complex conjugate. I have to get the solutions X of an equation Ze(X)=N(X)/D(X) with N and D are very-low order polynomial (orders 1 and 2). An alternative of using iterative root finding (for example with scipy.optimize.fsolve) is to expand the equation as a polynomial of order close to 2N. I do understand this approach might seem non-obvious but it is rather efficient for a low value of N (<10)... -- Fabrice Silva Laboratory of Mechanics and Acoustics - CNRS 31 chemin Joseph Aiguier, 13402 Marseille, France. From nwagner at iam.uni-stuttgart.de Mon Jul 6 11:13:10 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 06 Jul 2009 17:13:10 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: <1246892022.3573.54.camel@localhost.localdomain> References: <1246614525.2871.30.camel@localhost.localdomain> <1246873478.3573.25.camel@localhost.localdomain> <1246892022.3573.54.camel@localhost.localdomain> Message-ID: On Mon, 06 Jul 2009 16:53:42 +0200 Fabrice Silva wrote: > Le lundi 06 juillet 2009 ? 08:16 -0600, Charles R Harris >a ?crit : > >> Double precision breaks down at about degree 25 if >>things are well >> scaled, so that is suspicious in itself. Also, the >>companion matrix >> isn't Hermitean so that property of the roots isn't >>preserved by the >> algorithm. If it were Hermitian then eigh would be used >>instead of >> eig. That said, there are other ways of computing >>polynomial roots >> that might preserve the Hermitean property, but I don't >>know that that >> would solve your problem. > > I think there is a misunderstanding: I was referring to >the fact the > solution had to be real or complex conjugate, and not >that the companion > matrix would be a hermitian matrix (which it isn't due >to its > construction). IIRC, the coefficients of your polynomial are complex. So, you cannot guarantee that the roots are complex conjugate pairs. Nils > Something I forgot to tell is that the roots are complex > eigenfrequencies of a physical system, the real and >imaginary parts > expressing the damping and the frequency, respectively. >If a positive > frequency is solution then the opposite negative is >solution too but > with the ?same-signed? damping. > > > >> The problem is floating point round off error in >>representing the >> coefficients. Even if you know the coefficients exactly >>they can't >> generally be represented exactly in double precision. >>Any >> computational roundoff error just adds to that. If the >>coefficients >> were all integers I would have more confidence in the no >>error claim. >> >> Where do the coefficients come from? Perhaps there are >>options there. > > Here is the construction: > the coefficients are obtained from a modal analysis of a >subsystem of a > bigger system. A quantity called impedance depending of >a variable X is > the result of the combination of several terms: > Z(X) = C1/(X-X1)+C1*/(X-X1*)+...+CN/(X-XN)+CN*/(X-XN*) > where * denotes the complex conjugate. > I have to get the solutions X of an equation >Ze(X)=N(X)/D(X) with N and > D are very-low order polynomial (orders 1 and 2). An >alternative of > using iterative root finding (for example with >scipy.optimize.fsolve) is > to expand the equation as a polynomial of order close to >2N. > I do understand this approach might seem non-obvious but >it is rather > efficient for a low value of N (<10)... > -- >Fabrice Silva > Laboratory of Mechanics and Acoustics - CNRS > 31 chemin Joseph Aiguier, 13402 Marseille, France. > From silva at lma.cnrs-mrs.fr Mon Jul 6 11:57:56 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Mon, 06 Jul 2009 17:57:56 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: References: <1246614525.2871.30.camel@localhost.localdomain> <1246873478.3573.25.camel@localhost.localdomain> <1246892022.3573.54.camel@localhost.localdomain> Message-ID: <1246895876.3573.57.camel@localhost.localdomain> Le lundi 06 juillet 2009 ? 17:13 +0200, Nils Wagner a ?crit : > IIRC, the coefficients of your polynomial are complex. > So, you cannot guarantee that the roots are complex > conjugate pairs. Correct! If the construction is done with X1 and X1* treated separately, the coefficients appear to be complex. But if their processing is done simultaneously, the combination gives real coefficients. Let me modify this point and come back to tell you if it is the breakpoint... -- Fabrice Silva Laboratory of Mechanics and Acoustics - CNRS 31 chemin Joseph Aiguier, 13402 Marseille, France. From elaine.angelino at gmail.com Mon Jul 6 13:12:28 2009 From: elaine.angelino at gmail.com (Elaine Angelino) Date: Mon, 6 Jul 2009 13:12:28 -0400 Subject: [Numpy-discussion] subclassing from np.ndarray and np.rec.recarray Message-ID: <901520e20907061012t39acaff8gdda6d276c7c925e6@mail.gmail.com> Hi -- We are subclassing from np.rec.recarray and are confused about how some methods of np.rec.recarray relate to (differ from) analogous methods of its parent, np.ndarray. Below are specific questions about the __eq__, __getitem__ and view methods, we'd appreciate answers to our specific questions and/or more general points that we may be not understanding about subclassing from np.ndarray (and np.rec.recarray). --- 1) Suppose I have a recarray object, x. How come np.ndarray.__getitem__(x, 'column_name') returns a recarray object rather than a ndarray? e.g., In [230]: x = np.rec.fromrecords([(1,'dd'), (2,'cc')], names=['a','b']) In [231]: np.ndarray.__getitem__(x, 'a') Out[231]: rec.array([1, 2]) In [232]: np.ndarray.__getitem__(x, 'a').dtype Out[232]: dtype('int32') The returned object is a recarray but it does not have a structured dtype. This generally seems to be the case when passing the instance of a subclass of np.ndarray (such as a np.rec.recarray object) to np.ndarray.__getitem__ --- 2)a) When I use the __getitem__ method of recarray to get an individual column, the returned object is an ndarray when the column is a numeric type but it is a recarray when the column is a string type. Why doesn't __getitem__ always return an ndarray for an individual column? e.g., In [175]: x = np.rec.fromrecords([(1,'dd'), (2,'cc')], names=['a','b']) In [176]: x['a'] Out[176]: array([1, 2]) In [177]: x['b'] Out[177]: rec.array(['dd', 'cc'], dtype='|S2') 2)b) Suppose I have a subclass of recarray, NewRecarray, that attaches some new attribute, e.g. 'info'. x = NewRecarray(data, names = ['a','b'], formats = ' From d_l_goldsmith at yahoo.com Mon Jul 6 14:05:18 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Mon, 6 Jul 2009 11:05:18 -0700 (PDT) Subject: [Numpy-discussion] The baffling behavior that just won't die Message-ID: <726396.73289.qm@web52110.mail.re2.yahoo.com> Right. DG --- On Mon, 7/6/09, David Cournapeau wrote: > From: David Cournapeau > Subject: Re: [Numpy-discussion] The baffling behavior that just won't die > To: "Discussion of Numerical Python" > Date: Monday, July 6, 2009, 2:50 AM > David Goldsmith wrote: > > --- On Mon, 7/6/09, David Cournapeau > wrote: > > > >??? > >> avoid this. I can't understand why anyone would > got into > >> site-packages ? > >>? ??? > > > > Why, to look at the source, of course - I did it all > the time in Mac & Unix, too - must not have ever tried > to run anything while in there I guess; maybe I knew this > and forgot.? Oh well, don't think I'll forget this time > (just as I don't think I'll now forget what the qr > factorization is all about). :-) > >??? > > Ah, I get it now - since few people build from sources on > windows, > looking there is the only way to look at the sources. > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pgmdevlist at gmail.com Mon Jul 6 14:18:52 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 6 Jul 2009 14:18:52 -0400 Subject: [Numpy-discussion] subclassing from np.ndarray and np.rec.recarray In-Reply-To: <901520e20907061012t39acaff8gdda6d276c7c925e6@mail.gmail.com> References: <901520e20907061012t39acaff8gdda6d276c7c925e6@mail.gmail.com> Message-ID: <70D6435D-FD16-42EB-98B3-860A23D25173@gmail.com> On Jul 6, 2009, at 1:12 PM, Elaine Angelino wrote: > Hi -- We are subclassing from np.rec.recarray and are confused about > how some methods of np.rec.recarray relate to (differ from) > analogous methods of its parent, np.ndarray. Below are specific > questions about the __eq__, __getitem__ and view methods, we'd > appreciate answers to our specific questions and/or more general > points that we may be not understanding about subclassing from > np.ndarray (and np.rec.recarray). For generic information about subclassing, please refer to: http://www.scipy.org/Subclasses http://docs.scipy.org/doc/numpy/user/basics.subclassing.html > 1) Suppose I have a recarray object, x. How come > np.ndarray.__getitem__(x, 'column_name') returns a recarray object > rather than a ndarray? e.g., ndarray.__getitem__(x, item) calls x.__array_finalize__ if item is a basestring and not an integer. __array_finalize__ outputs an array of the same subtype as x (here, a recarray). > 2)a) When I use the __getitem__ method of recarray to get an > individual column, the returned object is an ndarray when the column > is a numeric type but it is a recarray when the column is a string > type. Why doesn't __getitem__ always return an ndarray for an > individual column? e.g., > > > In [175]: x = np.rec.fromrecords([(1,'dd'), (2,'cc')], > names=['a','b']) > In your example. >>> x.dtype dtype([('a', ' 2)b) Suppose I have a subclass of recarray, NewRecarray, that > attaches some new attribute, e.g. 'info'. > > x = NewRecarray(data, names = ['a','b'], formats = ' > Now say I want to use recarray's __getitem__ method to get an > individual column. Then > > x['a'] is an ndarray > x['b'] is a NewRecarray and x['b'].info == x.info > > Is this the expected / proper behavior? Is there something wrong > with the way I've subclassed recarray? No, that's expected behavior. Once again, calling getitem with a field name as input calls __array_finalize__ internally. __array_finalize__ transforms the output in an array w/ the same subclass as your input: that's why x['b'] is a NewRecArray/ However, if the dtype of the output is builtin, it's transformed back to a standard ndarray: that's why x['a'] is a standard ndarray. > --- > > 3)a) If I have two recarrays with the same len and column headers, > the __eq__ method returns the rich comparison. Why is the result a > recarray rather than an ndarray? > > In [162]: x = np.rec.fromrecords([(1,'dd'), (2,'cc')], > names=['a','b']) > In [163]: y = np.rec.fromrecords([(1,'dd'), (2,'cc')], > names=['a','b']) > In [164]: x == y > Out[164]: rec.array([ True, True], dtype=bool) OK, as far as I understand, here's what's going on: * First, we check whether the dtypes are compatible. * Then, each field of x is compared to the corresponding field of y, which calls a __array_finalize__ internally, and __array_wrap__ (because you call the 'equal' ufunc). * Then, a __array_finalize__ is called on the output, which transforms it back to a recarray. > 3)b) Suppose I have a subclass of recarray, NewRecarray, that > attaches some new attribute, e.g. 'info'. > > x = NewRecarray(data) > y = NewRecarray(data) > z = x == y > > Then z is a NewRecarray object and z.info = x.info. > > Is this the expected / proper behavior? Is there something wrong > with the way I've subclassed recarray? [Dan Yamins asked this a > couple days ago] To tell you whether there's something wrong, I'd need to see the code. I'm not especially surprised by this behavior... > --- > > 4) Suppose I have a subclass of np.ndarray, NewArray, that attaches > some new attribute, e.g. 'info'. When I view a NewArray object as a > ndarray, the result has no 'info' attribute. Is the memory > corresponding to the 'info' attribute garbage collected? What > happens to it? It's alive! No, seriously: when you take a view as a ndarray, you only access the portion of memory corresponding to the values of your ndarray and none of its extra info. Same thing as calling .__array__() on your object. So the information is still accessible, as long as the initial object exists (Correct me if I'm wrong on this one...) From silva at lma.cnrs-mrs.fr Mon Jul 6 14:33:31 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Mon, 06 Jul 2009 20:33:31 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: <1246895876.3573.57.camel@localhost.localdomain> References: <1246614525.2871.30.camel@localhost.localdomain> <1246873478.3573.25.camel@localhost.localdomain> <1246892022.3573.54.camel@localhost.localdomain> <1246895876.3573.57.camel@localhost.localdomain> Message-ID: <1246905211.3573.88.camel@localhost.localdomain> Le lundi 06 juillet 2009 ? 17:57 +0200, Fabrice Silva a ?crit : > Le lundi 06 juillet 2009 ? 17:13 +0200, Nils Wagner a ?crit : > > IIRC, the coefficients of your polynomial are complex. > > So, you cannot guarantee that the roots are complex > > conjugate pairs. > > Correct! If the construction is done with X1 and X1* treated separately, > the coefficients appear to be complex. But if their processing is done > simultaneously, the combination gives real coefficients. Let me modify > this point and come back to tell you if it is the breakpoint... Alright, the coefficients are now all real (I still wonder how it passed the check tests I have). You can find attached a simple test. Looking at the evolution of a.r, it seems there really is a trouble with the representation of high order polynomial. 'a' is initialized with its roots, but a.r is far from equal to the input roots... -- Fabrice Silva Laboratory of Mechanics and Acoustics - CNRS 31 chemin Joseph Aiguier, 13402 Marseille, France. -------------- next part -------------- A non-text attachment was scrubbed... Name: test_roots.py Type: text/x-python Size: 388 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: teste_roots.png Type: image/png Size: 16335 bytes Desc: not available URL: From washakie at gmail.com Mon Jul 6 17:08:08 2009 From: washakie at gmail.com (John [H2O]) Date: Mon, 6 Jul 2009 14:08:08 -0700 (PDT) Subject: [Numpy-discussion] upgrading numpy to 1.3 on ubuntu Message-ID: <24362835.post@talk.nabble.com> Hello, I run Ubuntu 9.04 which has python-numpy 1.2 installed through apt-get. I would like to upgrade to 1.3 in order to be able to use the scikits.timeseries package. However, I cannot seem to do it using apt-get/aptitude/synaptic or at least not that I've discovered. Currently: python -c "import numpy; print numpy.version.version" 1.2.1 I have been able to manually build and install 1.3, but the problem is, it only seems to be recognized by my python when I remove python-numpy using the package manager. Once that is done, then: python -c "import numpy; print numpy.version.version" 1.3.0 However, the process of removing python-numpy causes many other packages to be removed as well due to dependencies (matplotlib, scipy, etc...). What do I have to do to get numpy upgraded? Thanks in advance! -john -- View this message in context: http://www.nabble.com/upgrading-numpy-to-1.3-on-ubuntu-tp24362835p24362835.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From josef.pktd at gmail.com Mon Jul 6 17:38:32 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 6 Jul 2009 17:38:32 -0400 Subject: [Numpy-discussion] Bug in the F distribution? In-Reply-To: <20090703212142.6a38541f@ajackson.org> References: <20090703183801.7ec5fb56@ajackson.org> <20090703212142.6a38541f@ajackson.org> Message-ID: <1cd32cbb0907061438q349735d6hedd1fcb7b80be2f6@mail.gmail.com> On Fri, Jul 3, 2009 at 10:21 PM, Alan Jackson wrote: > I've tried the same scheme using R and it seems to give the right > answers > >> quantile( rf(10000000,10,10), .99) > ? ?99% > 4.84548 >> quantile( rf(10000000,11,10), .99) > ? ? 99% > 4.770002 >> quantile( rf(10000000,11,11), .99) > ? ? 99% > 4.465655 >> quantile( rf(10000000,10,11), .99) > ? ? 99% > 4.539423 > > >>> I either found a bug in the F distribution, or I'm really messed up. >>> >>> From a table I find >>> >>> dfnum ?dfden ?F(P<.01) >>> 10 ? ? ?10 ? ? 4.85 >>> 11 ? ? ?10 ? ? 4.78 >>> 11 ? ? ?11 ? ? 4.46 >>> 10 ? ? ?11 ? ? 4.54 >>> >>> So let's calculate the same quantities using numpy... >>> >>> import scipy.stats as stats >>> import numpy as np >>> In [89]: stats.scoreatpercentile(np.random.f(10,10,1000000), 99) >>> Out[89]: 4.8575912131878365 >>> In [90]: stats.scoreatpercentile(np.random.f(11,10,1000000), 99) >>> Out[90]: 5.2721528315236501 >>> In [91]: stats.scoreatpercentile(np.random.f(11,11,1000000), 99) >>> Out[91]: 4.4695161332631841 >>> In [92]: stats.scoreatpercentile(np.random.f(10,11,1000000), 99) >>> Out[92]: 4.1229323443042674 >>> >>> >>> So at 10,10 and 11,11 it works (maybe), but all the other values are clearly >>> off. I tried re-running the example I put into the documentation last summer, >>> which worked, and I don't get the right answer any longer. > > > -- > ----------------------------------------------------------------------- > | Alan K. Jackson ? ? ? ? ? ?| To see a World in a Grain of Sand ? ? ?| > | alan at ajackson.org ? ? ? ? ?| And a Heaven in a Wild Flower, ? ? ? ? | > | www.ajackson.org ? ? ? ? ? | Hold Infinity in the palm of your hand | > | Houston, Texas ? ? ? ? ? ? | And Eternity in an hour. - Blake ? ? ? | > ----------------------------------------------------------------------- > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I don't see any problem here. If you can replicate your results, we would need more information about the versions. Josef ''' >>> np.version.version '1.3.0' >>> scipy.version.version '0.8.0.dev5789' ''' import numpy as np from scipy import stats print stats.scoreatpercentile(np.random.f(10,10,1000000), 99) print stats.scoreatpercentile(np.random.f(11,10,1000000), 99) print stats.scoreatpercentile(np.random.f(11,11,1000000), 99) print stats.scoreatpercentile(np.random.f(10,11,1000000), 99) print stats.f.ppf(0.99,[10,11,11,10],[10,10,11,11]) ''' output of 3 runs 4.8481204521 4.76654256549 4.463383254 4.52687574568 [ 4.8491468 4.77151806 4.46243604 4.53928181] >>> 4.84318380095 4.76241120816 4.45605908509 4.52990235306 [ 4.8491468 4.77151806 4.46243604 4.53928181] >>> 4.82112829219 4.77673133327 4.46580427304 4.54572040722 [ 4.8491468 4.77151806 4.46243604 4.53928181] ''' for _ in range(10): rvsf = np.random.f(11,10,100000) print np.sum(rvsf>4.77151806)/100000. ''' output of loop 0.0098 0.01016 0.01045 0.00995 0.00999 0.01029 0.01007 0.00973 0.00964 0.01006 ''' From keflavich at gmail.com Mon Jul 6 19:00:41 2009 From: keflavich at gmail.com (keflavich) Date: Mon, 6 Jul 2009 16:00:41 -0700 (PDT) Subject: [Numpy-discussion] median object Message-ID: <24364327.post@talk.nabble.com> Hi, I want to be able to do something like: import numpy x=numpy.array([1,4,4,6,7,3,4,2]) x.median() rather than numpy.median(x) so that in a function, I can call x.median() and allow x to be a masked array or a numpy array. Using the ma.median version by default is something of a workaround, because it appears to work on both numpy arrays and masked arrays. My question, though, is: why is median() not implemented as an attribute of the array/masked array object, while min/max/mean/std are? Is there a better workaround? Thanks, Adam -- View this message in context: http://www.nabble.com/median-object-tp24364327p24364327.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From sccolbert at gmail.com Mon Jul 6 19:03:32 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 6 Jul 2009 19:03:32 -0400 Subject: [Numpy-discussion] upgrading numpy to 1.3 on ubuntu In-Reply-To: <24362835.post@talk.nabble.com> References: <24362835.post@talk.nabble.com> Message-ID: <7f014ea60907061603p3a11a92cm31a8d5120e351a55@mail.gmail.com> I ran into this same problem a few days ago. The issue is that Python imports from /usr/python2.6/dist-packages before /usr/local/python2.6/dist-packages causing your numpy 1.3 (assuming its installed there) to be hidden by the snaptic numpy. To solve the problem, I added this line to my ~/.bashrc file: export PYTHONPATH="${PYTHONPATH}:/usr/local/lib/python2.6/dist-packages" replace the /usr/local/lib/python2.6/dist-packages with whatever directory you installed your numpy 1.3 in. This path will be imported before any of the other standard directories. On Mon, Jul 6, 2009 at 5:08 PM, John [H2O] wrote: > > Hello, > > I run Ubuntu 9.04 which has python-numpy 1.2 installed through apt-get. I > would like to upgrade to 1.3 in order to be able to use the > scikits.timeseries package. However, I cannot seem to do it using > apt-get/aptitude/synaptic or at least not that I've discovered. > > Currently: > python -c "import numpy; print numpy.version.version" > 1.2.1 > > I have been able to manually build and install 1.3, but the problem is, it > only seems to be recognized by my python when I remove python-numpy using > the package manager. Once that is done, then: > python -c "import numpy; print numpy.version.version" > 1.3.0 > > However, the process of removing python-numpy causes many other packages to > be removed as well due to dependencies (matplotlib, scipy, etc...). > > What do I have to do to get numpy upgraded? > > Thanks in advance! > -john > > > > -- > View this message in context: > http://www.nabble.com/upgrading-numpy-to-1.3-on-ubuntu-tp24362835p24362835.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From washakie at gmail.com Mon Jul 6 19:29:23 2009 From: washakie at gmail.com (John [H2O]) Date: Mon, 6 Jul 2009 16:29:23 -0700 (PDT) Subject: [Numpy-discussion] upgrading numpy to 1.3 on ubuntu In-Reply-To: <7f014ea60907061603p3a11a92cm31a8d5120e351a55@mail.gmail.com> References: <24362835.post@talk.nabble.com> <7f014ea60907061603p3a11a92cm31a8d5120e351a55@mail.gmail.com> Message-ID: <24364617.post@talk.nabble.com> Great! Solved/ A question, however, any reason not to say replace the apt dist-packages/numpy with the v1.3? Chris Colbert wrote: > > I ran into this same problem a few days ago. The issue is that Python > imports from /usr/python2.6/dist-packages before > /usr/local/python2.6/dist-packages causing your numpy 1.3 (assuming its > installed there) to be hidden by the snaptic numpy. > > To solve the problem, I added this line to my ~/.bashrc file: > > export PYTHONPATH="${PYTHONPATH}:/usr/local/lib/python2.6/dist-packages" > > replace the /usr/local/lib/python2.6/dist-packages with whatever directory > you installed your numpy 1.3 in. This path will be imported before any of > the other standard directories. > > > > On Mon, Jul 6, 2009 at 5:08 PM, John [H2O] wrote: > >> >> Hello, >> >> I run Ubuntu 9.04 which has python-numpy 1.2 installed through apt-get. I >> would like to upgrade to 1.3 in order to be able to use the >> scikits.timeseries package. However, I cannot seem to do it using >> apt-get/aptitude/synaptic or at least not that I've discovered. >> >> Currently: >> python -c "import numpy; print numpy.version.version" >> 1.2.1 >> >> I have been able to manually build and install 1.3, but the problem is, >> it >> only seems to be recognized by my python when I remove python-numpy using >> the package manager. Once that is done, then: >> python -c "import numpy; print numpy.version.version" >> 1.3.0 >> >> However, the process of removing python-numpy causes many other packages >> to >> be removed as well due to dependencies (matplotlib, scipy, etc...). >> >> What do I have to do to get numpy upgraded? >> >> Thanks in advance! >> -john >> >> >> >> -- >> View this message in context: >> http://www.nabble.com/upgrading-numpy-to-1.3-on-ubuntu-tp24362835p24362835.html >> Sent from the Numpy-discussion mailing list archive at Nabble.com. >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- View this message in context: http://www.nabble.com/upgrading-numpy-to-1.3-on-ubuntu-tp24362835p24364617.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From sccolbert at gmail.com Mon Jul 6 19:38:16 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 6 Jul 2009 19:38:16 -0400 Subject: [Numpy-discussion] upgrading numpy to 1.3 on ubuntu In-Reply-To: <24364617.post@talk.nabble.com> References: <24362835.post@talk.nabble.com> <7f014ea60907061603p3a11a92cm31a8d5120e351a55@mail.gmail.com> <24364617.post@talk.nabble.com> Message-ID: <7f014ea60907061638l2394347ej1e0b96d4f3ebaea1@mail.gmail.com> ask the package maintainers.... I typically build numpy with threaded atlas support, so the repo version isn't good for me anyway. And, the single threaded atlas repo libs are busted anyway... On Mon, Jul 6, 2009 at 7:29 PM, John [H2O] wrote: > > Great! Solved/ > > A question, however, any reason not to say replace the apt > dist-packages/numpy with the v1.3? > > > Chris Colbert wrote: > > > > I ran into this same problem a few days ago. The issue is that Python > > imports from /usr/python2.6/dist-packages before > > /usr/local/python2.6/dist-packages causing your numpy 1.3 (assuming its > > installed there) to be hidden by the snaptic numpy. > > > > To solve the problem, I added this line to my ~/.bashrc file: > > > > export PYTHONPATH="${PYTHONPATH}:/usr/local/lib/python2.6/dist-packages" > > > > replace the /usr/local/lib/python2.6/dist-packages with whatever > directory > > you installed your numpy 1.3 in. This path will be imported before any of > > the other standard directories. > > > > > > > > On Mon, Jul 6, 2009 at 5:08 PM, John [H2O] wrote: > > > >> > >> Hello, > >> > >> I run Ubuntu 9.04 which has python-numpy 1.2 installed through apt-get. > I > >> would like to upgrade to 1.3 in order to be able to use the > >> scikits.timeseries package. However, I cannot seem to do it using > >> apt-get/aptitude/synaptic or at least not that I've discovered. > >> > >> Currently: > >> python -c "import numpy; print numpy.version.version" > >> 1.2.1 > >> > >> I have been able to manually build and install 1.3, but the problem is, > >> it > >> only seems to be recognized by my python when I remove python-numpy > using > >> the package manager. Once that is done, then: > >> python -c "import numpy; print numpy.version.version" > >> 1.3.0 > >> > >> However, the process of removing python-numpy causes many other packages > >> to > >> be removed as well due to dependencies (matplotlib, scipy, etc...). > >> > >> What do I have to do to get numpy upgraded? > >> > >> Thanks in advance! > >> -john > >> > >> > >> > >> -- > >> View this message in context: > >> > http://www.nabble.com/upgrading-numpy-to-1.3-on-ubuntu-tp24362835p24362835.html > >> Sent from the Numpy-discussion mailing list archive at Nabble.com. > >> > >> _______________________________________________ > >> Numpy-discussion mailing list > >> Numpy-discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > View this message in context: > http://www.nabble.com/upgrading-numpy-to-1.3-on-ubuntu-tp24362835p24364617.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan at ajackson.org Mon Jul 6 21:02:27 2009 From: alan at ajackson.org (Alan Jackson) Date: Mon, 6 Jul 2009 20:02:27 -0500 Subject: [Numpy-discussion] Bug in the F distribution? In-Reply-To: <1cd32cbb0907061438q349735d6hedd1fcb7b80be2f6@mail.gmail.com> References: <20090703183801.7ec5fb56@ajackson.org> <20090703212142.6a38541f@ajackson.org> <1cd32cbb0907061438q349735d6hedd1fcb7b80be2f6@mail.gmail.com> Message-ID: <20090706200227.045b3cc8@ajackson.org> >On Fri, Jul 3, 2009 at 10:21 PM, Alan Jackson wrote: >> > >I don't see any problem here. If you can replicate your results, we >would need more information about the versions. > >Josef > >''' >>>> np.version.version >'1.3.0' >>>> scipy.version.version >'0.8.0.dev5789' >''' > In [4]: numpy.version.version Out[4]: '1.1.1' In [7]: scipy.version.version Out[7]: '0.6.0' Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49) Ubuntu Intrepid Ibex Looks like it has been fixed. I guess I'll try out the instructions for upgrading... ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From geometrian at gmail.com Mon Jul 6 23:27:45 2009 From: geometrian at gmail.com (Ian Mallett) Date: Mon, 6 Jul 2009 20:27:45 -0700 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <9457e7c80907060318p532caf2aib6fc4f317023959a@mail.gmail.com> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> <9457e7c80907060318p532caf2aib6fc4f317023959a@mail.gmail.com> Message-ID: That didn't fix it. I messed around some more, but I couldn't get the spherical coordinates working. I decided to rework my first method. By raising the radius to the one third power, like for the other method, basically the same thing is accomplished. It's working now, thanks. :D vecs = numpy.random.standard_normal((size,size,3)) magnitudes = numpy.sqrt((vecs*vecs).sum(axis=-1)) vecs = vecs / magnitudes[...,numpy.newaxis] randlen = numpy.random.random((size,size)) randlen = randlen ** (1.0/3.0) randpoints = vecs*randlen[...,numpy.newaxis] rgb = ((randpoints+1.0)/2.0)*255.0 Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From pfeldman at verizon.net Tue Jul 7 01:34:52 2009 From: pfeldman at verizon.net (Dr. Phillip M. Feldman) Date: Mon, 6 Jul 2009 22:34:52 -0700 (PDT) Subject: [Numpy-discussion] transpose of a matrix should be another matrix Message-ID: <24367402.post@talk.nabble.com> I'm using the Enthought Python Distribution. When I define a matrix and transpose it, it appears that the result is no longer a matrix (see below). This is both surprising and disappointing. Any suggestions will be appreciated. In [16]: A=matrix([[1,2,3],[4,5,6],[7,8,9]]) In [17]: B=A.transpose In [18]: A Out[18]: matrix([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) In [19]: B Out[19]: -- View this message in context: http://www.nabble.com/transpose-of-a-matrix-should-be-another-matrix-tp24367402p24367402.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From sccolbert at gmail.com Tue Jul 7 01:42:21 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 7 Jul 2009 01:42:21 -0400 Subject: [Numpy-discussion] transpose of a matrix should be another matrix In-Reply-To: <24367402.post@talk.nabble.com> References: <24367402.post@talk.nabble.com> Message-ID: <7f014ea60907062242n229d2d9cge89b0792746ee607@mail.gmail.com> you actually have to call the method as transpose(). What you requested was the actual method. >>> import numpy as np >>> a = np. matrix([[1,2,3],[4,5,6],[7,8,9]]) >>> a matrix([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) >>> b = a.transpose() >>> b matrix([[1, 4, 7], [2, 5, 8], [3, 6, 9]]) >>> On Tue, Jul 7, 2009 at 1:34 AM, Dr. Phillip M. Feldman wrote: > > I'm using the Enthought Python Distribution. When I define a matrix and > transpose it, it appears that the result is no longer a matrix (see below). > This is both surprising and disappointing. Any suggestions will be > appreciated. > > In [16]: A=matrix([[1,2,3],[4,5,6],[7,8,9]]) > > In [17]: B=A.transpose > > In [18]: A > Out[18]: > matrix([[1, 2, 3], > [4, 5, 6], > [7, 8, 9]]) > > In [19]: B > Out[19]: > -- > View this message in context: > http://www.nabble.com/transpose-of-a-matrix-should-be-another-matrix-tp24367402p24367402.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Tue Jul 7 01:46:01 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 7 Jul 2009 01:46:01 -0400 Subject: [Numpy-discussion] transpose of a matrix should be another matrix In-Reply-To: <7f014ea60907062242n229d2d9cge89b0792746ee607@mail.gmail.com> References: <24367402.post@talk.nabble.com> <7f014ea60907062242n229d2d9cge89b0792746ee607@mail.gmail.com> Message-ID: <7f014ea60907062246s55c80c0cm92c1d8824775ab7e@mail.gmail.com> I should clarify, everything in python is an object. Even methods of classes. The syntax to invoke a method is the method name followed by the parenthese (). If you leave off the parentheses, python return the method object. This can be useful if you want to pass the method to another function or method which can then execute it. They are no "pointers" in python so to speak. So passing a function or method like this is the equivalent of a function pointer. disclaimer: all arguments to functions are passed as pointers in C, but just think of them as objects and you're better off. Chris On Tue, Jul 7, 2009 at 1:42 AM, Chris Colbert wrote: > you actually have to call the method as transpose(). What you requested was > the actual method. > > >>> import numpy as np > >>> a = np. matrix([[1,2,3],[4,5,6],[7,8,9]]) > >>> a > matrix([[1, 2, 3], > [4, 5, 6], > [7, 8, 9]]) > >>> b = a.transpose() > >>> b > matrix([[1, 4, 7], > [2, 5, 8], > [3, 6, 9]]) > > >>> > > On Tue, Jul 7, 2009 at 1:34 AM, Dr. Phillip M. Feldman < > pfeldman at verizon.net> wrote: > >> >> I'm using the Enthought Python Distribution. When I define a matrix and >> transpose it, it appears that the result is no longer a matrix (see >> below). >> This is both surprising and disappointing. Any suggestions will be >> appreciated. >> >> In [16]: A=matrix([[1,2,3],[4,5,6],[7,8,9]]) >> >> In [17]: B=A.transpose >> >> In [18]: A >> Out[18]: >> matrix([[1, 2, 3], >> [4, 5, 6], >> [7, 8, 9]]) >> >> In [19]: B >> Out[19]: >> -- >> View this message in context: >> http://www.nabble.com/transpose-of-a-matrix-should-be-another-matrix-tp24367402p24367402.html >> Sent from the Numpy-discussion mailing list archive at Nabble.com. >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Tue Jul 7 01:46:39 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 7 Jul 2009 01:46:39 -0400 Subject: [Numpy-discussion] transpose of a matrix should be another matrix In-Reply-To: <7f014ea60907062246s55c80c0cm92c1d8824775ab7e@mail.gmail.com> References: <24367402.post@talk.nabble.com> <7f014ea60907062242n229d2d9cge89b0792746ee607@mail.gmail.com> <7f014ea60907062246s55c80c0cm92c1d8824775ab7e@mail.gmail.com> Message-ID: <7f014ea60907062246u31d3c904l453fd0880eedf05a@mail.gmail.com> and my grammar just sucks tonight... On Tue, Jul 7, 2009 at 1:46 AM, Chris Colbert wrote: > I should clarify, everything in python is an object. Even methods of > classes. The syntax to invoke a method is the method name followed by the > parenthese (). If you leave off the parentheses, python return the method > object. This can be useful if you want to pass the method to another > function or method which can then execute it. > > They are no "pointers" in python so to speak. So passing a function or > method like this is the equivalent of a function pointer. > > disclaimer: all arguments to functions are passed as pointers in C, but > just think of them as objects and you're better off. > > Chris > > > On Tue, Jul 7, 2009 at 1:42 AM, Chris Colbert wrote: > >> you actually have to call the method as transpose(). What you requested >> was the actual method. >> >> >>> import numpy as np >> >>> a = np. matrix([[1,2,3],[4,5,6],[7,8,9]]) >> >>> a >> matrix([[1, 2, 3], >> [4, 5, 6], >> [7, 8, 9]]) >> >>> b = a.transpose() >> >>> b >> matrix([[1, 4, 7], >> [2, 5, 8], >> [3, 6, 9]]) >> >> >>> >> >> On Tue, Jul 7, 2009 at 1:34 AM, Dr. Phillip M. Feldman < >> pfeldman at verizon.net> wrote: >> >>> >>> I'm using the Enthought Python Distribution. When I define a matrix and >>> transpose it, it appears that the result is no longer a matrix (see >>> below). >>> This is both surprising and disappointing. Any suggestions will be >>> appreciated. >>> >>> In [16]: A=matrix([[1,2,3],[4,5,6],[7,8,9]]) >>> >>> In [17]: B=A.transpose >>> >>> In [18]: A >>> Out[18]: >>> matrix([[1, 2, 3], >>> [4, 5, 6], >>> [7, 8, 9]]) >>> >>> In [19]: B >>> Out[19]: >>> -- >>> View this message in context: >>> http://www.nabble.com/transpose-of-a-matrix-should-be-another-matrix-tp24367402p24367402.html >>> Sent from the Numpy-discussion mailing list archive at Nabble.com. >>> >>> _______________________________________________ >>> Numpy-discussion mailing list >>> Numpy-discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d_l_goldsmith at yahoo.com Tue Jul 7 02:21:22 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Mon, 6 Jul 2009 23:21:22 -0700 (PDT) Subject: [Numpy-discussion] transpose of a matrix should be another matrix Message-ID: <172355.99505.qm@web52104.mail.re2.yahoo.com> Also, there is a "non-method" way: the .T property: >>> a = np. matrix([[1,2,3],[4,5,6],[7,8,9]]) >>> a matrix([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) >>> b = a.T >>> b matrix([[1, 4, 7], [2, 5, 8], [3, 6, 9]]) DG --- On Mon, 7/6/09, Chris Colbert wrote: > From: Chris Colbert > Subject: Re: [Numpy-discussion] transpose of a matrix should be another matrix > To: "Discussion of Numerical Python" > Date: Monday, July 6, 2009, 10:42 PM > you actually have to call the method as > transpose(). What you requested was the actual method. > > >>> import numpy as np > >>> a = np. matrix([[1,2,3],[4,5,6],[7,8,9]]) > >>> a > matrix([[1, 2, 3], > > ??????? [4, 5, 6], > ??????? [7, 8, 9]]) > >>> b = a.transpose() > >>> b > matrix([[1, 4, 7], > ??????? [2, 5, 8], > ??????? [3, 6, 9]]) > >>> > > On Tue, Jul 7, 2009 at 1:34 AM, > Dr. Phillip M. Feldman > wrote: > > > > I'm using the Enthought Python Distribution. When I > define a matrix and > > transpose it, it appears that the result is no longer a > matrix (see below). > > This is both surprising and disappointing. ?Any > suggestions will be > > appreciated. > > > > In [16]: A=matrix([[1,2,3],[4,5,6],[7,8,9]]) > > > > In [17]: B=A.transpose > > > > In [18]: A > > Out[18]: > > matrix([[1, 2, 3], > > ? ? ? ?[4, 5, 6], > > ? ? ? ?[7, 8, 9]]) > > > > In [19]: B > > Out[19]: 0x00CF2A30> > > -- > > View this message in context: http://www.nabble.com/transpose-of-a-matrix-should-be-another-matrix-tp24367402p24367402.html > > > Sent from the Numpy-discussion mailing list archive at > Nabble.com. > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -----Inline Attachment Follows----- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From scott.sinclair.za at gmail.com Tue Jul 7 02:34:24 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Tue, 7 Jul 2009 08:34:24 +0200 Subject: [Numpy-discussion] median object In-Reply-To: <24364327.post@talk.nabble.com> References: <24364327.post@talk.nabble.com> Message-ID: <6a17e9ee0907062334l31c09b48l3e1990dab51c7099@mail.gmail.com> > 2009/7/7 keflavich : > > Hi, I want to be able to do something like: > > import numpy > x=numpy.array([1,4,4,6,7,3,4,2]) > x.median() > > rather than > numpy.median(x) > > so that in a function, I can call > x.median() > and allow x to be a masked array or a numpy array. > > Using the ma.median version by default is something of a workaround, because > it appears to work on both numpy arrays and masked arrays. ?My question, > though, is: why is median() not implemented as an attribute of the > array/masked array object, while min/max/mean/std are? ?Is there a better > workaround? I think the canonical answer is that it's easier to implement a median function using existing array methods than to implement median as a method of the array object. I guess no-one has wanted the feature badly enough to implement it. I don't know about a better workaround, the ma.median function looks pretty flexible. Cheers, Scott From pfeldman at verizon.net Tue Jul 7 11:52:49 2009 From: pfeldman at verizon.net (Phillip M. Feldman) Date: Tue, 07 Jul 2009 08:52:49 -0700 Subject: [Numpy-discussion] transpose of a matrix should be another matrix In-Reply-To: <172355.99505.qm@web52104.mail.re2.yahoo.com> References: <172355.99505.qm@web52104.mail.re2.yahoo.com> Message-ID: <4A536F51.6030608@verizon.net> Chris and David- I've verified that both of your solutions work. I have a lot to learn! Thanks! Phillip David Goldsmith wrote: > Also, there is a "non-method" way: the .T property: > > >>>> a = np. matrix([[1,2,3],[4,5,6],[7,8,9]]) >>>> a >>>> > matrix([[1, 2, 3], > [4, 5, 6], > [7, 8, 9]]) > >>>> b = a.T >>>> b >>>> > matrix([[1, 4, 7], > [2, 5, 8], > [3, 6, 9]]) > > DG > > --- On Mon, 7/6/09, Chris Colbert wrote: > > >> From: Chris Colbert >> Subject: Re: [Numpy-discussion] transpose of a matrix should be another matrix >> To: "Discussion of Numerical Python" >> Date: Monday, July 6, 2009, 10:42 PM >> you actually have to call the method as >> transpose(). What you requested was the actual method. >> >> >>>>> import numpy as np >>>>> a = np. matrix([[1,2,3],[4,5,6],[7,8,9]]) >>>>> a >>>>> >> matrix([[1, 2, 3], >> >> [4, 5, 6], >> [7, 8, 9]]) >> >>>>> b = a.transpose() >>>>> b >>>>> >> matrix([[1, 4, 7], >> [2, 5, 8], >> [3, 6, 9]]) >> >> On Tue, Jul 7, 2009 at 1:34 AM, >> Dr. Phillip M. Feldman >> wrote: >> >> >> >> I'm using the Enthought Python Distribution. When I >> define a matrix and >> >> transpose it, it appears that the result is no longer a >> matrix (see below). >> >> This is both surprising and disappointing. Any >> suggestions will be >> >> appreciated. >> >> >> >> In [16]: A=matrix([[1,2,3],[4,5,6],[7,8,9]]) >> >> >> >> In [17]: B=A.transpose >> >> >> >> In [18]: A >> >> Out[18]: >> >> matrix([[1, 2, 3], >> >> [4, 5, 6], >> >> [7, 8, 9]]) >> >> >> >> In [19]: B >> >> Out[19]: > 0x00CF2A30> >> >> -- >> >> View this message in context: http://www.nabble.com/transpose-of-a-matrix-should-be-another-matrix-tp24367402p24367402.html >> >> >> Sent from the Numpy-discussion mailing list archive at >> Nabble.com. >> >> >> >> _______________________________________________ >> >> Numpy-discussion mailing list >> >> Numpy-discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> >> -----Inline Attachment Follows----- >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenny.abernathy at gmail.com Tue Jul 7 13:18:14 2009 From: kenny.abernathy at gmail.com (Kenny Abernathy) Date: Tue, 7 Jul 2009 13:18:14 -0400 Subject: [Numpy-discussion] Creating PyArrayObject in C++ Message-ID: I've built an analysis framework for studying electrophysiogical data, and I'd like to add NumPy (and SciPy, although that's not germane to my current question) as an engine so users can more easily add their own functions to the program. I'm at the point now of trying to share the data between the C++ program and the python interpreter. Keep in mind, these can be large datasets (some of our data is sampled at 40,000 Hz), so I'd like to minimize copying data as much as possible. My first solution is to use PyArray_FromDimsAndData(). I know that it's not recommended, but the data is owned by an object that takes care of freeing it, so it would look like the following: class Unit() { public: Unit(int size, int *data) : data_(data), pyArray_(0) { pyArray_ = PyArrayFromDimsAndData(1, &size, PyArray_INT, (char*)data_); } } ~Unit() { Py_DECREF(pyArray_); delete[] data_; } private: int *data_; PyArrayObject* pyArray_; }; I can guarantee that all analysis will be finished before the Unit object is destroyed and delete[] is called, so I don't think that's a problem. However, what if there are modifications to the array from within a Python script (specifically changes to the number of elements in the array). I should be able to get the new size from PyArrayObject, but will it have any effect on the delete[] operation? Is there a chance that PyArrayObject.data could change from within a Python script, necessitating code such as: if (data_ == pyArray_.data) { delete[] data_; } else { delete[] data_; delete[] pyArray_.data); } Or is there a better way to do this without expensive copy operations that I haven't come across yet? Thanks, Kenny -------------- next part -------------- An HTML attachment was scrubbed... URL: From d_l_goldsmith at yahoo.com Tue Jul 7 14:24:51 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Tue, 7 Jul 2009 11:24:51 -0700 (PDT) Subject: [Numpy-discussion] transpose of a matrix should be another matrix Message-ID: <253682.39694.qm@web52108.mail.re2.yahoo.com> There is no one that doesn't (have a lot to learn, that is) - we're all here to help each other do that. ;-) DG --- On Tue, 7/7/09, Phillip M. Feldman wrote: > From: Phillip M. Feldman > Subject: Re: [Numpy-discussion] transpose of a matrix should be another matrix > To: "Discussion of Numerical Python" > Date: Tuesday, July 7, 2009, 8:52 AM > > > > > > > Chris and David- > > > > I've verified that both of your solutions work.? I > have a lot to > learn!? Thanks! > > > > Phillip > > > > David Goldsmith wrote: > > Also, there is a "non-method" way: the .T > property: > > > > > > a = np. matrix([[1,2,3],[4,5,6],[7,8,9]]) > a > > > > > matrix([[1, 2, 3], > [4, 5, 6], > [7, 8, 9]]) > > > > > b = a.T > b > > > > > matrix([[1, 4, 7], > [2, 5, 8], > [3, 6, 9]]) > > DG > > --- On Mon, 7/6/09, Chris Colbert > wrote: > > > > From: Chris Colbert > Subject: Re: [Numpy-discussion] transpose of a matrix > should be another matrix > To: "Discussion of Numerical Python" > Date: Monday, July 6, 2009, 10:42 PM > you actually have to call the method as > transpose(). What you requested was the actual method. > > > > > > import numpy as np > a = np. matrix([[1,2,3],[4,5,6],[7,8,9]]) > a > > > > > matrix([[1, 2, 3], > > ??????? [4, 5, 6], > ??????? [7, 8, 9]]) > > > > > b = a.transpose() > b > > > > > matrix([[1, 4, 7], > ??????? [2, 5, 8], > ??????? [3, 6, 9]]) > > On Tue, Jul 7, 2009 at 1:34 AM, > Dr. Phillip M. Feldman > wrote: > > > > I'm using the Enthought Python Distribution. When I > define a matrix and > > transpose it, it appears that the result is no longer a > matrix (see below). > > This is both surprising and disappointing. ?Any > suggestions will be > > appreciated. > > > > In [16]: A=matrix([[1,2,3],[4,5,6],[7,8,9]]) > > > > In [17]: B=A.transpose > > > > In [18]: A > > Out[18]: > > matrix([[1, 2, 3], > > ? ? ? ?[4, 5, 6], > > ? ? ? ?[7, 8, 9]]) > > > > In [19]: B > > Out[19]: 0x00CF2A30> > > -- > > View this message in context: http://www.nabble.com/transpose-of-a-matrix-should-be-another-matrix-tp24367402p24367402.html > > > Sent from the Numpy-discussion mailing list archive at > Nabble.com. > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -----Inline Attachment Follows----- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > -----Inline Attachment Follows----- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Tue Jul 7 16:37:34 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 7 Jul 2009 14:37:34 -0600 Subject: [Numpy-discussion] Question for David re npy_funcs Message-ID: David, Should any standard c functions used in loops.c.src be the npy_* version? I've been using fabs, but I'm wondering if that should be npy_fabs. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d_l_goldsmith at yahoo.com Tue Jul 7 17:43:31 2009 From: d_l_goldsmith at yahoo.com (d_l_goldsmith at yahoo.com) Date: Tue, 7 Jul 2009 14:43:31 -0700 (PDT) Subject: [Numpy-discussion] Question for David re npy_funcs Message-ID: <400743.48926.qm@web52106.mail.re2.yahoo.com> Hi, Charles.? Which David? We have at least three lurking on this list. ;-) I mean, I know I can't answer this, but... DG --- On Tue, 7/7/09, Charles R Harris wrote: > From: Charles R Harris > Subject: [Numpy-discussion] Question for David re npy_funcs > To: "numpy-discussion" > Date: Tuesday, July 7, 2009, 1:37 PM > David, > > Should any standard c functions used in loops.c.src be the > npy_* version? I've been using fabs, but I'm > wondering if that should be npy_fabs. > > Chuck > > > -----Inline Attachment Follows----- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From stefan at sun.ac.za Tue Jul 7 18:20:16 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 8 Jul 2009 00:20:16 +0200 Subject: [Numpy-discussion] Creating PyArrayObject in C++ In-Reply-To: References: Message-ID: <9457e7c80907071520s23ac8b87odaf230848d0f7987@mail.gmail.com> Hi Kenny 2009/7/7 Kenny Abernathy : > I can guarantee that all analysis will be finished before the Unit object is > destroyed and delete[] is called, so I don't think that's a problem. There is a neat trick to make sure things don't get deallocated out of order: http://blog.enthought.com/?p=62 > ?However, what if there are modifications to the array from within a Python > script (specifically changes to the number of elements in the array). ?I > should be able to get the new size from PyArrayObject, but will it have any > effect on the delete[] operation? NumPy won't reallocate any of your memory if you set the array flag OWNDATA to False, so it should be safe to deallocate as usual. Regards St?fan From charlesr.harris at gmail.com Tue Jul 7 18:21:06 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 7 Jul 2009 16:21:06 -0600 Subject: [Numpy-discussion] Question for David re npy_funcs In-Reply-To: <400743.48926.qm@web52106.mail.re2.yahoo.com> References: <400743.48926.qm@web52106.mail.re2.yahoo.com> Message-ID: On Tue, Jul 7, 2009 at 3:43 PM, wrote: > > Hi, Charles. Which David? We have at least three lurking on this list. > ;-) I mean, I know I can't answer this, but... > That would be David C. I figured the subject line would give it away ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From trevor at notcows.com Tue Jul 7 18:50:27 2009 From: trevor at notcows.com (Trevor Clarke) Date: Tue, 7 Jul 2009 18:50:27 -0400 Subject: [Numpy-discussion] custom allocation of numpy array Message-ID: <7bde5d400907071550u21538fdem3d84fe6bca21b4f4@mail.gmail.com> I'm embedding python and numpy in a C++ program and I want to share some data owned by C++. I'm able to get the allocation/deallocation and memory sharing working for a contiguous array. However, I have non-contiguous data (random sub-array locations, not a fixed skip factor) and I may not have all of the data in memory at any given time. The data is quite large (200gb is not uncommon) so I need to load and access on-demand. I've already got a paging system in-place which obtains a contiguous sub-array on-demand and I'd like to use this with numpy. I've looked at the memmap array code and that's not sufficient as the files are compressed so a simple memmap won't work. My initial thought is to tell numpy that the entire array is available and intercept the access requests and load as necessary. I'd rather load larger sub-arrays at a time instead of each individual access. Any thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at mit.edu Tue Jul 7 20:07:22 2009 From: pinto at mit.edu (Nicolas Pinto) Date: Tue, 7 Jul 2009 20:07:22 -0400 Subject: [Numpy-discussion] bug using array with a PIL Image (misc.imread) Message-ID: <954ae5aa0907071707n5be0c415w944174717794b0f4@mail.gmail.com> Dear all, For some reason I have a problem converting a specific png image using array(). Here is what I am getting (using numpy 1.3.0 and scipy 0.7.0). % wget http://files.getdropbox.com/u/167753/spiky_adj_023.png % python -c "import numpy as np; import Image; print np.array(Image.open('spiky_adj_023.png'))" % wget http://files.getdropbox.com/u/167753/spiky_norm_001.png % python -c "import numpy as np; import Image; print np.array(Image.open('spiky_norm_001.png'))" [[134 30 140 ..., 230 83 59] [ 99 202 233 ..., 160 63 133] [ 93 241 35 ..., 7 240 101] ..., [206 132 196 ..., 139 190 112] [218 21 217 ..., 121 152 109] [ 83 188 187 ..., 6 240 251]] I was initially using scipy's misc.imread when I found this bug. I am currently using the following workaround: % python -c "import numpy as np; import Image; print np.array(Image.open('spiky_adj_023.png').convert('RGB'))" Let me know what you think. Thanks in advance. Sincerely, -- Nicolas Pinto Ph.D. Candidate, Brain & Computer Sciences Massachusetts Institute of Technology, USA http://web.mit.edu/pinto -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Jul 7 20:54:40 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 7 Jul 2009 19:54:40 -0500 Subject: [Numpy-discussion] custom allocation of numpy array In-Reply-To: <7bde5d400907071550u21538fdem3d84fe6bca21b4f4@mail.gmail.com> References: <7bde5d400907071550u21538fdem3d84fe6bca21b4f4@mail.gmail.com> Message-ID: <3d375d730907071754u5319b2ebg8abf861b3fc18ac0@mail.gmail.com> On Tue, Jul 7, 2009 at 17:50, Trevor Clarke wrote: > I'm embedding python and numpy in a C++ program and I want to share some > data owned by C++. I'm able to get the allocation/deallocation and memory > sharing working for a contiguous array. However, I have non-contiguous data > (random sub-array locations, not a fixed skip factor) and I may not have all > of the data in memory at any given time. The data is quite large (200gb is > not uncommon) so I need to load and access on-demand. I've already got a > paging system in-place which obtains a contiguous sub-array on-demand and > I'd like to use this with numpy. I've looked at the memmap array code and > that's not sufficient as the files are compressed so a simple memmap won't > work. > > My initial thought is to tell numpy that the entire array is available and > intercept the access requests and load as necessary. This is not possible without special support from the virtual memory system like mmap. numpy's memory model requires that the data can be described by a valid pointer for the starting location and the shape and strides. This is a very regular structure. You cannot have multiple, arbitrarily-distributed "starting pointers". You will have to deal with the sub-arrays as sub-arrays. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Tue Jul 7 21:04:07 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 7 Jul 2009 20:04:07 -0500 Subject: [Numpy-discussion] median object In-Reply-To: <24364327.post@talk.nabble.com> References: <24364327.post@talk.nabble.com> Message-ID: <3d375d730907071804y2df25a29yf8f5be95b3df66e0@mail.gmail.com> On Mon, Jul 6, 2009 at 18:00, keflavich wrote: > > Hi, I want to be able to do something like: > > import numpy > x=numpy.array([1,4,4,6,7,3,4,2]) > x.median() > > rather than > numpy.median(x) > > so that in a function, I can call > x.median() > and allow x to be a masked array or a numpy array. > > Using the ma.median version by default is something of a workaround, because > it appears to work on both numpy arrays and masked arrays. ?My question, > though, is: why is median() not implemented as an attribute of the > array/masked array object, while min/max/mean/std are? ?Is there a better > workaround? You may want to consider using simplegeneric to make a generic function that will dispatch to the type-appropriate implementation. http://pypi.python.org/pypi/simplegeneric -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Tue Jul 7 21:07:02 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 7 Jul 2009 20:07:02 -0500 Subject: [Numpy-discussion] median object In-Reply-To: <6a17e9ee0907062334l31c09b48l3e1990dab51c7099@mail.gmail.com> References: <24364327.post@talk.nabble.com> <6a17e9ee0907062334l31c09b48l3e1990dab51c7099@mail.gmail.com> Message-ID: <3d375d730907071807n6dd931f5o3ec7e5bda5956ecd@mail.gmail.com> On Tue, Jul 7, 2009 at 01:34, Scott Sinclair wrote: >> 2009/7/7 keflavich : >> >> Hi, I want to be able to do something like: >> >> import numpy >> x=numpy.array([1,4,4,6,7,3,4,2]) >> x.median() >> >> rather than >> numpy.median(x) >> >> so that in a function, I can call >> x.median() >> and allow x to be a masked array or a numpy array. >> >> Using the ma.median version by default is something of a workaround, because >> it appears to work on both numpy arrays and masked arrays. ?My question, >> though, is: why is median() not implemented as an attribute of the >> array/masked array object, while min/max/mean/std are? ?Is there a better >> workaround? > > I think the canonical answer is that it's easier to implement a median > function using existing array methods than to implement median as a > method of the array object. I guess no-one has wanted the feature > badly enough to implement it. Mostly just history. Many, though certainly not all, single-argument functions from Numeric became ndarray methods. I believe that median was added after that event, and there is some resistance to adding yet another method to ndarray. The implementor of numpy.ma did not feel that way and found it easier to implement median as a method on masked arrays. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Tue Jul 7 21:27:31 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 7 Jul 2009 20:27:31 -0500 Subject: [Numpy-discussion] argwhere does not accept py list In-Reply-To: <9457e7c80907040313y766990c0r969e1ea43821489f@mail.gmail.com> References: <6a17e9ee0907030344o14d6a394q381952358b076d0e@mail.gmail.com> <9457e7c80907040313y766990c0r969e1ea43821489f@mail.gmail.com> Message-ID: <3d375d730907071827j12a4d537j46b32c8cca66da24@mail.gmail.com> 2009/7/4 St?fan van der Walt : > Thanks, Scott. ?This should now be fixed in SVN. You should probably change that to asanyarray() before the masked array crowd gets upset. :-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sccolbert at gmail.com Tue Jul 7 21:54:45 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 7 Jul 2009 21:54:45 -0400 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> <9457e7c80907060318p532caf2aib6fc4f317023959a@mail.gmail.com> Message-ID: <7f014ea60907071854y40bec379obcf7f59d7390d02e@mail.gmail.com> I figured I'd contribute something similar: the "even" distribution of points on a sphere. I can't take credit for the algorithm though, I got it from the following page and just vectorized it, and tweaked it so it uses the golden ration properly: http://www.xsi-blog.com/archives/115 It makes use of the the golden ratio to increment the longitude thus tracing a spiral out around the sphere. ############################################################ # A simple program to place evenly distributed points on a unit sphere import numpy as np from enthought.mayavi import mlab def pointsOnSphere(N): """ Generate even distributed N points on the unit sphere centered at the origin. Uses the 'Golded Spiral'""" x = np.array((N,), dtype='float') y = np.array((N,), dtype='float') z = np.array((N,), dtype='float') phi = (1 + np.sqrt(5)) / 2 # the golden ratio long_incr = 2*np.pi / phi # how much to increment the longitude dz = 2.0 / float(N) # a unit sphere has diameter 2 bands = np.arange(0, N, 1) # each band will have one point placed on it z = bands * dz - 1 + (dz/2) # the z location of each band/point r = np.sqrt(1 - z*z) # the radius can be directly determined from height az = bands * long_incr # the azimuth where to place the point x = r * np.cos(az) y = r * np.sin(az) return (x, y, z) if __name__=='__main__': x, y, z = pointsOnSphere(3000) pts = mlab.points3d(x, y, z) here's a picture of the result: www.therealstevencolbert.com/dump/spherepts.png Cheers, Chris On Mon, Jul 6, 2009 at 11:27 PM, Ian Mallett wrote: > That didn't fix it. I messed around some more, but I couldn't get the > spherical coordinates working. I decided to rework my first method. By > raising the radius to the one third power, like for the other method, > basically the same thing is accomplished. It's working now, thanks. :D > > vecs = numpy.random.standard_normal((size,size,3)) > magnitudes = numpy.sqrt((vecs*vecs).sum(axis=-1)) > vecs = vecs / magnitudes[...,numpy.newaxis] > randlen = numpy.random.random((size,size)) > randlen = randlen ** (1.0/3.0) > randpoints = vecs*randlen[...,numpy.newaxis] > rgb = ((randpoints+1.0)/2.0)*255.0 > > Ian > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Tue Jul 7 21:55:33 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 7 Jul 2009 22:55:33 -0300 Subject: [Numpy-discussion] Creating PyArrayObject in C++ In-Reply-To: <9457e7c80907071520s23ac8b87odaf230848d0f7987@mail.gmail.com> References: <9457e7c80907071520s23ac8b87odaf230848d0f7987@mail.gmail.com> Message-ID: 2009/7/7 St?fan van der Walt : > Hi Kenny > > 2009/7/7 Kenny Abernathy : >> I can guarantee that all analysis will be finished before the Unit object is >> destroyed and delete[] is called, so I don't think that's a problem. > > There is a neat trick to make sure things don't get deallocated out of order: > > http://blog.enthought.com/?p=62 > Moreover, if you use PyCObject_FromVoidPtr() http://docs.python.org/c-api/cobject.html, you can register a deallocation routine to be called at DECREF time, so no need at all of implementing your a brand new type. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From keflavich at gmail.com Tue Jul 7 22:08:39 2009 From: keflavich at gmail.com (keflavich) Date: Tue, 7 Jul 2009 19:08:39 -0700 (PDT) Subject: [Numpy-discussion] median object In-Reply-To: <3d375d730907071807n6dd931f5o3ec7e5bda5956ecd@mail.gmail.com> References: <24364327.post@talk.nabble.com> <6a17e9ee0907062334l31c09b48l3e1990dab51c7099@mail.gmail.com> <3d375d730907071807n6dd931f5o3ec7e5bda5956ecd@mail.gmail.com> Message-ID: <24384227.post@talk.nabble.com> Robert Kern-2 wrote: > > Mostly just history. Many, though certainly not all, single-argument > functions from Numeric became ndarray methods. I believe that median > was added after that event, and there is some resistance to adding yet > another method to ndarray. The implementor of numpy.ma did not feel > that way and found it easier to implement median as a method on masked > arrays. > I don't think masked arrays have median methods either, but if you use numpy.ma.median exclusively, it will work on both types. Thanks for the pointer to generic, that's a more elegant solution in general. Adam -- View this message in context: http://www.nabble.com/median-object-tp24364327p24384227.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From alan at ajackson.org Tue Jul 7 22:30:06 2009 From: alan at ajackson.org (alan at ajackson.org) Date: Tue, 7 Jul 2009 21:30:06 -0500 Subject: [Numpy-discussion] Mathematica vs Matlab vs Python Message-ID: <20090707213006.028eb12d@ajackson.org> Mathematica vs Matlab vs Python http://www.larssono.com/musings/matmatpy/index.html -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From dwf at cs.toronto.edu Tue Jul 7 22:32:50 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Tue, 7 Jul 2009 22:32:50 -0400 Subject: [Numpy-discussion] bug using array with a PIL Image (misc.imread) In-Reply-To: <954ae5aa0907071707n5be0c415w944174717794b0f4@mail.gmail.com> References: <954ae5aa0907071707n5be0c415w944174717794b0f4@mail.gmail.com> Message-ID: <8A175C97-C6B1-430C-B51E-5502465C3D06@cs.toronto.edu> That is some weird mojo; I'm guessing that numpy simply doesn't recognize the 'LA' type (luminance+alpha) as it's pretty uncommon. In the meantime you probably want to convert to RGBA, since that file giving you problems is grayscale+alpha channel, and a conversion to RGB might lose the alpha channel. FWIW that's pretty much what matplotlib does, see matplotlib.image.pil_to_array which does the conversion fine. David On 7-Jul-09, at 8:07 PM, Nicolas Pinto wrote: > Dear all, > > For some reason I have a problem converting a specific png image > using array(). > > Here is what I am getting (using numpy 1.3.0 and scipy 0.7.0). > > % wget http://files.getdropbox.com/u/167753/spiky_adj_023.png > % python -c "import numpy as np; import Image; print > np.array(Image.open('spiky_adj_023.png'))" > > > % wget http://files.getdropbox.com/u/167753/spiky_norm_001.png > % python -c "import numpy as np; import Image; print > np.array(Image.open('spiky_norm_001.png'))" > [[134 30 140 ..., 230 83 59] > [ 99 202 233 ..., 160 63 133] > [ 93 241 35 ..., 7 240 101] > ..., > [206 132 196 ..., 139 190 112] > [218 21 217 ..., 121 152 109] > [ 83 188 187 ..., 6 240 251]] > > > I was initially using scipy's misc.imread when I found this bug. I > am currently using the following workaround: > % python -c "import numpy as np; import Image; print > np.array(Image.open('spiky_adj_023.png').convert('RGB'))" > > Let me know what you think. > > Thanks in advance. > > Sincerely, > > -- > Nicolas Pinto > Ph.D. Candidate, Brain & Computer Sciences > Massachusetts Institute of Technology, USA > http://web.mit.edu/pinto > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From scott.sinclair.za at gmail.com Wed Jul 8 03:18:46 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Wed, 8 Jul 2009 09:18:46 +0200 Subject: [Numpy-discussion] argwhere does not accept py list In-Reply-To: <3d375d730907071827j12a4d537j46b32c8cca66da24@mail.gmail.com> References: <6a17e9ee0907030344o14d6a394q381952358b076d0e@mail.gmail.com> <9457e7c80907040313y766990c0r969e1ea43821489f@mail.gmail.com> <3d375d730907071827j12a4d537j46b32c8cca66da24@mail.gmail.com> Message-ID: <6a17e9ee0907080018x5106610p30678e30d7e87f20@mail.gmail.com> > 2009/7/8 Robert Kern : > 2009/7/4 St?fan van der Walt : >> Thanks, Scott. ?This should now be fixed in SVN. > > You should probably change that to asanyarray() before the masked > array crowd gets upset. :-) I hadn't thought about that, but I'm don't think it matters in this case. MaskedArray.nonzero() returns a tuple of ndarrays... >>> import numpy as np >>> a = np.ma.array([4,0,2,1,3]) >>> a masked_array(data = [4 0 2 1 3], mask = False, fill_value = 999999) >>> np.asarray(a) array([4, 0, 2, 1, 3]) >>> np.asarray(a).nonzero() (array([0, 2, 3, 4]),) >>> np.asanyarray(a) masked_array(data = [4 0 2 1 3], mask = False, fill_value = 999999) >>> np.asanyarray(a).nonzero() (array([0, 2, 3, 4]),) Cheers, Scott From pav at iki.fi Wed Jul 8 04:19:32 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 8 Jul 2009 08:19:32 +0000 (UTC) Subject: [Numpy-discussion] Mathematica vs Matlab vs Python References: <20090707213006.028eb12d@ajackson.org> Message-ID: Tue, 07 Jul 2009 21:30:06 -0500, alan kirjoitti: > Mathematica vs Matlab vs Python > > http://www.larssono.com/musings/matmatpy/index.html The Python code there is not very idiomatic Numpy code. It's written for Numeric, and fancy indexing etc. are not used. Seems like the author also left it as an exercise for the reader to implement the data file parser in Matlab. I doubt it can be easily done in 20 lines :) -- Pauli Virtanen From cournape at gmail.com Wed Jul 8 04:58:50 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Jul 2009 17:58:50 +0900 Subject: [Numpy-discussion] Question for David re npy_funcs In-Reply-To: References: Message-ID: <5b8d13220907080158i24fdddf8u762e5cefb6e24d60@mail.gmail.com> On Wed, Jul 8, 2009 at 5:37 AM, Charles R Harris wrote: > David, > > Should any standard c functions used in loops.c.src be the npy_* version? > I've been using fabs, but I'm wondering if that should be npy_fabs. Yes. Although fabs is available on any platform in theory, we should standardize on using npy_ everywhere (even if the function is available, we may use a different implementation for the npy_ one because the original is too buggy). It would be nice to enforce this, actually. David From washakie at gmail.com Wed Jul 8 07:03:58 2009 From: washakie at gmail.com (John [H2O]) Date: Wed, 8 Jul 2009 04:03:58 -0700 (PDT) Subject: [Numpy-discussion] Help with np.where and datetime functions Message-ID: <24389447.post@talk.nabble.com> Hello, I have several issues which require me to iterate through a fairly large array (300000+ records). The first case is calculating and hourly average from non-regularly sampled data. The second is screening one array, based on data in the second array. The functions are defined below, but inherent to each is the following snippet: ind = np.where( (t1 < X[:,0]) & (X[:,0] < t2) ) where X is a (n,2) array and X[:,0] = a vector of datetime objects. What I am trying to do (obviously?) is find all the values of X that fall within a time range. Specifically, one point I do not understand is why the following two methods fail: --> 196 ind = np.where( (t1 < Y[:,0] < t2) ) #same result with/without inner parens TypeError: can't compare datetime.datetime to numpy.ndarray OR trying the 'and' method: --> 196 ind = np.where( (Y[:,0]>t1) and (Y[:,0]t1) & (Y[:,0]t1) and (Y[:,0] low: if s < high: dflag = False else: print 'MISSING' else: print 'no data available' dflag = False if dflag: print '''%s ::: data: %s, OKAY''' % (X[i,0],screen_vals) newX.append([X[i,0],X[i,1]]) else: print '''%s ::: BAD: %s''' % (X[i,0],screen_vals) return np.array(newX) -- View this message in context: http://www.nabble.com/Help-with-np.where-and-datetime-functions-tp24389447p24389447.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From cournape at gmail.com Wed Jul 8 07:30:35 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Jul 2009 20:30:35 +0900 Subject: [Numpy-discussion] RFC: add a install_clib numpy.distutils command to install pure C libraries In-Reply-To: <4A3231EC.9050707@ar.media.kyoto-u.ac.jp> References: <4A3231EC.9050707@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220907080430i5494338x9cdd59351846f5a8@mail.gmail.com> On Fri, Jun 12, 2009 at 7:46 PM, David Cournapeau wrote: > Hi, > > ? ?I have finally spent some time so that we can install pure C > libraries using numpy.distutils. With this, one could imagine having a C > library for fft, special functions in numpy or scipy, so that the > library could be reused in another package at the C level. If someone > knowledgeable about numpy.distutils would like to review this branch, I > would be grateful: I reworked the branch, making it simpler to add installable libraries, and ensuring it works for both conventional install and in-place builds. The design is better, I believe: - libraries are installed through the add_installed_library function, which is like add_library but with an install_dir argument - build_clib now takes an in-place argument (so that installable libraries can be reused from an in-place build) - retrieving build info has been totally redesigned. Instead of hardcoding it, I know use .ini file for each installable library, with a syntax similar to .pc files as used by pkg-config. - build information to reuse the installable libraries is through the get_info function, which returns a dict compatible with the extra_info argument of add_library/add_extension. The main change is thus the .ini file. The infrastructure could be used to replace the rather inflexible system_info at some point. It does not support every option of pkg-config, but it can handle dependencies. For example, npymath depends on the math library, so querying the libs option of npymath will automatically pull out the libs option of the mlib as well. They can also be queried from the command line: python numpy/distutils/npy_pkg_config npymath --cflags python numpy/distutils/npy_pkg_config npymath --libs So that non distutils-based build systems can retrieve the info easily (cmake, in particular). I think it would be nice to have this in 1.4.0, so that we can rely on npymath in scipy, and I need this for windows 64 bits support, cheers, David From silva at lma.cnrs-mrs.fr Mon Jul 6 13:53:31 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Mon, 06 Jul 2009 19:53:31 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: <1246895876.3573.57.camel@localhost.localdomain> References: <1246614525.2871.30.camel@localhost.localdomain> <1246873478.3573.25.camel@localhost.localdomain> <1246892022.3573.54.camel@localhost.localdomain> <1246895876.3573.57.camel@localhost.localdomain> Message-ID: <1246902812.3573.84.camel@localhost.localdomain> Le lundi 06 juillet 2009 ? 17:57 +0200, Fabrice Silva a ?crit : > Le lundi 06 juillet 2009 ? 17:13 +0200, Nils Wagner a ?crit : > > IIRC, the coefficients of your polynomial are complex. > > So, you cannot guarantee that the roots are complex > > conjugate pairs. > > Correct! If the construction is done with X1 and X1* treated separately, > the coefficients appear to be complex. But if their processing is done > simultaneously, the combination gives real coefficients. Let me modify > this point and come back to tell you if it is the breakpoint... Alright, the coefficients are now all real (I still wonder how it passed the check tests I have). You can find attached a simple test. Looking at the evolution of a.r, it seems there really is a trouble with the representation of high order polynomial. 'a' is initialized with its roots, but a.r is far from equal to the input roots... -- Fabrice Silva Laboratory of Mechanics and Acoustics - CNRS 31 chemin Joseph Aiguier, 13402 Marseille, France. -------------- next part -------------- A non-text attachment was scrubbed... Name: test_roots.png Type: image/png Size: 70770 bytes Desc: URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_roots.py Type: text/x-python Size: 388 bytes Desc: not available URL: From silva at lma.cnrs-mrs.fr Mon Jul 6 13:55:52 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Mon, 06 Jul 2009 19:55:52 +0200 Subject: [Numpy-discussion] roots and high-order polynomial In-Reply-To: <1246895876.3573.57.camel@localhost.localdomain> References: <1246614525.2871.30.camel@localhost.localdomain> <1246873478.3573.25.camel@localhost.localdomain> <1246892022.3573.54.camel@localhost.localdomain> <1246895876.3573.57.camel@localhost.localdomain> Message-ID: <1246902952.3573.85.camel@localhost.localdomain> Le lundi 06 juillet 2009 ? 17:57 +0200, Fabrice Silva a ?crit : > Le lundi 06 juillet 2009 ? 17:13 +0200, Nils Wagner a ?crit : > > IIRC, the coefficients of your polynomial are complex. > > So, you cannot guarantee that the roots are complex > > conjugate pairs. > > Correct! If the construction is done with X1 and X1* treated separately, > the coefficients appear to be complex. But if their processing is done > simultaneously, the combination gives real coefficients. Let me modify > this point and come back to tell you if it is the breakpoint... Alright, the coefficients are now all real (I still wonder how it passed the check tests I have). You can find attached a simple test. Looking at the evolution of a.r, it seems there really is a trouble with the representation of high order polynomial. 'a' is initialized with its roots, but a.r is far from equal to the input roots... -- Fabrice Silva Laboratory of Mechanics and Acoustics - CNRS 31 chemin Joseph Aiguier, 13402 Marseille, France. -------------- next part -------------- A non-text attachment was scrubbed... Name: test_roots.py Type: text/x-python Size: 388 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_roots.png Type: image/png Size: 32222 bytes Desc: URL: From neilcrighton at gmail.com Wed Jul 8 12:30:31 2009 From: neilcrighton at gmail.com (Neil Crighton) Date: Wed, 8 Jul 2009 16:30:31 +0000 (UTC) Subject: [Numpy-discussion] Help with np.where and datetime functions References: <24389447.post@talk.nabble.com> Message-ID: John [H2O] gmail.com> writes: > What I am trying to do (obviously?) is find all the values of X that fall > within a time range. > > Specifically, one point I do not understand is why the following two methods > fail: > > --> 196 ind = np.where( (t1 < Y[:,0] < t2) ) #same result > with/without inner parens > TypeError: can't compare datetime.datetime to numpy.ndarray > > OR trying the 'and' method: > > --> 196 ind = np.where( (Y[:,0]>t1) and (Y[:,0] ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all() > Use (t1 < Y[:,0]) & (Y[:,0] < t2). The python keywords 'and' and 'or' can't be overloaded, but the bitwise operators can be (and are) overloaded by arrays. Conditionals like a < x < b are converted to (a < x) and (x < b), which is why they don't work either. There is a proposal to enable overloadable 'and' and 'or' methods (http://www.python.org/dev/peps/pep-0335/), but I don't think it's ever got enough support to be accepted. Also, if you don't need the indices, you can just use the conditional expression as a boolean mask: >>> condition = (t1 < Y[:,0]) & (Y[:,0] < t2) >>> Y[:,0][condition] Neil From lciti at essex.ac.uk Wed Jul 8 13:34:57 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Wed, 8 Jul 2009 18:34:57 +0100 Subject: [Numpy-discussion] performing operations in-place in numpy Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC0@sernt14.essex.ac.uk> Hello guys, I made a patch for numpy which allows performing operations in-place to save memory allocations. For example 'sqrt(a**2 + b**2)' can be performed allocating only two arrays instead of four. You find the details in ticket 1153 of numpy-core. I thought maybe you could be interested. I am impatient to have some feedback from the community. Best, Luca From kenny.abernathy at gmail.com Wed Jul 8 14:20:08 2009 From: kenny.abernathy at gmail.com (Kenny Abernathy) Date: Wed, 8 Jul 2009 14:20:08 -0400 Subject: [Numpy-discussion] Creating PyArrayObject in C++ In-Reply-To: References: <9457e7c80907071520s23ac8b87odaf230848d0f7987@mail.gmail.com> Message-ID: Thanks, you two. That helps alot. The PyCObject_FromVoidPtr() trick is good to know--I still have to have my class because it holds other data, but this definitely points me in the right direction. On Tue, Jul 7, 2009 at 9:55 PM, Lisandro Dalcin wrote: > 2009/7/7 St?fan van der Walt : > > Hi Kenny > > > > 2009/7/7 Kenny Abernathy : > >> I can guarantee that all analysis will be finished before the Unit > object is > >> destroyed and delete[] is called, so I don't think that's a problem. > > > > There is a neat trick to make sure things don't get deallocated out of > order: > > > > http://blog.enthought.com/?p=62 > > > > Moreover, if you use PyCObject_FromVoidPtr() > http://docs.python.org/c-api/cobject.html, you can register a > deallocation routine to be called at DECREF time, so no need at all of > implementing your a brand new type. > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Wed Jul 8 14:33:15 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 8 Jul 2009 14:33:15 -0400 Subject: [Numpy-discussion] argwhere does not accept py list In-Reply-To: <6a17e9ee0907080018x5106610p30678e30d7e87f20@mail.gmail.com> References: <6a17e9ee0907030344o14d6a394q381952358b076d0e@mail.gmail.com> <9457e7c80907040313y766990c0r969e1ea43821489f@mail.gmail.com> <3d375d730907071827j12a4d537j46b32c8cca66da24@mail.gmail.com> <6a17e9ee0907080018x5106610p30678e30d7e87f20@mail.gmail.com> Message-ID: <9B91F05F-58D2-46B1-9B60-B010F9654A11@gmail.com> On Jul 8, 2009, at 3:18 AM, Scott Sinclair wrote: >> 2009/7/8 Robert Kern : >> 2009/7/4 St?fan van der Walt : >>> Thanks, Scott. This should now be fixed in SVN. >> >> You should probably change that to asanyarray() before the masked >> array crowd gets upset. :-) > > I hadn't thought about that, but I'm don't think it matters in this > case. MaskedArray.nonzero() returns a tuple of ndarrays... Except that a masked value is considered zero in that case: >>> a = ma.array([0,1,2,3],mask=[0,0,1,0]) >>> a.nonzero() (array([1, 3]),) Taking np.asarray instead of np.asanyarray loses the mask, and you end up with the wrong result. From pgmdevlist at gmail.com Wed Jul 8 14:44:50 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 8 Jul 2009 14:44:50 -0400 Subject: [Numpy-discussion] Help with np.where and datetime functions In-Reply-To: <24389447.post@talk.nabble.com> References: <24389447.post@talk.nabble.com> Message-ID: On Jul 8, 2009, at 7:03 AM, John [H2O] wrote: > > Hello, > > I have several issues which require me to iterate through a fairly > large > array (300000+ records). > > The first case is calculating and hourly average from non-regularly > sampled > data. Would you like to give the scikits.timeseries package a try ? It's available at pytseries.sourceforge.net. Calculatng the hourly average should be straightforward. > The second is screening one array, based on data in the second array. > The functions are defined below, but inherent to each is the following > snippet: > > ind = np.where( (t1 < X[:,0]) & (X[:,0] < t2) ) > > where X is a (n,2) array and X[:,0] = a vector of datetime objects. > > What I am trying to do (obviously?) is find all the values of X that > fall > within a time range. Well, timeseries could help you on this one as well. > > Specifically, one point I do not understand is why the following two > methods > fail: > > --> 196 ind = np.where( (t1 < Y[:,0] < t2) ) #same result > with/without inner parens > TypeError: can't compare datetime.datetime to numpy.ndarray > > > OR trying the 'and' method: > > --> 196 ind = np.where( (Y[:,0]>t1) and (Y[:,0] ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all() As mentioned by Neil, you need to use & instead of and in your expression. You can also use the lengthier np.logical_and(Y[:,0]>t1, Y[:,0] References: <20090707213006.028eb12d@ajackson.org> Message-ID: On 8-Jul-09, at 4:19 AM, Pauli Virtanen wrote: > Tue, 07 Jul 2009 21:30:06 -0500, alan kirjoitti: >> Mathematica vs Matlab vs Python >> >> http://www.larssono.com/musings/matmatpy/index.html > > The Python code there is not very idiomatic Numpy code. It's written > for > Numeric, and fancy indexing etc. are not used. > > Seems like the author also left it as an exercise for the reader to > implement the data file parser in Matlab. I doubt it can be easily > done > in 20 lines :) Yup. It's not even very idiomatic Python. readlines() is probably a bad idea unless your file is trivial length, and even ignoring numpy.loadtxt(), all of this could be considerably simplified with the built-in csv module. David From matthew.brett at gmail.com Wed Jul 8 15:48:17 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 8 Jul 2009 12:48:17 -0700 Subject: [Numpy-discussion] Mathematica vs Matlab vs Python In-Reply-To: References: <20090707213006.028eb12d@ajackson.org> Message-ID: <1e2af89e0907081248s8fe819cm3c186e4765dec57@mail.gmail.com> Hi, > Yup. It's not even very idiomatic Python. readlines() is probably a > bad idea unless your file is trivial length, and even ignoring > numpy.loadtxt(), all of this could be considerably simplified with the > built-in csv module. or a 1-liner with scipy.io.loadmat ... Best, Matthew From gael.varoquaux at normalesup.org Wed Jul 8 15:50:04 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 8 Jul 2009 21:50:04 +0200 Subject: [Numpy-discussion] Mathematica vs Matlab vs Python In-Reply-To: <1e2af89e0907081248s8fe819cm3c186e4765dec57@mail.gmail.com> References: <20090707213006.028eb12d@ajackson.org> <1e2af89e0907081248s8fe819cm3c186e4765dec57@mail.gmail.com> Message-ID: <20090708195003.GA27701@phare.normalesup.org> On Wed, Jul 08, 2009 at 12:48:17PM -0700, Matthew Brett wrote: > > Yup. It's not even very idiomatic Python. readlines() is probably a > > bad idea unless your file is trivial length, and even ignoring > > numpy.loadtxt(), all of this could be considerably simplified with the > > built-in csv module. > or a 1-liner with scipy.io.loadmat ... No crack! Some dude built a matlab reader for Python. Crazy man! Ga?l From stefan at sun.ac.za Wed Jul 8 16:38:31 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 8 Jul 2009 22:38:31 +0200 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC0@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC0@sernt14.essex.ac.uk> Message-ID: <9457e7c80907081338m4d0c0e03v43848724035c0fe7@mail.gmail.com> Hi Luca 2009/7/8 Citi, Luca : > Hello guys, > I made a patch for numpy which allows performing > operations in-place to save memory allocations. > For example 'sqrt(a**2 + b**2)' can be performed > allocating only two arrays instead of four. > You find the details in ticket 1153 of numpy-core. > I thought maybe you could be interested. > I am impatient to have some feedback from the > community. In the ticket you wrote: "The approach works in most cases (it passes most of the numpy tests) but makes the assumption that ufuncs can work even if one of the inputs is used as output. Is this an acceptable constraint on ufuncs?". I don't think we can make that assumption. People doing c = a + b + d expect a, b and d to remain unchanged. If this functionality could somehow be put in a scikit, I'm sure it would be useful (and to some extent, that is what `numexpr` does). Regards St?fan From stefan at sun.ac.za Wed Jul 8 16:46:20 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 8 Jul 2009 22:46:20 +0200 Subject: [Numpy-discussion] argwhere does not accept py list In-Reply-To: <3d375d730907071827j12a4d537j46b32c8cca66da24@mail.gmail.com> References: <6a17e9ee0907030344o14d6a394q381952358b076d0e@mail.gmail.com> <9457e7c80907040313y766990c0r969e1ea43821489f@mail.gmail.com> <3d375d730907071827j12a4d537j46b32c8cca66da24@mail.gmail.com> Message-ID: <9457e7c80907081346q5a6db8f1k9da3a300de32b533@mail.gmail.com> 2009/7/8 Robert Kern : > 2009/7/4 St?fan van der Walt : >> Thanks, Scott. ?This should now be fixed in SVN. > > You should probably change that to asanyarray() before the masked > array crowd gets upset. :-) Thanks, fixed! St?fan From stefan at sun.ac.za Wed Jul 8 16:58:15 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 8 Jul 2009 22:58:15 +0200 Subject: [Numpy-discussion] Mathematica vs Matlab vs Python In-Reply-To: <20090708195003.GA27701@phare.normalesup.org> References: <20090707213006.028eb12d@ajackson.org> <1e2af89e0907081248s8fe819cm3c186e4765dec57@mail.gmail.com> <20090708195003.GA27701@phare.normalesup.org> Message-ID: <9457e7c80907081358p2b8f41daxce4d7bd5b6ae63fa@mail.gmail.com> 2009/7/8 Gael Varoquaux : > On Wed, Jul 08, 2009 at 12:48:17PM -0700, Matthew Brett wrote: >> > Yup. It's not even very idiomatic Python. readlines() is probably a >> > bad idea unless your file is trivial length, and even ignoring >> > numpy.loadtxt(), all of this could be considerably simplified with the >> > built-in csv module. > >> or a 1-liner with scipy.io.loadmat ... > > No crack! Some dude built a matlab reader for Python. Crazy man! Not even a Frenchman would be that crazy. Must've been a Brit... From fons at kokkinizita.net Wed Jul 8 17:00:25 2009 From: fons at kokkinizita.net (Fons Adriaensen) Date: Wed, 8 Jul 2009 23:00:25 +0200 Subject: [Numpy-discussion] Problem with ML subscriptions? In-Reply-To: <9457e7c80907041522m1268eb64s4c4f71767feab80e@mail.gmail.com> References: <9457e7c80907041522m1268eb64s4c4f71767feab80e@mail.gmail.com> Message-ID: <20090708210025.GB2950@zita2.kokkinizita.net> On Sun, Jul 05, 2009 at 12:22:37AM +0200, St?fan van der Walt wrote: > 2009/7/5 Pauli Virtanen : > > 2009-07-04 22:52 +0200, Fons Adriaensen wrote: > > [clip] > >> I subscribed to numpy-discussion almost two days ago. > >> I do receive messages from the list, but the ones I post > >> don't appear. I wrote to numpy-discussion-owner at scipy.org > >> but got no response so far. So it looks that either this > >> list doesn't want me or there is some other problem. > >> Anyway it seems impossible to reach the right people. > > > > I don't think I am able to help you here. Forwarding this to the ML, so > > the list admins know. > > I've CC'd Peter Wang, who is currently the most knowledgeable about that setup. Many thanks ! Five days after being subscribed to this list the situation has not changed. - My message (send now four times) still doesn't appear on the list. - I don't get any mail bounce messages. - I don't get any 'your message is waiting to be approved by the list moderator' messages, - I've got no reply at all from anyone in charge of this list. Apparently, whoever is managing numpy-discussion doesn't care very much about it. So I have no choice but to unsubscribe and forget about numpy. It would have been a nice framework for what I want to do - the messages I've received so far just confirm that. But with this level of support I'd better look for an alternative. Kind regards, -- Fons Adriaensen Io lo dico sempre: l'Italia ? troppo stretta e lunga. From charlesr.harris at gmail.com Wed Jul 8 17:16:31 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Jul 2009 15:16:31 -0600 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC0@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC0@sernt14.essex.ac.uk> Message-ID: On Wed, Jul 8, 2009 at 11:34 AM, Citi, Luca wrote: > Hello guys, > I made a patch for numpy which allows performing > operations in-place to save memory allocations. > For example 'sqrt(a**2 + b**2)' can be performed I think this particular function is already available as a ufunc. What might be interesting is to use generalized ufuncs to implement various norms/products so that no copies would be needed. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lciti at essex.ac.uk Wed Jul 8 17:57:43 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Wed, 8 Jul 2009 22:57:43 +0100 Subject: [Numpy-discussion] performing operations in-place in numpy References: Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> Hi Stefan, I am afraid I did not explain myself clear enough. Of course c = a + b + d leaves a, b, and d unchanged. The only array that is overwritten is (a+b) which is a temporary array that would be destroyed anyway. Normally the operation above is performed like this: 1) allocation of a temporary array f 2) execution of sum(a, b, out=f) 3) allocation of a temporary array g 4) execution of sum(f, d, out=g) 5) assignment of g to c 6) deallocation of f With my method it is performed like this: 1) allocation of a temporary array f 2) execution of sum(a, b, out=f) 3) execution of sum(f, d, out=f) 4) assignment of f to c When I write "The approach works in most cases (it passes most of the numpy tests) but makes the assumption that ufuncs can work even if one of the inputs is used as output." I mean that the core ufunc, the atomic operation working on the single array element, must be able to work even if one of the input is used as output. As far as I can tell, they already work like this. Can we expect someone implementing a new ufunc to keep that in mind? Actually with my last version of the patch, all tests are passed. If after further thorough testing, this approach is found to not break things, there is no reason not to include it. It is completely transparent to the python user. The only programmer that should be aware of it is the numpy C developer. I hope I answered your concerns. I am sorry if my description of the ticket is a bit confused. Best, Luca From numpy-discussion at maubp.freeserve.co.uk Wed Jul 8 18:01:55 2009 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Wed, 8 Jul 2009 23:01:55 +0100 Subject: [Numpy-discussion] Problem with ML subscriptions? In-Reply-To: <20090708210025.GB2950@zita2.kokkinizita.net> References: <9457e7c80907041522m1268eb64s4c4f71767feab80e@mail.gmail.com> <20090708210025.GB2950@zita2.kokkinizita.net> Message-ID: <320fb6e00907081501n2c8954c4q612951fa0f8acfdd@mail.gmail.com> On Wed, Jul 8, 2009 at 10:00 PM, Fons Adriaensen wrote: > > On Sun, Jul 05, 2009 at 12:22:37AM +0200, St?fan van der Walt wrote: > >> 2009/7/5 Pauli Virtanen : >> > 2009-07-04 22:52 +0200, Fons Adriaensen wrote: >> > [clip] >> >> I subscribed to numpy-discussion almost two days ago. >> >> I do receive messages from the list, but the ones I post >> >> don't appear. I wrote to numpy-discussion-owner at scipy.org >> >> but got no response so far. So it looks that either this >> >> list doesn't want me or there is some other problem. >> >> Anyway it seems impossible to reach the right people. >> > >> > I don't think I am able to help you here. Forwarding this to the ML, so >> > the list admins know. >> >> I've CC'd Peter Wang, who is currently the most knowledgeable about that setup. > > Many thanks ! > > Five days after being subscribed to this list the > situation has not changed. > > - My message (send now four times) still doesn't > appear on the list. This message did. > - I don't get any mail bounce messages. That could be your ISP's fault. > - I don't get any 'your message is waiting to be > approved by the list moderator' messages, Once you are subscribed, there is no moderation. > - I've got no reply at all from anyone in charge > of this list. > > Apparently, whoever is managing numpy-discussion > doesn't care very much about it. So I have no choice > but to unsubscribe and forget about numpy. It would > have been a nice framework for what I want to do - > the messages I've received so far just confirm that. > But with this level of support I'd better look for > an alternative. We don't know if the mailing list team ever got your emails. It's pretty hard to give someone support if you don't know about their request. Anyway - fingers crossed the list is working for you now... Peter (just another NumPy user) From charlesr.harris at gmail.com Wed Jul 8 18:10:31 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Jul 2009 16:10:31 -0600 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> Message-ID: On Wed, Jul 8, 2009 at 3:57 PM, Citi, Luca wrote: > Hi Stefan, > I am afraid I did not explain myself clear enough. > > Of course > c = a + b + d > leaves a, b, and d unchanged. > The only array that is overwritten is (a+b) which is a temporary > array that would be destroyed anyway. > Normally the operation above is performed like this: > 1) allocation of a temporary array f > 2) execution of sum(a, b, out=f) > 3) allocation of a temporary array g > 4) execution of sum(f, d, out=g) > 5) assignment of g to c > 6) deallocation of f > With my method it is performed like this: > 1) allocation of a temporary array f > 2) execution of sum(a, b, out=f) > 3) execution of sum(f, d, out=f) > 4) assignment of f to c > > When I write > "The approach works in most cases (it passes > most of the numpy tests) but makes the assumption that ufuncs can work > even if one of the inputs is used as output." > I mean that the core ufunc, the atomic operation working on the > single array element, must be able to work even if one of the > input is used as output. > As far as I can tell, they already work like this. > Can we expect someone implementing a new ufunc to keep that in mind? > Yes, the last thing the happens in a ufunc is the assignment to the output. This should probably be a documented and required behaviour. The compiler also has to assume that the pointer values can be aliased, so you should be safe against optimizations. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lciti at essex.ac.uk Wed Jul 8 18:10:43 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Wed, 8 Jul 2009 23:10:43 +0100 Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> @Charles R Harris >> For example 'sqrt(a**2 + b**2)' can be performed... > I think this particular function is already available as a ufunc. I am not sure it is implemented as ufunc. But in any case it was just an example. Another example is sin(2*pi*w+phi) that is currently implemented allocating a temporary vector for (2*pi*w), another temporary vector for (2*pi*w+phi) and another for sin(2*pi*w+phi). With my patch only the first temporary vector is allocated, then it is reused twice and finally returned. One allocation instead of three. All this happens under the hood and is completely transparent to the user. Best, Luca P.S. I am sorry I am messing up with the threads but I had the digest enabled and I did not know how to reply to a post From robert.kern at gmail.com Wed Jul 8 18:12:25 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Jul 2009 17:12:25 -0500 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> Message-ID: <3d375d730907081512k6c79d607wfb485098fedd126a@mail.gmail.com> On Wed, Jul 8, 2009 at 17:10, Citi, Luca wrote: > @Charles R Harris > >>> For example 'sqrt(a**2 + b**2)' can be performed... >> I think this particular function is already available as a ufunc. > > I am not sure it is implemented as ufunc. hypot(a, b) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Wed Jul 8 18:13:45 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Jul 2009 16:13:45 -0600 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> Message-ID: On Wed, Jul 8, 2009 at 4:10 PM, Citi, Luca wrote: > @Charles R Harris > > >> For example 'sqrt(a**2 + b**2)' can be performed... > > I think this particular function is already available as a ufunc. > > I am not sure it is implemented as ufunc. > But in any case it was just an example. > > Another example is > sin(2*pi*w+phi) > that is currently implemented allocating a temporary vector for > (2*pi*w), another temporary vector for (2*pi*w+phi) and another > for sin(2*pi*w+phi). > With my patch only the first temporary vector is allocated, > then it is reused twice and finally returned. > One allocation instead of three. > > All this happens under the hood and is completely transparent to the user. > On thing to keep in mind is that the inputs might be different views of the same array so the elements might accessed in an unexpected order. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav+sp at iki.fi Wed Jul 8 18:16:22 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Wed, 8 Jul 2009 22:16:22 +0000 (UTC) Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) Message-ID: Hi, Ticket #1143 points out that Numpy's reduction operations are not always cache friendly. I worked a bit on tuning them. Just to tickle some interest, a "pathological" case before optimization: In [1]: import numpy as np In [2]: x = np.zeros((80000, 256)) In [3]: %timeit x.sum(axis=0) 10 loops, best of 3: 850 ms per loop After optimization: In [1]: import numpy as np In [2]: x = np.zeros((80000, 256)) In [3]: %timeit x.sum(axis=0) 10 loops, best of 3: 78.5 ms per loop For comparison, a reduction operation on a contiguous array of the same size: In [4]: x = np.zeros((256, 80000)) In [5]: %timeit x.sum(axis=1) 10 loops, best of 3: 88.9 ms per loop Funnily enough, it's actually slower than the reduction over the axis with the larger stride. The improvement factor depends on the CPU and its cache size. The code is here: http://github.com/pv/numpy-work/ticket/1143-speedup-reduce A benchmark script numpy/core/src/bench-reduce.py is included. Just do ./bench-reduce.py ../../../build/lib.linux-i686-2.6 | tee output.txt to run it against the system Numpy, and ./bench-reduce.py plot < output.txt to plot the results. Here's one result set: http://www.iki.fi/pav/tmp/xeon.png http://www.iki.fi/pav/tmp/xeon.txt The y-axis is proportional to the number of elemental (addition) operations per second, x-axis the total size of the array. Vertical lines show how the result changed by the cache optimization. Factor of 2 improvement seems common in the test cases (typically, arrays that are "skinny" in some dimension). There are also some factors of 10, especially for arrays larger than the 6144 kb cache of this particular CPU. Also, for this CPU, there are no significant regressions. On an older CPU (slower, smaller cache), the situation is slightly different: http://www.iki.fi/pav/tmp/athlon.png http://www.iki.fi/pav/tmp/athlon.txt On average, it's still an improvement in many cases. However, now there are more regressions. The significant ones (factor of 1/2) are N-D arrays where the reduction runs over an axis with a small number of elements. The optimization is in evaluating the reduction operation with a blocked loop (pseudocode) for block_idx in loop_over_blocks_except_axis(arr): result[block_idx] = arr[block_idx,0] for j in xrange(1, N): result[block_idx] += arr[block_idx,j] } ufunc loop instead of the usual ufunc reduction loop for idx in loop_over_array_except_axis(arr): result[idx] = arr[idx,0] } for j in xrange(1, N): } ufunc loop result[idx] += arr[idx,j] } The blocks are chosen to minimize the stride of the inmost ufunc loop, maximizing reuse of elements fitting in a single cache line. This has to be balanced against avoiding inner loops over very small dimensions, which incur some overhead. For N-D arrays, I also attempt to lump several dimensions into the ufunc loop, if the data can be iterated over with a single stride. The decision heuristics are in ufunc_object.c:_construct_reduce, and the new outer loop NOBUFFER_UFUNCREDUCE_TRANSPOSE in ufunc_object.c:PyUFunc_Reduce. Unfortunately, improving the performance using the above scheme comes at the cost of some slightly murky heuristics. I didn't manage to come up with an optimal decision rule, so they are partly empirical. There is one parameter tuning the cross-over between minimizing stride and avoiding small dimensions. (This is more or less straightforward.) Another empirical decision is required in choosing whether to use the usual reduction loop, which is better in some cases, or the blocked loop. How to make this latter choice is not so clear to me. The present approach seems to mostly work, as far as the benchmark produces improvements and produces no severe regressions. However, I think the regressions for the older CPU point out that something still needs to be tuned. The main suspect is the latter decision rule, but I'm not sure what the correct solution here is. Comments? I'm not a cache optimization expert, so bright ideas about better decision rules and further optimizations would be appreciated. Also, I think this addition could be nice to have included in Numpy. The main value IMHO is in boosting the most cache-inefficient corner cases. I think many users do not realize that np.zeros((256, 80000)).sum(axis=1) is currently more efficient than np.zeros((80000, 256)).sum(axis=0). And as shown above, there is no real reason to have this performance difference... Cheers, Pauli From lciti at essex.ac.uk Wed Jul 8 18:17:51 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Wed, 8 Jul 2009 23:17:51 +0100 Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> > On thing to keep in mind is that the inputs might be different views of the > same array so the elements might accessed in an unexpected order. Only inputs owning their own data and with refcount 1 (i.e. no other array can be a view of it) are re-used as outputs. From charlesr.harris at gmail.com Wed Jul 8 18:23:03 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Jul 2009 16:23:03 -0600 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) In-Reply-To: References: Message-ID: On Wed, Jul 8, 2009 at 4:16 PM, Pauli Virtanen > wrote: > Hi, > > Ticket #1143 points out that Numpy's reduction operations are not > always cache friendly. I worked a bit on tuning them. > > > Just to tickle some interest, a "pathological" case before optimization: > > In [1]: import numpy as np > In [2]: x = np.zeros((80000, 256)) > In [3]: %timeit x.sum(axis=0) > 10 loops, best of 3: 850 ms per loop > > After optimization: > > In [1]: import numpy as np > In [2]: x = np.zeros((80000, 256)) > In [3]: %timeit x.sum(axis=0) > 10 loops, best of 3: 78.5 ms per loop > > For comparison, a reduction operation on a contiguous array of > the same size: > > In [4]: x = np.zeros((256, 80000)) > In [5]: %timeit x.sum(axis=1) > 10 loops, best of 3: 88.9 ms per loop > ;) > > Funnily enough, it's actually slower than the reduction over the > axis with the larger stride. The improvement factor depends on > the CPU and its cache size. > > How do the benchmarks compare with just making contiguous copies? Which is blocking of sort, I suppose. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jul 8 18:30:05 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Jul 2009 16:30:05 -0600 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> Message-ID: On Wed, Jul 8, 2009 at 4:17 PM, Citi, Luca wrote: > > > On thing to keep in mind is that the inputs might be different views of > the > > same array so the elements might accessed in an unexpected order. > > Only inputs owning their own data and with refcount 1 (i.e. no other array > can be a view of it) > are re-used as outputs. > _____ > In that case I don't see a problem offhand. That said, I haven't looked at the code yet. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Jul 8 18:38:49 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 9 Jul 2009 00:38:49 +0200 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) In-Reply-To: References: Message-ID: <9457e7c80907081538x11f5e63m2c030743f7921e78@mail.gmail.com> Hi Pauli 2009/7/9 Pauli Virtanen : > Unfortunately, improving the performance using the above scheme > comes at the cost of some slightly murky heuristics. ?I didn't > manage to come up with an optimal decision rule, so they are > partly empirical. There is one parameter tuning the cross-over > between minimizing stride and avoiding small dimensions. (This is > more or less straightforward.) ?Another empirical decision is > required in choosing whether to use the usual reduction loop, > which is better in some cases, or the blocked loop. How to make > this latter choice is not so clear to me. I know very little about cache optimality, so excuse the triviality of this question: Is it possible to design this loop optimally (taking into account certain build-time measurable parameters), or is it the kind of thing that can only be discovered by tuning at compile-time? ATNumPy... scary :-) Cheers St?fan From pav+sp at iki.fi Wed Jul 8 18:55:05 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Wed, 8 Jul 2009 22:55:05 +0000 (UTC) Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) References: Message-ID: On 2009-07-08, Charles R Harris wrote: [clip] > How do the benchmarks compare with just making contiguous copies? Which is > blocking of sort, I suppose. I think that's slower than just walking over the discontiguous array: 1) The new code: (on the Athlon machine) $ ./bench-reduce.py ../../../build/lib.linux-i686-2.6 8000,260 0,1 0 <8000> {260} [ <2080> {8} ] -15 vs. 0.876923 : 1 2 new 8000,260 0,1 0 27.7 29.5 28.9 30.2 30.3 2) Old code, with "x.transpose(1,0).copy().sum(axis=-1)": $ ./bench-reduce.py '' -o 8000,260 0,1 0 old 8000,260 0,1 0 5.03 5.13 5.11 5.07 4.87 3) Old code, with "x.sum(axis=0)" $ ./bench-reduce.py '' 8000,260 0,1 0 old 8000,260 0,1 0 18.6 17.3 16.6 18.9 17.2 Last five numbers on the rows are repeats per second, five repeats. (Above, the trailing dimension 260 was chosen so that it maximizes the speed of the old code: 260 is almost twice faster than 256... This seems to be a cache padding effect.) I think this is expected: the main bottleneck is probably the memory bandwidth, so when you make a copy and then reduce it, you round-trip the array twice through the CPU cache. This is likely to be more expensive than a single trip, even if it's discontiguous. Also, I'm not sure if copying a discontiguous array in Numpy actually has any cache optimization, or whether it just walks the array in virtual C-order copying the data. If so, then it probably hits similar cache problems as the non-optimized reduction operation. -- Pauli Virtanen From fons at kokkinizita.net Wed Jul 8 18:53:50 2009 From: fons at kokkinizita.net (Fons Adriaensen) Date: Thu, 9 Jul 2009 00:53:50 +0200 Subject: [Numpy-discussion] Problem with ML subscriptions? In-Reply-To: <320fb6e00907081501n2c8954c4q612951fa0f8acfdd@mail.gmail.com> References: <9457e7c80907041522m1268eb64s4c4f71767feab80e@mail.gmail.com> <20090708210025.GB2950@zita2.kokkinizita.net> <320fb6e00907081501n2c8954c4q612951fa0f8acfdd@mail.gmail.com> Message-ID: <20090708225350.GC2950@zita2.kokkinizita.net> On Wed, Jul 08, 2009 at 11:01:55PM +0100, Peter wrote: > On Wed, Jul 8, 2009 at 10:00 PM, Fons Adriaensen wrote: > > > > On Sun, Jul 05, 2009 at 12:22:37AM +0200, St?fan van der Walt wrote: > > > >> 2009/7/5 Pauli Virtanen : > >> > 2009-07-04 22:52 +0200, Fons Adriaensen wrote: > >> > [clip] > >> >> I subscribed to numpy-discussion almost two days ago. > >> >> I do receive messages from the list, but the ones I post > >> >> don't appear. I wrote to numpy-discussion-owner at scipy.org > >> >> but got no response so far. So it looks that either this > >> >> list doesn't want me or there is some other problem. > >> >> Anyway it seems impossible to reach the right people. > >> > > >> > I don't think I am able to help you here. Forwarding this to the ML, so > >> > the list admins know. > >> > >> I've CC'd Peter Wang, who is currently the most knowledgeable about that setup. > > > > Many thanks ! > > > > Five days after being subscribed to this list the > > situation has not changed. > > > > - My message (send now four times) still doesn't > > appear on the list. > > This message did. Strangely enough, yes. It wasn't meant for the list, I just replied to Stefan's message. The one I posted to the list one hour before is still 'somewhere'. Or isn't anymore at all. I verified the address in my outgoing mail logs, it *is* correct. No HTML. No attachments. No foul language. > > - I don't get any mail bounce messages. > > That could be your ISP's fault. Don't think so. All the years I've been using this service (HostEurope in Germany) I've received quite informative message for anything that did go wrong, either there or down the line. > > - I don't get any 'your message is waiting to be > > approved by the list moderator' messages, > > Once you are subscribed, there is no moderation. OK. Some lists will have the first few posts moderated, just to be sure it's not some spammer. > > - I've got no reply at all from anyone in charge > > of this list. > > > > Apparently, whoever is managing numpy-discussion > > doesn't care very much about it. So I have no choice > > but to unsubscribe and forget about numpy. It would > > have been a nice framework for what I want to do - > > the messages I've received so far just confirm that. > > But with this level of support I'd better look for > > an alternative. > > We don't know if the mailing list team ever got your > emails. It's pretty hard to give someone support if > you don't know about their request. Pauli Virtanen copied my message to him to the list, and Stefan van der Walt CC'd it to Peter Wang. Someone should have seen it. > Anyway - fingers crossed the list is working for you > now... I'm not convinced... Will try posting again. Ciao, -- FA Io lo dico sempre: l'Italia ? troppo stretta e lunga. From robert.kern at gmail.com Wed Jul 8 19:01:22 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Jul 2009 18:01:22 -0500 Subject: [Numpy-discussion] Problem with ML subscriptions? In-Reply-To: <20090708225350.GC2950@zita2.kokkinizita.net> References: <9457e7c80907041522m1268eb64s4c4f71767feab80e@mail.gmail.com> <20090708210025.GB2950@zita2.kokkinizita.net> <320fb6e00907081501n2c8954c4q612951fa0f8acfdd@mail.gmail.com> <20090708225350.GC2950@zita2.kokkinizita.net> Message-ID: <3d375d730907081601y7a7faf1p4c5525cfe408c845@mail.gmail.com> On Wed, Jul 8, 2009 at 17:53, Fons Adriaensen wrote: > On Wed, Jul 08, 2009 at 11:01:55PM +0100, Peter wrote: >> Anyway - fingers crossed the list is working for you >> now... > > I'm not convinced... Will try posting again. And here you are. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pav+sp at iki.fi Wed Jul 8 19:02:47 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Wed, 8 Jul 2009 23:02:47 +0000 (UTC) Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) References: <9457e7c80907081538x11f5e63m2c030743f7921e78@mail.gmail.com> Message-ID: On 2009-07-08, St?fan van der Walt wrote: > I know very little about cache optimality, so excuse the triviality of > this question: Is it possible to design this loop optimally (taking > into account certain build-time measurable parameters), or is it the > kind of thing that can only be discovered by tuning at compile-time? > ATNumPy... scary :-) I'm still kind of hoping that it's possible to make some minimal assumptions about CPU caches in general, and have a rule that decides a code path that is good enough, if not optimal. I don't think we want to go the ATNumPy route, or even have tunable parameters chosen at build or compile time. (Unless, of course, we want to bring a monster into the world -- think about cross-breeding distutils with the ATLAS build system :) -- Pauli Virtanen From robert.kern at gmail.com Wed Jul 8 19:06:55 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Jul 2009 18:06:55 -0500 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) In-Reply-To: References: <9457e7c80907081538x11f5e63m2c030743f7921e78@mail.gmail.com> Message-ID: <3d375d730907081606x47d9cf99j325ad3bce0e09449@mail.gmail.com> On Wed, Jul 8, 2009 at 18:02, Pauli Virtanen wrote: > On 2009-07-08, St?fan van der Walt wrote: >> I know very little about cache optimality, so excuse the triviality of >> this question: Is it possible to design this loop optimally (taking >> into account certain build-time measurable parameters), or is it the >> kind of thing that can only be discovered by tuning at compile-time? >> ATNumPy... scary :-) > > I'm still kind of hoping that it's possible to make some minimal > assumptions about CPU caches in general, and have a rule that > decides a code path that is good enough, if not optimal. > > I don't think we want to go the ATNumPy route, or even have > tunable parameters chosen at build or compile time. (Unless, of > course, we want to bring a monster into the world -- think about > cross-breeding distutils with the ATLAS build system :) I imagine that we could do some or all of this configuration at runtime. We have a dynamic language. ATLAS does not. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournape at gmail.com Wed Jul 8 19:20:43 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 9 Jul 2009 08:20:43 +0900 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) In-Reply-To: References: <9457e7c80907081538x11f5e63m2c030743f7921e78@mail.gmail.com> Message-ID: <5b8d13220907081620s5c1c9734k1873e0240a08b58d@mail.gmail.com> On Thu, Jul 9, 2009 at 8:02 AM, Pauli Virtanen wrote: > > I don't think we want to go the ATNumPy route, or even have > tunable parameters chosen at build or compile time. Detecting things like cache size at compile time should not be too difficult, at least for common platforms. Even detecting it at runtime should be relatively simple in some particular cases (x86). BTW, one good baseline for those summation is to use dot: np.ones((80000, 256)).sum(axis=0) vs np.dot(np.ones((1, 80000)), np.ones((80000, 256))) Assuming dot uses an optimized blas, this is generally one order of magnitude faster than sum. > (Unless, of > course, we want to bring a monster into the world -- think about > cross-breeding distutils with the ATLAS build system :) Kill me now :) David From fons at kokkinizita.net Wed Jul 8 19:15:45 2009 From: fons at kokkinizita.net (Fons Adriaensen) Date: Thu, 9 Jul 2009 01:15:45 +0200 Subject: [Numpy-discussion] ndarray and the buffer protocol Message-ID: <20090708231545.GD2950@zita2.kokkinizita.net> Hello all, (resending for the Nth time, as the previous attempts didn't make it to the list) I'm new to this list (and numpy is mostly new to me :-). Using python 2.6 and numpy 1.3. My plan is to write some C extensions that will perform rather specialised processing on multichannel digital audio data stored stored in numpy arrays. Sinc 2.6 python has a new buffer protocol, and referring to this, the numpy website states rather fuzzily: Of course, numpy makes a good interoperation library to support the buffer protocol. Numpy can both consume and produce protocol-supporting objects. A numpy ndarray can be constructed as a view of an object supporting the buffer protocol, and a numpy ndarray supports buffer object views of its data. I'd want the C code to be based on using the new buffer protocol - everything else is marked as 'deprecated', also in the numpy docs. But it remains quite unclear if numpy 1.3 does or does not support this, and if yes, how. As far as I've been able to find out, calling PyObject_CheckBuffer() on a numpy array returns zero, indicating that the object does not support the buffer protocol. To add to the confusion, the python 2.6 docs say that the new buffer protocol should also be used to access python's own array type. But also for this type PyObject_CheckBuffer() returns zero. I'd be very happy is someone could clear the fog... TIA, -- FA Io lo dico sempre: l'Italia ? troppo stretta e lunga. From charlesr.harris at gmail.com Wed Jul 8 19:24:52 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Jul 2009 17:24:52 -0600 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) In-Reply-To: References: <9457e7c80907081538x11f5e63m2c030743f7921e78@mail.gmail.com> Message-ID: On Wed, Jul 8, 2009 at 5:02 PM, Pauli Virtanen > wrote: > On 2009-07-08, St?fan van der Walt wrote: > > I know very little about cache optimality, so excuse the triviality of > > this question: Is it possible to design this loop optimally (taking > > into account certain build-time measurable parameters), or is it the > > kind of thing that can only be discovered by tuning at compile-time? > > ATNumPy... scary :-) > > I'm still kind of hoping that it's possible to make some minimal > assumptions about CPU caches in general, and have a rule that > decides a code path that is good enough, if not optimal. > > I don't think we want to go the ATNumPy route, or even have > tunable parameters chosen at build or compile time. (Unless, of > course, we want to bring a monster into the world -- think about > cross-breeding distutils with the ATLAS build system :) > Sort of the software version of the Human Fly. Sounds like next summer's blockbuster movie. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav+sp at iki.fi Wed Jul 8 19:28:16 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Wed, 8 Jul 2009 23:28:16 +0000 (UTC) Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> Message-ID: On 2009-07-08, Charles R Harris wrote: > In that case I don't see a problem offhand. That said, I haven't looked at > the code yet. I'm a bit worried about the problem that cropped up in the ticket with the complex ufuncs. As Luca noted in the ticket, obj3 = PyArray_Conjugate((PyAO *)obj1, NULL); needed to be replaced with Py_INCREF(obj1); obj3 = PyArray_Conjugate((PyAO *)obj1, NULL); Py_DECREF(obj1); to avoid overwriting obj1 when refcount of `obj1 == 1`. Now, having refcounts of 1 can be a common condition in arrays in C code, and PyArray_Conjugate is a public function in Numpy's C-API. So, as it stands now, this patch seems to potentially break existing C-extension modules that use the Numpy/Numeric API. This would need to be fixed. Perhaps the INCREF/DECREF pair can be moved inside PyArray_Conjugate. If I understand correctly similar issues can only crop up in functions calling the *GenericUnary* etc. ufunc functions directly. Are these functions part of the public API? If not, there's a chance of fixing their callers inside Numpy. If they are a part of the public API, then the issue seems to become more hairy. Anyway, it seems there is some auditing to do before this change can be safely considered. Also, the speedups obtained were fairly modest, 20%. Are they larger for more complicated expressions? (Ie. is there an expression whose execution time is halved?) -- Pauli Virtanen From robert.kern at gmail.com Wed Jul 8 19:34:57 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Jul 2009 18:34:57 -0500 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> Message-ID: <3d375d730907081634r61a4174cl850835a543511caf@mail.gmail.com> On Wed, Jul 8, 2009 at 18:28, Pauli Virtanen wrote: > On 2009-07-08, Charles R Harris wrote: >> In that case I don't see a problem offhand. That said, I haven't looked at >> the code yet. > > I'm a bit worried about the problem that cropped up in the ticket > with the complex ufuncs. As Luca noted in the ticket, > > ? ?obj3 = PyArray_Conjugate((PyAO *)obj1, NULL); > > needed to be replaced with > > ? ?Py_INCREF(obj1); > ? ?obj3 = PyArray_Conjugate((PyAO *)obj1, NULL); > ? ?Py_DECREF(obj1); > > to avoid overwriting obj1 when refcount of `obj1 == 1`. > > Now, having refcounts of 1 can be a common condition in arrays in > C code, and PyArray_Conjugate is a public function in Numpy's > C-API. So, as it stands now, this patch seems to potentially > break existing C-extension modules that use the Numpy/Numeric > API. This would need to be fixed. > > > Perhaps the INCREF/DECREF pair can be moved inside > PyArray_Conjugate. It's entirely possible that this should be done regardless. > If I understand correctly similar issues can > only crop up in functions calling the *GenericUnary* etc. ufunc > functions directly. Are these functions part of the public API? > If not, there's a chance of fixing their callers inside Numpy. The public API is exactly the set of function pointers in PyArray_API and PyUFunc_API. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From washakie at gmail.com Wed Jul 8 20:01:48 2009 From: washakie at gmail.com (John [H2O]) Date: Wed, 8 Jul 2009 17:01:48 -0700 (PDT) Subject: [Numpy-discussion] Help with np.where and datetime functions In-Reply-To: References: <24389447.post@talk.nabble.com> Message-ID: <24401595.post@talk.nabble.com> Pierre GM-2 wrote: > > > > Would you like to give the scikits.timeseries package a try ? It's > available at pytseries.sourceforge.net. > Calculatng the hourly average should be straightforward. > I would, in fact I have been investigating it, but I didn't have numpy1.3 up and running until just recently. Will it manage non-regularly sampled data? That was my concern, also, it wasn't clear to me how to use it, but I recently found more information on the web page (I started with the 'Examples' page, which as of yet isn't ready. > > Well, timeseries could help you on this one as well. > Great. sounds helpful. > As mentioned by Neil, you need to use & instead of and in your > expression. You can also use the lengthier np.logical_and(Y[:,0]>t1, > Y[:,0] > Also, using np.where without additional inputs > (np.where(condition,x,y)) is equivalent to .nonzero > Thanks for the helpful input! -- View this message in context: http://www.nabble.com/Help-with-np.where-and-datetime-functions-tp24389447p24401595.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From washakie at gmail.com Wed Jul 8 20:05:39 2009 From: washakie at gmail.com (John [H2O]) Date: Wed, 8 Jul 2009 17:05:39 -0700 (PDT) Subject: [Numpy-discussion] Help with np.where and datetime functions In-Reply-To: References: <24389447.post@talk.nabble.com> Message-ID: <24401626.post@talk.nabble.com> nhmc wrote: > > > > Also, if you don't need the indices, you can just use the conditional > expression as a boolean mask: > >>>> condition = (t1 < Y[:,0]) & (Y[:,0] < t2) >>>> Y[:,0][condition] > > Neil > 'condition' is not an index array? Wouldn't it just be the indices as well? Would it be possible to do this: Y[:,0][ (t1 < Y[:,0]) & (Y[:,0] < t2) ] I guess that IS what you're saying to do above, but I'm confused what 'condition' would be once returned... (I'll go play in the sandbox now...) -- View this message in context: http://www.nabble.com/Help-with-np.where-and-datetime-functions-tp24389447p24401626.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From washakie at gmail.com Wed Jul 8 20:11:32 2009 From: washakie at gmail.com (John [H2O]) Date: Wed, 8 Jul 2009 17:11:32 -0700 (PDT) Subject: [Numpy-discussion] Help with np.where and datetime functions In-Reply-To: <24401626.post@talk.nabble.com> References: <24389447.post@talk.nabble.com> <24401626.post@talk.nabble.com> Message-ID: <24401687.post@talk.nabble.com> Also, could someone please explain why: Tsub = T[ (T[:,0]>t1) & (T[:,0] References: <24389447.post@talk.nabble.com> <24401626.post@talk.nabble.com> <24401687.post@talk.nabble.com> Message-ID: <3d375d730907081733j5193d8e3i1185b4accdc0ec8@mail.gmail.com> On Wed, Jul 8, 2009 at 19:11, John [H2O] wrote: > > Also, could someone please explain why: > > Tsub = T[ (T[:,0]>t1) & (T[:,0] > Works, but: > > Tsub = T[ (t1 > Does not??? I'm not positive, but I think it boils down to this: Python tries to look up the method from the first operand before the second operand. TimeSeries know about datetime objects, but datetime objects don't know about TimeSeries. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lciti at essex.ac.uk Wed Jul 8 20:51:13 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Thu, 9 Jul 2009 01:51:13 +0100 Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> Hello The problem is not PyArray_Conjugate itself. The problem is that whenever you call a function from the C side and one of the inputs has ref_count 1, it can be overwritten. This is not a problem from the python side because if the ufunc sees a ref_count=1 it means that no python object is referencing to it. Let us consider an ufunc like sumdiff : c,d = sumdiff(a,b) <=> c,d = a+b,a-b if you implement it in python it is fine. If it is implemented in C, it has to be implemented like this: Py_INCREF(obj1); Py_INCREF(obj2); obj3 = PyArray_Sum((PyAO *)obj1, (PyAO *)obj2, NULL); Py_DECREF(obj1); Py_DECREF(obj2); obj4 = PyArray_Diff((PyAO *)obj1, (PyAO *)obj2, NULL); Without the Py_INCREF/Py_DECREF pair, if sumdiff is called with arrays not bounded to any python variable, such as sumdiff(arange(10), ones(10)), PyArray_Sum can overwrite one of the inputs and make the result of PyArray_Diff wrong. With the Py_INCREF/Py_DECREF pair, PyArray_Sum cannot overwrite its inputs (while PyArray_Diff can). Moving the Py_INCREF/Py_DECREF inside the ufuncs makes the patch unuseful because no array could have ref_count 1. You are right, existing C-extension modules that use the Numpy/Numeric API, might give wrong results. Best, Luca From alan at ajackson.org Wed Jul 8 22:39:12 2009 From: alan at ajackson.org (alan at ajackson.org) Date: Wed, 8 Jul 2009 21:39:12 -0500 Subject: [Numpy-discussion] Bug in the F distribution? In-Reply-To: <20090706200227.045b3cc8@ajackson.org> References: <20090703183801.7ec5fb56@ajackson.org> <20090703212142.6a38541f@ajackson.org> <1cd32cbb0907061438q349735d6hedd1fcb7b80be2f6@mail.gmail.com> <20090706200227.045b3cc8@ajackson.org> Message-ID: <20090708213912.29dcad6a@ajackson.org> > >>On Fri, Jul 3, 2009 at 10:21 PM, Alan Jackson wrote: >>> >> >>I don't see any problem here. If you can replicate your results, we >>would need more information about the versions. >> >>Josef >> >>''' >>>>> np.version.version >>'1.3.0' >>>>> scipy.version.version >>'0.8.0.dev5789' >>''' >> > >In [4]: numpy.version.version >Out[4]: '1.1.1' >In [7]: scipy.version.version >Out[7]: '0.6.0' > >Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49) > >Ubuntu Intrepid Ibex > >Looks like it has been fixed. I guess I'll try out the instructions for >upgrading... I compiled numpy 1.3.0 and it seems to work now. -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From magawake at gmail.com Thu Jul 9 01:12:16 2009 From: magawake at gmail.com (Mag Gam) Date: Thu, 9 Jul 2009 01:12:16 -0400 Subject: [Numpy-discussion] speeding up an array operation Message-ID: <1cbd6f830907082212x74193fd6t5bab749e9826fac@mail.gmail.com> Hey All I am reading thru a file and trying to store the values into another array, but instead of storing the values 1 by 1, I would like to store them in bulk sets for optimization purposes. Here is what I have, which does it 1x1: z={} #dictionary r=csv.reader(file) for i,row in enumerate(r): p="/MIT/"+row[1] if p not in z: z[p]=0: else: z[p]+=1 arr[p]['chem'][z[p]]=tuple(row) #this loads the array 1 x 1 I would like to avoid the 1x1 loading, instead I would like to bulk load the array. Lets say load up 5million lines into memory and then push into array. Any ideas on how to do that? TIA From scott.sinclair.za at gmail.com Thu Jul 9 01:39:51 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 9 Jul 2009 07:39:51 +0200 Subject: [Numpy-discussion] argwhere does not accept py list In-Reply-To: <9B91F05F-58D2-46B1-9B60-B010F9654A11@gmail.com> References: <6a17e9ee0907030344o14d6a394q381952358b076d0e@mail.gmail.com> <9457e7c80907040313y766990c0r969e1ea43821489f@mail.gmail.com> <3d375d730907071827j12a4d537j46b32c8cca66da24@mail.gmail.com> <6a17e9ee0907080018x5106610p30678e30d7e87f20@mail.gmail.com> <9B91F05F-58D2-46B1-9B60-B010F9654A11@gmail.com> Message-ID: <6a17e9ee0907082239h7f131dfdsf862484ed08b1deb@mail.gmail.com> > 2009/7/8 Pierre GM : > > On Jul 8, 2009, at 3:18 AM, Scott Sinclair wrote: > >>> 2009/7/8 Robert Kern : >>> 2009/7/4 St?fan van der Walt : >>>> Thanks, Scott. ?This should now be fixed in SVN. >>> >>> You should probably change that to asanyarray() before the masked >>> array crowd gets upset. :-) >> >> I hadn't thought about that, but I'm don't think it matters in this >> case. MaskedArray.nonzero() returns a tuple of ndarrays... > > Taking np.asarray instead of np.asanyarray loses the mask, and you end > up with the wrong result. Ouch. Obviously. Note to self - conduct extensive tests for unseasonal temperatures in hell before raising a KernError. Thanks for fixing St?fan. Cheers, Scott From josef.pktd at gmail.com Thu Jul 9 03:45:59 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Jul 2009 03:45:59 -0400 Subject: [Numpy-discussion] Bug in the F distribution? In-Reply-To: <20090708213912.29dcad6a@ajackson.org> References: <20090703183801.7ec5fb56@ajackson.org> <20090703212142.6a38541f@ajackson.org> <1cd32cbb0907061438q349735d6hedd1fcb7b80be2f6@mail.gmail.com> <20090706200227.045b3cc8@ajackson.org> <20090708213912.29dcad6a@ajackson.org> Message-ID: <1cd32cbb0907090045j4160fec4u7bb0b714bd625a6c@mail.gmail.com> On Wed, Jul 8, 2009 at 10:39 PM, wrote: >> >>>On Fri, Jul 3, 2009 at 10:21 PM, Alan Jackson wrote: >>>> >>> >>>I don't see any problem here. If you can replicate your results, we >>>would need more information about the versions. >>> >>>Josef >>> >>>''' >>>>>> np.version.version >>>'1.3.0' >>>>>> scipy.version.version >>>'0.8.0.dev5789' >>>''' >>> >> >>In [4]: numpy.version.version >>Out[4]: '1.1.1' >>In [7]: scipy.version.version >>Out[7]: '0.6.0' >> >>Python 2.5.2 (r252:60911, Oct ?5 2008, 19:24:49) >> >>Ubuntu Intrepid Ibex >> >>Looks like it has been fixed. I guess I'll try out the instructions for >>upgrading... > > I compiled numpy 1.3.0 and it seems to work now. > good, there were several bugfixes in numpy.random in the last year. Almost all distributions in numpy.random are tested as part of scipy.stats, although only for some parameter values, and all tests pass. Josef > > -- > ----------------------------------------------------------------------- > | Alan K. Jackson ? ? ? ? ? ?| To see a World in a Grain of Sand ? ? ?| > | alan at ajackson.org ? ? ? ? ?| And a Heaven in a Wild Flower, ? ? ? ? | > | www.ajackson.org ? ? ? ? ? | Hold Infinity in the palm of your hand | > | Houston, Texas ? ? ? ? ? ? | And Eternity in an hour. - Blake ? ? ? | > ----------------------------------------------------------------------- > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From matthieu.brucher at gmail.com Thu Jul 9 03:54:26 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 9 Jul 2009 09:54:26 +0200 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) In-Reply-To: References: <9457e7c80907081538x11f5e63m2c030743f7921e78@mail.gmail.com> Message-ID: 2009/7/9 Pauli Virtanen : > On 2009-07-08, St?fan van der Walt wrote: >> I know very little about cache optimality, so excuse the triviality of >> this question: Is it possible to design this loop optimally (taking >> into account certain build-time measurable parameters), or is it the >> kind of thing that can only be discovered by tuning at compile-time? >> ATNumPy... scary :-) > > I'm still kind of hoping that it's possible to make some minimal > assumptions about CPU caches in general, and have a rule that > decides a code path that is good enough, if not optimal. Unfortunately, this is not possible. We've been playing with blocking loops for a long time in finite difference schemes, and it is always compiler dependent (that is, the optimal size of the block is bandwidth dependent and even operation dependent). Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From matthieu.brucher at gmail.com Thu Jul 9 04:00:25 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 9 Jul 2009 10:00:25 +0200 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> Message-ID: 2009/7/9 Citi, Luca : > Hello > > The problem is not PyArray_Conjugate itself. > The problem is that whenever you call a function from the C side > and one of the inputs has ref_count 1, it can be overwritten. > This is not a problem from the python side because if the > ufunc sees a ref_count=1 it means that no python object is referencing to it. Does this also hold if you are using the Numpy API directly? Say I've decided to write some operation with the Numpy API, I will never have one of my arrays with ref_count == 1? Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From dagss at student.matnat.uio.no Thu Jul 9 04:19:33 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 9 Jul 2009 10:19:33 +0200 Subject: [Numpy-discussion] ndarray and the buffer protocol In-Reply-To: <20090708231545.GD2950@zita2.kokkinizita.net> References: <20090708231545.GD2950@zita2.kokkinizita.net> Message-ID: > Hello all, > > (resending for the Nth time, as the previous attempts > didn't make it to the list) > > I'm new to this list (and numpy is mostly new to me :-). > > Using python 2.6 and numpy 1.3. > > My plan is to write some C extensions that will perform > rather specialised processing on multichannel digital audio > data stored stored in numpy arrays. > > Sinc 2.6 python has a new buffer protocol, and referring > to this, the numpy website states rather fuzzily: > > Of course, numpy makes a good interoperation library to > support the buffer protocol. Numpy can both consume and > produce protocol-supporting objects. A numpy ndarray can > be constructed as a view of an object supporting the > buffer protocol, and a numpy ndarray supports buffer > object views of its data. > > I'd want the C code to be based on using the new buffer > protocol - everything else is marked as 'deprecated', also > in the numpy docs. > > But it remains quite unclear if numpy 1.3 does or does not > support this, and if yes, how. It does not support the new buffer protocol -- the stuff on the web page refers to the old buffer protocol. You might consider using Cython for at least smaller parts of your work -- it has support for the new buffer protocol as well as emulating it for NumPy arrays for any NumPy version. http://wiki.cython.org/enhancements/buffer As well as Cython/Includes/numpy.pxd within the Cython distribution. A short look at numpy.pxd will tell you how to write NumPy-specific access to get the same information that the buffer protocol would have given you, if you want to stay in C. Dag Sverre From dwf at cs.toronto.edu Thu Jul 9 04:41:14 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 9 Jul 2009 04:41:14 -0400 Subject: [Numpy-discussion] speeding up an array operation In-Reply-To: <1cbd6f830907082212x74193fd6t5bab749e9826fac@mail.gmail.com> References: <1cbd6f830907082212x74193fd6t5bab749e9826fac@mail.gmail.com> Message-ID: <6B070B64-D277-48F1-9F99-EFF112640240@cs.toronto.edu> On 9-Jul-09, at 1:12 AM, Mag Gam wrote: > Here is what I have, which does it 1x1: > > z={} #dictionary > r=csv.reader(file) > for i,row in enumerate(r): > p="/MIT/"+row[1] > > if p not in z: > z[p]=0: > else: > z[p]+=1 > > arr[p]['chem'][z[p]]=tuple(row) #this loads the array 1 x 1 > > > I would like to avoid the 1x1 loading, instead I would like to bulk > load the array. Lets say load up 5million lines into memory and then > push into array. Any ideas on how to do that? Depending on how big your data is, this looks like a job for e.g. numpy.loadtxt(), to give you one big array. Then sort the array on the second column, so that all the rows with the same 'p' appear one after the other. Then you can assign slices of this big array to be arr[p]['chem']. David From pav at iki.fi Thu Jul 9 04:41:51 2009 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 9 Jul 2009 08:41:51 +0000 (UTC) Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> Message-ID: Thu, 09 Jul 2009 10:00:25 +0200, Matthieu Brucher kirjoitti: > 2009/7/9 Citi, Luca : >> Hello >> >> The problem is not PyArray_Conjugate itself. The problem is that >> whenever you call a function from the C side and one of the inputs has >> ref_count 1, it can be overwritten. This is not a problem from the >> python side because if the ufunc sees a ref_count=1 it means that no >> python object is referencing to it. > > Does this also hold if you are using the Numpy API directly? Say I've > decided to write some operation with the Numpy API, I will never have > one of my arrays with ref_count == 1? Newly created arrays have a refcount of 1, so this is a problem for the C-API. What needs to be ensured in Numpy is that none of the C-API functions can cause unexpected modification of their input arguments. So one would have to go through the list of exported functions, and check that none of them is problematic. (We know already that at least PyArray_Conjugate is.) -- Pauli Virtanen From david at ar.media.kyoto-u.ac.jp Thu Jul 9 04:28:06 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 09 Jul 2009 17:28:06 +0900 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) In-Reply-To: References: <9457e7c80907081538x11f5e63m2c030743f7921e78@mail.gmail.com> Message-ID: <4A55AA16.50908@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > > Unfortunately, this is not possible. We've been playing with blocking > loops for a long time in finite difference schemes, and it is always > compiler dependent You mean CPU dependent, right ? I can't see how a reasonable optimizing compiler could make a big difference on cache effects ? @ Pauli: if (optionally) knowing a few cache info would help you, I could implement it. It should not be too difficult for most cases we care about, David From dwf at cs.toronto.edu Thu Jul 9 04:48:52 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 9 Jul 2009 04:48:52 -0400 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) In-Reply-To: References: Message-ID: On 8-Jul-09, at 6:16 PM, Pauli Virtanen wrote: > Just to tickle some interest, a "pathological" case before > optimization: > > In [1]: import numpy as np > In [2]: x = np.zeros((80000, 256)) > In [3]: %timeit x.sum(axis=0) > 10 loops, best of 3: 850 ms per loop > > After optimization: > > In [1]: import numpy as np > In [2]: x = np.zeros((80000, 256)) > In [3]: %timeit x.sum(axis=0) > 10 loops, best of 3: 78.5 ms per loop Not knowing a terrible lot about cache optimization, I have nothing to contribute but encouragement. :) Pauli, this is fantastic work! Just curious about regressions: have you tested on any non-x86 hardware? Being a frequent user of an older ppc machine I worry about such things (and plan to give your benchmark a try tomorrow on both ppc and ppc64 OS X). Cheers, David From matthieu.brucher at gmail.com Thu Jul 9 05:02:01 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 9 Jul 2009 11:02:01 +0200 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) In-Reply-To: <4A55AA16.50908@ar.media.kyoto-u.ac.jp> References: <9457e7c80907081538x11f5e63m2c030743f7921e78@mail.gmail.com> <4A55AA16.50908@ar.media.kyoto-u.ac.jp> Message-ID: 2009/7/9 David Cournapeau : > Matthieu Brucher wrote: >> >> Unfortunately, this is not possible. We've been playing with blocking >> loops for a long time in finite difference schemes, and it is always >> compiler dependent > > You mean CPU dependent, right ? I can't see how a reasonable optimizing > compiler could make a big difference on cache effects ? Yes, of course, CU dependent... > @ Pauli: if (optionally) knowing a few cache info would help you, I > could implement it. It should not be too difficult for most cases we > care about, > > David > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From lciti at essex.ac.uk Thu Jul 9 05:03:47 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Thu, 9 Jul 2009 10:03:47 +0100 Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk> Hello Pauli, excuse me if I insist, PyArray_Conjugate is not the problem. If when using the numpy API, it is accepted something like: ob1 = PyArray_CreateSomehowAnArray(....); obj2 = PyArray_DoSomethingWithArray(obj1,...); obj3 = PyArray_DoSomethingElseWithArray(obj1,...); Py_DECREF(obj1); then there is no way my patch is guaranteed to not break things. What happens with conjugate can happen with sin, round, sum, prod, ... The problem is that the philosophy of the patch is: "If I (function) have been called with an input with ref_count=1, that input is mine and I can do whatever I want with it. If you (caller) need it for later be sure you let me know by incrementing its ref_count." >From what I understand, this works fine from within numpy, by changing a few operations (like divmod and variance) but could break libraries using the numpy C API. I hope someone can find a way to have the cake and eat it too. Best, Luca From pav at iki.fi Thu Jul 9 05:07:45 2009 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 9 Jul 2009 09:07:45 +0000 (UTC) Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) References: <9457e7c80907081538x11f5e63m2c030743f7921e78@mail.gmail.com> Message-ID: Thu, 09 Jul 2009 09:54:26 +0200, Matthieu Brucher kirjoitti: > 2009/7/9 Pauli Virtanen : [clip] >> I'm still kind of hoping that it's possible to make some minimal >> assumptions about CPU caches in general, and have a rule that decides a >> code path that is good enough, if not optimal. > > Unfortunately, this is not possible. We've been playing with blocking > loops for a long time in finite difference schemes, and it is always > compiler dependent (that is, the optimal size of the block is bandwidth > dependent and even operation dependent). I'm not completely sure about this: the data access pattern in a reduce operation is in principle relatively simple, and the main focus would be in improving worst cases rather than being completely optimal. This could perhaps be achieved with a generic rule that tries to maximize data locality. Of course, I may be wrong here... -- Pauli Virtanen From pav at iki.fi Thu Jul 9 05:35:06 2009 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 9 Jul 2009 09:35:06 +0000 (UTC) Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC2@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk> Message-ID: Thu, 09 Jul 2009 10:03:47 +0100, Citi, Luca kirjoitti: [clip] > Excuse me if I insist, PyArray_Conjugate is not the problem. If when > using the numpy API, it is accepted something like: > > obj1 = PyArray_CreateSomehowAnArray(....); > obj2 = PyArray_DoSomethingWithArray(obj1,...); > obj3 = PyArray_DoSomethingElseWithArray(obj1,...); > Py_DECREF(obj1); > > then there is no way my patch is guaranteed to not break things. So it seems. PyUFunc_GenericFunction is a part of the public C-API, so I guess this discussion is moot. I doubt we can change ufunc semantics at this point, and I don't see ways in which one could distinguish between "temporary arrays" and refcount-1 arrays used in extension modules. -- Pauli Virtanen From magawake at gmail.com Thu Jul 9 07:14:20 2009 From: magawake at gmail.com (Mag Gam) Date: Thu, 9 Jul 2009 07:14:20 -0400 Subject: [Numpy-discussion] speeding up an array operation In-Reply-To: <6B070B64-D277-48F1-9F99-EFF112640240@cs.toronto.edu> References: <1cbd6f830907082212x74193fd6t5bab749e9826fac@mail.gmail.com> <6B070B64-D277-48F1-9F99-EFF112640240@cs.toronto.edu> Message-ID: <1cbd6f830907090414y304bc4f1v319a231f38e68c6f@mail.gmail.com> The problem is the array is very large. We are talking about 200+ million rows. On Thu, Jul 9, 2009 at 4:41 AM, David Warde-Farley wrote: > On 9-Jul-09, at 1:12 AM, Mag Gam wrote: > >> Here is what I have, which does it 1x1: >> >> z={} ?#dictionary >> r=csv.reader(file) >> for i,row in enumerate(r): >> ?p="/MIT/"+row[1] >> >> ?if p not in z: >> ? ?z[p]=0: >> ?else: >> ? ?z[p]+=1 >> >> ?arr[p]['chem'][z[p]]=tuple(row) #this loads the array 1 x 1 >> >> >> I would like to avoid the 1x1 loading, instead I would like to bulk >> load the array. Lets say load up 5million lines into memory and then >> push into array. Any ideas on how to do that? > > > Depending on how big your data is, this looks like a job for e.g. > numpy.loadtxt(), to give you one big array. > > Then sort the array on the second column, so that all the rows with > the same 'p' appear one after the other. Then you can assign slices of > this big array to be arr[p]['chem']. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From thrabe at googlemail.com Thu Jul 9 09:42:32 2009 From: thrabe at googlemail.com (Thomas Hrabe) Date: Thu, 9 Jul 2009 15:42:32 +0200 Subject: [Numpy-discussion] interpolation in numpy In-Reply-To: <4d06725d0907090624q60d7153fpe9aa6a79ea0a05c2@mail.gmail.com> References: <4d06725d0907090624q60d7153fpe9aa6a79ea0a05c2@mail.gmail.com> Message-ID: <4d06725d0907090642s4fc345e6i42591c7f0855dd3e@mail.gmail.com> Hi all, I am not a newbie to python and numpy, but however, I kinda do not find a proper solution for my interpolation problem without coding it explicitly myself. All I want to do is to increase the resolution of an tree dimensional array. I have a volume 'a' a = numpy.random.rand(3,3,3) now, I would like to expand a to a volume b of size say 10x10x10 I want b to have interpolated values from a. One could think of this procedure as "zooming" into the volume, similar for images or so. numpy.interp does such a thing for 1d, but is there such function for 3d, too? Thank you in advance for your help, Thomas FYI: I do not have scipy installed From lciti at essex.ac.uk Thu Jul 9 10:10:37 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Thu, 9 Jul 2009 15:10:37 +0100 Subject: [Numpy-discussion] interpolation in numpy References: <4d06725d0907090624q60d7153fpe9aa6a79ea0a05c2@mail.gmail.com> <4d06725d0907090642s4fc345e6i42591c7f0855dd3e@mail.gmail.com> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCC@sernt14.essex.ac.uk> Hi, you can have a look at the method interp2d of scipy.interpolate. I think it is what you are looking for. Best, Luca From zachary.pincus at yale.edu Thu Jul 9 10:22:11 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 9 Jul 2009 10:22:11 -0400 Subject: [Numpy-discussion] interpolation in numpy In-Reply-To: <4d06725d0907090642s4fc345e6i42591c7f0855dd3e@mail.gmail.com> References: <4d06725d0907090624q60d7153fpe9aa6a79ea0a05c2@mail.gmail.com> <4d06725d0907090642s4fc345e6i42591c7f0855dd3e@mail.gmail.com> Message-ID: You might want also to look into scipy.ndimage.zoom. Zach On Jul 9, 2009, at 9:42 AM, Thomas Hrabe wrote: > Hi all, > > I am not a newbie to python and numpy, but however, I kinda do not > find a proper solution for my interpolation problem without coding it > explicitly myself. > > All I want to do is to increase the resolution of an tree > dimensional array. > > I have a volume 'a' > > a = numpy.random.rand(3,3,3) > > now, I would like to expand a to a volume b of size say 10x10x10 > > I want b to have interpolated values from a. One could think of this > procedure as "zooming" into the volume, similar for images or so. > > numpy.interp does such a thing for 1d, but is there such function > for 3d, too? > > Thank you in advance for your help, > > Thomas > > FYI: I do not have scipy installed > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From gael.varoquaux at normalesup.org Thu Jul 9 10:51:25 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 9 Jul 2009 16:51:25 +0200 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk> Message-ID: <20090709145125.GC18518@phare.normalesup.org> On Thu, Jul 09, 2009 at 09:35:06AM +0000, Pauli Virtanen wrote: > Thu, 09 Jul 2009 10:03:47 +0100, Citi, Luca kirjoitti: > [clip] > > Excuse me if I insist, PyArray_Conjugate is not the problem. If when > > using the numpy API, it is accepted something like: > > obj1 = PyArray_CreateSomehowAnArray(....); > > obj2 = PyArray_DoSomethingWithArray(obj1,...); > > obj3 = PyArray_DoSomethingElseWithArray(obj1,...); > > Py_DECREF(obj1); > > then there is no way my patch is guaranteed to not break things. > So it seems. > PyUFunc_GenericFunction is a part of the public C-API, so I guess this > discussion is moot. I doubt we can change ufunc semantics at this point, > and I don't see ways in which one could distinguish between "temporary > arrays" and refcount-1 arrays used in extension modules. OK, so from what I understand, the proposed change what induce a major change on the C API. So it seems that we can't just merge it in. However, I really like the idea: I think it is a simple and clever optimisation, that is transparent to the user. Do you think it is possible to keep it around, think about it, and merge it in later, eg in numpy 2.0? Ga?l From jkington at wisc.edu Thu Jul 9 10:52:40 2009 From: jkington at wisc.edu (Joe Kington) Date: Thu, 9 Jul 2009 09:52:40 -0500 Subject: [Numpy-discussion] interpolation in numpy In-Reply-To: <4d06725d0907090642s4fc345e6i42591c7f0855dd3e@mail.gmail.com> References: <4d06725d0907090624q60d7153fpe9aa6a79ea0a05c2@mail.gmail.com> <4d06725d0907090642s4fc345e6i42591c7f0855dd3e@mail.gmail.com> Message-ID: scipy.ndimage.zoom is exactly what you're looking for, as Zach Pincus already said. As far as I know, numpy doesn't have any 3D interpolation routines, so you'll have to install scipy. Interp2d will only interpolate slices of your data, not the whole volume. -Joe On Thu, Jul 9, 2009 at 8:42 AM, Thomas Hrabe wrote: > Hi all, > > I am not a newbie to python and numpy, but however, I kinda do not > find a proper solution for my interpolation problem without coding it > explicitly myself. > > All I want to do is to increase the resolution of an tree dimensional > array. > > I have a volume 'a' > > a = numpy.random.rand(3,3,3) > > now, I would like to expand a to a volume b of size say 10x10x10 > > I want b to have interpolated values from a. One could think of this > procedure as "zooming" into the volume, similar for images or so. > > numpy.interp does such a thing for 1d, but is there such function for 3d, > too? > > Thank you in advance for your help, > > Thomas > > FYI: I do not have scipy installed > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thrabe at googlemail.com Thu Jul 9 10:57:24 2009 From: thrabe at googlemail.com (Thomas Hrabe) Date: Thu, 9 Jul 2009 16:57:24 +0200 Subject: [Numpy-discussion] interpolation in numpy In-Reply-To: References: <4d06725d0907090624q60d7153fpe9aa6a79ea0a05c2@mail.gmail.com> <4d06725d0907090642s4fc345e6i42591c7f0855dd3e@mail.gmail.com> Message-ID: <4d06725d0907090757w340cb6bfw94db781a962b98a8@mail.gmail.com> Yep, thats the one. Unfortunately, installing scipy makes my project dependant on another package. However, I installed it and it works... Thank you 2009/7/9 Joe Kington : > scipy.ndimage.zoom is exactly what you're looking for, as Zach Pincus > already said. > > As far as I know, numpy doesn't have any 3D interpolation routines, so > you'll have to install scipy. Interp2d will only interpolate slices of your > data, not the whole volume. > > -Joe > > On Thu, Jul 9, 2009 at 8:42 AM, Thomas Hrabe wrote: >> >> Hi all, >> >> I am not a newbie to python and numpy, but however, I kinda do not >> find a proper solution for my interpolation problem without coding it >> explicitly myself. >> >> All I want to do is to increase the resolution of an tree dimensional >> array. >> >> I have a volume 'a' >> >> a = numpy.random.rand(3,3,3) >> >> now, I would like to expand a to a volume b of size say 10x10x10 >> >> I want b to have interpolated values from a. One could think of this >> procedure as "zooming" into the volume, similar for images or so. >> >> numpy.interp does such a thing for 1d, but is there such function for 3d, >> too? >> >> Thank you in advance for your help, >> >> Thomas >> >> FYI: I do not have scipy installed >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From lciti at essex.ac.uk Thu Jul 9 11:30:11 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Thu, 9 Jul 2009 16:30:11 +0100 Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk> <20090709145125.GC18518@phare.normalesup.org> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCE@sernt14.essex.ac.uk> Hello Ga?l, I think it might be an option. Also one could have an internal flag which says whether or not is safe to overwrite inputs with ref_count=1. Then import_array() sets this flag to "unsafe" (i.e. current behaviour). If the user of the numpy C-api is aware of how the new feature works, he/she can enable it by switching the flag to "safe" and act accordingly (increase refcounts before / decrease after) whenever he/she needs an array for later reuse. If possible, when imported from python (is there a way to know it? is import_array() called anyway?) the flag could be set to "safe". Best, Luca From robert.kern at gmail.com Thu Jul 9 11:34:00 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Jul 2009 10:34:00 -0500 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCE@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC3@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk> <20090709145125.GC18518@phare.normalesup.org> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCE@sernt14.essex.ac.uk> Message-ID: <3d375d730907090834n24f33615yda3b2ec9c58e628a@mail.gmail.com> On Thu, Jul 9, 2009 at 10:30, Citi, Luca wrote: > Hello Ga?l, > > I think it might be an option. > > Also one could have an internal flag which says whether or not is safe > to overwrite inputs with ref_count=1. > Then import_array() sets this flag to "unsafe" (i.e. current behaviour). > If the user of the numpy C-api is aware of how the new feature works, > he/she can enable it by switching the flag to "safe" and act accordingly > (increase refcounts before / decrease after) whenever he/she needs an array > for later reuse. > If possible, when imported from python (is there a way to know it? is > import_array() called anyway?) the flag could be set to "safe". I'm not sure how best to explain why, but suffice it to say that this will not work. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Jul 9 11:41:38 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Jul 2009 10:41:38 -0500 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk> Message-ID: <3d375d730907090841t4be70ff1v8049daab59186ed7@mail.gmail.com> On Thu, Jul 9, 2009 at 04:35, Pauli Virtanen wrote: > Thu, 09 Jul 2009 10:03:47 +0100, Citi, Luca kirjoitti: > [clip] >> Excuse me if I insist, PyArray_Conjugate is not the problem. If when >> using the numpy API, it is accepted something like: >> >> ? ? obj1 = PyArray_CreateSomehowAnArray(....); >> ? ? obj2 = PyArray_DoSomethingWithArray(obj1,...); >> ? ? obj3 = PyArray_DoSomethingElseWithArray(obj1,...); >> ? ? Py_DECREF(obj1); >> >> then there is no way my patch is guaranteed to not break things. > > So it seems. > > PyUFunc_GenericFunction is a part of the public C-API, so I guess this > discussion is moot. I doubt we can change ufunc semantics at this point, > and I don't see ways in which one could distinguish between "temporary > arrays" and refcount-1 arrays used in extension modules. We could change ufunc_generic_call() (i.e. the C implementation of ufunc.__call__) to use a new function like PyUFunc_GenericFunction except with the refcount-1 semantics. This allows old C-level to remain unchanged but let Python expressions have the optimized semantics. Is there a hole in my reasoning? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nadavh at visionsense.com Thu Jul 9 12:25:38 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 9 Jul 2009 19:25:38 +0300 Subject: [Numpy-discussion] Test: My replies do not apear on the list Message-ID: <710F2847B0018641891D9A21602763605AD0B2@ex3.envision.co.il> From gael.varoquaux at normalesup.org Thu Jul 9 12:30:52 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 9 Jul 2009 18:30:52 +0200 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3d375d730907090841t4be70ff1v8049daab59186ed7@mail.gmail.com> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk> <3d375d730907090841t4be70ff1v8049daab59186ed7@mail.gmail.com> Message-ID: <20090709163052.GG23505@phare.normalesup.org> On Thu, Jul 09, 2009 at 10:41:38AM -0500, Robert Kern wrote: > We could change ufunc_generic_call() (i.e. the C implementation of > ufunc.__call__) to use a new function like PyUFunc_GenericFunction > except with the refcount-1 semantics. This allows old C-level to > remain unchanged but let Python expressions have the optimized > semantics. > Is there a hole in my reasoning? I don't know the C part of numpy well (actually, I know it very badly), but it looks to me as your suggestion is so far the cleanest I have heard. Ga?l From gokhansever at gmail.com Thu Jul 9 12:31:48 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Thu, 9 Jul 2009 11:31:48 -0500 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <7f014ea60907071854y40bec379obcf7f59d7390d02e@mail.gmail.com> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> <9457e7c80907060318p532caf2aib6fc4f317023959a@mail.gmail.com> <7f014ea60907071854y40bec379obcf7f59d7390d02e@mail.gmail.com> Message-ID: <49d6b3500907090931x6b7b37f7kff868c3380772aa3@mail.gmail.com> Still speaking of particles, has anyone seen Nvidia's Cuda powered particle systme demo? http://www.youtube.com/watch?v=RqduA7myZok I have a CUDA supported graphics card on my laptop, and tested on the actual hardware. Seems very cool. However has way too many lines of code for that piece of demo. I don't would the power of CUDA be harnessed using Python as well? -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Thu Jul 9 12:32:30 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 9 Jul 2009 19:32:30 +0300 Subject: [Numpy-discussion] interpolation in numpy References: <4d06725d0907090624q60d7153fpe9aa6a79ea0a05c2@mail.gmail.com> <4d06725d0907090642s4fc345e6i42591c7f0855dd3e@mail.gmail.com> <4d06725d0907090757w340cb6bfw94db781a962b98a8@mail.gmail.com> Message-ID: <710F2847B0018641891D9A21602763605AD0B3@ex3.envision.co.il> The ndimage package can be accessed as numpy.numarray.nd_image. scipy is not needed Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Thomas Hrabe ????: ? 09-????-09 17:57 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] interpolation in numpy Yep, thats the one. Unfortunately, installing scipy makes my project dependant on another package. However, I installed it and it works... Thank you 2009/7/9 Joe Kington : > scipy.ndimage.zoom is exactly what you're looking for, as Zach Pincus > already said. > > As far as I know, numpy doesn't have any 3D interpolation routines, so > you'll have to install scipy. Interp2d will only interpolate slices of your > data, not the whole volume. > > -Joe > > On Thu, Jul 9, 2009 at 8:42 AM, Thomas Hrabe wrote: >> >> Hi all, >> >> I am not a newbie to python and numpy, but however, I kinda do not >> find a proper solution for my interpolation problem without coding it >> explicitly myself. >> >> All I want to do is to increase the resolution of an tree dimensional >> array. >> >> I have a volume 'a' >> >> a = numpy.random.rand(3,3,3) >> >> now, I would like to expand a to a volume b of size say 10x10x10 >> >> I want b to have interpolated values from a. One could think of this >> procedure as "zooming" into the volume, similar for images or so. >> >> numpy.interp does such a thing for 1d, but is there such function for 3d, >> too? >> >> Thank you in advance for your help, >> >> Thomas >> >> FYI: I do not have scipy installed >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Thu Jul 9 12:43:13 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Jul 2009 11:43:13 -0500 Subject: [Numpy-discussion] interpolation in numpy In-Reply-To: <710F2847B0018641891D9A21602763605AD0B3@ex3.envision.co.il> References: <4d06725d0907090624q60d7153fpe9aa6a79ea0a05c2@mail.gmail.com> <4d06725d0907090642s4fc345e6i42591c7f0855dd3e@mail.gmail.com> <4d06725d0907090757w340cb6bfw94db781a962b98a8@mail.gmail.com> <710F2847B0018641891D9A21602763605AD0B3@ex3.envision.co.il> Message-ID: <3d375d730907090943h2450c8a0odb6bcadbf1a55c88@mail.gmail.com> 2009/7/9 Nadav Horesh : > The ndimage package can be accessed as numpy.numarray.nd_image. scipy is not needed numpy.numarray.nd_image just imports from scipy.ndimage or a standalone ndimage if you have built it so. numpy does not contain the ndimage code. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fons at kokkinizita.net Thu Jul 9 12:44:58 2009 From: fons at kokkinizita.net (Fons Adriaensen) Date: Thu, 9 Jul 2009 18:44:58 +0200 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCE@sernt14.essex.ac.uk> References: <20090709145125.GC18518@phare.normalesup.org> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCE@sernt14.essex.ac.uk> Message-ID: <20090709164458.GB2944@zita2.kokkinizita.net> On Thu, Jul 09, 2009 at 04:30:11PM +0100, Citi, Luca wrote: > Also one could have an internal flag which says whether or not is safe > to overwrite inputs with ref_count=1. > Then import_array() sets this flag to "unsafe" (i.e. current behaviour). > If the user of the numpy C-api is aware of how the new feature works, > he/she can enable it by switching the flag to "safe" and act accordingly > (increase refcounts before / decrease after) whenever he/she needs an array > for later reuse. > If possible, when imported from python (is there a way to know it? is > import_array() called anyway?) the flag could be set to "safe". Why complicate things ? There is a simple rule which says that if you use an object pointer as a function argument you must INCREF it. This is just the logical consequence of using refcounted objects. Creating exceptions to this will not simplify life, it complicates the rules and will just lead to more bugs and programmer frustration. Ciao, -- FA Io lo dico sempre: l'Italia ? troppo stretta e lunga. From sccolbert at gmail.com Thu Jul 9 12:54:50 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 9 Jul 2009 12:54:50 -0400 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <49d6b3500907090931x6b7b37f7kff868c3380772aa3@mail.gmail.com> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> <9457e7c80907060318p532caf2aib6fc4f317023959a@mail.gmail.com> <7f014ea60907071854y40bec379obcf7f59d7390d02e@mail.gmail.com> <49d6b3500907090931x6b7b37f7kff868c3380772aa3@mail.gmail.com> Message-ID: <7f014ea60907090954g5839ecacub9db5295503446e6@mail.gmail.com> there's a wrapper called PyCuda. but you actually write cuda code as a docstring and its compiled and exectuted at run time. I think it can be done more pythonic. On Thu, Jul 9, 2009 at 12:31 PM, G?khan SEVER wrote: > Still speaking of particles, has anyone seen Nvidia's Cuda powered particle > systme demo? > > http://www.youtube.com/watch?v=RqduA7myZok > > I have a CUDA supported graphics card on my laptop, and tested on the actual > hardware. Seems very cool. However has way too many lines of code for that > piece of demo. > > I don't would the power of CUDA be harnessed using Python as well? > > -- > G?khan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From robert.kern at gmail.com Thu Jul 9 13:00:23 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Jul 2009 12:00:23 -0500 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <20090709164458.GB2944@zita2.kokkinizita.net> References: <20090709145125.GC18518@phare.normalesup.org> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCE@sernt14.essex.ac.uk> <20090709164458.GB2944@zita2.kokkinizita.net> Message-ID: <3d375d730907091000y3609548dh8e22c5269353cc19@mail.gmail.com> On Thu, Jul 9, 2009 at 11:44, Fons Adriaensen wrote: > On Thu, Jul 09, 2009 at 04:30:11PM +0100, Citi, Luca wrote: > >> Also one could have an internal flag which says whether or not is safe >> to overwrite inputs with ref_count=1. >> Then import_array() sets this flag to "unsafe" (i.e. current behaviour). >> If the user of the numpy C-api is aware of how the new feature works, >> he/she can enable it by switching the flag to "safe" and act accordingly >> (increase refcounts before / decrease after) whenever he/she needs an array >> for later reuse. >> If possible, when imported from python (is there a way to know it? is >> import_array() called anyway?) the flag could be set to "safe". > > Why complicate things ? > > There is a simple rule which says that if you use an object > pointer as a function argument you must INCREF it. This is > just the logical consequence of using refcounted objects. That's not true. There are many functions even in the standard Python C API that "borrow" a reference. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ijstokes at crystal.harvard.edu Thu Jul 9 13:16:26 2009 From: ijstokes at crystal.harvard.edu (Ian Stokes-Rees) Date: Thu, 09 Jul 2009 13:16:26 -0400 Subject: [Numpy-discussion] Test: My replies do not apear on the list In-Reply-To: <710F2847B0018641891D9A21602763605AD0B2@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD0B2@ex3.envision.co.il> Message-ID: <4A5625EA.9090000@crystal.harvard.edu> This showed up on numpy-discussion. I posted this morning and it hasn't appeared yet. Ian Nadav Horesh wrote: > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Ian Stokes-Rees W: http://sbgrid.org ijstokes at crystal.harvard.edu T: +1 617 432-5608 x75 SBGrid, Harvard Medical School F: +1 617 432-5600 From ijstokes at crystal.harvard.edu Thu Jul 9 13:18:44 2009 From: ijstokes at crystal.harvard.edu (Ian Stokes-Rees) Date: Thu, 09 Jul 2009 13:18:44 -0400 Subject: [Numpy-discussion] Selecting "column" from array of tuples Message-ID: <4A562674.5000701@crystal.harvard.edu> [take 6 on sending this -- I'm subscribed to numpy-discuss, but this post refuses to show up] I have an large array consisting of 5-tuples. I'd like to select the first and second columns in order to produce a scatter plot. Each tuple consists of mixed types (floats and strings). The Matlab equivalent would be: data[:,1:2] 1. I cannot figure out how to specify "column" selection from an array of tuples; or 2. I cannot figure out how to use an array of lists instead of an array of tuples. Some code should illustrate this. What works: dtype = [("score", "f4"), ("rfac", "f4"), ("codefull", "a10"), ("code2", "a2"), ("subset","a4")] results = zeros((len(lines),), dtype=dtype) idx = 0 for line in lines: parts = line.split() codefull = parts[0] code2 = codefull[1:3] rfac = float(parts[12]) score = float(parts[13]) subset = parts[14] results[idx] = (score, rfac, codefull, code2, subset) idx += 1 What does not work: results = zeros((len(lines),len(dtype)), dtype=dtype) ... results[idx] = [score, rfac, codefull, code2, subset] or indexing into the array: results[:,0] results[:,1] results[:].take(0) results[:][0] # this works, but doesn't return the desired first column Any suggestions greatly appreciated. Ian -- Ian Stokes-Rees W: http://sbgrid.org ijstokes at crystal.harvard.edu T: +1 617 432-5608 x75 SBGrid, Harvard Medical School F: +1 617 432-5600 From lciti at essex.ac.uk Thu Jul 9 13:30:33 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Thu, 9 Jul 2009 18:30:33 +0100 Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk><3d375d730907090841t4be70ff1v8049daab59186ed7@mail.gmail.com> <20090709163052.GG23505@phare.normalesup.org> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCF@sernt14.essex.ac.uk> Let me see if I understand correctly... what you suggest is something like: 1) adding an argument flag to construct_arrays that enables/disables the feature 2) adding the same argument flag to construct_loop which is passed untouched to construct_arrays 3) set the flag to "disable" in the construct_loop call inside PyUFunc_GenericFunction 4) write an exact copy of PyUFunc_GenericFunction with the flag "enabled" 5) in ufunc_generic_call, call the latter instead of PyUFunc_GenericFunction Am I correct? Sounds doable to me as long as ufunc.__call__ is not directly or indirectly in the numpy C API. From mark.wendell at gmail.com Thu Jul 9 15:19:36 2009 From: mark.wendell at gmail.com (Mark Wendell) Date: Thu, 9 Jul 2009 13:19:36 -0600 Subject: [Numpy-discussion] Selecting "column" from array of tuples In-Reply-To: <4A562674.5000701@crystal.harvard.edu> References: <4A562674.5000701@crystal.harvard.edu> Message-ID: Make sure you send the message as plain text, rather than with rich formatting, and it should show up right away. Mark On Thu, Jul 9, 2009 at 11:18 AM, Ian Stokes-Rees wrote: > > [take 6 on sending this -- I'm subscribed to numpy-discuss, but this > post refuses to show up] > > From pav+sp at iki.fi Thu Jul 9 16:17:44 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Thu, 9 Jul 2009 20:17:44 +0000 (UTC) Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk> <3d375d730907090841t4be70ff1v8049daab59186ed7@mail.gmail.com> <20090709163052.GG23505@phare.normalesup.org> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCF@sernt14.essex.ac.uk> Message-ID: On 2009-07-09, Citi, Luca wrote: > Let me see if I understand correctly... > what you suggest is something like: > 1) adding an argument flag to construct_arrays > that enables/disables the feature > 2) adding the same argument flag to construct_loop which > is passed untouched to construct_arrays > 3) set the flag to "disable" in the construct_loop call inside > PyUFunc_GenericFunction > 4) write an exact copy of PyUFunc_GenericFunction with the flag > "enabled" > 5) in ufunc_generic_call, call the latter instead of > PyUFunc_GenericFunction > Am I correct? > > Sounds doable to me as long as ufunc.__call__ is not > directly or indirectly [used] in the numpy C API. It's conceivable that someone would call PyObject_Call(ufunc, args, kw) from C code. But this is already a bit contrived, given that PyUFunc_GenericFunction is there, and the args tuple probably usually has new references of the contained objects. [CPython itself creates a new tuple when doing a = (np.zeros((4,)), np.zeros((5,))) np.add(*a) so that also in this case the refcounts of the two arrays in the tuple are > 1.] -- Pauli Virtanen From fons at kokkinizita.net Thu Jul 9 16:20:44 2009 From: fons at kokkinizita.net (Fons Adriaensen) Date: Thu, 9 Jul 2009 22:20:44 +0200 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <3d375d730907091000y3609548dh8e22c5269353cc19@mail.gmail.com> References: <20090709145125.GC18518@phare.normalesup.org> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCE@sernt14.essex.ac.uk> <20090709164458.GB2944@zita2.kokkinizita.net> <3d375d730907091000y3609548dh8e22c5269353cc19@mail.gmail.com> Message-ID: <20090709202044.GC2944@zita2.kokkinizita.net> On Thu, Jul 09, 2009 at 12:00:23PM -0500, Robert Kern wrote: > On Thu, Jul 9, 2009 at 11:44, Fons Adriaensen wrote: > > > There is a simple rule which says that if you use an object > > pointer as a function argument you must INCREF it. This is > > just the logical consequence of using refcounted objects. > > That's not true. There are many functions even in the standard Python > C API that "borrow" a reference. Indeed. Still, when calling a function Python will INCREF the arguments. And that's only logical since these arguments become local variables (i.e. a new reference) inside the called function and that function can do whatever it wants with them, including deleting or re-assigning them. Creating a function argument is not different from assigning to a new variable. This is also the reason why sys.getrefcount() will return a value that is one higher than the one in the caller's context. *In principle* you need to do the same manually when using any PyObject* as a function argument in C code. Of course if you know that the called function will do no harm, you can call it without an INCREF. In that case you allow the function to use a borrowed reference. If you don't know what the function is going to do, the only safe way to call it is to INCREF the arguments. Ciao, -- FA Io lo dico sempre: l'Italia ? troppo stretta e lunga. From dagss at student.matnat.uio.no Thu Jul 9 16:26:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 9 Jul 2009 22:26:29 +0200 Subject: [Numpy-discussion] ndarray and the buffer protocol In-Reply-To: References: <20090708231545.GD2950@zita2.kokkinizita.net> Message-ID: <0795558612d8f111e79c17a5fd6147bd.squirrel@webmail.uio.no> >> Hello all, >> >> (resending for the Nth time, as the previous attempts >> didn't make it to the list) >> >> I'm new to this list (and numpy is mostly new to me :-). >> >> Using python 2.6 and numpy 1.3. >> >> My plan is to write some C extensions that will perform >> rather specialised processing on multichannel digital audio >> data stored stored in numpy arrays. >> >> Sinc 2.6 python has a new buffer protocol, and referring >> to this, the numpy website states rather fuzzily: >> >> Of course, numpy makes a good interoperation library to >> support the buffer protocol. Numpy can both consume and >> produce protocol-supporting objects. A numpy ndarray can >> be constructed as a view of an object supporting the >> buffer protocol, and a numpy ndarray supports buffer >> object views of its data. >> >> I'd want the C code to be based on using the new buffer >> protocol - everything else is marked as 'deprecated', also >> in the numpy docs. >> >> But it remains quite unclear if numpy 1.3 does or does not >> support this, and if yes, how. > > It does not support the new buffer protocol -- the stuff on the web page > refers to the old buffer protocol. > > You might consider using Cython for at least smaller parts of your work -- > it has support for the new buffer protocol as well as emulating it for > NumPy arrays for any NumPy version. > > http://wiki.cython.org/enhancements/buffer > > As well as Cython/Includes/numpy.pxd within the Cython distribution. > > A short look at numpy.pxd will tell you how to write NumPy-specific access > to get the same information that the buffer protocol would have given you, > if you want to stay in C. Looking closer at the webpage I must say I'm really puzzled too now -- it says rather unambigiously that it should work; but at least I can't get it to work either (under Py2.6). Somebody closer to it than me should really comment... Dag Sverre From robert.kern at gmail.com Thu Jul 9 16:30:00 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Jul 2009 15:30:00 -0500 Subject: [Numpy-discussion] performing operations in-place in numpy In-Reply-To: <20090709202044.GC2944@zita2.kokkinizita.net> References: <20090709145125.GC18518@phare.normalesup.org> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCE@sernt14.essex.ac.uk> <20090709164458.GB2944@zita2.kokkinizita.net> <3d375d730907091000y3609548dh8e22c5269353cc19@mail.gmail.com> <20090709202044.GC2944@zita2.kokkinizita.net> Message-ID: <3d375d730907091330i3e93a5b5jd13cc2f2834aaf47@mail.gmail.com> On Thu, Jul 9, 2009 at 15:20, Fons Adriaensen wrote: > On Thu, Jul 09, 2009 at 12:00:23PM -0500, Robert Kern wrote: > >> On Thu, Jul 9, 2009 at 11:44, Fons Adriaensen wrote: >> >> > There is a simple rule which says that if you use an object >> > pointer as a function argument you must INCREF it. This is >> > just the logical consequence of using refcounted objects. >> >> That's not true. There are many functions even in the standard Python >> C API that "borrow" a reference. > > Indeed. Still, when calling a function Python will > INCREF the arguments. And that's only logical since > these arguments become local variables (i.e. a new > reference) inside the called function and that function > can do whatever it wants with them, including deleting > or re-assigning them. Creating a function argument is > not different from assigning to a new variable. > This is also the reason why sys.getrefcount() will return > a value that is one higher than the one in the caller's > context. That's not quite true. Forming the argument tuple increments the count, not the call itself. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lciti at essex.ac.uk Thu Jul 9 21:35:16 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Fri, 10 Jul 2009 02:35:16 +0100 Subject: [Numpy-discussion] performing operations in-place in numpy References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC4@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DC8@sernt14.essex.ac.uk><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCA@sernt14.essex.ac.uk><3d375d730907090841t4be70ff1v8049daab59186ed7@mail.gmail.com><20090709163052.GG23505@phare.normalesup.org><3DA3B328CBC48B4EBB88484B8A5EA19106AF9DCF@sernt14.essex.ac.uk> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DD2@sernt14.essex.ac.uk> I tried to implement what you suggest. The patch is in the ticket page. From sccolbert at gmail.com Thu Jul 9 22:08:56 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 9 Jul 2009 22:08:56 -0400 Subject: [Numpy-discussion] transform an array of points efficiently? Message-ID: <7f014ea60907091908w67fc2a84q691216ac5df1d7ac@mail.gmail.com> say i have an Nx4 array of points and I want to dot every [n, :] 1x4 slice with a 4x4 matrix. Currently I am using apply_along_axis in the following manner: def func(slice, mat): return np.dot(mat, slice) np.apply_along_axis(func, arr, 1, mat) Is there a more efficient way of doing this that doesn't require a python function for each slice? Cheers, Chris From kwgoodman at gmail.com Thu Jul 9 22:22:40 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 9 Jul 2009 19:22:40 -0700 Subject: [Numpy-discussion] transform an array of points efficiently? In-Reply-To: <7f014ea60907091908w67fc2a84q691216ac5df1d7ac@mail.gmail.com> References: <7f014ea60907091908w67fc2a84q691216ac5df1d7ac@mail.gmail.com> Message-ID: On Thu, Jul 9, 2009 at 7:08 PM, Chris Colbert wrote: > say i have an Nx4 array of points and I want to dot every [n, :] 1x4 > slice with a 4x4 matrix. > > Currently I am using apply_along_axis in the following manner: > > def func(slice, mat): > ? ? return np.dot(mat, slice) > > np.apply_along_axis(func, arr, 1, mat) > > Is there a more efficient way of doing this that doesn't require a > python function for each slice? I'm sure I'm missing an important point, but can't you solve the whole problem with one dot: >> x = np.random.rand(3,4) >> y = np.random.rand(4,4) >> np.dot(x, y) array([[ 0.86488057, 0.23456114, 0.91592677, 0.89798689], [ 1.24197754, 0.39907686, 1.45453141, 1.13645076], [ 1.41419289, 0.81818818, 1.09768428, 1.32719635]]) >> np.dot(x[0,:], y) array([ 0.86488057, 0.23456114, 0.91592677, 0.89798689]) >> np.dot(x[1,:], y) array([ 1.24197754, 0.39907686, 1.45453141, 1.13645076]) >> np.dot(x[2,:], y) array([ 1.41419289, 0.81818818, 1.09768428, 1.32719635]) From sccolbert at gmail.com Thu Jul 9 22:36:58 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 9 Jul 2009 22:36:58 -0400 Subject: [Numpy-discussion] transform an array of points efficiently? In-Reply-To: References: <7f014ea60907091908w67fc2a84q691216ac5df1d7ac@mail.gmail.com> Message-ID: <7f014ea60907091936y10eb9b95we2ebab965f099c67@mail.gmail.com> no, because dot(x,y) != dot(y,x) >>> x = np.random.rand(3,4) >>> y = np.random.rand(4,4) >>> np.dot(x, y) array([[ 1.67624043, 1.66719374, 1.72465017, 1.20372021], [ 0.70046162, 0.60187869, 0.73094349, 0.4604766 ], [ 0.78707401, 1.01959666, 0.61617829, 0.43147398]]) >>> np.dot(y, x[0,: ... ]) array([ 1.44627767, 1.09332339, 2.66001046, 1.13972652]) >>> np.dot(y, x[1,:]) array([ 0.27854715, 0.56261048, 0.7793413 , 0.44260709]) >>> np.dot(y, x[2,:]) array([ 0.70468211, 0.42843143, 1.34022702, 0.53021987]) >>> hence I need xnew = [Transform]*[xold] and not [xold]*[Transform] On Thu, Jul 9, 2009 at 10:22 PM, Keith Goodman wrote: > On Thu, Jul 9, 2009 at 7:08 PM, Chris Colbert wrote: >> say i have an Nx4 array of points and I want to dot every [n, :] 1x4 >> slice with a 4x4 matrix. >> >> Currently I am using apply_along_axis in the following manner: >> >> def func(slice, mat): >> ? ? return np.dot(mat, slice) >> >> np.apply_along_axis(func, arr, 1, mat) >> >> Is there a more efficient way of doing this that doesn't require a >> python function for each slice? > > I'm sure I'm missing an important point, but can't you solve the whole > problem with one dot: > >>> x = np.random.rand(3,4) >>> y = np.random.rand(4,4) > >>> np.dot(x, y) > > array([[ 0.86488057, ?0.23456114, ?0.91592677, ?0.89798689], > ? ? ? [ 1.24197754, ?0.39907686, ?1.45453141, ?1.13645076], > ? ? ? [ 1.41419289, ?0.81818818, ?1.09768428, ?1.32719635]]) > >>> np.dot(x[0,:], y) > ? array([ 0.86488057, ?0.23456114, ?0.91592677, ?0.89798689]) >>> np.dot(x[1,:], y) > ? array([ 1.24197754, ?0.39907686, ?1.45453141, ?1.13645076]) >>> np.dot(x[2,:], y) > ? array([ 1.41419289, ?0.81818818, ?1.09768428, ?1.32719635]) > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwmsmith at gmail.com Thu Jul 9 23:45:08 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Thu, 9 Jul 2009 22:45:08 -0500 Subject: [Numpy-discussion] transform an array of points efficiently? In-Reply-To: <7f014ea60907091936y10eb9b95we2ebab965f099c67@mail.gmail.com> References: <7f014ea60907091908w67fc2a84q691216ac5df1d7ac@mail.gmail.com> <7f014ea60907091936y10eb9b95we2ebab965f099c67@mail.gmail.com> Message-ID: On Thu, Jul 9, 2009 at 9:36 PM, Chris Colbert wrote: > no, because dot(x,y) != dot(y,x) > Try this, then: In [24]: x = np.random.rand(10,4) In [25]: y = np.random.rand(4,4) In [26]: result = np.dot(y,x.T).T In [39]: for i, res in enumerate(result): ....: assert np.allclose(res, np.dot(y,x[i,:])) ....: ....: In [40]: >>>> x = np.random.rand(3,4) >>>> y = np.random.rand(4,4) >>>> np.dot(x, y) > array([[ 1.67624043, ?1.66719374, ?1.72465017, ?1.20372021], > ? ? ? [ 0.70046162, ?0.60187869, ?0.73094349, ?0.4604766 ], > ? ? ? [ 0.78707401, ?1.01959666, ?0.61617829, ?0.43147398]]) >>>> np.dot(y, x[0,: > ... ]) > array([ 1.44627767, ?1.09332339, ?2.66001046, ?1.13972652]) >>>> np.dot(y, x[1,:]) > array([ 0.27854715, ?0.56261048, ?0.7793413 , ?0.44260709]) >>>> np.dot(y, x[2,:]) > array([ 0.70468211, ?0.42843143, ?1.34022702, ?0.53021987]) >>>> > > > hence I need xnew = [Transform]*[xold] > > and not [xold]*[Transform] > > > > > On Thu, Jul 9, 2009 at 10:22 PM, Keith Goodman wrote: >> On Thu, Jul 9, 2009 at 7:08 PM, Chris Colbert wrote: >>> say i have an Nx4 array of points and I want to dot every [n, :] 1x4 >>> slice with a 4x4 matrix. >>> >>> Currently I am using apply_along_axis in the following manner: >>> >>> def func(slice, mat): >>> ? ? return np.dot(mat, slice) >>> >>> np.apply_along_axis(func, arr, 1, mat) >>> >>> Is there a more efficient way of doing this that doesn't require a >>> python function for each slice? >> >> I'm sure I'm missing an important point, but can't you solve the whole >> problem with one dot: >> >>>> x = np.random.rand(3,4) >>>> y = np.random.rand(4,4) >> >>>> np.dot(x, y) >> >> array([[ 0.86488057, ?0.23456114, ?0.91592677, ?0.89798689], >> ? ? ? [ 1.24197754, ?0.39907686, ?1.45453141, ?1.13645076], >> ? ? ? [ 1.41419289, ?0.81818818, ?1.09768428, ?1.32719635]]) >> >>>> np.dot(x[0,:], y) >> ? array([ 0.86488057, ?0.23456114, ?0.91592677, ?0.89798689]) >>>> np.dot(x[1,:], y) >> ? array([ 1.24197754, ?0.39907686, ?1.45453141, ?1.13645076]) >>>> np.dot(x[2,:], y) >> ? array([ 1.41419289, ?0.81818818, ?1.09768428, ?1.32719635]) >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sccolbert at gmail.com Thu Jul 9 23:51:24 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 9 Jul 2009 23:51:24 -0400 Subject: [Numpy-discussion] transform an array of points efficiently? In-Reply-To: References: <7f014ea60907091908w67fc2a84q691216ac5df1d7ac@mail.gmail.com> <7f014ea60907091936y10eb9b95we2ebab965f099c67@mail.gmail.com> Message-ID: <7f014ea60907092051n4bc3b8bcn44b88748a52a07df@mail.gmail.com> hey, great man! thanks! I had thought that it may have been possible with a single dot, but how to do it escaped me. Thanks again! Chris On Thu, Jul 9, 2009 at 11:45 PM, Kurt Smith wrote: > On Thu, Jul 9, 2009 at 9:36 PM, Chris Colbert wrote: >> no, because dot(x,y) != dot(y,x) >> > > Try this, then: > > In [24]: x = np.random.rand(10,4) > > In [25]: y = np.random.rand(4,4) > > In [26]: result = np.dot(y,x.T).T > > In [39]: for i, res in enumerate(result): > ? ....: ? ? assert np.allclose(res, np.dot(y,x[i,:])) > ? ....: > ? ....: > > In [40]: > > >>>>> x = np.random.rand(3,4) >>>>> y = np.random.rand(4,4) >>>>> np.dot(x, y) >> array([[ 1.67624043, ?1.66719374, ?1.72465017, ?1.20372021], >> ? ? ? [ 0.70046162, ?0.60187869, ?0.73094349, ?0.4604766 ], >> ? ? ? [ 0.78707401, ?1.01959666, ?0.61617829, ?0.43147398]]) >>>>> np.dot(y, x[0,: >> ... ]) >> array([ 1.44627767, ?1.09332339, ?2.66001046, ?1.13972652]) >>>>> np.dot(y, x[1,:]) >> array([ 0.27854715, ?0.56261048, ?0.7793413 , ?0.44260709]) >>>>> np.dot(y, x[2,:]) >> array([ 0.70468211, ?0.42843143, ?1.34022702, ?0.53021987]) >>>>> >> >> >> hence I need xnew = [Transform]*[xold] >> >> and not [xold]*[Transform] >> >> >> >> >> On Thu, Jul 9, 2009 at 10:22 PM, Keith Goodman wrote: >>> On Thu, Jul 9, 2009 at 7:08 PM, Chris Colbert wrote: >>>> say i have an Nx4 array of points and I want to dot every [n, :] 1x4 >>>> slice with a 4x4 matrix. >>>> >>>> Currently I am using apply_along_axis in the following manner: >>>> >>>> def func(slice, mat): >>>> ? ? return np.dot(mat, slice) >>>> >>>> np.apply_along_axis(func, arr, 1, mat) >>>> >>>> Is there a more efficient way of doing this that doesn't require a >>>> python function for each slice? >>> >>> I'm sure I'm missing an important point, but can't you solve the whole >>> problem with one dot: >>> >>>>> x = np.random.rand(3,4) >>>>> y = np.random.rand(4,4) >>> >>>>> np.dot(x, y) >>> >>> array([[ 0.86488057, ?0.23456114, ?0.91592677, ?0.89798689], >>> ? ? ? [ 1.24197754, ?0.39907686, ?1.45453141, ?1.13645076], >>> ? ? ? [ 1.41419289, ?0.81818818, ?1.09768428, ?1.32719635]]) >>> >>>>> np.dot(x[0,:], y) >>> ? array([ 0.86488057, ?0.23456114, ?0.91592677, ?0.89798689]) >>>>> np.dot(x[1,:], y) >>> ? array([ 1.24197754, ?0.39907686, ?1.45453141, ?1.13645076]) >>>>> np.dot(x[2,:], y) >>> ? array([ 1.41419289, ?0.81818818, ?1.09768428, ?1.32719635]) >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sccolbert at gmail.com Fri Jul 10 01:16:30 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 10 Jul 2009 01:16:30 -0400 Subject: [Numpy-discussion] an np.arange for arrays? Message-ID: <7f014ea60907092216k7a53f515v4df5373021c9e315@mail.gmail.com> If i have two arrays representing start points and end points, is there a function that will return a 2d array where each row is the range(start, end, n) where n is a fixed number of steps and is the same for all rows? From sccolbert at gmail.com Fri Jul 10 01:25:33 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 10 Jul 2009 01:25:33 -0400 Subject: [Numpy-discussion] an np.arange for arrays? In-Reply-To: <7f014ea60907092216k7a53f515v4df5373021c9e315@mail.gmail.com> References: <7f014ea60907092216k7a53f515v4df5373021c9e315@mail.gmail.com> Message-ID: <7f014ea60907092225k1590439am4964789042e490c8@mail.gmail.com> actually what would be better is if i can pass two 1d arrays X and Y both size Nx1 and get back a 2d array of size NxM where the [n,:] row is the linear interpolation of X[n] to Y[n] On Fri, Jul 10, 2009 at 1:16 AM, Chris Colbert wrote: > If i have two arrays representing start points and end points, is > there a function that will return a 2d array where each row is the > range(start, end, n) where n is a fixed number of steps and is the > same for all rows? > From robertwb at math.washington.edu Fri Jul 10 02:52:47 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 9 Jul 2009 23:52:47 -0700 Subject: [Numpy-discussion] Using __complex__ for complex dtype In-Reply-To: <9254E05D-BBC2-4ECD-A08C-58D5850EC5E4@math.washington.edu> References: <9254E05D-BBC2-4ECD-A08C-58D5850EC5E4@math.washington.edu> Message-ID: <1136D021-C844-4A09-87ED-95185ED6A20F@math.washington.edu> Nevermind, I just found http://bugs.python.org/issue1675423 . On Jul 9, 2009, at 1:41 AM, Robert Bradshaw wrote: > I know using __complex__ has been discussed before, but it would be > really nice if it were at least used to convert object to the > complex dtypes. > > - Robert > > > In [1]: class Foo: > ...: def __float__(self): return 2.0 > ...: > ...: > > In [2]: import numpy > > In [3]: numpy.array([Foo()]) > Out[3]: array([<__main__.Foo instance at 0x17b6da0>], dtype=object) > > In [4]: numpy.array([Foo()], dtype=float) > Out[4]: array([ 2.]) > > In [5]: numpy.array([Foo()], dtype=complex) > Out[5]: array([ 2.+0.j]) > > In [6]: class FooC: > ...: def __complex__(self): return 1+1j > ...: > ...: > > In [7]: numpy.array([Foo()]) > Out[7]: array([<__main__.Foo instance at 0x17b6f08>], dtype=object) > > In [8]: numpy.array([FooC()], dtype=complex) > ---------------------------------------------------------------------- > ----- > TypeError Traceback (most recent > call last) > > /Users/robert/ in () > > TypeError: a float is required > From markus.rosenstihl at physik.tu-darmstadt.de Fri Jul 10 04:04:29 2009 From: markus.rosenstihl at physik.tu-darmstadt.de (Markus Rosenstihl) Date: Fri, 10 Jul 2009 10:04:29 +0200 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <7f014ea60907071854y40bec379obcf7f59d7390d02e@mail.gmail.com> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> <9457e7c80907060318p532caf2aib6fc4f317023959a@mail.gmail.com> <7f014ea60907071854y40bec379obcf7f59d7390d02e@mail.gmail.com> Message-ID: <5FE76576-3A3E-4BB7-BD84-FFD0F4544F6F@physik.tu-darmstadt.de> Here is a different one, based on the ZCW (Zaremba, Conroy, Wolfsberg) algorithm used for powder spectra to equally distribute points over a sphere surface: This paper gives an nice overview (I think I took the actual algorithm from there): Computer simulations in solid-state NMR. III. Powder averaging M. Ed?n Concepts in Magnetic Resonance Part A 18A 24-55 (2003) http://dx.doi.org/10.1002/cmr.a.10065 ############# Fibonacci number generator ################## def fib(n): """ From http://en.literateprograms.org/Fibonacci_numbers_(Python) Returns (n+2)-th and n-th fibonacci number """ if n > 90: raise ValueError,"Can't represent higher numbers!" start = N.array([[1,1],[1,0]],dtype='int64') temp = start[:] for i in xrange(n): temp = N.dot(start,temp) return temp[0,0],temp[0,1],temp[1,1] ############# ZCW Angles ########## def zcw_angles(m): """ Getting N=fibonacchi(m) numbers of ZCW angles """ samples, fib_1, fib_2 = fib(m) #fib_2 = fib_1 print "Using %i Samples"%samples js = N.arange(samples, dtype='Float64')/samples c_full = (1.,2.,1.) c_hemi = (-1.,1.,1.) c_oct = (2.,1.,8.) c = c_full j_samples = fib_2*js # alpha phi = 2*N.pi/c[2] * N.mod(j_samples,1.0) # beta theta = N.arccos(c[0]*(c[1]*N.mod(js,1.0) - 1)) # weights weight = N.ones(samples)/samples return phi,theta,weight Here is a interesting one, REPULSION: REPULSION, A Novel Approach to Efficient Powder Averaging in Solid- State NMR M. Bak and N. C. Nielsen Journal of Magnetic Resonance 125 132--139 (1997) http://www.sciencedirect.com/science/article/B6WJX-45N4XS2-J/1/58bb5f6971e8a76ab01a17e2794f98b7 A novel approach to efficient powder averaging in magnetic resonance is presented. The method relies on a simple numerical procedure which based on a random set of crystallite orientations through simulation of fictive intercrystallite repulsive forces iteratively determines a set of orientations uniformly distributed over the unit sphere. The so- called REPULSION partition scheme is compared to earlier methods with respect to the distribution of crystallite orientations, solid angles, and powder averaging efficiency. It is demonstrated that powder averaging using REPULSION converges faster than previous methods with respect to the number of crystallite orientations involved in the averaging. This feature renders REPULSION particularly attractive for calculation of magic-angle-spinning solid-state NMR spectra using a minimum of crystallite orientations. For numerical simulation of powder spectra, the reduced number of required crystallite orientations translates into shorter computation times and simulations less prone to systematic errors induced by finite sets of nonuniformly distributed crystallite orientations. -- "A program that produces incorrect results twice as fast is infinitely slower." -- John Osterhout From dwf at cs.toronto.edu Fri Jul 10 04:49:44 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 10 Jul 2009 04:49:44 -0400 Subject: [Numpy-discussion] an np.arange for arrays? In-Reply-To: <7f014ea60907092225k1590439am4964789042e490c8@mail.gmail.com> References: <7f014ea60907092216k7a53f515v4df5373021c9e315@mail.gmail.com> <7f014ea60907092225k1590439am4964789042e490c8@mail.gmail.com> Message-ID: On 10-Jul-09, at 1:25 AM, Chris Colbert wrote: > actually what would be better is if i can pass two 1d arrays X and Y > both size Nx1 > and get back a 2d array of size NxM where the [n,:] row is the linear > interpolation of X[n] to Y[n] This could be more efficient, but here's a solution using mgrid and broadcasting: def interp_grid_thingie(X, Y, M): frac = np.mgrid[0:1:M*1j] return (1 - frac[np.newaxis, :]) * X[:, np.newaxis] + frac[np.newaxis, :] * Y[:, np.newaxis] In [11]: interp_grid_thingie(arange(1,6), arange(0,5), 5) Out[11]: array([[ 1. , 0.75, 0.5 , 0.25, 0. ], [ 2. , 1.75, 1.5 , 1.25, 1. ], [ 3. , 2.75, 2.5 , 2.25, 2. ], [ 4. , 3.75, 3.5 , 3.25, 3. ], [ 5. , 4.75, 4.5 , 4.25, 4. ]]) David From washakie at gmail.com Fri Jul 10 08:40:08 2009 From: washakie at gmail.com (John [H2O]) Date: Fri, 10 Jul 2009 05:40:08 -0700 (PDT) Subject: [Numpy-discussion] np.concatenate and np.column_stack shouldn't they be the same? Message-ID: <24426795.post@talk.nabble.com> Can someone explain: x = np.arange(20) y = np.arange(20) z = np.vstack((x,y)).T is equal to: z = np.column_stack((x,y)) but this does not do the same: z = np.concatenate((x,y),axis=0) # or with axis=1 Seems I should be able to use concatenate to make a column stack?? Thanks! -- View this message in context: http://www.nabble.com/np.concatenate-and-np.column_stack-shouldn%27t-they-be-the-same--tp24426795p24426795.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From bsouthey at gmail.com Fri Jul 10 09:28:07 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 10 Jul 2009 08:28:07 -0500 Subject: [Numpy-discussion] transform an array of points efficiently? Message-ID: <4A5741E7.5000201@gmail.com> > > hey, > > great man! thanks! > > I had thought that it may have been possible with a single dot, but > how to do it escaped me. > > Thanks again! > > Chris > Hi, When dot is not what you want, often numpy.inner() and numpy.outer() do what you want. So try using numpy.inner(x,y)... http://www.scipy.org/Numpy_Example_List_With_Doc#inner import numpy as np x = np.random.rand(10,4) y = np.random.rand(4,4) print 'x\n', x, '\ny\n', y print '\nresult\n', np.dot(y,x.T).T print '\ninner\n', np.inner(x,y) Bruce From sccolbert at gmail.com Fri Jul 10 10:44:03 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 10 Jul 2009 10:44:03 -0400 Subject: [Numpy-discussion] an np.arange for arrays? In-Reply-To: References: <7f014ea60907092216k7a53f515v4df5373021c9e315@mail.gmail.com> <7f014ea60907092225k1590439am4964789042e490c8@mail.gmail.com> Message-ID: <7f014ea60907100744k4fea806fy1abbbc399d21f3a4@mail.gmail.com> Oh cool, I couldn't figure out with mgrid. here's what ended up with using broadcasting: >>> import numpy as np >>> X = np.zeros((10)) >>> Y = np.arange(10, 20) >>> M = 10 >>> increments = np.arange(1, M+1) >>> delta = Y - X >>> dl = (delta / M).reshape(-1, 1) >>> interps = dl * increments >>> lines = X + interps >>> lines array([[ 1. , 2. , 3. , 4. , 5. , 6. , 7. , 8. , 9. , 10. ], [ 1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8, 9.9, 11. ], [ 1.2, 2.4, 3.6, 4.8, 6. , 7.2, 8.4, 9.6, 10.8, 12. ], [ 1.3, 2.6, 3.9, 5.2, 6.5, 7.8, 9.1, 10.4, 11.7, 13. ], [ 1.4, 2.8, 4.2, 5.6, 7. , 8.4, 9.8, 11.2, 12.6, 14. ], [ 1.5, 3. , 4.5, 6. , 7.5, 9. , 10.5, 12. , 13.5, 15. ], [ 1.6, 3.2, 4.8, 6.4, 8. , 9.6, 11.2, 12.8, 14.4, 16. ], [ 1.7, 3.4, 5.1, 6.8, 8.5, 10.2, 11.9, 13.6, 15.3, 17. ], [ 1.8, 3.6, 5.4, 7.2, 9. , 10.8, 12.6, 14.4, 16.2, 18. ], [ 1.9, 3.8, 5.7, 7.6, 9.5, 11.4, 13.3, 15.2, 17.1, 19. ]]) On Fri, Jul 10, 2009 at 4:49 AM, David Warde-Farley wrote: > > On 10-Jul-09, at 1:25 AM, Chris Colbert wrote: > >> actually what would be better is if i can pass two 1d arrays X and Y >> both size Nx1 >> and get back a 2d array of size NxM where the [n,:] row is the linear >> interpolation of X[n] to Y[n] > > This could be more efficient, but here's a solution using mgrid and > broadcasting: > > def interp_grid_thingie(X, Y, M): > ? ? ? ?frac = np.mgrid[0:1:M*1j] > ? ? ? ?return (1 - frac[np.newaxis, :]) * X[:, np.newaxis] + > frac[np.newaxis, :] * Y[:, np.newaxis] > > In [11]: interp_grid_thingie(arange(1,6), arange(0,5), 5) > Out[11]: > array([[ 1. ?, ?0.75, ?0.5 , ?0.25, ?0. ?], > ? ? ? ?[ 2. ?, ?1.75, ?1.5 , ?1.25, ?1. ?], > ? ? ? ?[ 3. ?, ?2.75, ?2.5 , ?2.25, ?2. ?], > ? ? ? ?[ 4. ?, ?3.75, ?3.5 , ?3.25, ?3. ?], > ? ? ? ?[ 5. ?, ?4.75, ?4.5 , ?4.25, ?4. ?]]) > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From lciti at essex.ac.uk Fri Jul 10 10:55:58 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Fri, 10 Jul 2009 15:55:58 +0100 Subject: [Numpy-discussion] speed up np.diag Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DD7@sernt14.essex.ac.uk> Hello, I happened to have a look at the code for np.diag and found it more complicated that necessary. I think it can be rewritten more cleanly and efficiently. Appended you can find both versions. The speed improvement is significant: In [145]: x = S.rand(1000,1300) In [146]: assert all(diag(x,-13) == diag2(x,-13)) In [147]: %timeit diag(x,-13) 10000 loops, best of 3: 37.4 ?s per loop In [148]: %timeit diag2(x,-13) 100000 loops, best of 3: 14.1 ?s per loop In [149]: x = x.T In [150]: assert all(diag(x,-13) == diag2(x,-13)) In [151]: %timeit diag(x,-13) 10000 loops, best of 3: 81.1 ?s per loop In [152]: %timeit diag2(x,-13) 100000 loops, best of 3: 14.8 ?s per loop It takes slightly more than one third for the C-contiguous input and slightly more than one sixth for the F-contiguous input. Best, Luca ## ORIGINAL def diag(v, k=0): v = asarray(v) s = v.shape if len(s)==1: n = s[0]+abs(k) res = zeros((n,n), v.dtype) if (k>=0): i = arange(0,n-k) fi = i+k+i*n else: i = arange(0,n+k) fi = i+(i-k)*n res.flat[fi] = v return res elif len(s)==2: N1,N2 = s if k >= 0: M = min(N1,N2-k) i = arange(0,M) fi = i+k+i*N2 else: M = min(N1+k,N2) i = arange(0,M) fi = i + (i-k)*N2 return v.flat[fi] else: raise ValueError, "Input must be 1- or 2-d." ## SUGGESTED def diag(v, k=0): v = asarray(v) s = v.shape if len(s) == 1: n = s[0]+abs(k) res = zeros((n,n), v.dtype) i = k if k >= 0 else (-k) * s[1] res.flat[i::n+1] = v return res elif len(s) == 2: if v.flags.f_contiguous: v, k, s = v.T, -k, s[::-1] i = k if k >= 0 else (-k) * s[1] return v.flat[i::s[1]+1] else: raise ValueError, "Input must be 1- or 2-d." From pav at iki.fi Fri Jul 10 11:11:27 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 10 Jul 2009 15:11:27 +0000 (UTC) Subject: [Numpy-discussion] speed up np.diag References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DD7@sernt14.essex.ac.uk> Message-ID: Fri, 10 Jul 2009 15:55:58 +0100, Citi, Luca kirjoitti: [clip] > ## SUGGESTED > def diag(v, k=0): > v = asarray(v) > s = v.shape > if len(s) == 1: [clip] > elif len(s) == 2: > if v.flags.f_contiguous: > v, k, s = v.T, -k, s[::-1] Is this correct? The .flat iterator always traverses the array in virtual C-order, not in the order it's laid out in memory. -- Pauli Virtanen From tgrav at mac.com Fri Jul 10 11:19:59 2009 From: tgrav at mac.com (Tommy Grav) Date: Fri, 10 Jul 2009 11:19:59 -0400 Subject: [Numpy-discussion] numpy for 2.6 on mac os x Message-ID: <452AFC6D-A596-4CB1-85C2-20A0ABC50C81@mac.com> The current dmg on the numpy download pages is buildt against 2.5. Is there any plans to make one for 2.6 or do I have to compile from the source? Cheers Tommy From lciti at essex.ac.uk Fri Jul 10 11:16:14 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Fri, 10 Jul 2009 16:16:14 +0100 Subject: [Numpy-discussion] speed up np.diag References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DD7@sernt14.essex.ac.uk> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DD8@sernt14.essex.ac.uk> >> if v.flags.f_contiguous: >> v, k, s = v.T, -k, s[::-1] >Is this correct? The .flat iterator always traverses the array in virtual >C-order, not in the order it's laid out in memory. The code could work (and gives the same results) even without the two lines above which in theory do nothing: taking the k-th diagonal of an array or the (-k)-th of its transpose should be the same. But in this case ndarray.flat works faster if the array is C-contiguous. Best, Luca From robert.kern at gmail.com Fri Jul 10 11:28:23 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 10 Jul 2009 10:28:23 -0500 Subject: [Numpy-discussion] np.concatenate and np.column_stack shouldn't they be the same? In-Reply-To: <24426795.post@talk.nabble.com> References: <24426795.post@talk.nabble.com> Message-ID: <3d375d730907100828w69104eeft22b9bd10d08c091b@mail.gmail.com> On Fri, Jul 10, 2009 at 07:40, John [H2O] wrote: > > Can someone explain: > > x = np.arange(20) > y = np.arange(20) > > z = np.vstack((x,y)).T > > is equal to: > > z = np.column_stack((x,y)) > > but this does not do the same: > > z = np.concatenate((x,y),axis=0) ?# or with axis=1 > > Seems I should be able to use concatenate to make a column stack?? z = np.concatenate((x[:,np.newaxis], y[:,np.newaxis]), axis=1) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From d_l_goldsmith at yahoo.com Fri Jul 10 13:26:22 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Fri, 10 Jul 2009 10:26:22 -0700 (PDT) Subject: [Numpy-discussion] an np.arange for arrays? Message-ID: <246522.48048.qm@web52105.mail.re2.yahoo.com> It's also 5 LOSC using linspace: def interp_grid(x, y, nrows): ''' A linearly-interpolated (across rows) 2-D grid generator ''' ## Put data quality checking here - confirm len(x) == len(y), ## nrows > 2, decide on behavior if inputs bad ncols = len(x) grid = np.zeros((nrows, ncols)) # allocate memory for i in range(ncols): grid[:,i] = np.linspace(x[i], y[i], nrows) return grid >>> nrows=10 >>> ncols = len(x) >>> x; y; nrows, ncols array([1, 2, 3, 4, 5, 6, 7, 8, 9]) array([10, 11, 12, 13, 14, 15, 16, 17, 18]) (10, 9) >>> def interp_grid(x, y, nrows): ... ncols = len(x) ... grid = np.zeros((nrows, ncols)) # allocate memory ... for i in range(ncols): ... grid[:,i] = np.linspace(x[i], y[i], nrows) ... return grid ... >>> interp_grid(x, y, nrows) array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9.], [ 2., 3., 4., 5., 6., 7., 8., 9., 10.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18.]]) Or, if you prefer one-liners, using list comprehension: grid = np.array([np.linspace(x[i],y[i],nrows) for i in range(len(x))]).T (the transposing at the end is necessary 'cause the result of linspace is interpreted as a row vector, but you want the interpolated values in columns; if you can have your data be in the first and last columns and the interpolation be in the rows, you can omit the final .T) >>> grid = np.array([np.linspace(x[i],y[i],nrows) for i in range(len(x))]).T >>> grid array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9.], [ 2., 3., 4., 5., 6., 7., 8., 9., 10.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18.]]) DG --- On Fri, 7/10/09, Chris Colbert wrote: > From: Chris Colbert > Subject: Re: [Numpy-discussion] an np.arange for arrays? > To: "Discussion of Numerical Python" > Date: Friday, July 10, 2009, 7:44 AM > Oh cool, I couldn't figure out with > mgrid. > here's what ended up with using broadcasting: > > >>> import numpy as np > >>> X = np.zeros((10)) > >>> Y = np.arange(10, 20) > >>> M = 10 > >>> increments = np.arange(1, M+1) > >>> delta = Y - X > >>> dl = (delta / M).reshape(-1, 1) > >>> interps = dl * increments > >>> lines = X + interps > >>> lines > array([[? 1. ,???2. > ,???3. ,???4. > ,???5. ,???6. > ,???7. ,???8. > ,???9. ,? 10. ], > ? ? ???[? > 1.1,???2.2,???3.3,???4.4,???5.5,???6.6,???7.7,???8.8,???9.9,? > 11. ], > ? ? ???[? > 1.2,???2.4,???3.6,???4.8,???6. > ,???7.2,???8.4,???9.6,? > 10.8,? 12. ], > ? ? ???[? > 1.3,???2.6,???3.9,???5.2,???6.5,???7.8,???9.1,? > 10.4,? 11.7,? 13. ], > ? ? ???[? > 1.4,???2.8,???4.2,???5.6,???7. > ,???8.4,???9.8,? > 11.2,? 12.6,? 14. ], > ? ? ???[? > 1.5,???3. > ,???4.5,???6. > ,???7.5,???9. ,? > 10.5,? 12. ,? 13.5,? 15. ], > ? ? ???[? > 1.6,???3.2,???4.8,???6.4,???8. > ,???9.6,? 11.2,? 12.8,? > 14.4,? 16. ], > ? ? ???[? > 1.7,???3.4,???5.1,???6.8,???8.5,? > 10.2,? 11.9,? 13.6,? 15.3,? 17. ], > ? ? ???[? > 1.8,???3.6,???5.4,???7.2,???9. > ,? 10.8,? 12.6,? 14.4,? 16.2,? 18. > ], > ? ? ???[? > 1.9,???3.8,???5.7,???7.6,???9.5,? > 11.4,? 13.3,? 15.2,? 17.1,? 19. ]]) > > > > > On Fri, Jul 10, 2009 at 4:49 AM, David Warde-Farley > wrote: > > > > On 10-Jul-09, at 1:25 AM, Chris Colbert wrote: > > > >> actually what would be better is if i can pass two > 1d arrays X and Y > >> both size Nx1 > >> and get back a 2d array of size NxM where the > [n,:] row is the linear > >> interpolation of X[n] to Y[n] > > > > This could be more efficient, but here's a solution > using mgrid and > > broadcasting: > > > > def interp_grid_thingie(X, Y, M): > > ? ? ? ?frac = np.mgrid[0:1:M*1j] > > ? ? ? ?return (1 - frac[np.newaxis, :]) * X[:, > np.newaxis] + > > frac[np.newaxis, :] * Y[:, np.newaxis] > > > > In [11]: interp_grid_thingie(arange(1,6), arange(0,5), > 5) > > Out[11]: > > array([[ 1. ?, ?0.75, ?0.5 , ?0.25, ?0. ?], > > ? ? ? ?[ 2. ?, ?1.75, ?1.5 , ?1.25, ?1. ?], > > ? ? ? ?[ 3. ?, ?2.75, ?2.5 , ?2.25, ?2. ?], > > ? ? ? ?[ 4. ?, ?3.75, ?3.5 , ?3.25, ?3. ?], > > ? ? ? ?[ 5. ?, ?4.75, ?4.5 , ?4.25, ?4. > ?]]) > > > > David > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From nouiz at nouiz.org Fri Jul 10 13:57:11 2009 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Fri, 10 Jul 2009 13:57:11 -0400 Subject: [Numpy-discussion] speeding up an array operation In-Reply-To: <1cbd6f830907090414y304bc4f1v319a231f38e68c6f@mail.gmail.com> References: <1cbd6f830907082212x74193fd6t5bab749e9826fac@mail.gmail.com> <6B070B64-D277-48F1-9F99-EFF112640240@cs.toronto.edu> <1cbd6f830907090414y304bc4f1v319a231f38e68c6f@mail.gmail.com> Message-ID: <2d1d7fe70907101057n7344579ej30d52b875960c317@mail.gmail.com> Can you do it by chunk instead of by row? If the chunk is not too big the sort could be faster then the access to the multiple dictionnary access. But don't forget, you change an algo of O(n), by O(nlogn) with a lower constant. So the n should not be too big. Just try different value. Fr?d?ric Bastien On Thu, Jul 9, 2009 at 7:14 AM, Mag Gam wrote: > The problem is the array is very large. We are talking about 200+ million > rows. > > > On Thu, Jul 9, 2009 at 4:41 AM, David Warde-Farley > wrote: > > On 9-Jul-09, at 1:12 AM, Mag Gam wrote: > > > >> Here is what I have, which does it 1x1: > >> > >> z={} #dictionary > >> r=csv.reader(file) > >> for i,row in enumerate(r): > >> p="/MIT/"+row[1] > >> > >> if p not in z: > >> z[p]=0: > >> else: > >> z[p]+=1 > >> > >> arr[p]['chem'][z[p]]=tuple(row) #this loads the array 1 x 1 > >> > >> > >> I would like to avoid the 1x1 loading, instead I would like to bulk > >> load the array. Lets say load up 5million lines into memory and then > >> push into array. Any ideas on how to do that? > > > > > > Depending on how big your data is, this looks like a job for e.g. > > numpy.loadtxt(), to give you one big array. > > > > Then sort the array on the second column, so that all the rows with > > the same 'p' appear one after the other. Then you can assign slices of > > this big array to be arr[p]['chem']. > > > > David > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Fri Jul 10 16:06:52 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 10 Jul 2009 16:06:52 -0400 Subject: [Numpy-discussion] an np.arange for arrays? In-Reply-To: <246522.48048.qm@web52105.mail.re2.yahoo.com> References: <246522.48048.qm@web52105.mail.re2.yahoo.com> Message-ID: <2075B3EA-F6AF-4379-B24B-3AE6A839EA29@cs.toronto.edu> On 10-Jul-09, at 1:26 PM, David Goldsmith wrote: > grid = np.array([np.linspace(x[i],y[i],nrows) for i in > range(len(x))]).T Indeed, linspace will work, but careful with Python loops though, it'll be 2x to 6x slower (based on my empirical fiddling) than the solution involving mgrid. In [31]: X = arange(10); Y = arange(10)+1 In [32]: timeit grid = np.array([np.linspace(X[i],Y[i],20) for i in range(len(X))]).T 1000 loops, best of 3: 424 us per loop In [33]: timeit interp_grid_thingie(X, Y, 20) 10000 loops, best of 3: 79.9 us per loop David From geometrian at gmail.com Fri Jul 10 16:09:43 2009 From: geometrian at gmail.com (Ian Mallett) Date: Fri, 10 Jul 2009 13:09:43 -0700 Subject: [Numpy-discussion] Even Sphere Volume In-Reply-To: <5FE76576-3A3E-4BB7-BD84-FFD0F4544F6F@physik.tu-darmstadt.de> References: <9457e7c80907051350s1add96fele3e4fbd51c42c9c5@mail.gmail.com> <9457e7c80907051640h67dfb58y2002fd8d63b6664a@mail.gmail.com> <49d6b3500907051708i2f8bd3e7vaf25397b2e5be063@mail.gmail.com> <9457e7c80907060318p532caf2aib6fc4f317023959a@mail.gmail.com> <7f014ea60907071854y40bec379obcf7f59d7390d02e@mail.gmail.com> <5FE76576-3A3E-4BB7-BD84-FFD0F4544F6F@physik.tu-darmstadt.de> Message-ID: These are all great algorithms, thanks for the help! -------------- next part -------------- An HTML attachment was scrubbed... URL: From d_l_goldsmith at yahoo.com Fri Jul 10 16:35:25 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Fri, 10 Jul 2009 13:35:25 -0700 (PDT) Subject: [Numpy-discussion] an np.arange for arrays? Message-ID: <393466.93149.qm@web52111.mail.re2.yahoo.com> Touche. DG --- On Fri, 7/10/09, David Warde-Farley wrote: > From: David Warde-Farley > Subject: Re: [Numpy-discussion] an np.arange for arrays? > To: "Discussion of Numerical Python" > Date: Friday, July 10, 2009, 1:06 PM > On 10-Jul-09, at 1:26 PM, David > Goldsmith wrote: > > > grid = np.array([np.linspace(x[i],y[i],nrows) for i > in? > > range(len(x))]).T > > Indeed, linspace will work, but careful with Python loops > though,? > it'll be 2x to 6x slower (based on my empirical fiddling) > than the? > solution involving mgrid. > > In [31]: X = arange(10); Y = arange(10)+1 > In [32]: timeit grid = np.array([np.linspace(X[i],Y[i],20) > for i in? > range(len(X))]).T > 1000 loops, best of 3: 424 us per loop > > In [33]: timeit interp_grid_thingie(X, Y, 20) > 10000 loops, best of 3: 79.9 us per loop > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dalcinl at gmail.com Fri Jul 10 17:48:49 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 10 Jul 2009 18:48:49 -0300 Subject: [Numpy-discussion] Using __complex__ for complex dtype In-Reply-To: <1136D021-C844-4A09-87ED-95185ED6A20F@math.washington.edu> References: <9254E05D-BBC2-4ECD-A08C-58D5850EC5E4@math.washington.edu> <1136D021-C844-4A09-87ED-95185ED6A20F@math.washington.edu> Message-ID: On Fri, Jul 10, 2009 at 3:52 AM, Robert Bradshaw wrote: > Nevermind, I just found http://bugs.python.org/issue1675423 . > Nevermind? Perhaps NumPy should handle this gotcha for Python < 2.6 ? - > On Jul 9, 2009, at 1:41 AM, Robert Bradshaw wrote: > >> I know using __complex__ has been discussed before, but it would be >> really nice if it were at least used to convert object to the >> complex dtypes. >> >> - Robert >> >> >> In [1]: class Foo: >> ? ?...: ? ? def __float__(self): return 2.0 >> ? ?...: >> ? ?...: >> >> In [2]: import numpy >> >> In [3]: numpy.array([Foo()]) >> Out[3]: array([<__main__.Foo instance at 0x17b6da0>], dtype=object) >> >> In [4]: numpy.array([Foo()], dtype=float) >> Out[4]: array([ 2.]) >> >> In [5]: numpy.array([Foo()], dtype=complex) >> Out[5]: array([ 2.+0.j]) >> >> In [6]: class FooC: >> ? ?...: ? ? def __complex__(self): return 1+1j >> ? ?...: >> ? ?...: >> >> In [7]: numpy.array([Foo()]) >> Out[7]: array([<__main__.Foo instance at 0x17b6f08>], dtype=object) >> >> In [8]: numpy.array([FooC()], dtype=complex) >> ---------------------------------------------------------------------- >> ----- >> TypeError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Traceback (most recent >> call last) >> >> /Users/robert/ in () >> >> TypeError: a float is required >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From cournape at gmail.com Fri Jul 10 22:15:05 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Jul 2009 11:15:05 +0900 Subject: [Numpy-discussion] numpy for 2.6 on mac os x In-Reply-To: <452AFC6D-A596-4CB1-85C2-20A0ABC50C81@mac.com> References: <452AFC6D-A596-4CB1-85C2-20A0ABC50C81@mac.com> Message-ID: <5b8d13220907101915s478ea649l3dbcc07ce1718baa@mail.gmail.com> On Sat, Jul 11, 2009 at 12:19 AM, Tommy Grav wrote: > The current dmg on the numpy download pages is buildt against 2.5. Is > there any plans > to make one for 2.6 or do I have to compile from the source? There are plans :) I am building the 0.7.1 binaries right now, and mac os x binaries will be available for both 2.6 and 2.5. I will add the corresponding 2.6 numpy binary at the same time. David From magawake at gmail.com Sat Jul 11 00:47:41 2009 From: magawake at gmail.com (Mag Gam) Date: Sat, 11 Jul 2009 00:47:41 -0400 Subject: [Numpy-discussion] speeding up an array operation In-Reply-To: <2d1d7fe70907101057n7344579ej30d52b875960c317@mail.gmail.com> References: <1cbd6f830907082212x74193fd6t5bab749e9826fac@mail.gmail.com> <6B070B64-D277-48F1-9F99-EFF112640240@cs.toronto.edu> <1cbd6f830907090414y304bc4f1v319a231f38e68c6f@mail.gmail.com> <2d1d7fe70907101057n7344579ej30d52b875960c317@mail.gmail.com> Message-ID: <1cbd6f830907102147h63943a11ha61f4f1c7772dfc5@mail.gmail.com> Hey Frederic: thanks for the response. I really want it to do it your way but I am a bad programmer. Do you have any sample code? your method seems correct 2009/7/10 Fr?d?ric Bastien : > Can you do it by chunk instead of by row? If the chunk is not too big the > sort could be faster then the access to the multiple dictionnary access. But > don't forget, you change an algo of O(n), by O(nlogn) with a lower constant. > So the n should not be too big. Just try different value. > > Fr?d?ric Bastien > > On Thu, Jul 9, 2009 at 7:14 AM, Mag Gam wrote: >> >> The problem is the array is very large. We are talking about 200+ million >> rows. >> >> >> On Thu, Jul 9, 2009 at 4:41 AM, David Warde-Farley >> wrote: >> > On 9-Jul-09, at 1:12 AM, Mag Gam wrote: >> > >> >> Here is what I have, which does it 1x1: >> >> >> >> z={} ?#dictionary >> >> r=csv.reader(file) >> >> for i,row in enumerate(r): >> >> ?p="/MIT/"+row[1] >> >> >> >> ?if p not in z: >> >> ? ?z[p]=0: >> >> ?else: >> >> ? ?z[p]+=1 >> >> >> >> ?arr[p]['chem'][z[p]]=tuple(row) #this loads the array 1 x 1 >> >> >> >> >> >> I would like to avoid the 1x1 loading, instead I would like to bulk >> >> load the array. Lets say load up 5million lines into memory and then >> >> push into array. Any ideas on how to do that? >> > >> > >> > Depending on how big your data is, this looks like a job for e.g. >> > numpy.loadtxt(), to give you one big array. >> > >> > Then sort the array on the second column, so that all the rows with >> > the same 'p' appear one after the other. Then you can assign slices of >> > this big array to be arr[p]['chem']. >> > >> > David >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From millman at berkeley.edu Sat Jul 11 04:32:21 2009 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 11 Jul 2009 01:32:21 -0700 Subject: [Numpy-discussion] ANN: SciPy 2009 early registration extended to July 17th Message-ID: The early registration deadline for SciPy 2009 has been extended for one week to July 17, 2009. Please register ( http://conference.scipy.org/to_register ) by this date to take advantage of the reduced early registration rate. About the conference -------------------- SciPy 2009, the 8th Python in Science conference, will be held from August 18-23, 2009 at Caltech in Pasadena, CA, USA. The conference starts with two days of tutorials to the scientific Python tools. There will be two tracks, one for introduction of the basic tools to beginners, and one for more advanced tools. The tutorials will be followed by two days of talks. Both days of talks will begin with a keynote address. The first day?s keynote will be given by Peter Norvig, the Director of Research at Google; while, the second keynote will be delivered by Jon Guyer, a Materials Scientist in the Thermodynamics and Kinetics Group at NIST. The program committee will select the remaining talks from submissions to our call for papers. All selected talks will be included in our conference proceedings edited by the program committee. After the talks each day we will provide several rooms for impromptu birds of a feather discussions. Finally, the last two days of the conference will be used for a number of coding sprints on the major software projects in our community. For the 8th consecutive year, the conference will bring together the developers and users of the open source software stack for scientific computing with Python. Attendees have the opportunity to review the available tools and how they apply to specific problems. By providing a forum for developers to share their Python expertise with the wider commercial, academic, and research communities, this conference fosters collaboration and facilitates the sharing of software components, techniques, and a vision for high level language use in scientific computing. For further information, please visit the conference homepage: http://conference.scipy.org. Important Dates --------------- * Friday, July 3: Abstracts Due * Wednesday, July 15: Announce accepted talks, post schedule * Friday, July 17: Early Registration ends * Tuesday-Wednesday, August 18-19: Tutorials * Thursday-Friday, August 20-21: Conference * Saturday-Sunday, August 22-23: Sprints * Friday, September 4: Papers for proceedings due Executive Committee ------------------- * Jarrod Millman, UC Berkeley, USA (Conference Chair) * Ga?l Varoquaux, INRIA Saclay, France (Program Co-Chair) * St?fan van der Walt, University of Stellenbosch, South Africa (Program Co-Chair) * Fernando P?rez, UC Berkeley, USA (Tutorial Chair) From sidhant at gmail.com Sat Jul 11 06:20:13 2009 From: sidhant at gmail.com (f0X_in_s0X) Date: Sat, 11 Jul 2009 03:20:13 -0700 (PDT) Subject: [Numpy-discussion] argmin and argmax Message-ID: <24428341.post@talk.nabble.com> Do argmin and argmax skip nan values? I cant find it anywhere in the numpy documentation. In any case, if I want to find max and min values in an array that may contain nan values also, what would the be the most efficient way to do that? Any pointers?? -- View this message in context: http://www.nabble.com/argmin-and-argmax-tp24428341p24428341.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From charlesr.harris at gmail.com Sat Jul 11 10:04:35 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 11 Jul 2009 08:04:35 -0600 Subject: [Numpy-discussion] ANN: SciPy 2009 early registration extended to July 17th In-Reply-To: References: Message-ID: On Sat, Jul 11, 2009 at 2:32 AM, Jarrod Millman wrote: > The early registration deadline for SciPy 2009 has been extended > for one week to July 17, 2009. Please register ( > http://conference.scipy.org/to_register ) > by this date to take advantage of the reduced early registration rate. Why isn't there an announcement and link on the scipy homepage? You would never know anything was happening to look at it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Sat Jul 11 10:09:57 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 11 Jul 2009 07:09:57 -0700 Subject: [Numpy-discussion] argmin and argmax In-Reply-To: <24428341.post@talk.nabble.com> References: <24428341.post@talk.nabble.com> Message-ID: On Sat, Jul 11, 2009 at 3:20 AM, f0X_in_s0X wrote: > > Do argmin and argmax skip nan values? I cant find it anywhere in the numpy > documentation. > > In any case, if I want to find max and min values in an array that may > contain nan values also, what would the be the most efficient way to do > that? Any pointers?? >> x = np.array([np.nan, 1, 2, np.nan]) >> x.argmax() 0 :( >> x.argmin() 0 :( >> np.__version__ '1.2.1' >> >> np.nanargmax(x) 2 :) >> np.nanargmin(x) 1 :) From kwgoodman at gmail.com Sat Jul 11 10:12:44 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 11 Jul 2009 07:12:44 -0700 Subject: [Numpy-discussion] argmin and argmax In-Reply-To: References: <24428341.post@talk.nabble.com> Message-ID: On Sat, Jul 11, 2009 at 7:09 AM, Keith Goodman wrote: > On Sat, Jul 11, 2009 at 3:20 AM, f0X_in_s0X wrote: >> >> Do argmin and argmax skip nan values? I cant find it anywhere in the numpy >> documentation. >> >> In any case, if I want to find max and min values in an array that may >> contain nan values also, what would the be the most efficient way to do >> that? Any pointers?? > >>> x = np.array([np.nan, 1, 2, np.nan]) >>> x.argmax() > ? 0 ? :( >>> x.argmin() > ? 0 ? :( >>> np.__version__ > ? '1.2.1' >>> >>> np.nanargmax(x) > ? 2 ? :) >>> np.nanargmin(x) > ? 1 ? :) Oh, you only want the values? Then take a look at np.nanmax and np.nanmin. From lciti at essex.ac.uk Sat Jul 11 14:00:58 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Sat, 11 Jul 2009 19:00:58 +0100 Subject: [Numpy-discussion] speed up np.diag References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DD7@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DD8@sernt14.essex.ac.uk> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDA@sernt14.essex.ac.uk> I have submitted Ticket #1167 with a patch to speed up diag and eye. On average the code is 3 times faster (but up to 9!). From kwgoodman at gmail.com Sat Jul 11 14:19:12 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 11 Jul 2009 11:19:12 -0700 Subject: [Numpy-discussion] speed up np.diag In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDA@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DD7@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DD8@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDA@sernt14.essex.ac.uk> Message-ID: On Sat, Jul 11, 2009 at 11:00 AM, Citi, Luca wrote: > I have submitted Ticket #1167 with a patch > to speed up diag and eye. > On average the code is 3 times faster (but > up to 9!). Wow! That's great. With your speed ups it won't be long before I use np.diag(np.eye(10)) instead of np.ones(10) ;) From gael.varoquaux at normalesup.org Sat Jul 11 15:55:42 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 11 Jul 2009 21:55:42 +0200 Subject: [Numpy-discussion] Small fix to nosetester Message-ID: <20090711195542.GO20169@phare.normalesup.org> Right now the logics to detect package name in nosetester is fairly simple, and fails for non-standard instal path. I have opened up a ticket for that, with a fix: http://projects.scipy.org/numpy/ticket/1168 Ga?l From ralf.gommers at googlemail.com Sun Jul 12 01:16:58 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 12 Jul 2009 01:16:58 -0400 Subject: [Numpy-discussion] find_common_type broken? Message-ID: Hi, While documenting find_common_type I found two problems. As I understand it, "common type" is a type to which all input types can be cast without loss of precision. 1. using any array types always returns "dtype('object')": >>> np.find_common_type([np.ndarray], []) dtype('object') >>> np.find_common_type([np.ndarray, np.ma.MaskedArray, np.recarray], []) dtype('object') 2. the second example below seems to be wrong, it should return dtype('float64'): >>> np.find_common_type([], [np.int64, np.float64]) dtype('float64') >>> np.find_common_type([], [np.int64, np.float32]) dtype('int64') One other question, why do type comparisons for numpy types and python built-ins do the opposite: >>> np.int32 > np.float32 False >>> np.int64 > np.float64 False >>> int > float True The numpy result makes more sense to me, what's going on with the builtins? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From lciti at essex.ac.uk Sun Jul 12 06:54:47 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Sun, 12 Jul 2009 11:54:47 +0100 Subject: [Numpy-discussion] find_common_type broken? References: Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDD@sernt14.essex.ac.uk> Hi, I am not very confident with types but I will try to give you my opinion. As for the first part of the question > >>> np.find_common_type([np.ndarray, np.ma.MaskedArray, np.recarray], []) > dtype('object') I think that the first argument of np.find_common_type should be the type of an _element_ of the array, not the type of the array. In your case you are asking np.find_common_type the common type between an array of arrays, an array of masked arrays, and an array of record arrays. Therefore the best thing np can do is to find object as common type. Correctly: >>> np.find_common_type([np.float64, np.int32], []) dtype('float64') As for the second part of the question np.find_common_type internally uses np.dtype(t) for each type in input. While the comparison between types work as expected: >>> np.complex128 > np.complex64 > np.float64 > np.float32 > np.int64 > np.int32 True the comparison between dtype(t) gives different results: >>> np.dtype(np.float64) > np.dtype(np.int64) True >>> np.dtype(np.float32) > np.dtype(np.int64) False >>> np.dtype(np.float32) > np.dtype(np.int32) False >>> np.dtype(np.float32) > np.dtype(np.int16) True At first I thought the comparison was made based on the number of bits in the mantissa or the highest integer N for which N-1 was still representable. But then I could not explain the first result. What is surprising is that >>> np.dtype(np.float32) > np.dtype(np.int32) False >>> np.dtype(np.float32) < np.dtype(np.int32) False >>> np.dtype(np.float32) == np.dtype(np.int32) False therefore the max() function in np.find_common_type cannot tell which to return, and returns the first. In fact: >>> np.find_common_type([], [np.int64, np.float32]) dtype('int64') but >>> np.find_common_type([], [np.float32, np.int64]) dtype('float32') which is unexpected. Best, Luca From david at ar.media.kyoto-u.ac.jp Sun Jul 12 07:36:22 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 12 Jul 2009 20:36:22 +0900 Subject: [Numpy-discussion] numpy for 2.6 on mac os x In-Reply-To: <5b8d13220907101915s478ea649l3dbcc07ce1718baa@mail.gmail.com> References: <452AFC6D-A596-4CB1-85C2-20A0ABC50C81@mac.com> <5b8d13220907101915s478ea649l3dbcc07ce1718baa@mail.gmail.com> Message-ID: <4A59CAB6.7090302@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > On Sat, Jul 11, 2009 at 12:19 AM, Tommy Grav wrote: > >> The current dmg on the numpy download pages is buildt against 2.5. Is >> there any plans >> to make one for 2.6 or do I have to compile from the source? >> A 2.6 dmg is now available on sourceforge. David From ralf.gommers at googlemail.com Sun Jul 12 11:54:07 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 12 Jul 2009 11:54:07 -0400 Subject: [Numpy-discussion] find_common_type broken? In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDD@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDD@sernt14.essex.ac.uk> Message-ID: On Sun, Jul 12, 2009 at 6:54 AM, Citi, Luca wrote: > Hi, > I am not very confident with types but I will try to give you > my opinion. > > As for the first part of the question > > > >>> np.find_common_type([np.ndarray, np.ma.MaskedArray, np.recarray], []) > > dtype('object') > > I think that the first argument of np.find_common_type should be > the type of an _element_ of the array, not the type of the array. > In your case you are asking np.find_common_type the common type > between an array of arrays, an array of masked arrays, and > an array of record arrays. Therefore the best thing np can do > is to find object as common type. That is what I thought at first, but then what is the difference between array_types and scalar_types? Function signature is: *find_common_type(array_types, scalar_types)* np.float64, np.int32 etc are scalar types so I thought they should go in the second argument. Maybe something else is supposed to go into the array_types list, but I have no clue what if not actual array types. > > > Correctly: > > >>> np.find_common_type([np.float64, np.int32], []) > dtype('float64') > > > > As for the second part of the question np.find_common_type > internally uses np.dtype(t) for each type in input. > > While the comparison between types work as expected: > > >>> np.complex128 > np.complex64 > np.float64 > np.float32 > np.int64 > > np.int32 > True > yes, this makes sense. > > the comparison between dtype(t) gives different results: > > >>> np.dtype(np.float64) > np.dtype(np.int64) > True > >>> np.dtype(np.float32) > np.dtype(np.int64) > False > >>> np.dtype(np.float32) > np.dtype(np.int32) > False > >>> np.dtype(np.float32) > np.dtype(np.int16) > True > > At first I thought the comparison was made based on the > number of bits in the mantissa or the highest integer > N for which N-1 was still representable. > But then I could not explain the first result. > > What is surprising is that > > >>> np.dtype(np.float32) > np.dtype(np.int32) > False > >>> np.dtype(np.float32) < np.dtype(np.int32) > False > >>> np.dtype(np.float32) == np.dtype(np.int32) > False that is confusing. so I guess the dtype(t) conversion should not happen? > > > therefore the max() function in np.find_common_type > cannot tell which to return, and returns the first. > > In fact: > > >>> np.find_common_type([], [np.int64, np.float32]) > dtype('int64') > > but > > >>> np.find_common_type([], [np.float32, np.int64]) > dtype('float32') > > which is unexpected. ah, missed that part. thanks, Luca. Ralf > > > Best, > Luca > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lciti at essex.ac.uk Sun Jul 12 13:24:55 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Sun, 12 Jul 2009 18:24:55 +0100 Subject: [Numpy-discussion] find_common_type broken? References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDD@sernt14.essex.ac.uk> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDF@sernt14.essex.ac.uk> > That is what I thought at first, but then what is the difference between > array_types and scalar_types? Function signature is: > *find_common_type(array_types, scalar_types)* As I understand it, the difference is that in the following case: np.choose(range(5), [np.arange(1,6), np.zeros(5, dtype=np.uint8), 1j*np.arange(5), 22, 1.5]) one should call: find_common_type([np.int64,np.uint8,np.complex128], [int,float]) I had a look at the code and it looks like dtype1 < dtype2 if dtype1 can safely be broadcasted to dtype2 As this is not the case, in either direction, for int32 and float32, then neither dtype(int32) < dtype(float32) nor dtype(int32) > dtype(float32) and this causes the problem you highlighted. I think in this case find_common_type should return float64. The same problem arises with: >>> np.find_common_type([np.int8,np.uint8], []) dtype('int8') >>> np.find_common_type([np.uint8,np.int8], []) dtype('uint8') here too, I think find_common_type should return e third type which is the "smallest" to which both can be safely broadcasted: int16. Best, Luca From pav at iki.fi Mon Jul 13 04:00:41 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 13 Jul 2009 08:00:41 +0000 (UTC) Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) References: Message-ID: Wed, 08 Jul 2009 22:16:22 +0000, Pauli Virtanen kirjoitti: [clip] > On an older CPU (slower, smaller cache), the situation is slightly > different: > > http://www.iki.fi/pav/tmp/athlon.png > http://www.iki.fi/pav/tmp/athlon.txt > > On average, it's still an improvement in many cases. However, now there > are more regressions. The significant ones (factor of 1/2) are N-D > arrays where the reduction runs over an axis with a small number of > elements. Part of this seemed (thanks, Valgrind!) to be because of L2 cache misses, which came from forgetting to evaluate also the first reduction iteration in blocks. Fixed -- the regressions are now less severe (most are ~0.8), although for this machine there are still some... -- Pauli Virtanen From lciti at essex.ac.uk Mon Jul 13 04:28:21 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Mon, 13 Jul 2009 09:28:21 +0100 Subject: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.) References: Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DE1@sernt14.essex.ac.uk> Hi Pauli, in my PC I have tried this and some of the regressions disappear, maybe you can give it a try. At the present state is compiler- and architecture-dependent, therefore not the best choice. But it may be worth trying. Best, Luca /* My additions are unindented */ /* * "Vectorized" reduction along an axis * * Evaluating the inner loop in smaller blocks interleaved with the * reduction loop aims to avoid cache misses in the loop->ret array. */ { typedef unsigned long long ticks; __inline__ ticks getticks(void) { unsigned a, d; /* return clock();*/ /* asm("cpuid");*/ asm volatile("rdtsc" : "=a" (a), "=d" (d)); return (((ticks)a) | (((ticks)d) << 32)); } npy_intp new_block_size; ticks t0, t1; int delta = 8; int speed, speed_p; /*t0 = getticks(); t0 = getticks();*/ t0 = getticks(); speed_p = 0.; block_size = 2 + (loop->bufsize / loop->outsize / 2); new_block_size = block_size; /*printf("was %d", block_size);*/ for (k = 0; k < loop->size; k += block_size) { char *bufptr[3]; block_size = new_block_size; /*printf(" then %d (speed_p %d)", block_size, speed_p);*/ bufptr[0] = loop->bufptr[0] + k * loop->steps[0]; bufptr[1] = loop->bufptr[1] + k * loop->steps[1]; bufptr[2] = loop->bufptr[2] + k * loop->steps[2]; if (k + block_size > loop->size) { block_size = loop->size - k; } for (i = i0; i <= loop->N; ++i) { bufptr[1] += loop->instrides; loop->function((char **)bufptr, &block_size, loop->steps, loop->funcdata); UFUNC_CHECK_ERROR(loop); } t1 = getticks(); speed = (block_size << 12) / (t1 - t0); if (speed < speed_p) delta = -delta; new_block_size = (1 + ((block_size * (128 + delta)) >> 10)) << 3; speed_p = speed; t0 = t1; } /*printf(" is %d (speed_p %d)\n", block_size, speed_p);*/ } PyArray_ITER_NEXT(loop->it); PyArray_ITER_NEXT(loop->rit); } From tsyu80 at gmail.com Mon Jul 13 13:50:50 2009 From: tsyu80 at gmail.com (Tony Yu) Date: Mon, 13 Jul 2009 13:50:50 -0400 Subject: [Numpy-discussion] Using interpolate with zero-rank array raises error In-Reply-To: <51B1E494-7722-400E-A586-A1E0D523823B@mit.edu> References: <51B1E494-7722-400E-A586-A1E0D523823B@mit.edu> Message-ID: <67526143-4424-4E06-BFE5-435091C78878@gmail.com> (Sorry if this is a duplicate; I think sent this from the wrong email the first time) When using interpolate with a zero-rank array, I get "ValueError: object of too small depth for desired array". The following code reproduces this issue >>> import numpy as np >>> x0 = np.array(0.1) >>> x = np.linspace(0, 1) >>> y = np.linspace(0, 1) >>> np.interp(x0, x, y) I hesitate to call this behavior a bug (when I've done this in the past, I find out I'm just doing something wrong), but I find the error unnecessary (and the error output a bit obscure). Below is a (poorly tested) fix, which seems to work for me. (Sorry I'm not on svn, so it isn't a proper diff) Cheers, -Tony Python 2.5.1 numpy 1.3.0 numpy/lib/function_base.py: line 1144 ============================= if isinstance(x, (float, int, number)): return compiled_interp([x], xp, fp, left, right).item() + elif isinstance(x, np.ndarray) and x.ndim == 0: + return compiled_interp(x[np.newaxis], xp, fp, left, right)[0] else: return compiled_interp(x, xp, fp, left, right) ============================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Jul 13 14:54:29 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 13 Jul 2009 14:54:29 -0400 Subject: [Numpy-discussion] find_common_type broken? In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDF@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDD@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDF@sernt14.essex.ac.uk> Message-ID: On Sun, Jul 12, 2009 at 1:24 PM, Citi, Luca wrote: > > That is what I thought at first, but then what is the difference between > > array_types and scalar_types? Function signature is: > > *find_common_type(array_types, scalar_types)* > As I understand it, the difference is that in the following case: > np.choose(range(5), [np.arange(1,6), np.zeros(5, dtype=np.uint8), > 1j*np.arange(5), 22, 1.5]) > one should call: > find_common_type([np.int64,np.uint8,np.complex128], [int,float]) > > I had a look at the code and it looks like > dtype1 < dtype2 if dtype1 can safely be broadcasted to dtype2 > > As this is not the case, in either direction, for int32 and float32, > then neither dtype(int32) < dtype(float32) nor dtype(int32) > > dtype(float32) > and this causes the problem you highlighted. > > I think in this case find_common_type should return float64. > The same problem arises with: > > >>> np.find_common_type([np.int8,np.uint8], []) > dtype('int8') > >>> np.find_common_type([np.uint8,np.int8], []) > dtype('uint8') > > here too, I think find_common_type should return e third type > which is the "smallest" to which both can be safely > broadcasted: int16. > > Best, > Luca > find_common_type() was added after a problem with r_ was reported in ticket 728. r_ still has a problem as well: >>> np.r_[1+1e-10, np.arange(2, dtype=np.float32)] - 1 array([ 0., -1., 0.], dtype=float32) I summarized this discussion on the ticket and reopened it. Cheers, Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsdale24 at gmail.com Mon Jul 13 17:18:27 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 13 Jul 2009 17:18:27 -0400 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: Message-ID: I've put together a first cut at implementing __array_prepare__, which appears to work, and I would like to request feedback. Here is an overview of the approach: Once the ufunc machinery has created the output arrays, it is time to offer subclasses a chance to initialize the output arrays and determine metadata and/or perform whatever other operations may be desired before the ufunc actually performs the computation. In the construct_arrays function in umath.c, I added a function _find_array_prepare, which attempts to find an __array_prepare__ method to call from the inputs, almost identical to the existing _find_array_wrap. The default implementation of __array_prepare__ is currently identical to the default implementation of __array_wrap__, in methods.c. I think that bit of code fits better in __array_prepare__, but maybe there is a good reason to keep it in __array_wrap__ and make the default __array_prepare__ just pass through the output array. So now that the output arrays have been created by the ufunc machinery, and those arrays have been initialized by __array_prepare__ (which has the same call signature as __array_wrap__), the ufunc can continue as usual. Classes that already rely on __array_wrap__ can continue to do so, implementing __array_prepare__ is entirely optional. But other classes like MA and Quantity can set the output array type, determine a mask, perform units analysis checks, and update some metadata in advance of the ufunc, and they can still update metadata after the ufunc using __array_wrap__. The implementation is included in the attached patch. I ran np.test and got 1 known failure and 11 skipped tests. I am using a quantities branch for testing (bzr branch lp:~dsdale24/python-quantities/quantities-array-prepare), and after simply moving my units analysis out of __array_wrap__ and into __array_prepare__, quantities.test() does not yield any errors. Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: array_prepare.patch Type: application/octet-stream Size: 8184 bytes Desc: not available URL: From stefan at sun.ac.za Mon Jul 13 18:09:50 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 14 Jul 2009 00:09:50 +0200 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: Message-ID: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> Hi Darren 2009/7/13 Darren Dale : > I've put together a first cut at implementing __array_prepare__, which > appears to work, and I would like to request feedback. Here is an overview > of the approach: This is pretty neat! Do you have a quick snippet at hand illustrating its use? Regards St?fan From geometrian at gmail.com Mon Jul 13 19:11:13 2009 From: geometrian at gmail.com (Ian Mallett) Date: Mon, 13 Jul 2009 16:11:13 -0700 Subject: [Numpy-discussion] Optimizing a pure Python Workaround Message-ID: Hello, I have some code that makes vertex buffer object terrain. Because the setup for this (a series of triangle strips) is a bit clunky, I just implemented the tricky parts in Python. The code works, but it's slow. How should I go about optimizing it? Thanks, Ian size = [#int_something,#int_something_else] fsize = map(float,size) array = [] for x in xrange(size[0]): for y in xrange(size[1]): array.append(np.array([ x /fsize[0],y/fsize[1],0.0],"f")) array.append(np.array([(x+1)/fsize[0],y/fsize[1],0.0],"f")) array = np.array(array) #"array" is the final product. I need this final product generated more quickly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsdale24 at gmail.com Mon Jul 13 19:12:34 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 13 Jul 2009 19:12:34 -0400 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> Message-ID: 2009/7/13 St?fan van der Walt > Hi Darren > > 2009/7/13 Darren Dale : > > I've put together a first cut at implementing __array_prepare__, which > > appears to work, and I would like to request feedback. Here is an > overview > > of the approach: > > This is pretty neat! Do you have a quick snippet at hand illustrating its > use? That would be helpful, wouldn't it? The attached script is a modified version of RealisticInfoArray from http://docs.scipy.org/doc/numpy/user/basics.subclassing.html . It should yield the following output: starting with [0 1 2 3 4] which is of type and has info attribute = "information" subtracting 3 from [0 1 2 3 4] subtract calling __array_prepare__ on [0 1 2 3 4] input output array is now of type output array values are still uninitialized: [139911601789568 39578752 139911614885536 39254560 48] __array_prepare__ is updating info attribute on output __array_prepare__ finished, subtract ufunc is taking over subtract calling __array_wrap__ on [0 1 2 3 4] input output array has initial value: [-3 -2 -1 0 1] __array_wrap__ is setting output endpoints to 0 yielding [ 0 -2 -1 0 0] which is of type and has info attribute = "new_information" Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: array_prepare.py Type: text/x-python Size: 1836 bytes Desc: not available URL: From robert.kern at gmail.com Tue Jul 14 00:26:01 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 13 Jul 2009 23:26:01 -0500 Subject: [Numpy-discussion] Optimizing a pure Python Workaround In-Reply-To: References: Message-ID: <3d375d730907132126g4ff5361bge8072890e26973c1@mail.gmail.com> On Mon, Jul 13, 2009 at 18:11, Ian Mallett wrote: > Hello, > > I have some code that makes vertex buffer object terrain.? Because the setup > for this (a series of triangle strips) is a bit clunky, I just implemented > the tricky parts in Python. > > The code works, but it's slow.? How should I go about optimizing it? > > Thanks, > > Ian > > size = [#int_something,#int_something_else] > fsize = map(float,size) > array = [] > for x in xrange(size[0]): > ??? for y in xrange(size[1]): > ??????? array.append(np.array([ x?? /fsize[0],y/fsize[1],0.0],"f")) > ??????? array.append(np.array([(x+1)/fsize[0],y/fsize[1],0.0],"f")) > array = np.array(array) > #"array" is the final product.? I need this final product generated more > quickly. Untested: x = np.arange(size[0]) x2 = np.column_stack([x, x+1]).reshape([-1,1]) x3 = x2.repeat(size[1], axis=1).flatten() y = np.arange(size[1]).repeat(size[0] * 2) array = np.zeros([len(y), 3]) array[:,0] = x3 array[:,1] = y -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dave.hirschfeld at gmail.com Tue Jul 14 09:16:31 2009 From: dave.hirschfeld at gmail.com (Dave) Date: Tue, 14 Jul 2009 13:16:31 +0000 (UTC) Subject: [Numpy-discussion] Integer Overflow? Message-ID: I got stung when taking an ordinary python integer to the power of a numpy integer - the result wasn't what I was expecting (see below)! Taking a wild guess I expect this is due to integer overflow (since it doesn't show up with int64). When working with an int32 type one has to be aware of such issues and I'm guessing the casting behaviour is a design decision, nevertheless I thought I'd post to make others aware of such considerations. HTH, Dave Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] Type "copyright", "credits" or "license" for more information. IPython 0.10.bzr.r1163 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [2]: import numpy as np In [3]: np.__version__ Out[3]: '1.4.0.dev7056' In [4]: 10**-10 Out[4]: 1e-010 In [5]: 10**np.float32(-10) Out[5]: 1e-010 In [6]: 10**np.int32(-10) Out[6]: 7.0918695992859933e-010 ???????????????????????????????????? In [7]: 10**np.int32(-1) Out[7]: 0.10000000000000001 In [8]: 10**np.int32(-2) Out[8]: 0.01 In [9]: 10**np.int32(-3) Out[9]: 0.001 In [10]: 10**np.int32(-4) Out[10]: 0.0001 In [11]: 10**np.int32(-5) Out[11]: 1.0000000000000001e-005 In [12]: 10**np.int32(-6) Out[12]: 9.9999999999999995e-007 In [13]: 10**np.int32(-7) Out[13]: 9.9999999999999995e-008 In [14]: 10**np.int32(-8) Out[14]: 1e-008 In [15]: 10**np.int32(-9) Out[15]: 1.0000000000000001e-009 From jed.ludlow at gmail.com Tue Jul 14 10:42:56 2009 From: jed.ludlow at gmail.com (Jed Ludlow) Date: Tue, 14 Jul 2009 08:42:56 -0600 Subject: [Numpy-discussion] Integer Overflow? In-Reply-To: References: Message-ID: Dave wrote: > I got stung when taking an ordinary python integer to the power of a numpy > integer - the result wasn't what I was expecting (see below)! From the results below, it seems to be okay if the base is a long. Note the type of the returned result in each case. Does it seem inconsistent to return a numpy.float64 in one instance and a pure python float in another? Jed Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. Option: import all optional modules >>> import numpy as np >>> np.__version__ '1.3.0' >>> out = 10**-10 >>> out.__class__ >>> out 1e-010 >>> out = 10**np.int32(-10) >>> out.__class__ >>> out 7.0918695992859933e-010 >>> out = 10L**np.int32(-10) >>> out.__class__ >>> out 1e-010 >>> From pgmdevlist at gmail.com Tue Jul 14 14:45:11 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 14 Jul 2009 14:45:11 -0400 Subject: [Numpy-discussion] np.isfinite on structured arrays Message-ID: <102AE848-878B-410F-B810-3FC91BB47EF6@gmail.com> All, Consider the following code: >>> a = np.array(zip(np.arange(3)),dtype=[('a',float)]) >>> np.isfinite(a) NotImplemented That is, when the input is a structured array, np.isfinite returns an object of type NotImplementedType. I would have expected it to raise a NotImplementedError exception. Can anybody cast some light on this issue ? Thanks a lot in advance P. From kwgoodman at gmail.com Tue Jul 14 15:07:55 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 14 Jul 2009 12:07:55 -0700 Subject: [Numpy-discussion] performance matrix multiplication vs. matlab In-Reply-To: <1244368329.4377.18.camel@gabriel-desktop> References: <20090604143638.xrqzx8w8v4ggckkk@www.sms.ed.ac.uk> <4A28F16C.3070607@ar.media.kyoto-u.ac.jp> <4A2955F5.9030006@hawaii.edu> <4A295A7D.1040407@ar.media.kyoto-u.ac.jp> <7f014ea60906051437p6002c6f7jb2d77374a220d2aa@mail.gmail.com> <7f014ea60906060959s6570cc32l277c5ab423f0b9ed@mail.gmail.com> <1244362840.4377.10.camel@gabriel-desktop> <1244368329.4377.18.camel@gabriel-desktop> Message-ID: On Sun, Jun 7, 2009 at 2:52 AM, Gabriel Beckers wrote: > OK, perhaps I drank that beer too soon... > > Now, numpy.test() hangs at: > > test_pinv (test_defmatrix.TestProperties) ... > > So perhaps something is wrong with ATLAS, even though the building went > fine, and "make check" and "make ptcheck" reported no errors. I ran into the same problem on 32-bit debian squeeze. From gfiske at whrc.org Tue Jul 14 15:33:54 2009 From: gfiske at whrc.org (Greg Fiske) Date: Tue, 14 Jul 2009 15:33:54 -0400 Subject: [Numpy-discussion] mean of two or more arrays while ignoring a specific value Message-ID: <00a401ca04ba$06493690$12dba3b0$@org> Dear list, I'm learning to work with numpy arrays. Can somebody explain how to get the average of two separate arrays while ignoring a user defined value in one array? For example: >>>a = numpy.array([1,5,4,99]) >>>b = numpy.array([3,7,2,8]) Ignoring the value 99, the result should be an array like c= ([2,6,3,8]) Thanks for any advice, Greg Greg Fiske Research Associate/GIS Manager The Woods Hole Research Center 149 Woods Hole Road Falmouth MA, 02540 508-540-9900x139 gfiske at whrc.org http://whrc.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Tue Jul 14 15:36:11 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 14 Jul 2009 15:36:11 -0400 Subject: [Numpy-discussion] mean of two or more arrays while ignoring a specific value In-Reply-To: <00a401ca04ba$06493690$12dba3b0$@org> References: <00a401ca04ba$06493690$12dba3b0$@org> Message-ID: <7f014ea60907141236q1bdd4688p9044ecd690df63c@mail.gmail.com> index with a boolean array? >>> import numpy as np >>> a = np.array([3, 3, 3, 4, 4, 4]) >>> a array([3, 3, 3, 4, 4, 4]) >>> np.average(a) 3.5 >>> b = a != 3 >>> b array([False, False, False, True, True, True], dtype=bool) >>> np.average(a[b]) 4.0 >>> On Tue, Jul 14, 2009 at 3:33 PM, Greg Fiske wrote: > Dear list, > > > > I?m learning to work with numpy arrays. ?Can somebody explain how to get the > average of two separate arrays while ignoring a user defined value in one > array? > > > > For example: > >>>>a = numpy.array([1,5,4,99]) > >>>>b = numpy.array([3,7,2,8]) > > > > Ignoring the value 99, the result should be an array like c= ([2,6,3,8]) > > > > Thanks for any advice, > > > > Greg > > > > > > Greg Fiske > > Research Associate/GIS Manager > > The Woods Hole Research Center > > 149 Woods Hole Road > > Falmouth MA, 02540 > > 508-540-9900x139 > > gfiske at whrc.org > > http://whrc.org > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From sccolbert at gmail.com Tue Jul 14 15:42:54 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 14 Jul 2009 15:42:54 -0400 Subject: [Numpy-discussion] mean of two or more arrays while ignoring a specific value In-Reply-To: <7f014ea60907141236q1bdd4688p9044ecd690df63c@mail.gmail.com> References: <00a401ca04ba$06493690$12dba3b0$@org> <7f014ea60907141236q1bdd4688p9044ecd690df63c@mail.gmail.com> Message-ID: <7f014ea60907141242x2565cef5s7ac6cde9358d4a05@mail.gmail.com> for your particular case: >>> a = np.array([1, 5, 4, 99], 'f') >>> b = np.array([3, 7, 2, 8], 'f') >>> c = b.copy() >>> d = a!=99 >>> c[d] = (a[d] + b[d])/2. >>> c array([ 2., 6., 3., 8.], dtype=float32) >>> On Tue, Jul 14, 2009 at 3:36 PM, Chris Colbert wrote: > index with a boolean array? > >>>> import numpy as np >>>> a = np.array([3, 3, 3, 4, 4, 4]) >>>> a > array([3, 3, 3, 4, 4, 4]) >>>> np.average(a) > 3.5 >>>> b = a != 3 >>>> b > array([False, False, False, ?True, ?True, ?True], dtype=bool) >>>> np.average(a[b]) > 4.0 >>>> > > > On Tue, Jul 14, 2009 at 3:33 PM, Greg Fiske wrote: >> Dear list, >> >> >> >> I?m learning to work with numpy arrays. ?Can somebody explain how to get the >> average of two separate arrays while ignoring a user defined value in one >> array? >> >> >> >> For example: >> >>>>>a = numpy.array([1,5,4,99]) >> >>>>>b = numpy.array([3,7,2,8]) >> >> >> >> Ignoring the value 99, the result should be an array like c= ([2,6,3,8]) >> >> >> >> Thanks for any advice, >> >> >> >> Greg >> >> >> >> >> >> Greg Fiske >> >> Research Associate/GIS Manager >> >> The Woods Hole Research Center >> >> 149 Woods Hole Road >> >> Falmouth MA, 02540 >> >> 508-540-9900x139 >> >> gfiske at whrc.org >> >> http://whrc.org >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From amcmorl at gmail.com Tue Jul 14 15:47:48 2009 From: amcmorl at gmail.com (Angus McMorland) Date: Tue, 14 Jul 2009 15:47:48 -0400 Subject: [Numpy-discussion] mean of two or more arrays while ignoring a specific value In-Reply-To: <00a401ca04ba$06493690$12dba3b0$@org> References: <00a401ca04ba$06493690$12dba3b0$@org> Message-ID: 2009/7/14 Greg Fiske : > Dear list, > > I?m learning to work with numpy arrays. ?Can somebody explain how to get the > average of two separate arrays while ignoring a user defined value in one > array? > > For example: > >>>>a = numpy.array([1,5,4,99]) >>>>b = numpy.array([3,7,2,8]) > > Ignoring the value 99, the result should be an array like c= ([2,6,3,8]) Using scipy as well, you could use nans and the nanmean function: import scipy.stats both = np.vstack((a, b)).astype(float) both[both == 99] = np.nan scipy.stats.nanmean(both, axis=0) -> array([ 2., 6., 3., 8.]) -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From robert.kern at gmail.com Tue Jul 14 15:47:53 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 14 Jul 2009 14:47:53 -0500 Subject: [Numpy-discussion] mean of two or more arrays while ignoring a specific value In-Reply-To: <7f014ea60907141242x2565cef5s7ac6cde9358d4a05@mail.gmail.com> References: <00a401ca04ba$06493690$12dba3b0$@org> <7f014ea60907141236q1bdd4688p9044ecd690df63c@mail.gmail.com> <7f014ea60907141242x2565cef5s7ac6cde9358d4a05@mail.gmail.com> Message-ID: <3d375d730907141247l17a97dadq6ced9450aaffa180@mail.gmail.com> On Tue, Jul 14, 2009 at 14:42, Chris Colbert wrote: > for your particular case: > >>>> a = np.array([1, 5, 4, 99], 'f') >>>> b = np.array([3, 7, 2, 8], 'f') >>>> c = b.copy() >>>> d = a!=99 >>>> c[d] = (a[d] + b[d])/2. >>>> c > array([ 2., ?6., ?3., ?8.], dtype=float32) >>>> A more general answer is to use masked arrays. In [5]: a = np.array([1, 5, 4, 99], 'f') In [6]: b = np.array([3, 7, 2, 8], 'f') In [7]: c = np.vstack([a,b]) In [8]: d = np.ma.masked_equal(c, 99.0) In [9]: d Out[9]: 8 masked_array(data = [[1.0 5.0 4.0 --] [3.0 7.0 2.0 8.0]], mask = [[False False False True] [False False False False]], fill_value = 1e+20) In [10]: d.mean(axis=0) Out[10]: 4 masked_array(data = [2.0 6.0 3.0 8.0], mask = [False False False False], fill_value = 1e+20) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dwf at cs.toronto.edu Tue Jul 14 15:59:01 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Tue, 14 Jul 2009 15:59:01 -0400 Subject: [Numpy-discussion] mean of two or more arrays while ignoring a specific value In-Reply-To: <00a401ca04ba$06493690$12dba3b0$@org> References: <00a401ca04ba$06493690$12dba3b0$@org> Message-ID: <9DC0E878-AD70-4299-B18A-DDAA8EB43F26@cs.toronto.edu> On 14-Jul-09, at 3:33 PM, Greg Fiske wrote: > Dear list, > > > > I'm learning to work with numpy arrays. Can somebody explain how to > get the > average of two separate arrays while ignoring a user defined value > in one > array? > > > > For example: > >>>> a = numpy.array([1,5,4,99]) > >>>> b = numpy.array([3,7,2,8]) > > > > Ignoring the value 99, the result should be an array like c= > ([2,6,3,8]) The simplest way I can think of doing it: In [60]: c = concatenate((a[newaxis,:], b[newaxis,:]),axis=0) In [61]: c = where(c == 99, nan, float64(c)) In [62]: nansum(c,axis=0) / (~isnan(c)).sum(axis=0) Out[62]: array([ 2., 6., 3., 8.]) The last line could be replaced with scipy.stats.nanmean(c, axis=0) if you have scipy installed. David From pav at iki.fi Wed Jul 15 04:23:12 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Jul 2009 08:23:12 +0000 (UTC) Subject: [Numpy-discussion] np.isfinite on structured arrays References: <102AE848-878B-410F-B810-3FC91BB47EF6@gmail.com> Message-ID: Tue, 14 Jul 2009 14:45:11 -0400, Pierre GM kirjoitti: > Consider the following code: > >>> a = np.array(zip(np.arange(3)),dtype=[('a',float)]) np.isfinite(a) > NotImplemented > > That is, when the input is a structured array, np.isfinite returns an > object of type NotImplementedType. I would have expected it to raise a > NotImplementedError exception. > > Can anybody cast some light on this issue ? Thanks a lot in advance Seems like a bug. As I understand, NotImplemented is intended to be returned only from __lt__ etc. comparison methods. -- Pauli Virtanen From gfiske at whrc.org Wed Jul 15 09:40:29 2009 From: gfiske at whrc.org (Greg Fiske) Date: Wed, 15 Jul 2009 09:40:29 -0400 Subject: [Numpy-discussion] mean of two or more arrays while ignoring a specific value Message-ID: <003901ca0551$d26673e0$77335ba0$@org> Dear list, I'm learning to work with numpy arrays. Can somebody explain how to get the average of two separate arrays while ignoring a user defined value in one array? For example: >>>a = numpy.array([1,5,4,99]) >>>b = numpy.array([3,7,2,8]) Ignoring the value 99, the result should be an array like c= ([2,6,3,8]) Thanks for any advice, Greg Greg Fiske Research Associate/GIS Manager The Woods Hole Research Center 149 Woods Hole Road Falmouth MA, 02540 508-540-9900x139 gfiske at whrc.org http://whrc.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Wed Jul 15 10:05:01 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 15 Jul 2009 10:05:01 -0400 Subject: [Numpy-discussion] mean of two or more arrays while ignoring a specific value In-Reply-To: <003901ca0551$d26673e0$77335ba0$@org> References: <003901ca0551$d26673e0$77335ba0$@org> Message-ID: <57E7494D-FFCD-4B92-97B0-EA73C7D31E77@yale.edu> Might want to look into masked arrays: numpy.ma.array. a = numpy.array([1,5,4,99]) b = numpy.array([3,7,2,8]) arr = numpy.array([a, b]) masked = numpy.ma.array(arr, mask = arr==99) masked.mean(axis=0) masked_array(data = [2.0 6.0 3.0 8.0], mask = [False False False False], fill_value = 1e+20) Zach On Jul 15, 2009, at 9:40 AM, Greg Fiske wrote: > Dear list, > > I?m learning to work with numpy arrays. Can somebody explain how to > get the average of two separate arrays while ignoring a user defined > value in one array? > > For example: > >>>a = numpy.array([1,5,4,99]) > >>>b = numpy.array([3,7,2,8]) > > Ignoring the value 99, the result should be an array like c= > ([2,6,3,8]) > > Thanks for any advice, > > Greg > > > Greg Fiske > Research Associate/GIS Manager > The Woods Hole Research Center > 149 Woods Hole Road > Falmouth MA, 02540 > 508-540-9900x139 > gfiske at whrc.org > http://whrc.org > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ndbecker2 at gmail.com Wed Jul 15 10:05:12 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 15 Jul 2009 10:05:12 -0400 Subject: [Numpy-discussion] saving complex vector Message-ID: Simple question. I want to save a complex vector as text in format real_0 imag_0\n real_1 imag_1\n ... I thought to use something like: np.savetxt ('gen_qpsk.txt', (mod_out.real, mod_out.imag), fmt='%g %g\n') I need a way to reinterpret the complex data as an array with 2 columns to make this work. Ideas? From dave.hirschfeld at gmail.com Wed Jul 15 11:43:16 2009 From: dave.hirschfeld at gmail.com (Dave) Date: Wed, 15 Jul 2009 15:43:16 +0000 (UTC) Subject: [Numpy-discussion] saving complex vector References: Message-ID: Neal Becker gmail.com> writes: > > Simple question. I want to save a complex vector as text in format > > real_0 imag_0\n > real_1 imag_1\n > ... > > I thought to use something like: > np.savetxt ('gen_qpsk.txt', (mod_out.real, mod_out.imag), fmt='%g %g\n') > > I need a way to reinterpret the complex data as an array with 2 columns to > make this work. Ideas? > Something like? gen_qpsk = (array([[1,1j]])*np.loadtxt('gen_qpsk.txt')).sum(1) HTH, Dave From robert.kern at gmail.com Wed Jul 15 11:49:03 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Jul 2009 10:49:03 -0500 Subject: [Numpy-discussion] saving complex vector In-Reply-To: References: Message-ID: <3d375d730907150849k72b7a7e7x54178fc99f5056e3@mail.gmail.com> On Wed, Jul 15, 2009 at 09:05, Neal Becker wrote: > Simple question. ?I want to save a complex vector as text in format > > real_0 imag_0\n > real_1 imag_1\n > ... > > I thought to use something like: > np.savetxt ('gen_qpsk.txt', (mod_out.real, mod_out.imag), fmt='%g %g\n') > > I need a way to reinterpret the complex data as an array with 2 columns to > make this work. ?Ideas? np.column_stack([mod_out.real, mod_out.imag]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pav at iki.fi Wed Jul 15 11:50:33 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Jul 2009 15:50:33 +0000 (UTC) Subject: [Numpy-discussion] saving complex vector References: Message-ID: Wed, 15 Jul 2009 10:05:12 -0400, Neal Becker kirjoitti: > Simple question. I want to save a complex vector as text in format > > real_0 imag_0\n > real_1 imag_1\n > ... > > I thought to use something like: > np.savetxt ('gen_qpsk.txt', (mod_out.real, mod_out.imag), fmt='%g %g\n') > > I need a way to reinterpret the complex data as an array with 2 columns > to make this work. Ideas? Try np.c_[mod_out.real, mod_out.imag] From pgmdevlist at gmail.com Wed Jul 15 12:20:22 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 15 Jul 2009 12:20:22 -0400 Subject: [Numpy-discussion] np.isfinite on structured arrays In-Reply-To: References: <102AE848-878B-410F-B810-3FC91BB47EF6@gmail.com> Message-ID: <08CCEB52-4655-4F4A-8167-6209987F437A@gmail.com> On Jul 15, 2009, at 4:23 AM, Pauli Virtanen wrote: > Tue, 14 Jul 2009 14:45:11 -0400, Pierre GM kirjoitti: >> Consider the following code: >>>>> a = np.array(zip(np.arange(3)),dtype=[('a',float)]) np.isfinite(a) >> NotImplemented > > Seems like a bug. As I understand, NotImplemented is intended to be > returned only from __lt__ etc. comparison methods. OK, I filed a ticket (#1175), just in case... From rowen at uw.edu Wed Jul 15 14:05:15 2009 From: rowen at uw.edu (Russell E. Owen) Date: Wed, 15 Jul 2009 11:05:15 -0700 Subject: [Numpy-discussion] record array question Message-ID: Is it straightforward to generate a record array (preferably a standard numpy.ndarray, not the numpy.rec variant) where some named fields contain pairs of numbers, for example: named field pos contains pairs of floats named field rot contains floats Any pointers to relevant documentation would be appreciated. I found a basic intro to record arrays on the scipy web site (which was quite useful for generating simple record arrays), and the numpy book has a bit of info, but I've not found anything very comprehensive. -- Russell From robert.kern at gmail.com Wed Jul 15 14:26:08 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Jul 2009 13:26:08 -0500 Subject: [Numpy-discussion] record array question In-Reply-To: References: Message-ID: <3d375d730907151126g6c43156fu78288271324e5822@mail.gmail.com> On Wed, Jul 15, 2009 at 13:05, Russell E. Owen wrote: > Is it straightforward to generate a record array (preferably a standard > numpy.ndarray, not the numpy.rec variant) where some named fields > contain pairs of numbers, for example: > > named field pos contains pairs of floats > named field rot contains floats dt = np.dtype([('pos', float, [2]), ('rot', float)]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rowen at uw.edu Wed Jul 15 14:26:57 2009 From: rowen at uw.edu (Russell E. Owen) Date: Wed, 15 Jul 2009 11:26:57 -0700 Subject: [Numpy-discussion] record array question References: Message-ID: In article , "Russell E. Owen" wrote: > Is it straightforward to generate a record array (preferably a standard > numpy.ndarray, not the numpy.rec variant) where some named fields > contain pairs of numbers, for example: > > named field pos contains pairs of floats > named field rot contains floats > > Any pointers to relevant documentation would be appreciated. I found a > basic intro to record arrays on the scipy web site (which was quite > useful for generating simple record arrays), and the numpy book has a > bit of info, but I've not found anything very comprehensive. > > -- Russell Never mind. I found it today based on a posting; the kind of array I was interested in is a "structured array" and using that term the docs are easy to find, e.g.: In particular: numpy.zeros(shape, dtype=[("pos", float, (2,)), ("rot", float)]) -- Russell From vebjorn at ljosa.com Wed Jul 15 15:19:29 2009 From: vebjorn at ljosa.com (Vebjorn Ljosa) Date: Wed, 15 Jul 2009 15:19:29 -0400 Subject: [Numpy-discussion] Record arrays, nesting, and assignment Message-ID: Suppose I have a record array where one of the fields is a nested array: >>> from numpy import * >>> desc = dtype([('point', 'i4', 3), ('unimportant', 'S3')]) >>> a = array([((1,2,3), 'foo'), ((7,8,9), 'bar')], dtype=desc) >>> a array([([1, 2, 3], 'foo'), ([7, 8, 9], 'bar')], dtype=[('point', '>> a[0] ([1, 2, 3], 'foo') If I try to assign to a[0]['point'], it only works partially: >>> a[0]['point'] = array([4, 5, 6]) >>> a[0] ([4, 2, 3], 'foo') However, if I assign to a['point'][0], it works: >>> a['point'][0] = array([10, 11, 12]) >>> a[0] ([10, 11, 12], 'foo') I should obviously remember to do it the second way ... but why does the first way not work? Thanks, Vebjorn P.S. In case it matters, numpy.__version__ == '1.2.1'. From robert.kern at gmail.com Wed Jul 15 15:28:40 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Jul 2009 14:28:40 -0500 Subject: [Numpy-discussion] Record arrays, nesting, and assignment In-Reply-To: References: Message-ID: <3d375d730907151228v654039abq21134c64b21103bd@mail.gmail.com> On Wed, Jul 15, 2009 at 14:19, Vebjorn Ljosa wrote: > Suppose I have a record array where one of the fields is a nested array: > > ?>>> from numpy import * > ?>>> desc = dtype([('point', 'i4', 3), ('unimportant', 'S3')]) > ?>>> a = array([((1,2,3), 'foo'), ((7,8,9), 'bar')], dtype=desc) > ?>>> a > array([([1, 2, 3], 'foo'), ([7, 8, 9], 'bar')], > ? ? ?dtype=[('point', ' ?>>> a[0] > ([1, 2, 3], 'foo') > > If I try to assign to a[0]['point'], it only works partially: > > ?>>> a[0]['point'] = array([4, 5, 6]) > ?>>> a[0] > ([4, 2, 3], 'foo') > > However, if I assign to a['point'][0], it works: > > ?>>> a['point'][0] = array([10, 11, 12]) > ?>>> a[0] > ([10, 11, 12], 'foo') > > I should obviously remember to do it the second way ... but why does > the first way not work? Generally, scalars are never views. a[0] pulls out a record scalar. Assigning into that scalar does not affect the original memory. a['point'] creates an array that is a view and assigning to the [0] element of that array modifies the original memory. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From vebjorn at ljosa.com Wed Jul 15 15:31:22 2009 From: vebjorn at ljosa.com (Vebjorn Ljosa) Date: Wed, 15 Jul 2009 15:31:22 -0400 Subject: [Numpy-discussion] Record arrays, nesting, and assignment In-Reply-To: <3d375d730907151228v654039abq21134c64b21103bd@mail.gmail.com> References: <3d375d730907151228v654039abq21134c64b21103bd@mail.gmail.com> Message-ID: <90219E40-ECAF-4829-AEC4-3FE48999DA50@ljosa.com> On Jul 15, 2009, at 15:28, Robert Kern wrote: > > Generally, scalars are never views. a[0] pulls out a record scalar. Thank you, that explains it. I should have noticed that a[0] was just a tuple. Vebjorn From pav+sp at iki.fi Wed Jul 15 15:38:36 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Wed, 15 Jul 2009 19:38:36 +0000 (UTC) Subject: [Numpy-discussion] Record arrays, nesting, and assignment References: <3d375d730907151228v654039abq21134c64b21103bd@mail.gmail.com> Message-ID: On 2009-07-15, Robert Kern wrote: > On Wed, Jul 15, 2009 at 14:19, Vebjorn Ljosa wrote: >> Suppose I have a record array where one of the fields is a nested array: >> >> ?>>> from numpy import * >> ?>>> desc = dtype([('point', 'i4', 3), ('unimportant', 'S3')]) >> ?>>> a = array([((1,2,3), 'foo'), ((7,8,9), 'bar')], dtype=desc) >> ?>>> a >> array([([1, 2, 3], 'foo'), ([7, 8, 9], 'bar')], >> ? ? ?dtype=[('point', '> ?>>> a[0] >> ([1, 2, 3], 'foo') >> >> If I try to assign to a[0]['point'], it only works partially: >> >> ?>>> a[0]['point'] = array([4, 5, 6]) >> ?>>> a[0] >> ([4, 2, 3], 'foo') > [clip] > > Generally, scalars are never views. a[0] pulls out a record scalar. > Assigning into that scalar does not affect the original memory. > a['point'] creates an array that is a view and assigning to the [0] > element of that array modifies the original memory. But then, why does it alter the first element of the sub-array? This seems like a bug... -- Pauli Virtanen From robert.kern at gmail.com Wed Jul 15 15:46:32 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Jul 2009 14:46:32 -0500 Subject: [Numpy-discussion] Record arrays, nesting, and assignment In-Reply-To: References: <3d375d730907151228v654039abq21134c64b21103bd@mail.gmail.com> Message-ID: <3d375d730907151246i2515fb44k6ecd54679a0ee514@mail.gmail.com> On Wed, Jul 15, 2009 at 14:38, Pauli Virtanen wrote: > On 2009-07-15, Robert Kern wrote: >> On Wed, Jul 15, 2009 at 14:19, Vebjorn Ljosa wrote: >>> Suppose I have a record array where one of the fields is a nested array: >>> >>> ?>>> from numpy import * >>> ?>>> desc = dtype([('point', 'i4', 3), ('unimportant', 'S3')]) >>> ?>>> a = array([((1,2,3), 'foo'), ((7,8,9), 'bar')], dtype=desc) >>> ?>>> a >>> array([([1, 2, 3], 'foo'), ([7, 8, 9], 'bar')], >>> ? ? ?dtype=[('point', '>> ?>>> a[0] >>> ([1, 2, 3], 'foo') >>> >>> If I try to assign to a[0]['point'], it only works partially: >>> >>> ?>>> a[0]['point'] = array([4, 5, 6]) >>> ?>>> a[0] >>> ([4, 2, 3], 'foo') >> > [clip] >> >> Generally, scalars are never views. a[0] pulls out a record scalar. >> Assigning into that scalar does not affect the original memory. >> a['point'] creates an array that is a view and assigning to the [0] >> element of that array modifies the original memory. > > But then, why does it alter the first element of the sub-array? > This seems like a bug... Hmm, it's worse than I feared. The scalars are indeed views; it's just that the assignment to an array field is buggy. But then, I don't think the records should be mutable at all. In [18]: dt = dtype([('foo', float, 2), ('bar', int)]) In [19]: a = zeros(3, dt) In [20]: a Out[20]: array([([0.0, 0.0], 0), ([0.0, 0.0], 0), ([0.0, 0.0], 0)], dtype=[('foo', ' Here's an interesting problem, and it might've already been solved by some of you folks that do image processing: Suppose I have an 8-bit integer 2-d array, X, and I want a 256x256 matrix that tells me how many times a pixel value v was horizontally (along second dimension) adjacent to a pixel value b. So, in an array [[1, 2, 1],[1,1,2],[1,0,0]] the upper left 3x3 square would be [[1, 1, 0], [1, 1, 3], [0, 3, 0]] with the rest of the matrix 0. Is there an efficient way to do such a thing with numpy operations? I can't think of a way. David From vebjorn at ljosa.com Wed Jul 15 18:40:54 2009 From: vebjorn at ljosa.com (Vebjorn Ljosa) Date: Wed, 15 Jul 2009 18:40:54 -0400 Subject: [Numpy-discussion] Neighbour-frequency matrix In-Reply-To: <9B9311B9-364D-43CD-9C91-C68683A9BAA6@cs.toronto.edu> References: <9B9311B9-364D-43CD-9C91-C68683A9BAA6@cs.toronto.edu> Message-ID: <0B1EB7ED-8884-4697-9CE8-52C3FFAE6F21@ljosa.com> On Jul 15, 2009, at 17:51, David Warde-Farley wrote: > Suppose I have an 8-bit integer 2-d array, X, and I want a 256x256 > matrix that tells me how many times a pixel value v was horizontally > (along second dimension) adjacent to a pixel value b > Is there an efficient way to do such a thing with numpy operations? I > can't think of a way. I wrote the following cooccurrence matrix function as part of computing Haralick texture features. It's not exactly what you ask for, but close enough that it may give you some ideas. Vebjorn # From https://svnrepos.broad.mit.edu/CellProfiler/trunk/CellProfiler/pyCellProfiler/cellprofiler/cpmath/haralick.py import numpy as np import scipy.ndimage as scind def cooccurrence(quantized_image, labels, scale=3): """Calculates co-occurrence matrices for all the objects in the image. Return an array P of shape (nobjects, nlevels, nlevels) such that P[o, :, :] is the cooccurence matrix for object o. quantized_image -- a numpy array of integer type labels -- a numpy array of integer type scale -- an integer For each object O, the cooccurrence matrix is defined as follows. Given a row number I in the matrix, let A be the set of pixels in O with gray level I, excluding pixels in the rightmost S columns of the image. Let B be the set of pixels in O that are S pixels to the right of a pixel in A. Row I of the cooccurence matrix is the gray-level histogram of the pixels in B. """ nlevels = quantized_image.max() + 1 nobjects = labels.max() image_a = quantized_image[:, :-scale] image_b = quantized_image[:, scale:] labels_ab = labels[:, :-scale] equilabel = ((labels[:, :-scale] == labels[:, scale:]) & (labels[:,:-scale] > 0)) P, bins_P = np.histogramdd([labels_ab[equilabel]-1, image_a[equilabel], image_b[equilabel]], (nobjects, nlevels, nlevels)) pixel_count = fix(scind.sum(equilabel, labels[:,:-scale], np.arange(nobjects)+1)) pixel_count = np.tile(pixel_count[:,np.newaxis,np.newaxis], (1,nlevels,nlevels)) return (P.astype(float) / pixel_count.astype(float), nlevels) def fix(whatever_it_returned): """Convert a result from scipy.ndimage to a numpy array scipy.ndimage has the annoying habit of returning a single, bare value instead of an array if the indexes passed in are of length 1. For instance: scind.maximum(image, labels, [1]) returns a float but scind.maximum(image, labels, [1,2]) returns a list """ if getattr(whatever_it_returned,"__getitem__",False): return np.array(whatever_it_returned) else: return np.array([whatever_it_returned]) From doutriaux1 at llnl.gov Wed Jul 15 19:27:22 2009 From: doutriaux1 at llnl.gov (=?UTF-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=) Date: Wed, 15 Jul 2009 16:27:22 -0700 Subject: [Numpy-discussion] quick C api array creation question Message-ID: <7060CEBE-0211-4088-8AD4-B1C08305FCBE@llnl.gov> Hi, I'm using a C api to create numpy array I create them in my C as follow: np_ow = (PyArrayObject *)PyArray_SimpleNew(4, dims, NPY_DOUBLE); np_osfc = (PyArrayObject *)PyArray_SimpleNew(3, dims, NPY_DOUBLE); np_ospc = (PyArrayObject *)PyArray_SimpleNew(3, dims, NPY_DOUBLE); PyArray_FILLWBYTE(np_ow,0); PyArray_FILLWBYTE(np_osfc,0); PyArray_FILLWBYTE(np_ospc,0); ... return Py_BuildValue("OOO",np_ow,np_osfc,np_ospc); It seems to work and they get back nicely to the python, no problem The only "little" issue I have is that the memory is never released! And quickly my code runs out of memory Any pointer on what I'm doing wrong? Do I need a decref somewhere? C. From dwf at cs.toronto.edu Wed Jul 15 19:37:10 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 15 Jul 2009 19:37:10 -0400 Subject: [Numpy-discussion] Neighbour-frequency matrix In-Reply-To: <0B1EB7ED-8884-4697-9CE8-52C3FFAE6F21@ljosa.com> References: <9B9311B9-364D-43CD-9C91-C68683A9BAA6@cs.toronto.edu> <0B1EB7ED-8884-4697-9CE8-52C3FFAE6F21@ljosa.com> Message-ID: <3638937A-948B-439F-8652-FB0CD2B7DCFD@cs.toronto.edu> On 15-Jul-09, at 6:40 PM, Vebjorn Ljosa wrote: > On Jul 15, 2009, at 17:51, David Warde-Farley wrote: > >> Suppose I have an 8-bit integer 2-d array, X, and I want a 256x256 >> matrix that tells me how many times a pixel value v was horizontally >> (along second dimension) adjacent to a pixel value b > >> Is there an efficient way to do such a thing with numpy operations? I >> can't think of a way. > > I wrote the following cooccurrence matrix function as part of > computing Haralick texture features. It's not exactly what you ask > for, but close enough that it may give you some ideas. The Haralick-style features are exactly what I was thinking about. Thanks for the pointer to that code. :) Just curious - I noticed in the comments that you (or someone) said there's a problem in the original paper's definition. Do you have any idea which feature that concerned? David From doutriaux1 at llnl.gov Wed Jul 15 19:58:56 2009 From: doutriaux1 at llnl.gov (=?UTF-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=) Date: Wed, 15 Jul 2009 16:58:56 -0700 Subject: [Numpy-discussion] quick C api array creation question In-Reply-To: <7060CEBE-0211-4088-8AD4-B1C08305FCBE@llnl.gov> References: <7060CEBE-0211-4088-8AD4-B1C08305FCBE@llnl.gov> Message-ID: <7342E43A-20DC-45E6-B61F-DC542E408C9C@llnl.gov> I think i can answer my own question I need to return Py_BuildValue("(NNN)") C. On Jul 15, 2009, at 4:27 PM, Charles ???? Doutriaux wrote: > Hi, > > I'm using a C api to create numpy array > > I create them in my C as follow: > > np_ow = (PyArrayObject *)PyArray_SimpleNew(4, dims, NPY_DOUBLE); > np_osfc = (PyArrayObject *)PyArray_SimpleNew(3, dims, NPY_DOUBLE); > np_ospc = (PyArrayObject *)PyArray_SimpleNew(3, dims, NPY_DOUBLE); > PyArray_FILLWBYTE(np_ow,0); > PyArray_FILLWBYTE(np_osfc,0); > PyArray_FILLWBYTE(np_ospc,0); > ... > return Py_BuildValue("OOO",np_ow,np_osfc,np_ospc); > > It seems to work and they get back nicely to the python, no problem > > The only "little" issue I have is that the memory is never released! > And quickly my code runs out of memory > > Any pointer on what I'm doing wrong? > > Do I need a decref somewhere? > > C. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://*mail.scipy.org/mailman/listinfo/numpy-discussion > From vebjorn at ljosa.com Wed Jul 15 20:45:59 2009 From: vebjorn at ljosa.com (Vebjorn Ljosa) Date: Wed, 15 Jul 2009 20:45:59 -0400 Subject: [Numpy-discussion] Neighbour-frequency matrix In-Reply-To: <3638937A-948B-439F-8652-FB0CD2B7DCFD@cs.toronto.edu> References: <9B9311B9-364D-43CD-9C91-C68683A9BAA6@cs.toronto.edu> <0B1EB7ED-8884-4697-9CE8-52C3FFAE6F21@ljosa.com> <3638937A-948B-439F-8652-FB0CD2B7DCFD@cs.toronto.edu> Message-ID: On Jul 15, 2009, at 19:37, David Warde-Farley wrote: > > Just curious - I noticed in the comments that you (or someone) said > there's a problem in the original paper's definition. Do you have any > idea which feature that concerned? The comments in our previous (Matlab) version of CellProfiler [1] claims that it's H7 (sum of variance). Unfortunately I don't have any more details than that. Vebjorn [1] https://svnrepos.broad.mit.edu/CellProfiler/trunk/CellProfiler/Modules/MeasureTexture.m From gael.varoquaux at normalesup.org Wed Jul 15 22:13:11 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 16 Jul 2009 04:13:11 +0200 Subject: [Numpy-discussion] [ANN] Announcing the SciPy conference schedule Message-ID: <20090716021311.GA21642@phare.normalesup.org> The SciPy conference committee is pleased to announce the schedule of the conference: http://conference.scipy.org/schedule This year?s program is very rich. In order to limit the number of interesting talks that we had to turn down, we decided to reduce the length of talks. Although this results in many short talks, we hope that it will foster discussions, and give new ideas. Many subjects are covered, both varying technical subject in the scientific computing spectrum, and covering a lot of different research areas. I would personally like to thank the members of the program committee, who spent time reviewing the proposed abstracts and giving the chairs feedback. Fernando Perez and the tutorial presenters are hard at work finishing planning all the details of the two-day tutorial session that will precede the conference. An introduction tutorial track and an advanced tutorial track, both covering various aspect of scientific computing in Python, presented by experts in the field, should help many people getting up to speed on the amazing technology driving this community. The SciPy 2009 program committee * Co-Chair Ga?l Varoquaux, Applied Mathematics and Neuroscience, * Neurospin, CEA - INRIA Saclay (France) * Co-Chair St?fan van der Walt, Applied Mathematics, University of * Stellenbosch (South Africa) * Michael Aivazis, Center for Advanced Computing Research, California * Institute of Technology (USA) * Brian Granger, Physics Department, California Polytechnic State * University, San Luis Obispo (USA) * Aric Hagberg, Theoretical Division, Los Alamos National Laboratory * (USA) * Konrad Hinsen, Centre de Biophysique Mol?culaire, CNRS Orl?ans * (France) * Randall LeVeque, Mathematics, University of Washington, Seattle * (USA) * Travis Oliphant, Enthought (USA) * Prabhu Ramachandran, Department of Aerospace Engineering, IIT * Bombay (India) * Raphael Ritz, International Neuroinformatics Coordinating Facility * (Sweden) * William Stein, Mathematics, University of Washington, Seattle (USA) Conference Chair: Jarrod Millman, Neuroscience Institute, UC Berkeley (USA) From vel.accel at gmail.com Wed Jul 15 23:29:27 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Wed, 15 Jul 2009 23:29:27 -0400 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings Message-ID: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> Hi all, I sure wish I was able to attend this year's event. I'm wondering, and really hoping, if/that the lectures will be recorded and then posted for the whole community's benefit? thanks, ~Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Thu Jul 16 00:39:44 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 15 Jul 2009 23:39:44 -0500 Subject: [Numpy-discussion] find_common_type broken? In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDD@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDF@sernt14.essex.ac.uk> Message-ID: <6E65B91C-9D2B-4652-95C6-84EF33AE8F9F@enthought.com> On Jul 13, 2009, at 1:54 PM, Ralf Gommers wrote: > > > On Sun, Jul 12, 2009 at 1:24 PM, Citi, Luca wrote: > > That is what I thought at first, but then what is the difference > between > > array_types and scalar_types? Function signature is: > > *find_common_type(array_types, scalar_types)* > As I understand it, the difference is that in the following case: > np.choose(range(5), [np.arange(1,6), np.zeros(5, dtype=np.uint8), > 1j*np.arange(5), 22, 1.5]) > one should call: > find_common_type([np.int64,np.uint8,np.complex128], [int,float]) > > I had a look at the code and it looks like > dtype1 < dtype2 if dtype1 can safely be broadcasted to dtype2 > > As this is not the case, in either direction, for int32 and float32, > then neither dtype(int32) < dtype(float32) nor dtype(int32) > > dtype(float32) > and this causes the problem you highlighted. > > I think in this case find_common_type should return float64. > The same problem arises with: > > >>> np.find_common_type([np.int8,np.uint8], []) > dtype('int8') > >>> np.find_common_type([np.uint8,np.int8], []) > dtype('uint8') > > here too, I think find_common_type should return e third type > which is the "smallest" to which both can be safely > broadcasted: int16. > > > Best, > Luca > > > find_common_type() was added after a problem with r_ was reported in > ticket 728. r_ still has a problem as well: > >>> np.r_[1+1e-10, np.arange(2, dtype=np.float32)] - 1 > array([ 0., -1., 0.], dtype=float32) This is not a problem with r_. This is correct behavior. A scalar "float" will not cause an array "float32" to be upcast. Nonetheless, the OP did point out a flaw in find_common_type that has been fixed in r7133. Best regards, -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Jul 16 01:59:56 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 16 Jul 2009 01:59:56 -0400 Subject: [Numpy-discussion] find_common_type broken? In-Reply-To: <6E65B91C-9D2B-4652-95C6-84EF33AE8F9F@enthought.com> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDD@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDF@sernt14.essex.ac.uk> <6E65B91C-9D2B-4652-95C6-84EF33AE8F9F@enthought.com> Message-ID: On Thu, Jul 16, 2009 at 12:39 AM, Travis Oliphant wrote: > > On Jul 13, 2009, at 1:54 PM, Ralf Gommers wrote: > > > > On Sun, Jul 12, 2009 at 1:24 PM, Citi, Luca wrote: > >> > That is what I thought at first, but then what is the difference between >> > array_types and scalar_types? Function signature is: >> > *find_common_type(array_types, scalar_types)* >> As I understand it, the difference is that in the following case: >> np.choose(range(5), [np.arange(1,6), np.zeros(5, dtype=np.uint8), >> 1j*np.arange(5), 22, 1.5]) >> one should call: >> find_common_type([np.int64,np.uint8,np.complex128], [int,float]) >> >> I had a look at the code and it looks like >> dtype1 < dtype2 if dtype1 can safely be broadcasted to dtype2 >> >> As this is not the case, in either direction, for int32 and float32, >> then neither dtype(int32) < dtype(float32) nor dtype(int32) > >> dtype(float32) >> and this causes the problem you highlighted. >> >> I think in this case find_common_type should return float64. >> The same problem arises with: >> >> >>> np.find_common_type([np.int8,np.uint8], []) >> dtype('int8') >> >>> np.find_common_type([np.uint8,np.int8], []) >> dtype('uint8') >> >> here too, I think find_common_type should return e third type >> which is the "smallest" to which both can be safely >> broadcasted: int16. >> > >> Best, >> Luca >> > > > find_common_type() was added after a problem with r_ was reported in ticket > 728. r_ still has a problem as well: > >>> np.r_[1+1e-10, np.arange(2, dtype=np.float32)] - 1 > array([ 0., -1., 0.], dtype=float32) > > > > This is not a problem with r_. This is correct behavior. A scalar > "float" will not cause an array "float32" to be upcast. > This was at first counter-intuitive but I found the reason for it in Guide to Numpy now: "Mixed scalar-array operations use a different set of casting rules that ensure that a scalar cannot upcast an array unless the scalar is of a fundamentally different kind of data (i.e. under a different hierarchy in the data type hierarchy) then the array. This rule enables you to use scalar constants in your code (which as Python types are interpreted accordingly in ufuncs) without worrying about whether the precision of the scalar constant will cause upcasting on your large (small precision) array." Makes sense. > > Nonetheless, the OP did point out a flaw in find_common_type that has been > fixed in r7133. > Great, it works for me now. There is still one rule I do not understand the reason for. Out of curiosity, what is the reason for this: In [16]: can_cast(int32, float32) Out[16]: False In [17]: can_cast(int64, float64) Out[17]: True Thanks, Ralf > Best regards, > > -Travis > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Jul 16 05:19:04 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 16 Jul 2009 18:19:04 +0900 Subject: [Numpy-discussion] Adding new functions to Numpy In-Reply-To: References: Message-ID: <5b8d13220907160219t50bf7cf2iac91a7914f611105@mail.gmail.com> Hi Pauli, On Sat, Jul 4, 2009 at 9:59 PM, Pauli Virtanen wrote: > Hi, > > When you add new functions to Numpy, please include > > ? ?.. versionadded:: 1.4.0 What is the best way to do this in the reference guide directly as well (for C API). For example, I added the function npy_copysign for 1.4.0, documented in the c-api.coremath.rst, but I don't know where to put the versionadded. More generally, I would like to follow our docstring format when documenting C functions, but how can I put the different sections in there (Parameters, Notes, etc...) cheers, David From gael.varoquaux at normalesup.org Thu Jul 16 11:05:29 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 16 Jul 2009 17:05:29 +0200 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> Message-ID: <20090716150529.GK13779@phare.normalesup.org> On Wed, Jul 15, 2009 at 11:29:27PM -0400, Peter Alexander wrote: > I sure wish I was able to attend this year's event. > I'm wondering, and really hoping, if/that the lectures will be recorded > and then posted for the whole community's benefit? The problem is that this requires actually a lot of work, to get something useful out of the recordings. I don't think it will happen :(. Ga?l From ijstokes at crystal.harvard.edu Thu Jul 16 11:20:22 2009 From: ijstokes at crystal.harvard.edu (Ian Stokes-Rees) Date: Thu, 16 Jul 2009 11:20:22 -0400 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <20090716150529.GK13779@phare.normalesup.org> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <20090716150529.GK13779@phare.normalesup.org> Message-ID: <4A5F4536.9010907@crystal.harvard.edu> I too am interested in this option, but for understandable reasons the conference committee aren't able to organize this extra work. I had approached them about it a month ago, but was told some committed volunteer would need to take this on. Since there aren't parallel sessions, a "low-tech" way to do this would be for some conference attendee to volunteer to do this with EVO http://evo.vrvs.org/ and a web-cam. EVO produces files that are about 1 MB/minute (although this is tunable depending on resolution and frame rate), but because they combine multiple streams in a custom format, they can only be played back by EVO Player (re-broadcast is fine). EVO can also do live webcasting, chat, and screen sharing, if it is used by the session presenter. EVO and EVO player are both free and are Java apps that run on any platform (JNLP/web-start based). The trickier part is figuring out how to securely distribute this (or enable access to live webcasts), and to be sure all presenters are OK with their work being recorded and redistributed in this way.\ The good news is that EVO is developed by a team at Caltech where the conference is taking place. Ian -- Ian Stokes-Rees W: http://sbgrid.org ijstokes at crystal.harvard.edu T: +1 617 432-5608 x75 SBGrid, Harvard Medical School F: +1 617 432-5600 From gokhansever at gmail.com Thu Jul 16 11:32:30 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Thu, 16 Jul 2009 10:32:30 -0500 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> Message-ID: <49d6b3500907160832y1c7a1967g57245c34e01f81a1@mail.gmail.com> On Wed, Jul 15, 2009 at 10:29 PM, Peter Alexander wrote: > Hi all, > > I sure wish I was able to attend this year's event. > I'm wondering, and really hoping, if/that the lectures will be recorded and > then posted for the whole community's benefit? > > thanks, > ~Peter > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > There are some interesting non-professionally recorded but quality records from this year's EuroPython09 and SageDays16 http://www.blip.tv/search?q=europython http://wiki.sagemath.org/days16 To me, they seem feasible for the SciPy09, too. With a dvd recording camera or flash-disk embedded recorder might do the job --of course with some volunteers to fix the camera or follow the scene :) I remember they were recording the talks during the AGU (www.agu.org) conference professional setups. However it has been almost two months but no video has ever appeared on their web-site yet. Seemingly, amateur initiatives give more quicker and productive results when it comes to session recording. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Thu Jul 16 11:51:35 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 16 Jul 2009 17:51:35 +0200 Subject: [Numpy-discussion] Neighbour-frequency matrix In-Reply-To: <9B9311B9-364D-43CD-9C91-C68683A9BAA6@cs.toronto.edu> References: <9B9311B9-364D-43CD-9C91-C68683A9BAA6@cs.toronto.edu> Message-ID: <9457e7c80907160851h28a6c6c9t4ab4086797f351a3@mail.gmail.com> 2009/7/15 David Warde-Farley : > Here's an interesting problem, and it might've already been solved by > some of you folks that do image processing: > > Suppose I have an 8-bit integer 2-d array, X, and I want a 256x256 > matrix that tells me how many times a pixel value v was horizontally > (along second dimension) adjacent to a pixel value b. > > So, ?in an array [[1, 2, 1],[1,1,2],[1,0,0]] the upper left 3x3 square > would be [[1, 1, 0], [1, 1, 3], [0, 3, 0]] with the rest of the matrix > 0. > > Is there an efficient way to do such a thing with numpy operations? I > can't think of a way. You can also have a look at the gray-level co-occurrence matrix code here: http://mentat.za.net/cgi-bin/hgwebdir.cgi/greycomatrix/ Regards St?fan From vel.accel at gmail.com Thu Jul 16 12:00:16 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Thu, 16 Jul 2009 12:00:16 -0400 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <49d6b3500907160832y1c7a1967g57245c34e01f81a1@mail.gmail.com> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <49d6b3500907160832y1c7a1967g57245c34e01f81a1@mail.gmail.com> Message-ID: <1e52e0880907160900r25d38002q15961fd41c1f3ae3@mail.gmail.com> On Thu, Jul 16, 2009 at 11:32 AM, G?khan SEVER wrote: > > > On Wed, Jul 15, 2009 at 10:29 PM, Peter Alexander wrote: > >> Hi all, >> >> I sure wish I was able to attend this year's event. >> I'm wondering, and really hoping, if/that the lectures will be recorded >> and then posted for the whole community's benefit? >> >> thanks, >> ~Peter >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > There are some interesting non-professionally recorded but quality records > from this year's EuroPython09 and SageDays16 > > http://www.blip.tv/search?q=europython > http://wiki.sagemath.org/days16 > > To me, they seem feasible for the SciPy09, too. With a dvd recording > camera or flash-disk embedded recorder might do the job --of course with > some volunteers to fix the camera or follow the scene :) > > I remember they were recording the talks during the AGU (www.agu.org) > conference professional setups. However it has been almost two months but no > video has ever appeared on their web-site yet. Seemingly, amateur > initiatives give more quicker and productive results when it comes to > session recording. > > > -- > G?khan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > I really hope someone will take the initiative. This, for me, is the most interesting of programming conferences; including outside the python community. I can't imagine this to be that technically challenging.. I would think. The open source community is all about volunteering and with that being said, I wouldn't expect the conference organizers to necessarily be involved in the cooridination of such an initiative. I almost feel its a duty of someone to pick up the ball and run with it. respectfully, pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From tsyu80 at gmail.com Thu Jul 16 12:07:54 2009 From: tsyu80 at gmail.com (Tony Yu) Date: Thu, 16 Jul 2009 12:07:54 -0400 Subject: [Numpy-discussion] Using interpolate with zero-rank array raises error In-Reply-To: References: Message-ID: <43C34D87-0E30-4A78-9A06-1C84B01C7EE0@gmail.com> Sorry, I don't know if its proper mailing-list-etiquette to bump my own post... Are there any comments on whether this interp error is expected behavior? Thanks, -Tony > Date: Mon, 13 Jul 2009 13:50:50 -0400 > From: Tony Yu > Subject: [Numpy-discussion] Using interpolate with zero-rank array > raises error > To: Discussion of Numerical Python > Message-ID: <67526143-4424-4E06-BFE5-435091C78878 at gmail.com> > Content-Type: text/plain; charset="us-ascii" > > (Sorry if this is a duplicate; I think sent this from the wrong email > the first time) > > When using interpolate with a zero-rank array, I get "ValueError: > object of too small depth for desired array". The following code > reproduces this issue > >>>> import numpy as np >>>> x0 = np.array(0.1) >>>> x = np.linspace(0, 1) >>>> y = np.linspace(0, 1) >>>> np.interp(x0, x, y) > > I hesitate to call this behavior a bug (when I've done this in the > past, I find out I'm just doing something wrong), but I find the error > unnecessary (and the error output a bit obscure). > > Below is a (poorly tested) fix, which seems to work for me. (Sorry I'm > not on svn, so it isn't a proper diff) > > Cheers, > -Tony > > Python 2.5.1 > numpy 1.3.0 > > numpy/lib/function_base.py: line 1144 > ============================= > if isinstance(x, (float, int, number)): > return compiled_interp([x], xp, fp, left, right).item() > + elif isinstance(x, np.ndarray) and x.ndim == 0: > + return compiled_interp(x[np.newaxis], xp, fp, left, right)[0] > else: > return compiled_interp(x, xp, fp, left, right) > ============================= > From ijstokes at crystal.harvard.edu Thu Jul 16 12:27:03 2009 From: ijstokes at crystal.harvard.edu (Ian Stokes-Rees) Date: Thu, 16 Jul 2009 12:27:03 -0400 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <1e52e0880907160900r25d38002q15961fd41c1f3ae3@mail.gmail.com> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <49d6b3500907160832y1c7a1967g57245c34e01f81a1@mail.gmail.com> <1e52e0880907160900r25d38002q15961fd41c1f3ae3@mail.gmail.com> Message-ID: <4A5F54D7.7010702@crystal.harvard.edu> > The open source community is all about volunteering and with that > being said, I wouldn't expect the conference organizers to necessarily > be involved in the cooridination of such an initiative. > > I almost feel its a duty of someone to pick up the ball and run with it. But keep in mind there are costs to running the event, and an associated registration fee. It seems only fair that any webcasting or recording only be made available at some cost to the viewers. I would be happy to pay the full conference + tutorial fee for access to this -- it is the food + hotel + flight that means I cannot attend. I'm sure others are in the same boat these days. Ian From william.ratcliff at gmail.com Thu Jul 16 14:12:39 2009 From: william.ratcliff at gmail.com (william ratcliff) Date: Thu, 16 Jul 2009 14:12:39 -0400 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <4A5F54D7.7010702@crystal.harvard.edu> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <49d6b3500907160832y1c7a1967g57245c34e01f81a1@mail.gmail.com> <1e52e0880907160900r25d38002q15961fd41c1f3ae3@mail.gmail.com> <4A5F54D7.7010702@crystal.harvard.edu> Message-ID: <827183970907161112q3d49ede9mec14adc95e41be58@mail.gmail.com> I think for PyCon, a lot of the seminars are online for free (even for non attendees). Also, when I've run a workshop, I just asked the speakers to sign a release form to authorize distribution of videos, etc. This would be best to be agreed upon in advance. Personally, for me, conferences are more than just the talks.... Cheers, William On Thu, Jul 16, 2009 at 12:27 PM, Ian Stokes-Rees < ijstokes at crystal.harvard.edu> wrote: > > > The open source community is all about volunteering and with that > > being said, I wouldn't expect the conference organizers to necessarily > > be involved in the cooridination of such an initiative. > > > > I almost feel its a duty of someone to pick up the ball and run with it. > > But keep in mind there are costs to running the event, and an associated > registration fee. It seems only fair that any webcasting or recording > only be made available at some cost to the viewers. I would be happy to > pay the full conference + tutorial fee for access to this -- it is the > food + hotel + flight that means I cannot attend. I'm sure others are > in the same boat these days. > > Ian > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pfeldman at verizon.net Thu Jul 16 14:43:55 2009 From: pfeldman at verizon.net (Phillip M. Feldman) Date: Thu, 16 Jul 2009 18:43:55 +0000 (UTC) Subject: [Numpy-discussion] alternative mechanism for initializing an array Message-ID: numpy.array understands V= array([[1,2,3,4],[4,3,2,1]]) but not V= array([1,2,3,4],[4,3,2,1]) It would be more convenient if it could handle either form. From dwf at cs.toronto.edu Thu Jul 16 15:04:32 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 16 Jul 2009 15:04:32 -0400 Subject: [Numpy-discussion] alternative mechanism for initializing an array In-Reply-To: References: Message-ID: <4EF4FE84-FBFF-4D40-ADAC-4037B7E588CF@cs.toronto.edu> On 16-Jul-09, at 2:43 PM, Phillip M. Feldman wrote: > numpy.array understands > > V= array([[1,2,3,4],[4,3,2,1]]) > > but not > > V= array([1,2,3,4],[4,3,2,1]) > > It would be more convenient if it could handle either form. That may be so, but the second argument is dtype. This would break that. David From kwgoodman at gmail.com Thu Jul 16 15:09:03 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 16 Jul 2009 12:09:03 -0700 Subject: [Numpy-discussion] alternative mechanism for initializing an array In-Reply-To: References: Message-ID: On Thu, Jul 16, 2009 at 11:43 AM, Phillip M. Feldman wrote: > numpy.array understands > > V= array([[1,2,3,4],[4,3,2,1]]) > > but not > > V= array([1,2,3,4],[4,3,2,1]) > > It would be more convenient if it could handle either form. You could do something like this: def myarray(*args): return np.array([z for z in args]) or def myarray(*args, **kwargs): return np.array([z for z in args], **kwargs) Then >> myarray([1,2,3], [4,5,6]) array([[1, 2, 3], [4, 5, 6]]) >> myarray([1,2,3], [4,5,6], dtype=np.float64) array([[ 1., 2., 3.], [ 4., 5., 6.]]) From pav+sp at iki.fi Thu Jul 16 15:11:06 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Thu, 16 Jul 2009 19:11:06 +0000 (UTC) Subject: [Numpy-discussion] Adding new functions to Numpy References: <5b8d13220907160219t50bf7cf2iac91a7914f611105@mail.gmail.com> Message-ID: On 2009-07-16, David Cournapeau wrote: > Hi Pauli, > > On Sat, Jul 4, 2009 at 9:59 PM, Pauli Virtanen wrote: >> Hi, >> >> When you add new functions to Numpy, please include >> >> ? ?.. versionadded:: 1.4.0 > > What is the best way to do this in the reference guide directly as > well (for C API). For example, I added the function npy_copysign for > 1.4.0, documented in the c-api.coremath.rst, but I don't know where to > put the versionadded. In the Python docs, the versionadded:: typically is the last item inside the function:: directive. > More generally, I would like to follow our > docstring format when documenting C functions, but how can I put the > different sections in there (Parameters, Notes, etc...) Yeah, currently there's no way to do that. I can probably add numpy*function/cfunction/... etc. directives that allow this. An alternative is to use .. rubric:: Notes to make the headings to appear. -- Pauli Virtanen From dpeterson at enthought.com Thu Jul 16 15:27:54 2009 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 16 Jul 2009 14:27:54 -0500 Subject: [Numpy-discussion] ANNOUNCEMENT: Enthought Tool Suite (ETS) v3.3.0 released Message-ID: <4A5F7F3A.4090101@enthought.com> Hello, I'm pleased to announce that Enthought Tool Suite (ETS) version 3.3.0 has been tagged and released! Please see below for a partial list of changes for this release. PyPi has been updated with the release, including the availability of both Windows binaries (.egg) and source distributions (.tar.gz). A full install of ETS can be done using a command like: easy_install -U "ETS[nonets] == 3.3.0" HOWEVER, it is important to note that there are still package dependencies that are outside the scope of easy_install. Therefore, we recommend that you have the following installed prior to installing ETS: setuptools: minimum version 0.6c9 VTK: minimum version 5.0, recommended 5.2 or later And at least one of: wxPython: minimum version 2.8.7.1 PyQt: minimum version 4.4 For additional installation information, see: https://svn.enthought.com/enthought/wiki/Install What Is ETS? =========== The Enthought Tool Suite (ETS) is a collection of components developed by Enthought and the open-source community, which we use every day to construct scientific applications. It includes a wide variety of components, including: * an extensible application framework * application building blocks * 2-D and 3-D graphics libraries * scientific and math libraries * developer tools The cornerstone on which these tools rest is the Traits package, which provides explicit type declarations in Python; its features include initialization, validation, delegation, notification, and visualization of typed attributes. More information on ETS is available from the development home page: http://code.enthought.com/projects/index.php ETS 3.3.0 is a feature-added update to ETS 3.2.0, including numerous bug-fixes. Some of the notable changes include (sub-projects listed in alphabetical order): Chaco 3.2.0 (July 15, 2009) =========================== Enhancements ------------ * Bounded grids - Horizontal and Vertical grid line start and end points can now be set to a fixed value in data space, or to be the return value of an arbitrary callable. The start and end tick can also be configured via the data_min and data_max traits. * Added dictionary interface to ArrayPlotData * Added a Traits UI view to the ScalesAxis * Added a new chaco.overlays subpackage and a new overlay, the DataBox. * Improved appearance of PlotToolbar * Changed add_default_axis() in the plot_factory module to take an axis class as a keyword argument. * Refactored contour plots into a common base class and implemented proper event handling when their colormaps or data ranges change. * Changed default colormap on img_plot() and contour_plot() method of Plot to be Spectral instead of Jet. * Added two more color palettes to the auto color palette, and created a default_colors module. * Added CandlePlot renderer * Changed Plot Labels to able to display inside the plot area, instead of only on the outside * Added optional title attribute to plot legends * Updated all containers to respect and use the new fixed_preferred_size trait on enable.Component * New Examples: * Interval trait editor as a Chaco example (from Stefan van der Walt) * embedding an interactive Chaco plot inside a VTK RenderWindow using the new Enable VTK backend * lasso tool on an image plot * bounded grid * candle plot Fixes ----- * Fixed call signature of ShowAllTickGenerator.get_ticks() * Plot.title_font is now a delegate to the underlying PlotLabel object (from Chris Colbert) * Fixed mouse event handling bug in RangeSelection (from Stefan van der Walt) * ImagePlots now redraw when their input data changes. * Fixed cached image invalidation in colormapped image plots * Optimized ScatterPlot.map_index() when index_only is True and the index data is sorted * Changed ColormappedScatterPlot to no longer incorrectly apply the fill_alpha to the outline color * Improved date ticking heuristics in chaco.scales subpackage, specifically eliminating the bug where all times between, midnight and 1am would be formatted at too course of a time resolution. * Cleanup of various examples (titles, appearance) * The spectrum.py (audio spectrograph) example now properly closes the audio stream. Enable 3.2.0 (July 15th, 2009) ============================== enthought.enable Enhancements ----------------------------- * Added Slider and Compass widgets * Added an OverlayContainer (almost identical to the one in Chaco) * Added ImageGraphicsContextEnable class so that one can always import a Kiva Image backend-based GraphicsContextEnable * renaming marker_trait to MarkerTrait (the old name is still permitted for backwards compatibility, but should be avoided) * Moved the scatter_markers module from Chaco to Enable, so that Enable components can use MarkerTrait * Added an experimental VTK backend for Enable, along with an example * Changed SVGButtonEditor toggle to draw a SVG under the button SVG instead of drawing a plain box * Added labels for SVGButton * Improving backbuffering performance on the Mac by creating the layer context from the window context instead of from a bitmap. * Adding a "fixed_preferred_size" trait to Components, so that relative size preferences can be expressed amongst different components in a container enthought.enable Fixes ---------------------- * Improved the backend selection to match the Traits UI backend unless ETSConfig.enable_toolkit is explicitly set * Fixed demo_main() in example_support.py so that it doesn't crash IPython * Fixed RGBAColorTrait so it can be used with the null toolkit * Changed the "sys_window" color to be the same as the Traits UI "WindowColor" constant * Fixed backend_cairo's get_text_extent() implementation to match other backends enthought.kiva Enhancements --------------------------- * Added basic gradients to Kiva enthought.kiva Fixes -------------------- * Fixed Python 2.6 datatype errors * Fixed memory leak as reported in ticket 1815 * The macport test is only run on Darwin systems * Removed deprecated calls to old numpy APIs Traits 3.2.0 ============ * Implemented editable_labels attribute in the TabularEditor for enabling editing of the labels (i.e. the first column) * Saving/restoring window positions works with multiple displays of different sizes * New ProgressEditor * Changed default colors for TableEditor * Added support for HTMLEditor for QT backend using QtWebKit * Improved support for opening links in external browser from HTMLEditor * Added support for TabularEditor for QT backend * Added support for marking up the CodeEditor, including adding squiggles and dimming lines * Added SearchEditor * Improved unicode support * Changed behavior of RangeEditor text box to not auto-set * Added support in RangeEditor for specifying the method to evaluate new values. * Add DefaultOverride editor factory courtesy St?fan van der Walt * Removed sys.exit() call from SaveHandler.exit() TraitsBackendQt 3.2.0 (July 15, 2009) ===================================== * Fixed a plethora of layout bugs * Implemented RGBColor trait * Fixed events not fired for 'custom' TextEditor * Improved the method by which the QT backend dispatches trait handlers when dispatch='ui'. Before, the code only worked when on the main application thread or a QThread. Now it works for regular Python threads too. * Fixed events not being fired correctly in TableEditor * Added support or 'click' and 'dclick' factory attributes to the TableEditor * TableEditor instances now editable * Improved FileEditor to look and act like the WX editor * Fixed missing minimize/maximize buttons for resizable dialogs * New HTMLEditor using QtWebKit * New TabularEditor * Added support for panel buttons * New SearchEditor * Added support for clipboard * Now responds to SIGINT correctly rather than raising KeyboardInterrupt TraitsBackendWX 3.2.0 (July 15, 2009) ===================================== * Fixed bug in DateEditor which would not display Feb correctly if the current date was visible and greater than the number of days in Feb * Reduced border_size from 1 to 4 for Group instances * Fixed layout issues: * Windows are now resized if they are larger than the desktop * Windows are now put in a valid location if they were opened off-screen * Windows with smaller parent are no longer opened at invalid positions with negative y values * Implemented editable_labels attribute in the TabularEditor for enabling editing of the labels (i.e. the first column) * Fix bug in ListEditor where a trait change listener would be fired when intermediate traits changed (when extended_name was of the type item1.item2.item3..) leading to a traceback. * Saving/restoring windows now works with multiple displays of different sizes * New ProgressDialog * Improved window colors to match desktop colors more closely * Replaced calls of wx.Yield() with wx.GetApp().Yield(True) * Changed default font to use system font * Fixed TabularEditor compatibility problem with wx 2.6 regarding the page-down key * Fixed bug in propagating click events in the TabularEditor to parent windows * DateEditor wx 2.6 compatability fixed * TableEditor scrollbar fixed * Improved support for opening links in external browser from HTMLEditor * Reduced the number of update events the PythonEditor fired * moved grid package from TraitsGui egg into enthought.pyface.ui.wx * moved clipboard from enthought.util.wx into pyface TraitsGUI 3.1.0 (July 15, 2009) =============================== * Removed Theming support from DockWindows. Borders and tabs are now drawn using lines instead of stretching images. * Changed default font to use the system font * Moved enthought.util.wx.clipboard to Pyface * Moved the grid package out of pyface and into pyface.ui.wx, left deprecated warnings * Improved info shown to the user if toolkits don't work as expected From dwf at cs.toronto.edu Thu Jul 16 16:49:27 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 16 Jul 2009 16:49:27 -0400 Subject: [Numpy-discussion] alternative mechanism for initializing an array In-Reply-To: References: Message-ID: <0D1A7105-9D85-4070-B4F0-2F1B37AE69E1@cs.toronto.edu> On 16-Jul-09, at 3:09 PM, Keith Goodman wrote: > def myarray(*args, **kwargs): > return np.array([z for z in args], **kwargs) This version is better IMHO, because then you can still specify the dtype by keyword. although (just to be a pedant) > return np.array(args, **kwargs) works as well. :) David From kwgoodman at gmail.com Thu Jul 16 16:57:18 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 16 Jul 2009 13:57:18 -0700 Subject: [Numpy-discussion] alternative mechanism for initializing an array In-Reply-To: <0D1A7105-9D85-4070-B4F0-2F1B37AE69E1@cs.toronto.edu> References: <0D1A7105-9D85-4070-B4F0-2F1B37AE69E1@cs.toronto.edu> Message-ID: On Thu, Jul 16, 2009 at 1:49 PM, David Warde-Farley wrote: > On 16-Jul-09, at 3:09 PM, Keith Goodman wrote: > >> def myarray(*args, **kwargs): >> ? ?return np.array([z for z in args], **kwargs) > > > This version is better IMHO, because then you can still specify the > dtype by keyword. although (just to be a pedant) > >> return np.array(args, **kwargs) > > works as well. :) Oh, of course. Nice. From dwf at cs.toronto.edu Thu Jul 16 17:03:25 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 16 Jul 2009 17:03:25 -0400 Subject: [Numpy-discussion] Minor doc crossref problem Message-ID: <4F291D61-EA63-48B9-9FB9-40119AC12A19@cs.toronto.edu> I noticed a few days ago that the cross-reference to 'nonzero' on this: http://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html points to the C API rather than the Python API. I had a quick gander at the Sphinx docs but didn't see anything about how to remedy this. Anyone? Regards, David From dwf at cs.toronto.edu Thu Jul 16 17:05:09 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 16 Jul 2009 17:05:09 -0400 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <20090716150529.GK13779@phare.normalesup.org> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <20090716150529.GK13779@phare.normalesup.org> Message-ID: <864CB368-465F-4A3A-ACF5-68801FACEE6A@cs.toronto.edu> On 16-Jul-09, at 11:05 AM, Gael Varoquaux wrote: > On Wed, Jul 15, 2009 at 11:29:27PM -0400, Peter Alexander wrote: >> I sure wish I was able to attend this year's event. >> I'm wondering, and really hoping, if/that the lectures will be >> recorded >> and then posted for the whole community's benefit? > > The problem is that this requires actually a lot of work, to get > something useful out of the recordings. I don't think it will > happen :(. > > Ga?l I think Travis V. had some recording gear at last year's, but the problem was editing and posting all of these things. David From fons at kokkinizita.net Thu Jul 16 17:05:12 2009 From: fons at kokkinizita.net (Fons Adriaensen) Date: Thu, 16 Jul 2009 23:05:12 +0200 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <4A5F54D7.7010702@crystal.harvard.edu> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <49d6b3500907160832y1c7a1967g57245c34e01f81a1@mail.gmail.com> <1e52e0880907160900r25d38002q15961fd41c1f3ae3@mail.gmail.com> <4A5F54D7.7010702@crystal.harvard.edu> Message-ID: <20090716210512.GB2959@zita2.kokkinizita.net> On Thu, Jul 16, 2009 at 12:27:03PM -0400, Ian Stokes-Rees wrote: > But keep in mind there are costs to running the event, and an associated > registration fee. It seems only fair that any webcasting or recording > only be made available at some cost to the viewers. I would be happy to > pay the full conference + tutorial fee for access to this -- it is the > food + hotel + flight that means I cannot attend. I'm sure others are > in the same boat these days. I've no idea of the size of this event, and a lot would depend on that. I organised this year's Linux Audio Conference in Parma, Italy, and managed to keep it free (as free beer) - there was no registration fee. All presentations were also streamed (audio and video) live without any cost to whoever would want to watch them. Attendants did pay only for optional things such as the concerts and the (very informal) conference dinner. And of course for their own transport and accomodation in Parma. But yes, this depends on a lot of dedication and hard work of a team of volunteers, on some sponsoring, and on an institution willing to host the event because it fits both to their mission and within their schedule. Ciao, -- FA Io lo dico sempre: l'Italia ? troppo stretta e lunga. From fperez.net at gmail.com Thu Jul 16 17:30:39 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 16 Jul 2009 14:30:39 -0700 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <1e52e0880907160900r25d38002q15961fd41c1f3ae3@mail.gmail.com> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <49d6b3500907160832y1c7a1967g57245c34e01f81a1@mail.gmail.com> <1e52e0880907160900r25d38002q15961fd41c1f3ae3@mail.gmail.com> Message-ID: Hi all, On Thu, Jul 16, 2009 at 9:00 AM, Peter Alexander wrote: > I really hope someone will take the initiative. This, for me, is the most > interesting of programming conferences; including outside the python > community. > I can't imagine this to be that technically challenging.. I would think. > > The open source community is all about volunteering and with that being > said, I wouldn't expect the conference organizers to necessarily be involved > in the cooridination of such an initiative. > I think this would be great to have, and for example the 2-days of advanced tutorials would lend themselves well to this. I can offer to bring a sturdy tripod (I'm driving, so weight is not an issue), but I have no video recording equipment of any kind at all. If someone has a camera that can do the recordings in a format that can then be directly recompressed at the command line with something like mencoder, that would be great. From experience, any recording mode that requires later manual editing sounds great in principle, and then nobody finds the time to do all the work. So I'd much rather have something that does: - low tech but acceptable quality, single-file-per-talk recordings. - Upon return we run overnight a shell script to reencode at lower resolution - We plop the lot onto the site unmodified. If anyone offers real fancy video editing support beyond the above it would be great, but I'm willing to settle for 'basic that exists' rather than "hollywood we won't have" :) So let me know if I should bring the sturdy tripod... Cheers, f From gokhansever at gmail.com Thu Jul 16 18:39:39 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Thu, 16 Jul 2009 17:39:39 -0500 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <49d6b3500907160832y1c7a1967g57245c34e01f81a1@mail.gmail.com> <1e52e0880907160900r25d38002q15961fd41c1f3ae3@mail.gmail.com> Message-ID: <49d6b3500907161539w28c7d76bke87711a80f524d8@mail.gmail.com> On Thu, Jul 16, 2009 at 4:30 PM, Fernando Perez wrote: > > If someone has a camera that can do the recordings in a format that > can then be directly recompressed at the command line with something > like mencoder, that would be great. From experience, any recording > mode that requires later manual editing sounds great in principle, and > then nobody finds the time to do all the work. > > There is a one great approach posted here at http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1246925071 Seems like his one man army approach worked very nicely. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arokem at berkeley.edu Thu Jul 16 21:13:22 2009 From: arokem at berkeley.edu (Ariel Rokem) Date: Thu, 16 Jul 2009 18:13:22 -0700 Subject: [Numpy-discussion] ANNOUNCEMENT: Enthought Tool Suite (ETS) v3.3.0 released In-Reply-To: <4A5F7F3A.4090101@enthought.com> References: <4A5F7F3A.4090101@enthought.com> Message-ID: <43958ee60907161813w28620411y124568e7c4fdf78d@mail.gmail.com> Hi - a first for me - upgrading ETS worked - hurray! Just a couple of comments - running: arokem$ easy_install -U "ETS[nonets] == 3.3.0" resulted in the following error: "pkg_resources.UnknownExtra: CodeTools 3.0.0 has no such extra feature 'nonets'" Then, I ran it, calling instead arokem$ easy_install -U "ETS == 3.3.0" withouth "nonets". This resulted in a sandbox violation error - but I kept on running easy_install several times, as above, until it finally did work. I am very excited to play with version 3.3 of mayavi :-) Thanks a lot for all the good work - Ariel On Thu, Jul 16, 2009 at 12:27 PM, Dave Peterson wrote: > Hello, > > I'm pleased to announce that Enthought Tool Suite (ETS) version 3.3.0 > has been tagged and released! Please see below for a partial list of > changes for this release. > > PyPi has been updated with the release, including the availability of > both Windows binaries (.egg) and source distributions (.tar.gz). A full > install of ETS can be done using a command like: > easy_install -U "ETS[nonets] == 3.3.0" > > HOWEVER, it is important to note that there are still package > dependencies that are outside the scope of easy_install. Therefore, we > recommend that you have the following installed prior to installing ETS: > setuptools: minimum version 0.6c9 > VTK: minimum version 5.0, recommended 5.2 or later > And at least one of: > wxPython: minimum version 2.8.7.1 > PyQt: minimum version 4.4 > > For additional installation information, see: > https://svn.enthought.com/enthought/wiki/Install > > > > What Is ETS? > =========== > > The Enthought Tool Suite (ETS) is a collection of components developed > by Enthought and the open-source community, which we use every day to > construct scientific applications. It includes a wide variety of > components, including: > * an extensible application framework > * application building blocks > * 2-D and 3-D graphics libraries > * scientific and math libraries > * developer tools > The cornerstone on which these tools rest is the Traits package, which > provides explicit type declarations in Python; its features include > initialization, validation, delegation, notification, and visualization > of typed attributes. > > More information on ETS is available from the development home page: > http://code.enthought.com/projects/index.php > > ETS 3.3.0 is a feature-added update to ETS 3.2.0, including numerous > bug-fixes. Some of the notable changes include (sub-projects listed in > alphabetical order): > > > Chaco 3.2.0 (July 15, 2009) > =========================== > > Enhancements > ------------ > > * Bounded grids - Horizontal and Vertical grid line start and end > points can now be set to a fixed value in data space, or to be the > return value of an arbitrary callable. The start and end tick can also > be configured via the data_min and data_max traits. > > * Added dictionary interface to ArrayPlotData > > * Added a Traits UI view to the ScalesAxis > > * Added a new chaco.overlays subpackage and a new overlay, the DataBox. > > * Improved appearance of PlotToolbar > > * Changed add_default_axis() in the plot_factory module to take an > axis class as a keyword argument. > > * Refactored contour plots into a common base class and implemented > proper event handling when their colormaps or data ranges change. > > * Changed default colormap on img_plot() and contour_plot() method of > Plot to be Spectral instead of Jet. > > * Added two more color palettes to the auto color palette, and created > a default_colors module. > > * Added CandlePlot renderer > > * Changed Plot Labels to able to display inside the plot area, instead > of only on the outside > > * Added optional title attribute to plot legends > > * Updated all containers to respect and use the new > fixed_preferred_size trait on enable.Component > > * New Examples: > * Interval trait editor as a Chaco example (from Stefan van der Walt) > * embedding an interactive Chaco plot inside a VTK RenderWindow > using the new Enable VTK backend > * lasso tool on an image plot > * bounded grid > * candle plot > > > Fixes > ----- > > * Fixed call signature of ShowAllTickGenerator.get_ticks() > > * Plot.title_font is now a delegate to the underlying PlotLabel object > (from Chris Colbert) > > * Fixed mouse event handling bug in RangeSelection (from Stefan van > der Walt) > > * ImagePlots now redraw when their input data changes. > > * Fixed cached image invalidation in colormapped image plots > > * Optimized ScatterPlot.map_index() when index_only is True and the > index data is sorted > > * Changed ColormappedScatterPlot to no longer incorrectly apply the > fill_alpha to the outline color > > * Improved date ticking heuristics in chaco.scales subpackage, > specifically eliminating the bug where all times between, midnight and > 1am would be formatted at too course of a time resolution. > > * Cleanup of various examples (titles, appearance) > > * The spectrum.py (audio spectrograph) example now properly closes the > audio stream. > > > > Enable 3.2.0 (July 15th, 2009) > ============================== > > enthought.enable Enhancements > ----------------------------- > > * Added Slider and Compass widgets > > * Added an OverlayContainer (almost identical to the one in Chaco) > > * Added ImageGraphicsContextEnable class so that one can always import > a Kiva Image backend-based GraphicsContextEnable > > * renaming marker_trait to MarkerTrait (the old name is still permitted > for backwards compatibility, but should be avoided) > > * Moved the scatter_markers module from Chaco to Enable, so that Enable > components can use MarkerTrait > > * Added an experimental VTK backend for Enable, along with an example > > * Changed SVGButtonEditor toggle to draw a SVG under the button SVG > instead of drawing a plain box > > * Added labels for SVGButton > > * Improving backbuffering performance on the Mac by creating the layer > context from the window context instead of from a bitmap. > > * Adding a "fixed_preferred_size" trait to Components, so that relative > size preferences can be expressed amongst different components in a > container > > > enthought.enable Fixes > ---------------------- > > * Improved the backend selection to match the Traits UI backend unless > ETSConfig.enable_toolkit is explicitly set > > * Fixed demo_main() in example_support.py so that it doesn't crash IPython > > * Fixed RGBAColorTrait so it can be used with the null toolkit > > * Changed the "sys_window" color to be the same as the Traits UI > "WindowColor" constant > > * Fixed backend_cairo's get_text_extent() implementation to match other > backends > > > enthought.kiva Enhancements > --------------------------- > > * Added basic gradients to Kiva > > > enthought.kiva Fixes > -------------------- > > * Fixed Python 2.6 datatype errors > > * Fixed memory leak as reported in ticket 1815 > > * The macport test is only run on Darwin systems > > * Removed deprecated calls to old numpy APIs > > > > Traits 3.2.0 > ============ > > * Implemented editable_labels attribute in the TabularEditor for > enabling editing of the labels (i.e. the first column) * > Saving/restoring window positions works with multiple displays of > different sizes > > * New ProgressEditor > > * Changed default colors for TableEditor > > * Added support for HTMLEditor for QT backend using QtWebKit > > * Improved support for opening links in external browser from HTMLEditor > > * Added support for TabularEditor for QT backend > > * Added support for marking up the CodeEditor, including adding > squiggles and dimming lines > > * Added SearchEditor > > * Improved unicode support > > * Changed behavior of RangeEditor text box to not auto-set > > * Added support in RangeEditor for specifying the method to evaluate > new values. > > * Add DefaultOverride editor factory courtesy St?fan van der Walt > > * Removed sys.exit() call from SaveHandler.exit() > > > TraitsBackendQt 3.2.0 (July 15, 2009) > ===================================== > > * Fixed a plethora of layout bugs > > * Implemented RGBColor trait > > * Fixed events not fired for 'custom' TextEditor > > * Improved the method by which the QT backend dispatches trait handlers > when dispatch='ui'. Before, the code only worked when on the main > application thread or a QThread. Now it works for regular Python threads > too. > > * Fixed events not being fired correctly in TableEditor > > * Added support or 'click' and 'dclick' factory attributes to the > TableEditor > > * TableEditor instances now editable > > * Improved FileEditor to look and act like the WX editor > > * Fixed missing minimize/maximize buttons for resizable dialogs > > * New HTMLEditor using QtWebKit > > * New TabularEditor > > * Added support for panel buttons > > * New SearchEditor > > * Added support for clipboard > > * Now responds to SIGINT correctly rather than raising KeyboardInterrupt > > > > TraitsBackendWX 3.2.0 (July 15, 2009) > ===================================== > > * Fixed bug in DateEditor which would not display Feb correctly if the > current date was visible and greater than the number of days in Feb > > * Reduced border_size from 1 to 4 for Group instances > > * Fixed layout issues: > * Windows are now resized if they are larger than the desktop > * Windows are now put in a valid location if they were opened off-screen > * Windows with smaller parent are no longer opened at invalid > positions with negative y values > > * Implemented editable_labels attribute in the TabularEditor for > enabling editing of the labels (i.e. the first column) * Fix bug in > ListEditor where a trait change listener would be fired when > intermediate traits changed (when extended_name was of the type > item1.item2.item3..) leading to a traceback. * Saving/restoring windows > now works with multiple displays of different sizes > > * New ProgressDialog > > * Improved window colors to match desktop colors more closely > > * Replaced calls of wx.Yield() with wx.GetApp().Yield(True) > > * Changed default font to use system font > > * Fixed TabularEditor compatibility problem with wx 2.6 regarding the > page-down key > > * Fixed bug in propagating click events in the TabularEditor to parent > windows > > * DateEditor wx 2.6 compatability fixed > > * TableEditor scrollbar fixed > > * Improved support for opening links in external browser from HTMLEditor > > * Reduced the number of update events the PythonEditor fired > > * moved grid package from TraitsGui egg into enthought.pyface.ui.wx > > * moved clipboard from enthought.util.wx into pyface > > > > TraitsGUI 3.1.0 (July 15, 2009) > =============================== > > * Removed Theming support from DockWindows. Borders and tabs are now > drawn using lines instead of stretching images. * Changed default font > to use the system font > > * Moved enthought.util.wx.clipboard to Pyface > > * Moved the grid package out of pyface and into pyface.ui.wx, left > deprecated warnings > > * Improved info shown to the user if toolkits don't work as expected > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Ariel Rokem Helen Wills Neuroscience Institute University of California, Berkeley http://argentum.ucbso.berkeley.edu/ariel -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Thu Jul 16 21:15:59 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 17 Jul 2009 03:15:59 +0200 Subject: [Numpy-discussion] ANNOUNCEMENT: Enthought Tool Suite (ETS) v3.3.0 released In-Reply-To: <43958ee60907161813w28620411y124568e7c4fdf78d@mail.gmail.com> References: <4A5F7F3A.4090101@enthought.com> <43958ee60907161813w28620411y124568e7c4fdf78d@mail.gmail.com> Message-ID: <20090717011559.GG17409@phare.normalesup.org> On Thu, Jul 16, 2009 at 06:13:22PM -0700, Ariel Rokem wrote: > a first for me - upgrading ETS worked - hurray! Dave, _thank you_ for doing the work of releasing and building binaries. Ga?l From pfeldman at verizon.net Thu Jul 16 21:34:33 2009 From: pfeldman at verizon.net (Phillip M. Feldman) Date: Thu, 16 Jul 2009 18:34:33 -0700 Subject: [Numpy-discussion] alternative mechanism for initializing an array Message-ID: <4A5FD529.7070403@verizon.net> This does the right thing sometimes, but not always. Out[2] and Out[4] are fine, but Out[3] is not (note the extra set of braces). Probably the only right way to fix this is to modify numpy itself. Phillip In [1]: def myarray(*args, **kwargs): return np.array([z for z in args], **kwargs) ...: In [2]: myarray([1,2,3],[4,5,6]) Out[2]: array([[1, 2, 3], [4, 5, 6]]) In [3]: myarray([1,2,3]) Out[3]: array([[1, 2, 3]]) In [4]: np.array([1,2,3]) Out[4]: array([1, 2, 3]) From: Keith Goodman gmail.com> Subject: Re: alternative mechanism for initializing an array Newsgroups: gmane.comp.python.numeric.general Date: 2009-07-16 20:57:18 GMT (4 hours and 29 minutes ago) On Thu, Jul 16, 2009 at 1:49 PM, David Warde-Farley cs.toronto.edu> wrote: > On 16-Jul-09, at 3:09 PM, Keith Goodman wrote: > >> def myarray(*args, **kwargs): >> return np.array([z for z in args], **kwargs) > > > This version is better IMHO, because then you can still specify the > dtype by keyword. although (just to be a pedant) > >> return np.array(args, **kwargs) > > works as well. :) Oh, of course. Nice. From robert.kern at gmail.com Thu Jul 16 21:37:45 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 16 Jul 2009 20:37:45 -0500 Subject: [Numpy-discussion] alternative mechanism for initializing an array In-Reply-To: <4A5FD529.7070403@verizon.net> References: <4A5FD529.7070403@verizon.net> Message-ID: <3d375d730907161837x6683e5b8n711a25ce1ff1dd2@mail.gmail.com> On Thu, Jul 16, 2009 at 20:34, Phillip M. Feldman wrote: > This does the right thing sometimes, but not always. ?Out[2] and Out[4] > are fine, but Out[3] is not (note the extra set of braces). ?Probably > the only right way to fix this is to modify numpy itself. Modifying numpy.array() would have exactly the same ambiguity. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dwf at cs.toronto.edu Thu Jul 16 23:22:33 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 16 Jul 2009 23:22:33 -0400 Subject: [Numpy-discussion] alternative mechanism for initializing an array In-Reply-To: <4A5FD529.7070403@verizon.net> References: <4A5FD529.7070403@verizon.net> Message-ID: <79AD496E-B1F7-4CCF-A54C-09125CBA8A28@cs.toronto.edu> On 16-Jul-09, at 9:34 PM, Phillip M. Feldman wrote: > This does the right thing sometimes, but not always. Out[2] and > Out[4] > are fine, but Out[3] is not (note the extra set of braces). Probably > the only right way to fix this is to modify numpy itself. yes, to achieve out[3] you'd need myarray(1,2,3). A .squeeze() after it would fix that, but it would also make it impossible to create the array [[[0]]] if you really wanted it for some reason. Basically the current syntax is the only ambiguity-free way I can think of that makes any sense. And NumPy was designed by some very smart people so I doubt I'll be able to top their best idea. :P One idea would be an object like mgrid that creates arrays via __getitem__ so that you could do foo[[1,2,3]] to get [[1,2,3]], foo[1,2,3] to get [1,2,3]... That'd get rid of a set of parens but it'd also be mighty confusing. David From ralf.gommers at googlemail.com Thu Jul 16 23:37:58 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 16 Jul 2009 23:37:58 -0400 Subject: [Numpy-discussion] Using interpolate with zero-rank array raises error In-Reply-To: <43C34D87-0E30-4A78-9A06-1C84B01C7EE0@gmail.com> References: <43C34D87-0E30-4A78-9A06-1C84B01C7EE0@gmail.com> Message-ID: On Thu, Jul 16, 2009 at 12:07 PM, Tony Yu wrote: > Sorry, I don't know if its proper mailing-list-etiquette to bump my > own post... I don't see a problem as long as you wait at least a few days before bumping (which you did). > > > Are there any comments on whether this interp error is expected > behavior? It seems to me that there are quite a few other functions that will give errors with 0-D arrays (apply_along/over_axis are two that come to mind). There is nothing to interpolate so I'm not surprised. Unless there is a policy in Numpy that all funcs should work with 0-D arrays I would expect that it's up to the user to not do things with them they're not intended for. Disclaimer: i don't know much about the reasons for 0-D arrays existing, just what I learned from http://projects.scipy.org/numpy/wiki/ZeroRankArray Cheers, Ralf > > Thanks, > -Tony > > > Date: Mon, 13 Jul 2009 13:50:50 -0400 > > From: Tony Yu > > Subject: [Numpy-discussion] Using interpolate with zero-rank array > > raises error > > To: Discussion of Numerical Python > > Message-ID: <67526143-4424-4E06-BFE5-435091C78878 at gmail.com> > > Content-Type: text/plain; charset="us-ascii" > > > > (Sorry if this is a duplicate; I think sent this from the wrong email > > the first time) > > > > When using interpolate with a zero-rank array, I get "ValueError: > > object of too small depth for desired array". The following code > > reproduces this issue > > > >>>> import numpy as np > >>>> x0 = np.array(0.1) > >>>> x = np.linspace(0, 1) > >>>> y = np.linspace(0, 1) > >>>> np.interp(x0, x, y) > > > > I hesitate to call this behavior a bug (when I've done this in the > > past, I find out I'm just doing something wrong), but I find the error > > unnecessary (and the error output a bit obscure). > > > > Below is a (poorly tested) fix, which seems to work for me. (Sorry I'm > > not on svn, so it isn't a proper diff) > > > > Cheers, > > -Tony > > > > Python 2.5.1 > > numpy 1.3.0 > > > > numpy/lib/function_base.py: line 1144 > > ============================= > > if isinstance(x, (float, int, number)): > > return compiled_interp([x], xp, fp, left, right).item() > > + elif isinstance(x, np.ndarray) and x.ndim == 0: > > + return compiled_interp(x[np.newaxis], xp, fp, left, right)[0] > > else: > > return compiled_interp(x, xp, fp, left, right) > > ============================= > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pfeldman at verizon.net Fri Jul 17 01:36:40 2009 From: pfeldman at verizon.net (Phillip M. Feldman) Date: Thu, 16 Jul 2009 22:36:40 -0700 Subject: [Numpy-discussion] alternative mechanism for initializing an array Message-ID: <4A600DE8.30607@verizon.net> I think that I finally figured it out. See code below. Phillip *** Start of myarray.py *** import numpy def array(*args, **kwargs): """This function acts as an interface to numpy.array, accepting multi-row matrices with or without an outer set of enclosing brackets. Usage examples: In [1]: from myarray import array In [2]: x=array([1,2,3],[4,5,6],dtype=int) In [3]: x/2 Out[3]: array([[0, 1, 1], [2, 2, 3]]) Dr. Phillip M. Feldman 16 July 2009""" args1= [] args2= [] for arg in args: if isinstance(arg,list): args1.append(arg) else: args2.append(arg) if len(args1)==1: args1= args1[0] if len(args2)==1: args2= args2[0] if len(args2): return numpy.array(args1, args2, **kwargs) else: return numpy.array(args1, **kwargs) *** End of myarray.py *** From: Keith Goodman gmail.com> Subject: Re: alternative mechanism for initializing an array Newsgroups: gmane.comp.python.numeric.general Date: 2009-07-16 20:57:18 GMT (4 hours and 29 minutes ago) From william.ratcliff at gmail.com Fri Jul 17 03:38:43 2009 From: william.ratcliff at gmail.com (william ratcliff) Date: Fri, 17 Jul 2009 03:38:43 -0400 Subject: [Numpy-discussion] [ANN] Announcing the SciPy conference schedule In-Reply-To: <20090716021311.GA21642@phare.normalesup.org> References: <20090716021311.GA21642@phare.normalesup.org> Message-ID: <827183970907170038jef07198r867b086a284e8b4a@mail.gmail.com> A humble suggestion--for the March meeting of the american physical society, there is a roommate finder for splitting hotel rooms. This could be useful in keeping expenses down for some. There should be a way to do it without liability.... Cheers, William On Wed, Jul 15, 2009 at 10:13 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > The SciPy conference committee is pleased to announce the schedule of the > conference: > > http://conference.scipy.org/schedule > > This year?s program is very rich. In order to limit the number of > interesting talks that we had to turn down, we decided to reduce the > length of talks. Although this results in many short talks, we hope that > it will foster discussions, and give new ideas. Many subjects are > covered, both varying technical subject in the scientific computing > spectrum, and covering a lot of different research areas. > > I would personally like to thank the members of the program committee, > who spent time reviewing the proposed abstracts and giving the chairs > feedback. > > Fernando Perez and the tutorial presenters are hard at work finishing > planning all the details of the two-day tutorial session that will > precede the conference. An introduction tutorial track and an advanced > tutorial track, both covering various aspect of scientific computing in > Python, presented by experts in the field, should help many people > getting up to speed on the amazing technology driving this community. > > The SciPy 2009 program committee > > * Co-Chair Ga?l Varoquaux, Applied Mathematics and Neuroscience, > * Neurospin, CEA - INRIA Saclay (France) > * Co-Chair St?fan van der Walt, Applied Mathematics, University of > * Stellenbosch (South Africa) > * Michael Aivazis, Center for Advanced Computing Research, California > * Institute of Technology (USA) > * Brian Granger, Physics Department, California Polytechnic State > * University, San Luis Obispo (USA) > * Aric Hagberg, Theoretical Division, Los Alamos National Laboratory > * (USA) > * Konrad Hinsen, Centre de Biophysique Mol?culaire, CNRS Orl?ans > * (France) > * Randall LeVeque, Mathematics, University of Washington, Seattle > * (USA) > * Travis Oliphant, Enthought (USA) > * Prabhu Ramachandran, Department of Aerospace Engineering, IIT > * Bombay (India) > * Raphael Ritz, International Neuroinformatics Coordinating Facility > * (Sweden) > * William Stein, Mathematics, University of Washington, Seattle (USA) > > Conference Chair: Jarrod Millman, Neuroscience Institute, UC Berkeley > (USA) > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lciti at essex.ac.uk Fri Jul 17 04:29:02 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Fri, 17 Jul 2009 09:29:02 +0100 Subject: [Numpy-discussion] alternative mechanism for initializing anarray References: <4A600DE8.30607@verizon.net> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E0C@sernt14.essex.ac.uk> Hello, all these alternative mechanisms for initializing arrays risk to break current code. Isn't it? Then one would need to specify the data type with a kw argument while with the current implementation the second argument is the data type irregardless of whether or not it is given with the dtype keyword. >>> np.array(['AB','S4']) array(['AB', 'S4'], dtype='|S2') >>> np.array('AB','S4') array('AB', dtype='|S4') Best, Luca From vel.accel at gmail.com Fri Jul 17 05:06:25 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Fri, 17 Jul 2009 05:06:25 -0400 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <49d6b3500907161539w28c7d76bke87711a80f524d8@mail.gmail.com> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <49d6b3500907160832y1c7a1967g57245c34e01f81a1@mail.gmail.com> <1e52e0880907160900r25d38002q15961fd41c1f3ae3@mail.gmail.com> <49d6b3500907161539w28c7d76bke87711a80f524d8@mail.gmail.com> Message-ID: <1e52e0880907170206lfdb793eu4d900a58450485c2@mail.gmail.com> On Thu, Jul 16, 2009 at 6:39 PM, G?khan SEVER wrote: > > > On Thu, Jul 16, 2009 at 4:30 PM, Fernando Perez wrote: > >> >> If someone has a camera that can do the recordings in a format that >> can then be directly recompressed at the command line with something >> like mencoder, that would be great. From experience, any recording >> mode that requires later manual editing sounds great in principle, and >> then nobody finds the time to do all the work. >> >> > There is a one great approach posted here at > http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1246925071 > > Seems like his one man army approach worked very nicely. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I wish I was free to do a beautiful production quality video recording for the community. I would really enjoy doing that. I also feel it is quite important considering lectures of this kind are so, so instructive. If it is too much work to do video, that's perfectly ok. Not preferable, but definately O.K. Besides I'd rather have a rich audio recording then haphazard video/audio If someone could just get a decent recording of the audio, then the slides and audio can be edited together as a screencast on a website. I could imagine a pipe of the stage microphone to a (possibly spare) computer. I could then see it then being uploaded to a storage server for the future editing. The talk given, provides so much more insight than the slides (pdf) alone. Just think how enticing it is for scipy new comers to have screencasts to watch. ~Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From dfnsonfsduifb at gmx.de Fri Jul 17 09:10:53 2009 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Fri, 17 Jul 2009 15:10:53 +0200 Subject: [Numpy-discussion] Mirror/flip numpy array? Message-ID: <4A60785D.80707@gmx.de> Hello list, I have a really simple newbie question: How can I mirror/flip a numpy.ndarray? I.e. mirror switches the colums (leftmost becomes rightmost and so on), flip changes the rows (top becomes bottom and so on)? Kind regards, Joe From gruben at bigpond.net.au Fri Jul 17 09:21:51 2009 From: gruben at bigpond.net.au (Gary Ruben) Date: Fri, 17 Jul 2009 23:21:51 +1000 Subject: [Numpy-discussion] Mirror/flip numpy array? In-Reply-To: <4A60785D.80707@gmx.de> References: <4A60785D.80707@gmx.de> Message-ID: <4A607AEF.7090205@bigpond.net.au> In [1]: a=array([1,2,3]) In [2]: a[::-1] Out[2]: array([3, 2, 1]) Johannes Bauer wrote: > Hello list, > > I have a really simple newbie question: How can I mirror/flip a > numpy.ndarray? I.e. mirror switches the colums (leftmost becomes > rightmost and so on), flip changes the rows (top becomes bottom and so on)? > > Kind regards, > Joe From kwgoodman at gmail.com Fri Jul 17 09:30:36 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 17 Jul 2009 06:30:36 -0700 Subject: [Numpy-discussion] Mirror/flip numpy array? In-Reply-To: <4A607AEF.7090205@bigpond.net.au> References: <4A60785D.80707@gmx.de> <4A607AEF.7090205@bigpond.net.au> Message-ID: On Fri, Jul 17, 2009 at 6:21 AM, Gary Ruben wrote: > In [1]: a=array([1,2,3]) > > In [2]: a[::-1] > Out[2]: array([3, 2, 1]) > > > Johannes Bauer wrote: >> Hello list, >> >> I have a really simple newbie question: How can I mirror/flip a >> numpy.ndarray? I.e. mirror switches the colums (leftmost becomes >> rightmost and so on), flip changes the rows (top becomes bottom and so on)? There's also np.fliplr (left right) and np.flipud (up down). The last line of these functions are "return m[:, ::-1]" and "return m[::-1,...]", respectively. From tsyu80 at gmail.com Fri Jul 17 09:51:51 2009 From: tsyu80 at gmail.com (Tony Yu) Date: Fri, 17 Jul 2009 09:51:51 -0400 Subject: [Numpy-discussion] Using interpolate with zero-rank array raises error In-Reply-To: References: Message-ID: > Date: Thu, 16 Jul 2009 23:37:58 -0400 > From: Ralf Gommers > > It seems to me that there are quite a few other functions that will > give > errors with 0-D arrays (apply_along/over_axis are two that come to > mind). > There is nothing to interpolate so I'm not surprised. Hmm, I don't quite understand. In the example below, the 0-D array (`x0`) gives the x-value(s) where you want interpolated values. This shouldn't require a non-scalar, and in fact, interp currently accepts python scalars (but not Numpy scalars). If the 0-D array replaced `x` and `y`---the known data points--- then, I agree there would be nothing to interpolate. I believe the example functions you cite are similar to replacing `x` and `y` below with scalar values. ... or am I just missing something? Thanks, -Tony >>> >>> When using interpolate with a zero-rank array, I get "ValueError: >>> object of too small depth for desired array". The following code >>> reproduces this issue >>> >>>>>> import numpy as np >>>>>> x0 = np.array(0.1) >>>>>> x = np.linspace(0, 1) >>>>>> y = np.linspace(0, 1) >>>>>> np.interp(x0, x, y) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsdale24 at gmail.com Fri Jul 17 10:03:01 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Fri, 17 Jul 2009 10:03:01 -0400 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> Message-ID: On Mon, Jul 13, 2009 at 7:12 PM, Darren Dale wrote: > 2009/7/13 St?fan van der Walt > >> Hi Darren >> >> 2009/7/13 Darren Dale : >> > I've put together a first cut at implementing __array_prepare__, which >> > appears to work, and I would like to request feedback. Here is an >> overview >> > of the approach: >> >> This is pretty neat! Do you have a quick snippet at hand illustrating its >> use? > > > That would be helpful, wouldn't it? The attached script is a modified > version of RealisticInfoArray from > http://docs.scipy.org/doc/numpy/user/basics.subclassing.html . It should > yield the following output: > > > starting with [0 1 2 3 4] > which is of type > and has info attribute = "information" > subtracting 3 from [0 1 2 3 4] > subtract calling __array_prepare__ on [0 1 2 3 4] input > output array is now of type > output array values are still uninitialized: > [139911601789568 39578752 139911614885536 39254560 > 48] > __array_prepare__ is updating info attribute on output > __array_prepare__ finished, subtract ufunc is taking over > subtract calling __array_wrap__ on [0 1 2 3 4] input > output array has initial value: [-3 -2 -1 0 1] > __array_wrap__ is setting output endpoints to 0 > yielding [ 0 -2 -1 0 0] > which is of type > and has info attribute = "new_information" > This is a gentle ping, hoping to get some feedback so this feature has a chance of being included in the next release. Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: From meine at informatik.uni-hamburg.de Fri Jul 17 10:54:21 2009 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Fri, 17 Jul 2009 16:54:21 +0200 Subject: [Numpy-discussion] VIGRA, NumPy and Fortran-order (again) Message-ID: <200907171654.22107.meine@informatik.uni-hamburg.de> Hi, as I mentioned in the past [1], we considered refactoring our VIGRA (an image analysis library [2]) python bindings to be based on NumPy [3]. However, we have the problem that VIGRA uses Fortran-order indexing (i.e. there's operator()(x, y) in C++), and this should of course be the same in Python. (It is more important to us to have the same indexing in VIGRA's python bindings and in VIGRA itself, than to have the same indexing as in e.g. MPL or PIL.) As discussing in-depth in [1], numpy does not support Fortran order very well. First, there are performance issues, but even more important: the order is not preserved when performing simple operations. :-( In [8]: numpy.isfortran(a) Out[8]: True In [9]: numpy.isfortran(a + 4) Out[9]: False In [10]: numpy.isfortran(a.copy()) Out[10]: False In the worst case, this means that VIGRA functions exported to python only for unstrided images cannot be called on the results of any numpy function call. Do you agree that this is a bug^H^H^Hmissing feature and how difficult would it be to implement that? The specs would be: preserve the *ordering* of the strides (i.e. we're using mixed-order for RGB images to be able to write image[x, y] = (r, g, b)), and in the case of multiple input arguments, use the same rules (i.e. array priority) as for the output type determination. If I understood Travis' comments in the above-mentioned thread [1] correctly, this would already fix some of the performance issues along the way (since it would suddenly allow the use of special, optimized code paths). Have a nice day, Hans [1] http://mail.scipy.org/pipermail/numpy-discussion/2007-November/029837.html [2] http://hci.iwr.uni-heidelberg.de/vigra/ [3] https://mailhost.informatik.uni-hamburg.de/pipermail/vigra/2009- May/000610.html From robert.kern at gmail.com Fri Jul 17 11:21:34 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 17 Jul 2009 10:21:34 -0500 Subject: [Numpy-discussion] [SciPy-user] [ANN] Announcing the SciPy conference schedule In-Reply-To: <827183970907170038jef07198r867b086a284e8b4a@mail.gmail.com> References: <20090716021311.GA21642@phare.normalesup.org> <827183970907170038jef07198r867b086a284e8b4a@mail.gmail.com> Message-ID: <3d375d730907170821l32139208ob3410bd9c9520c01@mail.gmail.com> On Fri, Jul 17, 2009 at 02:38, william ratcliff wrote: > A humble suggestion--for the March meeting of the american physical society, > there is a roommate finder for splitting hotel rooms. ?This could be useful > in keeping expenses down for some. ?There should be a way to do it without > liability.... A wiki page would probably be the best thing given the short time frame. I recommend either the Saga or the Vagabond hotels for keeping costs down and staying close to campus. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dsdale24 at gmail.com Fri Jul 17 11:44:55 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Fri, 17 Jul 2009 11:44:55 -0400 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> Message-ID: On Fri, Jul 17, 2009 at 10:03 AM, Darren Dale wrote: > On Mon, Jul 13, 2009 at 7:12 PM, Darren Dale wrote: > >> 2009/7/13 St?fan van der Walt >> >>> Hi Darren >>> >>> 2009/7/13 Darren Dale : >>> > I've put together a first cut at implementing __array_prepare__, which >>> > appears to work, and I would like to request feedback. Here is an >>> overview >>> > of the approach: >>> >>> This is pretty neat! Do you have a quick snippet at hand illustrating >>> its use? >> >> >> That would be helpful, wouldn't it? The attached script is a modified >> version of RealisticInfoArray from >> http://docs.scipy.org/doc/numpy/user/basics.subclassing.html . It should >> yield the following output: >> >> >> starting with [0 1 2 3 4] >> which is of type >> and has info attribute = "information" >> subtracting 3 from [0 1 2 3 4] >> subtract calling __array_prepare__ on [0 1 2 3 4] input >> output array is now of type >> output array values are still uninitialized: >> [139911601789568 39578752 139911614885536 39254560 >> 48] >> __array_prepare__ is updating info attribute on output >> __array_prepare__ finished, subtract ufunc is taking over >> subtract calling __array_wrap__ on [0 1 2 3 4] input >> output array has initial value: [-3 -2 -1 0 1] >> __array_wrap__ is setting output endpoints to 0 >> yielding [ 0 -2 -1 0 0] >> which is of type >> and has info attribute = "new_information" >> > > This is a gentle ping, hoping to get some feedback so this feature has a > chance of being included in the next release. > I have a question about the C-api. If I want to make the default implementation of __array_prepare__ (or __array_wrap__, is anyone out there?) simply pass through the output array: static PyObject * array_preparearray(PyArrayObject *self, PyObject *args) { PyObject *arr; if (PyTuple_Size(args) < 1) { PyErr_SetString(PyExc_TypeError, "only accepts 1 argument"); return NULL; } arr = PyTuple_GET_ITEM(args, 0); if (!PyArray_Check(arr)) { PyErr_SetString(PyExc_TypeError, "can only be called with ndarray object"); return NULL; } return arr; } Is this sufficient, or do I need to worry about calling Py_INCREF? Thanks, Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jul 17 12:04:05 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 17 Jul 2009 10:04:05 -0600 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> Message-ID: On Fri, Jul 17, 2009 at 9:44 AM, Darren Dale wrote: > On Fri, Jul 17, 2009 at 10:03 AM, Darren Dale wrote: > >> On Mon, Jul 13, 2009 at 7:12 PM, Darren Dale wrote: >> >>> 2009/7/13 St?fan van der Walt >>> >>>> Hi Darren >>>> >>>> 2009/7/13 Darren Dale : >>>> > I've put together a first cut at implementing __array_prepare__, which >>>> > appears to work, and I would like to request feedback. Here is an >>>> overview >>>> > of the approach: >>>> >>>> This is pretty neat! Do you have a quick snippet at hand illustrating >>>> its use? >>> >>> >>> That would be helpful, wouldn't it? The attached script is a modified >>> version of RealisticInfoArray from >>> http://docs.scipy.org/doc/numpy/user/basics.subclassing.html . It should >>> yield the following output: >>> >>> >>> starting with [0 1 2 3 4] >>> which is of type >>> and has info attribute = "information" >>> subtracting 3 from [0 1 2 3 4] >>> subtract calling __array_prepare__ on [0 1 2 3 4] input >>> output array is now of type >>> output array values are still uninitialized: >>> [139911601789568 39578752 139911614885536 39254560 >>> 48] >>> __array_prepare__ is updating info attribute on output >>> __array_prepare__ finished, subtract ufunc is taking over >>> subtract calling __array_wrap__ on [0 1 2 3 4] input >>> output array has initial value: [-3 -2 -1 0 1] >>> __array_wrap__ is setting output endpoints to 0 >>> yielding [ 0 -2 -1 0 0] >>> which is of type >>> and has info attribute = "new_information" >>> >> >> This is a gentle ping, hoping to get some feedback so this feature has a >> chance of being included in the next release. >> > > I have a question about the C-api. If I want to make the default > implementation of __array_prepare__ (or __array_wrap__, is anyone out > there?) simply pass through the output array: > > static PyObject * > array_preparearray(PyArrayObject *self, PyObject *args) > { > PyObject *arr; > > if (PyTuple_Size(args) < 1) { > PyErr_SetString(PyExc_TypeError, > "only accepts 1 argument"); > return NULL; > } > arr = PyTuple_GET_ITEM(args, 0); > if (!PyArray_Check(arr)) { > PyErr_SetString(PyExc_TypeError, > "can only be called with ndarray object"); > return NULL; > } > return arr; > } > > Is this sufficient, or do I need to worry about calling Py_INCREF? > PyObject* *PyTuple_GetItem*(PyObject *p, Py_ssize_t pos) Return value: Borrowed reference. Return the object at position pos in the tuple pointed to by p. If pos is out of bounds, return NULL and sets an IndexError exception. It's a borrowed reference so you need to call Py_INCREF on it. I find this Python C-API documentation useful. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Fri Jul 17 12:24:40 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 17 Jul 2009 18:24:40 +0200 Subject: [Numpy-discussion] Universal file format Message-ID: Hi all, Is there a Python tool to read and write files in the so-called universal format ? I found a Matlab implementation http://www.mathworks.com/matlabcentral/fileexchange/6395 Any pointer would be appreciated. Thanks in advance Nils http://www.sdrl.uc.edu/universal-file-formats-for-modal-analysis-testing-1 http://zone.ni.com/devzone/cda/tut/p/id/4463 From ralf.gommers at googlemail.com Fri Jul 17 13:27:25 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 17 Jul 2009 13:27:25 -0400 Subject: [Numpy-discussion] Using interpolate with zero-rank array raises error In-Reply-To: References: Message-ID: On Fri, Jul 17, 2009 at 9:51 AM, Tony Yu wrote: > > Date: Thu, 16 Jul 2009 23:37:58 -0400 > From: Ralf Gommers > > It seems to me that there are quite a few other functions that will give > errors with 0-D arrays (apply_along/over_axis are two that come to mind). > There is nothing to interpolate so I'm not surprised. > > > Hmm, I don't quite understand. In the example below, the 0-D array (`x0`) > gives the x-value(s) where you want interpolated values. This shouldn't > require a non-scalar, and in fact, interp currently accepts python scalars > (but not Numpy scalars). > > If the 0-D array replaced `x` and `y`---the known data points--- then, I > agree there would be nothing to interpolate. I believe the example functions > you cite are similar to replacing `x` and `y` below with scalar values. > You're right, sorry for the confusion, i did swap the roles of x/y with that of x0 in my mind. That's what I get for writing emails at midnight. If it works with scalars it should work with 0-D arrays I think. So you should probably open a ticket and attach your patch. Cheers, Ralf > ... or am I just missing something? > > Thanks, > -Tony > > > When using interpolate with a zero-rank array, I get "ValueError: > > object of too small depth for desired array". The following code > > reproduces this issue > > > import numpy as np > > x0 = np.array(0.1) > > x = np.linspace(0, 1) > > y = np.linspace(0, 1) > > np.interp(x0, x, y) > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Fri Jul 17 13:37:09 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_SEVER?=) Date: Fri, 17 Jul 2009 12:37:09 -0500 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <20090716150529.GK13779@phare.normalesup.org> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <20090716150529.GK13779@phare.normalesup.org> Message-ID: <49d6b3500907171037w7764c89eoe7265fda46d3d163@mail.gmail.com> On Thu, Jul 16, 2009 at 10:05 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Wed, Jul 15, 2009 at 11:29:27PM -0400, Peter Alexander wrote: > > I sure wish I was able to attend this year's event. > > I'm wondering, and really hoping, if/that the lectures will be > recorded > > and then posted for the whole community's benefit? > > The problem is that this requires actually a lot of work, to get > something useful out of the recordings. I don't think it will happen :(. > > Ga?l > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Ga?l, Your recently posted mayavi tutorial video and Fernando's previous py4science recordings in very good quality. I have learnt a great amount of information just by watching those videos. For me, most important gains, hearing the developer lingo and usage techniques that can't be easily find in any documentation as well as presentation skills. I think, it would be great to have a similar equipment setup during the SciPy09. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Fri Jul 17 14:21:54 2009 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 17 Jul 2009 11:21:54 -0700 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <49d6b3500907171037w7764c89eoe7265fda46d3d163@mail.gmail.com> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <20090716150529.GK13779@phare.normalesup.org> <49d6b3500907171037w7764c89eoe7265fda46d3d163@mail.gmail.com> Message-ID: On Fri, Jul 17, 2009 at 10:37 AM, G?khan SEVER wrote: > I think, it would be great to have a similar equipment setup during the > SciPy09. Absolutely. It would be *great* to have the tutorials and talks recorded. If anyone steps up to bring equipment, record the talks, and post them, everyone would be very appreciative. If no one offers to do this, it won't happen. If anyone wants to volunteer to take care of this, feel free to contact Gael, Stefan, or I off-list. Jarrod From jonathan.taylor at utoronto.ca Fri Jul 17 14:29:57 2009 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Fri, 17 Jul 2009 14:29:57 -0400 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <20090716150529.GK13779@phare.normalesup.org> <49d6b3500907171037w7764c89eoe7265fda46d3d163@mail.gmail.com> Message-ID: <463e11f90907171129w35ae2e57nab40a1544fa019a2@mail.gmail.com> I would also like to see this. Have we looked at using videolectures.net? In my field, the people from videolectures.net takes care of filming and editing. I am not sure, but they might actually be free as they have a big grant from the EU to do this sort of thing. It might be worth pinging them to find out if it would be possible and how much, if any, it would cost. Best, Jonathan. On Fri, Jul 17, 2009 at 2:21 PM, Jarrod Millman wrote: > On Fri, Jul 17, 2009 at 10:37 AM, G?khan SEVER wrote: >> I think, it would be great to have a similar equipment setup during the >> SciPy09. > > Absolutely. ?It would be *great* to have the tutorials and talks > recorded. ?If anyone steps up to bring equipment, record the talks, > and post them, everyone would be very appreciative. ?If no one offers > to do this, it won't happen. ?If anyone wants to volunteer to take > care of this, feel free to contact Gael, Stefan, or I off-list. > > Jarrod > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From fperez.net at gmail.com Fri Jul 17 15:18:04 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 17 Jul 2009 12:18:04 -0700 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <463e11f90907171129w35ae2e57nab40a1544fa019a2@mail.gmail.com> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <20090716150529.GK13779@phare.normalesup.org> <49d6b3500907171037w7764c89eoe7265fda46d3d163@mail.gmail.com> <463e11f90907171129w35ae2e57nab40a1544fa019a2@mail.gmail.com> Message-ID: On Fri, Jul 17, 2009 at 11:29 AM, Jonathan Taylor wrote: > I would also like to see this. ?Have we looked at using videolectures.net? > > In my field, the people from videolectures.net takes care of filming > and editing. ?I am not sure, but they might actually be free as they > have a big grant from the EU to do this sort of thing. ?It might be > worth pinging them to find out if it would be possible and how much, > if any, it would cost. That's a great idea. BTW, there's an excellent set of lectures with lots of python stuff from last year's NIPS up there, including a very nice talk by John on matplotlib: http://videolectures.net/mloss08_whistler/ It would be *fantastic* if anyone from our community could take up contacting the videolectures guys... Cheers, f From fperez.net at gmail.com Fri Jul 17 15:19:22 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 17 Jul 2009 12:19:22 -0700 Subject: [Numpy-discussion] Scipy Conference 2009 Lecture Recordings In-Reply-To: <49d6b3500907171037w7764c89eoe7265fda46d3d163@mail.gmail.com> References: <1e52e0880907152029w194bf1b0y106c74cff6141282@mail.gmail.com> <20090716150529.GK13779@phare.normalesup.org> <49d6b3500907171037w7764c89eoe7265fda46d3d163@mail.gmail.com> Message-ID: On Fri, Jul 17, 2009 at 10:37 AM, G?khan SEVER wrote: > Your recently posted mayavi tutorial video and Fernando's previous > py4science recordings in very good quality. BTW, for those interested, mine are a 2-day intro course on python/science (similar in spirit to our upcoming introductory tutorial track at the conference) that I taught here at UC Berkeley: http://www.archive.org/search.php?query=Fernando+Perez+scientific+python Cheers, f From jonathan.taylor at utoronto.ca Fri Jul 17 15:57:08 2009 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Fri, 17 Jul 2009 15:57:08 -0400 Subject: [Numpy-discussion] performance matrix multiplication vs. matlab In-Reply-To: <7f014ea60906060959s6570cc32l277c5ab423f0b9ed@mail.gmail.com> References: <20090604143638.xrqzx8w8v4ggckkk@www.sms.ed.ac.uk> <4A28EC37.10205@ar.media.kyoto-u.ac.jp> <4A28F16C.3070607@ar.media.kyoto-u.ac.jp> <4A2955F5.9030006@hawaii.edu> <4A295A7D.1040407@ar.media.kyoto-u.ac.jp> <7f014ea60906051437p6002c6f7jb2d77374a220d2aa@mail.gmail.com> <7f014ea60906060959s6570cc32l277c5ab423f0b9ed@mail.gmail.com> Message-ID: <463e11f90907171257m2f5ff9d6v34924e0200145f4@mail.gmail.com> Following these instructions I have the following problem when I import numpy. Does anyone know why this might be? Thanks, Jonathan. >>> import numpy Traceback (most recent call last): File "", line 1, in File "/home/jtaylor/lib/python2.5/site-packages/numpy/__init__.py", line 130, in import add_newdocs File "/home/jtaylor/lib/python2.5/site-packages/numpy/add_newdocs.py", line 9, in from lib import add_newdoc File "/home/jtaylor/lib/python2.5/site-packages/numpy/lib/__init__.py", line 13, in from polynomial import * File "/home/jtaylor/lib/python2.5/site-packages/numpy/lib/polynomial.py", line 18, in from numpy.linalg import eigvals, lstsq File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/__init__.py", line 47, in from linalg import * File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/linalg.py", line 22, in from numpy.linalg import lapack_lite ImportError: /usr/local/lib/libptcblas.so: undefined symbol: ATL_cpttrsm On Sat, Jun 6, 2009 at 12:59 PM, Chris Colbert wrote: > since there is demand, and someone already emailed me, I'll put what I > did in this post. It pretty much follows whats on the scipy website, > with a couple other things I gleaned from reading the ATLAS install > guide: > > and here it goes, this is valid for Ubuntu 9.04 64-bit ?(# starts a > comment when working in the terminal) > > > download lapack 3.2.1 http://www.netlib.org/lapack/lapack.tgz > download atlas 3.8.3 > http://sourceforge.net/project/downloading.php?group_id=23725&filename=atlas3.8.3.tar.bz2&a=65663372 > > create folder ?/home/your-user-name/build/atlas ? #this is where we build > create folder /home/your-user-name/build/lapack #atlas and lapack > > extract the folder lapack-3.2.1 to /home/your-user-name/build/lapack > extract the contents of atlas to /home/your-user-name/build/atlas > > > > now in the terminal: > > # remove g77 and get stuff we need > sudo apt-get remove g77 > sudo apt-get install gfortran > sudo apt-get install build-essential > sudo apt-get install python-dev > sudo apt-get install python-setuptools > sudo easy_install nose > > > # build lapack > cd /home/your-user-name/build/lapack/lapack-3.2.1 > cp INSTALL/make.inc.gfortran make.inc > > gedit make.inc > ################# > #in the make.inc file make sure the line ? OPTS = -O2 -fPIC -m64 > #and ? ?NOOPTS = -O0 -fPIC -m64 > #the -m64 flags build 64-bit code, if you want 32-bit, simply leave > #the -m64 flags out > ################# > > cd SRC > > #this should build lapack without error > make > > > > # build atlas > > cd /home/your-user-name/build/atlas > > #this is simply where we will build the atlas > #libs, you can name it what you want > mkdir Linux_X64SSE2 > > cd Linux_X64SSE2 > > #need to turn off cpu-throttling > sudo cpufreq-selector -g performance > > #if you don't want 64bit code remove the -b 64 flag. replace the > #number 2400 with your CPU frequency in MHZ > #i.e. my cpu is 2.53 GHZ so i put 2530 > ../configure -b 64 -D c -DPentiumCPS=2400 -Fa ?-alg -fPIC > --with-netlib-lapack=/home/your-user-name/build/lapack/lapack-3.2.1/Lapack_LINUX.a > > #the configure step takes a bit, and should end without errors > > ?#this takes a long time, go get some coffee, it should end without error > make build > > #this will verify the build, also long running > make check > > #this will test the performance of your build and give you feedback on > #it. your numbers should be close to the test numbers at the end > make time > > cd lib > > #builds single threaded .so's > make shared > > #builds multithreaded .so's > make ptshared > > #copies all of the atlas libs (and the lapack lib built with atlas) > #to our lib dir > sudo ?cp ?*.so ?/usr/local/lib/ > > > > #now we need to get and build numpy > > download numpy 1.3.0 > http://sourceforge.net/project/downloading.php?group_id=1369&filename=numpy-1.3.0.tar.gz&a=93506515 > > extract the folder numpy-1.3.0 to /home/your-user-name/build > > #in the terminal > > cd /home/your-user-name/build/numpy-1.3.0 > cp site.cfg.example site.cfg > > gedit site.cfg > ############################################### > # in site.cfg uncomment the following lines and make them look like these > [DEFAULT] > library_dirs = /usr/local/lib > include_dirs = /usr/local/include > > [blas_opt] > libraries = ptf77blas, ptcblas, atlas > > [lapack_opt] > libraries = lapack, ptf77blas, ptcblas, atlas > ################################################### > #if you want single threaded libs, uncomment those lines instead > > > #build numpy- should end without error > python setup.py build > > #install numpy > python setup.py install > > cd /home > > sudo ldconfig > > python >>>import numpy >>>numpy.test() ? #this should run with no errors (skipped tests and known-fails are ok) >>>a = numpy.random.randn(6000, 6000) >>>numpy.dot(a, a) ? ? # look at your cpu monitor and verify all cpu cores are at 100% if you built with threads > > > Celebrate with a beer! > > > Cheers! > > Chris > > > > > > On Sat, Jun 6, 2009 at 10:42 AM, Keith Goodman wrote: >> On Fri, Jun 5, 2009 at 2:37 PM, Chris Colbert wrote: >>> I'll caution anyone from using Atlas from the repos in Ubuntu 9.04? as the >>> package is broken: >>> >>> https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510 >>> >>> >>> just build Atlas yourself, you get better performance AND threading. >>> Building it is not the nightmare it sounds like. I think i've done it a >>> total of four times now, both 32-bit and 64-bit builds. >>> >>> If you need help with it,? just email me off list. >> >> That's a nice offer. I tried building ATLAS on Debian a year or two >> ago and got stuck. >> >> Clear out your inbox! >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From millman at berkeley.edu Fri Jul 17 16:01:58 2009 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 17 Jul 2009 13:01:58 -0700 Subject: [Numpy-discussion] ANN: SciPy 2009 early registration extended to July 22nd Message-ID: The early registration deadline for SciPy 2009 has been extended until Wednesday, July 22, 2009. Please register ( http://conference.scipy.org/to_register ) by this date to take advantage of the reduced early registration rate. Since we just announced the conference schedule, I was asked to provide extra time for people to register. Fortunately, we were able to get a few extra days from our vendors. But we will have to place orders next Thursday, so this is the last time we will be able to extend the deadline for registration. The conference schedule is available here: http://conference.scipy.org/schedule About the conference -------------------- SciPy 2009, the 8th Python in Science conference, will be held from August 18-23, 2009 at Caltech in Pasadena, CA, USA. The conference starts with two days of tutorials to the scientific Python tools. There will be two tracks, one for introduction of the basic tools to beginners, and one for more advanced tools. The tutorials will be followed by two days of talks. Both days of talks will begin with a keynote address. The first day?s keynote will be given by Peter Norvig, the Director of Research at Google; while, the second keynote will be delivered by Jon Guyer, a Materials Scientist in the Thermodynamics and Kinetics Group at NIST. The program committee will select the remaining talks from submissions to our call for papers. All selected talks will be included in our conference proceedings edited by the program committee. After the talks each day we will provide several rooms for impromptu birds of a feather discussions. Finally, the last two days of the conference will be used for a number of coding sprints on the major software projects in our community. For the 8th consecutive year, the conference will bring together the developers and users of the open source software stack for scientific computing with Python. Attendees have the opportunity to review the available tools and how they apply to specific problems. By providing a forum for developers to share their Python expertise with the wider commercial, academic, and research communities, this conference fosters collaboration and facilitates the sharing of software components, techniques, and a vision for high level language use in scientific computing. For further information, please visit the conference homepage: http://conference.scipy.org. Important Dates --------------- * Friday, July 3: Abstracts Due * Wednesday, July 15: Announce accepted talks, post schedule * Wednesday, July 22: Early Registration ends * Tuesday-Wednesday, August 18-19: Tutorials * Thursday-Friday, August 20-21: Conference * Saturday-Sunday, August 22-23: Sprints * Friday, September 4: Papers for proceedings due Executive Committee ------------------- * Jarrod Millman, UC Berkeley, USA (Conference Chair) * Ga?l Varoquaux, INRIA Saclay, France (Program Co-Chair) * St?fan van der Walt, University of Stellenbosch, South Africa (Program Co-Chair) * Fernando P?rez, UC Berkeley, USA (Tutorial Chair) From tsyu80 at gmail.com Fri Jul 17 16:07:46 2009 From: tsyu80 at gmail.com (Tony Yu) Date: Fri, 17 Jul 2009 16:07:46 -0400 Subject: [Numpy-discussion] Using interpolate with zero-rank array raises error In-Reply-To: References: Message-ID: > Date: Fri, 17 Jul 2009 13:27:25 -0400 > From: Ralf Gommers > Subject: Re: [Numpy-discussion] Using interpolate with zero-rank array > raises error [snip] > If it works with scalars it should work with 0-D arrays I think. So > you > should probably open a ticket and attach your patch. Thanks for your responses and suggestion. I've opened up Ticket #1177 on Trac to address this issue. Best, -Tony -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav+sp at iki.fi Fri Jul 17 16:15:31 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Fri, 17 Jul 2009 20:15:31 +0000 (UTC) Subject: [Numpy-discussion] VIGRA, NumPy and Fortran-order (again) References: <200907171654.22107.meine@informatik.uni-hamburg.de> Message-ID: On 2009-07-17, Hans Meine wrote: [clip] > As discussing in-depth in [1], numpy does not support Fortran order very well. > First, there are performance issues, but even more important: the order is not > preserved when performing simple operations. :-( [clip] > The specs would be: preserve the *ordering* of the strides (i.e. we're using > mixed-order for RGB images to be able to write image[x, y] = (r, g, b)), and > in the case of multiple input arguments, use the same rules (i.e. array > priority) as for the output type determination. > > If I understood Travis' comments in the above-mentioned thread [1] correctly, > this would already fix some of the performance issues along the way (since it > would suddenly allow the use of special, optimized code paths). I was wondering about this too, when working on improving the cache coherency of the reduction operations. Also these would be more efficient if the striding of the output array could be chosen freely. I wonder if it would be OK to make this change... -- Pauli Virtanen From dwf at cs.toronto.edu Fri Jul 17 16:20:52 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 17 Jul 2009 16:20:52 -0400 Subject: [Numpy-discussion] performance matrix multiplication vs. matlab In-Reply-To: <463e11f90907171257m2f5ff9d6v34924e0200145f4@mail.gmail.com> References: <20090604143638.xrqzx8w8v4ggckkk@www.sms.ed.ac.uk> <4A28EC37.10205@ar.media.kyoto-u.ac.jp> <4A28F16C.3070607@ar.media.kyoto-u.ac.jp> <4A2955F5.9030006@hawaii.edu> <4A295A7D.1040407@ar.media.kyoto-u.ac.jp> <7f014ea60906051437p6002c6f7jb2d77374a220d2aa@mail.gmail.com> <7f014ea60906060959s6570cc32l277c5ab423f0b9ed@mail.gmail.com> <463e11f90907171257m2f5ff9d6v34924e0200145f4@mail.gmail.com> Message-ID: <817E9DA4-A61B-4161-9339-B0DC6A615E35@cs.toronto.edu> On 17-Jul-09, at 3:57 PM, Jonathan Taylor wrote: > File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/ > __init__.py", > line 47, in > from linalg import * > File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/ > linalg.py", > line 22, in > from numpy.linalg import lapack_lite > ImportError: /usr/local/lib/libptcblas.so: undefined symbol: > ATL_cpttrsm It doesn't look like you ATLAS is linked together properly, specifically fblas. What fortran compiler are you using? What does ldd /usr/local/lib/libptcblas.so say? I seem to recall this sort of thing happening when g77 and gfortran get mixed up together... David From dwf at cs.toronto.edu Fri Jul 17 16:29:08 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 17 Jul 2009 16:29:08 -0400 Subject: [Numpy-discussion] performance matrix multiplication vs. matlab In-Reply-To: <817E9DA4-A61B-4161-9339-B0DC6A615E35@cs.toronto.edu> References: <20090604143638.xrqzx8w8v4ggckkk@www.sms.ed.ac.uk> <4A28EC37.10205@ar.media.kyoto-u.ac.jp> <4A28F16C.3070607@ar.media.kyoto-u.ac.jp> <4A2955F5.9030006@hawaii.edu> <4A295A7D.1040407@ar.media.kyoto-u.ac.jp> <7f014ea60906051437p6002c6f7jb2d77374a220d2aa@mail.gmail.com> <7f014ea60906060959s6570cc32l277c5ab423f0b9ed@mail.gmail.com> <463e11f90907171257m2f5ff9d6v34924e0200145f4@mail.gmail.com> <817E9DA4-A61B-4161-9339-B0DC6A615E35@cs.toronto.edu> Message-ID: <82CE7E00-FD2F-4380-B69C-F2F1D64EA1E7@cs.toronto.edu> On 17-Jul-09, at 4:20 PM, David Warde-Farley wrote: > It doesn't look like you ATLAS is linked together properly, > specifically fblas. What fortran compiler are you using? > > ImportError: /usr/local/lib/libptcblas.so: undefined symbol: ATL_cpttrsm Errr, nevermind. I seem to have very selective vision and saw that as 'ptf77blas.so'. Suffice it to say it's an ATLAS build problem and you seem to be doing everything right given the commands. You remembered ldconfig? David From dpgrote at lbl.gov Fri Jul 17 17:42:30 2009 From: dpgrote at lbl.gov (Dave Grote) Date: Fri, 17 Jul 2009 14:42:30 -0700 Subject: [Numpy-discussion] VIGRA, NumPy and Fortran-order (again) In-Reply-To: References: <200907171654.22107.meine@informatik.uni-hamburg.de> Message-ID: <4A60F046.9050509@lbl.gov> An HTML attachment was scrubbed... URL: From amenity at enthought.com Fri Jul 10 15:46:03 2009 From: amenity at enthought.com (Amenity Applewhite) Date: Fri, 10 Jul 2009 14:46:03 -0500 Subject: [Numpy-discussion] Scientific Computing with Python: July 17th webinar Message-ID: <5B10F30E-1E9B-4832-95AC-C841D8610F56@enthought.com> Greetings! We are pleased to announce our third Scientific Computing with Python webinar! July 17th: Step-by-step Chaco Next week, on Friday, July 17th, we'll be taking a look at Chaco, an component of our open-source Enthought Tool Suite. It seems that this powerful tool impressed participants at last month's webinar, as numerous attendees responded with "Cool!! But how can I do that?" So Peter Wang, Enthought's Director of Technical Architecture, has offered to pick up where he left off last month and give a step-by step introduction to Chaco. This will include a closer examination of the code and more in-depth guidance on how use it for visualizing your data with 2D plotting. This event will take place on Friday, July 17th at 1:00pm CDT (6pm UTC) and will last 60 to 90 minutes, depending on the questions asked. If you would like to participate, please register at https://www1.gotomeeting.com/register/158182457 . Can't get enough scientific computing with Python? This Scientific Computing with Python series focuses on the free and open-source Enthought Tool Suite and and should not be confused with our EPD webinar series. The latter is meant to provide customized support for our EPD Basic or above subscribers, and we initially intended for it to be closed to the public. Many non-subscribers have expressed interest in the series, however, so we have decided to allow non-subscribers to add their name to a waiting list for each EPD webinar. While EPD subscribers will be guaranteed entry and attendance will be capped at 35, we'll draw names from the waiting lists to available seats. These webinars are more intimate, with participatory demonstrations and VOIP question and answer available to attendees, and we think it's a great opportunity for us to support the broader SciPy community. For details, seehttp://www.enthought.com/training/webinars.php . Thanks, and see you Friday! The Enthought Team Forward email This email was sent to staff at enthought.com by amenity at enthought.com. Update Profile/Email Address | Instant removal with SafeUnsubscribe? | Privacy Policy. Enthought, Inc. | 515 Congress Ave. | Suite 2100 | Austin | TX | 78701 _______________________________________________ Staff mailing list Staff at mail.enthought.com https://mail.enthought.com/mailman/listinfo/staff -- Amenity Applewhite Enthought, Inc. Scientific Computing Solutions www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6.jpg Type: image/jpeg Size: 84868 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: safe_unsubscribe_logo.gif Type: image/gif Size: 218 bytes Desc: not available URL: From cournape at gmail.com Sat Jul 18 09:45:13 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 18 Jul 2009 22:45:13 +0900 Subject: [Numpy-discussion] Adding new functions to Numpy In-Reply-To: References: <5b8d13220907160219t50bf7cf2iac91a7914f611105@mail.gmail.com> Message-ID: <5b8d13220907180645h503c148u3e6c02d1582a8c2e@mail.gmail.com> On Fri, Jul 17, 2009 at 4:11 AM, Pauli Virtanen wrote: > On 2009-07-16, David Cournapeau wrote: >> Hi Pauli, >> >> On Sat, Jul 4, 2009 at 9:59 PM, Pauli Virtanen wrote: >>> Hi, >>> >>> When you add new functions to Numpy, please include >>> >>> ? ?.. versionadded:: 1.4.0 >> >> What is the best way to do this in the reference guide directly as >> well (for C API). For example, I added the function npy_copysign for >> 1.4.0, documented in the c-api.coremath.rst, but I don't know where to >> put the versionadded. > > In the Python docs, the versionadded:: typically is the last item > inside the function:: directive. > >> More generally, I would like to follow our >> docstring format when ?documenting C functions, but how can I put the >> different sections in there (Parameters, Notes, etc...) > > Yeah, currently there's no way to do that. I can probably add > numpy*function/cfunction/... etc. directives that allow this. > > An alternative is to use > > ? ? ? ?.. rubric:: Notes > > to make the headings to appear. Ok, that's better than nothing for the time being. I would really like to have a plugin to have autodoc for C code as well as python code, though. One more thing on the TODO list :) David From geometrian at gmail.com Sat Jul 18 14:56:03 2009 From: geometrian at gmail.com (Ian Mallett) Date: Sat, 18 Jul 2009 11:56:03 -0700 Subject: [Numpy-discussion] Optimizing a pure Python Workaround In-Reply-To: <3d375d730907132126g4ff5361bge8072890e26973c1@mail.gmail.com> References: <3d375d730907132126g4ff5361bge8072890e26973c1@mail.gmail.com> Message-ID: Hi, Sorry, I've been away in Oregon... The result isn't quite the same. The arrays must be in the range [0,1], so I just have it divide x3 and y. I also have it add 1 to size[1], as I realized that was also necessary for that behavior: x = np.arange(size[0]) x2 = np.column_stack([x,x+1]).reshape([-1,1]) x3 = np.array(x2.repeat(size[1]+1,axis=1).flatten(),"f") y = np.array(np.arange(size[1]+1).repeat(size[0]*2),"f") array = np.zeros([len(y),3]) array[:,0] = x3/size[0] array[:,1] = y/size[1] array = np.array(array,"f") When size is [3,2], the result from this code is: [[ 0. 0. 0. ] [ 0. 0. 0. ] [ 0. 0. 0. ] [ 0.33333334 0. 0. ] [ 0.33333334 0. 0. ] [ 0.33333334 0. 0. ] [ 0.33333334 0.5 0. ] [ 0.33333334 0.5 0. ] [ 0.33333334 0.5 0. ] [ 0.66666669 0.5 0. ] [ 0.66666669 0.5 0. ] [ 0.66666669 0.5 0. ] [ 0.66666669 1. 0. ] [ 0.66666669 1. 0. ] [ 0.66666669 1. 0. ] [ 1. 1. 0. ] [ 1. 1. 0. ] [ 1. 1. 0. ]] The correct output is: [[ 0. 0. 0. ] [ 0.33333334 0. 0. ] [ 0. 0.5 0. ] [ 0.33333334 0.5 0. ] [ 0. 1. 0. ] [ 0.33333334 1. 0. ] [ 0.33333334 0. 0. ] [ 0.66666669 0. 0. ] [ 0.33333334 0.5 0. ] [ 0.66666669 0.5 0. ] [ 0.33333334 1. 0. ] [ 0.66666669 1. 0. ] [ 0.66666669 0. 0. ] [ 1. 0. 0. ] [ 0.66666669 0.5 0. ] [ 1. 0.5 0. ] [ 0.66666669 1. 0. ] [ 1. 1. 0. ]] Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav+sp at iki.fi Sat Jul 18 19:11:21 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Sat, 18 Jul 2009 23:11:21 +0000 (UTC) Subject: [Numpy-discussion] Adding new functions to Numpy References: <5b8d13220907160219t50bf7cf2iac91a7914f611105@mail.gmail.com> <5b8d13220907180645h503c148u3e6c02d1582a8c2e@mail.gmail.com> Message-ID: On 2009-07-18, David Cournapeau wrote: > On Fri, Jul 17, 2009 at 4:11 AM, Pauli Virtanen wrote: [clip] >> Yeah, currently there's no way to do that. I can probably add >> numpy*function/cfunction/... etc. directives that allow this. >> >> An alternative is to use >> >> ? ? ? ?.. rubric:: Notes >> >> to make the headings to appear. > > Ok, that's better than nothing for the time being. I would really like > to have a plugin to have autodoc for C code as well as python code, > though. One more thing on the TODO list :) Well, you can now use ``np-cfunction::`` for this. I'm not 100% sure this is a wise route to go, but oh well... -- Pauli Virtanen From wright at esrf.fr Sun Jul 19 06:41:20 2009 From: wright at esrf.fr (Jon Wright) Date: Sun, 19 Jul 2009 12:41:20 +0200 Subject: [Numpy-discussion] [Jepp-users] win64 numpy hangs during tests using embedded interpreter In-Reply-To: <4A62EEBF.6090006@esrf.fr> References: <5b8d13220907160219t50bf7cf2iac91a7914f611105@mail.gmail.com> <5b8d13220907180645h503c148u3e6c02d1582a8c2e@mail.gmail.com> <4A62EEBF.6090006@esrf.fr> Message-ID: <4A62F850.5090502@esrf.fr> Ah, bugger, found the answer and it is bad news, from http://docs.python.org/c-api/init.html "Note that the PyGILState_*() functions assume there is only one global interpreter (created automatically by Py_Initialize() ). Python still supports the creation of additional interpreters (using Py_NewInterpreter() ), but mixing multiple interpreters and the PyGILState_*() API is unsupported." Since jepp is using Py_NewInterpreter() to allow many interpreters then it means it won't play nicely with extension modules who use PyGILState, like numpy. I can get around it by disabling threads in numpy altogether, but I'd rather have them working in these days of multi-core CPUs. Any ideas? Thanks, Jon Jon Wright wrote: > Hi There, > > I am trying to use numpy with jepp (from jepp.sourceforge.net) which is > an embedded python interpreter on win64 (vista). With the sourceforge > binaries I get intermittent crashes on import which went away when I > compiled svn trunk with msvc9. Suggests there is something incompatible > about them? > > Jepp has been compiled with Py_OptimizeFlag = 0 in pyembed_startup and > also removing PyRun_SimpleString("__builtins__.__import__ = > jep.jimport\n"); > from pyembed_thread_init (otherwise numpy is not found). > > Using the python.org win64 interpreter it seems to work fine, so I'm > looking for advice on how to debug the problem. Compiling with debugging > on (edit distutils/msvc9compiler.py as described here > http://mail.python.org/pipermail/python-list/2003-October/229543.html ) > I found it is blocking here: > > > > umath.pyd!_error_handler(int method=2, _object * > errobj=0x000000004d774d88, char * errtype=0x000000004c06c280, int > retstatus=1, int * first=0x00000000024f8de0) Line 69 + > > ALLOW_C_API; > 000000004C04A783 call qword ptr [__imp_PyGILState_Ensure > (4C0684C8h)] > -> 000000004C04A789 mov dword ptr [__save__],eax > switch(method) { > > Way up the call stack it seems jepp has done this to call numpy: > > PyEval_AcquireLock(); > prevThread = PyThreadState_Swap(jepThread->tstate); > ... > result = PyRun_String(str, /* = numpy.test() */ > Py_single_input, > jepThread->globals, > jepThread->globals); > > So it seems there is some sort of a threading problem. Does anyone have > any ideas? > > Thanks for any tips, > > Jon > > > ------------------------------------------------------------------------------ > Enter the BlackBerry Developer Challenge > This is your chance to win up to $100,000 in prizes! For a limited time, > vendors submitting new applications to BlackBerry App World(TM) will have > the opportunity to enter the BlackBerry Developer Challenge. See full prize > details at: http://p.sf.net/sfu/Challenge > _______________________________________________ > Jepp-users mailing list > Jepp-users at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/jepp-users > From mark.wendell at gmail.com Sun Jul 19 12:22:08 2009 From: mark.wendell at gmail.com (Mark Wendell) Date: Sun, 19 Jul 2009 10:22:08 -0600 Subject: [Numpy-discussion] list dtype? Message-ID: Is it possible to create a dtype that is a list of arbitrary length? More specifically, let's say I have a custom dtype like this: d = dtype( {'names':['index','xcoord','ycoord','weight'], 'formats': [ uint16, uint16, uint16, float ] } ) Now I'd like to create a 2D array where each entry is a LIST of items of type d. Each list may contain only one instance, or many, and I intend to write routines that will access and extend that list. So I guess the more general question is... how do I create a dtype that is a list, and can the length of that list vary from entry to entry in a 2D array? I hope I've explained this well enough. Thanks in advance. Mark -- -- Mark Wendell "I guess songs are just interesting things to do with the air." Tom Waits From robert.kern at gmail.com Sun Jul 19 16:37:10 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 19 Jul 2009 15:37:10 -0500 Subject: [Numpy-discussion] list dtype? In-Reply-To: References: Message-ID: <3d375d730907191337y4cd0a67fge94ed5f6e2c095c5@mail.gmail.com> On Sun, Jul 19, 2009 at 11:22, Mark Wendell wrote: > Is it possible to create a dtype that is a list of arbitrary length? Use the object dtype. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From mark.wendell at gmail.com Sun Jul 19 18:19:04 2009 From: mark.wendell at gmail.com (Mark Wendell) Date: Sun, 19 Jul 2009 16:19:04 -0600 Subject: [Numpy-discussion] list dtype? In-Reply-To: <3d375d730907191337y4cd0a67fge94ed5f6e2c095c5@mail.gmail.com> References: <3d375d730907191337y4cd0a67fge94ed5f6e2c095c5@mail.gmail.com> Message-ID: Sorry for being a newbie about it, but could you provide an example? I was able to do this: d = np.dtype( {'names':['mylist'], 'formats':[np.object]}) but I'm not sure if that's the correct way to produce a python list-type dtype. Further, how do I then define a new array with that as a dtype? When I do this: a = np.array( (5,5), dtype = d) I get the error: ValueError: size of tuple must match number of fields. Any help appreciated. Thanks! Mark On Sun, Jul 19, 2009 at 2:37 PM, Robert Kern wrote: > On Sun, Jul 19, 2009 at 11:22, Mark Wendell wrote: >> Is it possible to create a dtype that is a list of arbitrary length? > > Use the object dtype. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- -- Mark Wendell "I guess songs are just interesting things to do with the air." Tom Waits From d_l_goldsmith at yahoo.com Sun Jul 19 18:50:15 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Sun, 19 Jul 2009 15:50:15 -0700 (PDT) Subject: [Numpy-discussion] list dtype? Message-ID: <901116.33983.qm@web52107.mail.re2.yahoo.com> Robert'll correct me if I'm wrong, but I think what he meant is that all you have to do is: array_of_lists = numpy.empty(, dtype=object) and you'll be able to fill array_of_lists with whatever you want: >>> aol = np.empty((2,2),dtype=object) >>> aol[:,:] = [[[1, 2, 3], ['a','b','c']], [[(), (), ()], [{}, {}, {}]]] >>> aol array([[[1, 2, 3], ['a', 'b', 'c']], [[(), (), ()], [{}, {}, {}]]], dtype=object) However, you do have to specify the shape ahead of time: >>> aol = np.empty((),dtype=object) >>> aol[:,:] = [list([5, 6, 7]),list(['a', 'b', 'c'])] Traceback (most recent call last): File "", line 1, in IndexError: 0-d arrays can't be indexed. >>> aol.shape = (2,2) Traceback (most recent call last): File "", line 1, in ValueError: total size of new array must be unchanged (Of course, if you need to change its size and shape, you can always instantiate a new array, initializing the "old" portion of it w/ the existing array and filling in the rest w/ the new values.) DG --- On Sun, 7/19/09, Mark Wendell wrote: > From: Mark Wendell > Subject: Re: [Numpy-discussion] list dtype? > To: "Discussion of Numerical Python" > Date: Sunday, July 19, 2009, 3:19 PM > Sorry for being a newbie about it, > but could you provide an example? > > I was able to do this: > > d = np.dtype( {'names':['mylist'], 'formats':[np.object]}) > but I'm not sure if that's the correct way to produce a > python list-type dtype. > > Further, how do I then define a new array with that as a > dtype? > > When I do this: > a = np.array( (5,5), dtype = d) > I get the error: > ValueError: size of tuple must match number of fields. > > > Any help appreciated. Thanks! > Mark > > On Sun, Jul 19, 2009 at 2:37 PM, Robert Kern > wrote: > > On Sun, Jul 19, 2009 at 11:22, Mark Wendell > wrote: > >> Is it possible to create a dtype that is a list of > arbitrary length? > > > > Use the object dtype. > > > > -- > > Robert Kern > > > > "I have come to believe that the whole world is an > enigma, a harmless > > enigma that is made terrible by our own mad attempt to > interpret it as > > though it had an underlying truth." > > ?-- Umberto Eco > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > -- > Mark Wendell > "I guess songs are just interesting things to do with the > air." Tom Waits > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From mark.wendell at gmail.com Sun Jul 19 18:59:52 2009 From: mark.wendell at gmail.com (Mark Wendell) Date: Sun, 19 Jul 2009 16:59:52 -0600 Subject: [Numpy-discussion] list dtype? In-Reply-To: <901116.33983.qm@web52107.mail.re2.yahoo.com> References: <901116.33983.qm@web52107.mail.re2.yahoo.com> Message-ID: Excellent! I guess I was over-thinking it. Many thanks. Mark On Sun, Jul 19, 2009 at 4:50 PM, David Goldsmith wrote: > > Robert'll correct me if I'm wrong, but I think what he meant is that all you have to do is: > > array_of_lists = numpy.empty(, dtype=object) > > and you'll be able to fill array_of_lists with whatever you want: > >>>> aol = np.empty((2,2),dtype=object) >>>> aol[:,:] = [[[1, 2, 3], ['a','b','c']], [[(), (), ()], [{}, {}, {}]]] >>>> aol > array([[[1, 2, 3], ['a', 'b', 'c']], > ? ? ? [[(), (), ()], [{}, {}, {}]]], dtype=object) > > However, you do have to specify the shape ahead of time: > >>>> aol = np.empty((),dtype=object) >>>> aol[:,:] = [list([5, 6, 7]),list(['a', 'b', 'c'])] > Traceback (most recent call last): > ?File "", line 1, in > IndexError: 0-d arrays can't be indexed. >>>> aol.shape = (2,2) > Traceback (most recent call last): > ?File "", line 1, in > ValueError: total size of new array must be unchanged > > (Of course, if you need to change its size and shape, you can always instantiate a new array, initializing the "old" portion of it w/ the existing array and filling in the rest w/ the new values.) > > DG > > --- On Sun, 7/19/09, Mark Wendell wrote: > >> From: Mark Wendell >> Subject: Re: [Numpy-discussion] list dtype? >> To: "Discussion of Numerical Python" >> Date: Sunday, July 19, 2009, 3:19 PM >> Sorry for being a newbie about it, >> but could you provide an example? >> >> I was able to do this: >> >> d = np.dtype( {'names':['mylist'], 'formats':[np.object]}) >> but I'm not sure if that's the correct way to produce a >> python list-type dtype. >> >> Further, how do I then define a new array with that as a >> dtype? >> >> When I do this: >> a = np.array( (5,5), dtype = d) >> I get the error: >> ValueError: size of tuple must match number of fields. >> >> >> Any help appreciated. Thanks! >> Mark >> >> On Sun, Jul 19, 2009 at 2:37 PM, Robert Kern >> wrote: >> > On Sun, Jul 19, 2009 at 11:22, Mark Wendell >> wrote: >> >> Is it possible to create a dtype that is a list of >> arbitrary length? >> > >> > Use the object dtype. >> > >> > -- >> > Robert Kern >> > >> > "I have come to believe that the whole world is an >> enigma, a harmless >> > enigma that is made terrible by our own mad attempt to >> interpret it as >> > though it had an underlying truth." >> > ?-- Umberto Eco >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> >> >> >> -- >> -- >> Mark Wendell >> "I guess songs are just interesting things to do with the >> air." Tom Waits >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- -- Mark Wendell From d_l_goldsmith at yahoo.com Sun Jul 19 19:05:47 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Sun, 19 Jul 2009 16:05:47 -0700 (PDT) Subject: [Numpy-discussion] list dtype? Message-ID: <114844.66776.qm@web52102.mail.re2.yahoo.com> 99 times out of 100, the way to do things in Numpy (as in Python) is simpler than what you thought - that's why we love 'em. ;-) DG --- On Sun, 7/19/09, Mark Wendell wrote: > From: Mark Wendell > Subject: Re: [Numpy-discussion] list dtype? > To: "Discussion of Numerical Python" > Date: Sunday, July 19, 2009, 3:59 PM > Excellent! I guess I was > over-thinking it. > > Many thanks. > Mark > > On Sun, Jul 19, 2009 at 4:50 PM, David Goldsmith > wrote: > > > > Robert'll correct me if I'm wrong, but I think what he > meant is that all you have to do is: > > > > array_of_lists = numpy.empty(, > dtype=object) > > > > and you'll be able to fill array_of_lists with > whatever you want: > > > >>>> aol = np.empty((2,2),dtype=object) > >>>> aol[:,:] = [[[1, 2, 3], ['a','b','c']], > [[(), (), ()], [{}, {}, {}]]] > >>>> aol > > array([[[1, 2, 3], ['a', 'b', 'c']], > > ? ? ? [[(), (), ()], [{}, {}, {}]]], dtype=object) > > > > However, you do have to specify the shape ahead of > time: > > > >>>> aol = np.empty((),dtype=object) > >>>> aol[:,:] = [list([5, 6, 7]),list(['a', > 'b', 'c'])] > > Traceback (most recent call last): > > ?File "", line 1, in > > IndexError: 0-d arrays can't be indexed. > >>>> aol.shape = (2,2) > > Traceback (most recent call last): > > ?File "", line 1, in > > ValueError: total size of new array must be unchanged > > > > (Of course, if you need to change its size and shape, > you can always instantiate a new array, initializing the > "old" portion of it w/ the existing array and filling in the > rest w/ the new values.) > > > > DG > > > > --- On Sun, 7/19/09, Mark Wendell > wrote: > > > >> From: Mark Wendell > >> Subject: Re: [Numpy-discussion] list dtype? > >> To: "Discussion of Numerical Python" > >> Date: Sunday, July 19, 2009, 3:19 PM > >> Sorry for being a newbie about it, > >> but could you provide an example? > >> > >> I was able to do this: > >> > >> d = np.dtype( {'names':['mylist'], > 'formats':[np.object]}) > >> but I'm not sure if that's the correct way to > produce a > >> python list-type dtype. > >> > >> Further, how do I then define a new array with > that as a > >> dtype? > >> > >> When I do this: > >> a = np.array( (5,5), dtype = d) > >> I get the error: > >> ValueError: size of tuple must match number of > fields. > >> > >> > >> Any help appreciated. Thanks! > >> Mark > >> > >> On Sun, Jul 19, 2009 at 2:37 PM, Robert Kern > >> wrote: > >> > On Sun, Jul 19, 2009 at 11:22, Mark > Wendell > >> wrote: > >> >> Is it possible to create a dtype that is > a list of > >> arbitrary length? > >> > > >> > Use the object dtype. > >> > > >> > -- > >> > Robert Kern > >> > > >> > "I have come to believe that the whole world > is an > >> enigma, a harmless > >> > enigma that is made terrible by our own mad > attempt to > >> interpret it as > >> > though it had an underlying truth." > >> > ?-- Umberto Eco > >> > > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > >> > >> > >> -- > >> -- > >> Mark Wendell > >> "I guess songs are just interesting things to do > with the > >> air." Tom Waits > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > -- > Mark Wendell > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pinto at mit.edu Sun Jul 19 23:35:49 2009 From: pinto at mit.edu (Nicolas Pinto) Date: Sun, 19 Jul 2009 23:35:49 -0400 Subject: [Numpy-discussion] performance matrix multiplication vs. matlab In-Reply-To: <463e11f90907171257m2f5ff9d6v34924e0200145f4@mail.gmail.com> References: <20090604143638.xrqzx8w8v4ggckkk@www.sms.ed.ac.uk> <4A28F16C.3070607@ar.media.kyoto-u.ac.jp> <4A2955F5.9030006@hawaii.edu> <4A295A7D.1040407@ar.media.kyoto-u.ac.jp> <7f014ea60906051437p6002c6f7jb2d77374a220d2aa@mail.gmail.com> <7f014ea60906060959s6570cc32l277c5ab423f0b9ed@mail.gmail.com> <463e11f90907171257m2f5ff9d6v34924e0200145f4@mail.gmail.com> Message-ID: <954ae5aa0907192035i6b48b54rf2b4d1621720d8e6@mail.gmail.com> Jonathan, What does "ldd /home/jtaylor/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so" say ? You need to make sure that it's using the libraries in /usr/local/lib. You can remove the ones in /usr/lib or "export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH". Hope it helps. Best, N On Fri, Jul 17, 2009 at 3:57 PM, Jonathan Taylor wrote: > Following these instructions I have the following problem when I > import numpy. ?Does anyone know why this might be? > > Thanks, > Jonathan. > >>>> import numpy > Traceback (most recent call last): > ?File "", line 1, in > ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/__init__.py", > line 130, in > ? ?import add_newdocs > ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/add_newdocs.py", > line 9, in > ? ?from lib import add_newdoc > ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/lib/__init__.py", > line 13, in > ? ?from polynomial import * > ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/lib/polynomial.py", > line 18, in > ? ?from numpy.linalg import eigvals, lstsq > ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/__init__.py", > line 47, in > ? ?from linalg import * > ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/linalg.py", > line 22, in > ? ?from numpy.linalg import lapack_lite > ImportError: /usr/local/lib/libptcblas.so: undefined symbol: ATL_cpttrsm > > > On Sat, Jun 6, 2009 at 12:59 PM, Chris Colbert wrote: >> since there is demand, and someone already emailed me, I'll put what I >> did in this post. It pretty much follows whats on the scipy website, >> with a couple other things I gleaned from reading the ATLAS install >> guide: >> >> and here it goes, this is valid for Ubuntu 9.04 64-bit ?(# starts a >> comment when working in the terminal) >> >> >> download lapack 3.2.1 http://www.netlib.org/lapack/lapack.tgz >> download atlas 3.8.3 >> http://sourceforge.net/project/downloading.php?group_id=23725&filename=atlas3.8.3.tar.bz2&a=65663372 >> >> create folder ?/home/your-user-name/build/atlas ? #this is where we build >> create folder /home/your-user-name/build/lapack #atlas and lapack >> >> extract the folder lapack-3.2.1 to /home/your-user-name/build/lapack >> extract the contents of atlas to /home/your-user-name/build/atlas >> >> >> >> now in the terminal: >> >> # remove g77 and get stuff we need >> sudo apt-get remove g77 >> sudo apt-get install gfortran >> sudo apt-get install build-essential >> sudo apt-get install python-dev >> sudo apt-get install python-setuptools >> sudo easy_install nose >> >> >> # build lapack >> cd /home/your-user-name/build/lapack/lapack-3.2.1 >> cp INSTALL/make.inc.gfortran make.inc >> >> gedit make.inc >> ################# >> #in the make.inc file make sure the line ? OPTS = -O2 -fPIC -m64 >> #and ? ?NOOPTS = -O0 -fPIC -m64 >> #the -m64 flags build 64-bit code, if you want 32-bit, simply leave >> #the -m64 flags out >> ################# >> >> cd SRC >> >> #this should build lapack without error >> make >> >> >> >> # build atlas >> >> cd /home/your-user-name/build/atlas >> >> #this is simply where we will build the atlas >> #libs, you can name it what you want >> mkdir Linux_X64SSE2 >> >> cd Linux_X64SSE2 >> >> #need to turn off cpu-throttling >> sudo cpufreq-selector -g performance >> >> #if you don't want 64bit code remove the -b 64 flag. replace the >> #number 2400 with your CPU frequency in MHZ >> #i.e. my cpu is 2.53 GHZ so i put 2530 >> ../configure -b 64 -D c -DPentiumCPS=2400 -Fa ?-alg -fPIC >> --with-netlib-lapack=/home/your-user-name/build/lapack/lapack-3.2.1/Lapack_LINUX.a >> >> #the configure step takes a bit, and should end without errors >> >> ?#this takes a long time, go get some coffee, it should end without error >> make build >> >> #this will verify the build, also long running >> make check >> >> #this will test the performance of your build and give you feedback on >> #it. your numbers should be close to the test numbers at the end >> make time >> >> cd lib >> >> #builds single threaded .so's >> make shared >> >> #builds multithreaded .so's >> make ptshared >> >> #copies all of the atlas libs (and the lapack lib built with atlas) >> #to our lib dir >> sudo ?cp ?*.so ?/usr/local/lib/ >> >> >> >> #now we need to get and build numpy >> >> download numpy 1.3.0 >> http://sourceforge.net/project/downloading.php?group_id=1369&filename=numpy-1.3.0.tar.gz&a=93506515 >> >> extract the folder numpy-1.3.0 to /home/your-user-name/build >> >> #in the terminal >> >> cd /home/your-user-name/build/numpy-1.3.0 >> cp site.cfg.example site.cfg >> >> gedit site.cfg >> ############################################### >> # in site.cfg uncomment the following lines and make them look like these >> [DEFAULT] >> library_dirs = /usr/local/lib >> include_dirs = /usr/local/include >> >> [blas_opt] >> libraries = ptf77blas, ptcblas, atlas >> >> [lapack_opt] >> libraries = lapack, ptf77blas, ptcblas, atlas >> ################################################### >> #if you want single threaded libs, uncomment those lines instead >> >> >> #build numpy- should end without error >> python setup.py build >> >> #install numpy >> python setup.py install >> >> cd /home >> >> sudo ldconfig >> >> python >>>>import numpy >>>>numpy.test() ? #this should run with no errors (skipped tests and known-fails are ok) >>>>a = numpy.random.randn(6000, 6000) >>>>numpy.dot(a, a) ? ? # look at your cpu monitor and verify all cpu cores are at 100% if you built with threads >> >> >> Celebrate with a beer! >> >> >> Cheers! >> >> Chris >> >> >> >> >> >> On Sat, Jun 6, 2009 at 10:42 AM, Keith Goodman wrote: >>> On Fri, Jun 5, 2009 at 2:37 PM, Chris Colbert wrote: >>>> I'll caution anyone from using Atlas from the repos in Ubuntu 9.04? as the >>>> package is broken: >>>> >>>> https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510 >>>> >>>> >>>> just build Atlas yourself, you get better performance AND threading. >>>> Building it is not the nightmare it sounds like. I think i've done it a >>>> total of four times now, both 32-bit and 64-bit builds. >>>> >>>> If you need help with it,? just email me off list. >>> >>> That's a nice offer. I tried building ATLAS on Debian a year or two >>> ago and got stuck. >>> >>> Clear out your inbox! >>> _______________________________________________ >>> Numpy-discussion mailing list >>> Numpy-discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Nicolas Pinto Ph.D. Candidate, Brain & Computer Sciences Massachusetts Institute of Technology, USA http://web.mit.edu/pinto From pinto at mit.edu Mon Jul 20 02:42:38 2009 From: pinto at mit.edu (Nicolas Pinto) Date: Mon, 20 Jul 2009 02:42:38 -0400 Subject: [Numpy-discussion] ANN: line_profiler 1.0b2 released In-Reply-To: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com> References: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com> Message-ID: <954ae5aa0907192342h7222db91u5b69d71174c9f43e@mail.gmail.com> Thanks a lot Robert for this very useful tool! I was wondering if there is a way to make it work with Cython code (see below) ? Sincerely, Nicolas fib.pyx: @profile def fib(n): """Print the Fibonacci series up to n.""" a, b = 0, 1 while b < n: a, b = b, a + b test.py: import pyximport pyximport.install() import fib fib.fib(100) % python kernprof.py -vl test.py Wrote profile results to test.py.lprof Timer unit: 1e-06 s Traceback (most recent call last): File "kernprof.py", line 167, in sys.exit(main(sys.argv)) File "kernprof.py", line 155, in main execfile(script_file, ns, ns) File "test.py", line 5, in import fib File "/usr/lib/python2.5/site-packages/Cython-0.11.2-py2.5-linux-x86_64.egg/pyximport/pyximport.py", line 288, in load_module self.pyxbuild_dir) File "/usr/lib/python2.5/site-packages/Cython-0.11.2-py2.5-linux-x86_64.egg/pyximport/pyximport.py", line 154, in load_module raise ImportError("Building module failed: %s" % e) ImportError: Building module failed: 'builtin_function_or_method' object has no attribute '__dict__' On Thu, Feb 19, 2009 at 5:21 PM, Robert Kern wrote: > http://pypi.python.org/pypi/line_profiler/ > http://packages.python.org/line_profiler/ > > This release fixes the "negative timings" issue on Windows. > > Future announcements will occur on python-announce. I just wanted to > make sure my earliest users here who ran into this bug are aware of > the fix. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Nicolas Pinto Ph.D. Candidate, Brain & Computer Sciences Massachusetts Institute of Technology, USA http://web.mit.edu/pinto -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjhnson at gmail.com Mon Jul 20 06:27:59 2009 From: tjhnson at gmail.com (T J) Date: Mon, 20 Jul 2009 03:27:59 -0700 Subject: [Numpy-discussion] Repeated Dot Operations Message-ID: Hi, Is there a good way to perform dot on an arbitrary list of arrays which avoids using a loop? Here is what I'd like to avoid: # m1, m2, m3 ?are arrays >>> out = np.(m1.shape[0]) >>> prod = [m1, m2, m3, m1, m2, m3, m3, m2] >>> for m in prod: ... out = np.dot(out, m) ... I was hoping for something like np.dot.reduce(prod). From nwagner at iam.uni-stuttgart.de Mon Jul 20 06:34:20 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 20 Jul 2009 12:34:20 +0200 Subject: [Numpy-discussion] String manipulation In-Reply-To: <4A083AAE.1010209@american.edu> References: <200905111425.46475.faltet@pytables.org> <4A083AAE.1010209@american.edu> Message-ID: On Mon, 11 May 2009 10:48:14 -0400 Alan G Isaac wrote: > On 5/11/2009 8:36 AM Nils Wagner apparently wrote: >> I would like to split strings made of digits after eight >> characters each. > > > [l[i*8:(i+1)*8] for i in range(len(l)/8)] > > Alan Isaac > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion 1.000000E+00 0.000000E+00 1.000000E+00 1.000000E+00 1.000000E+00 1.000000E+00 -1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00 1.250000E+00 1.250000E+00 ifile = open('mydata','r') lines = ifile.readlines() for line in lines: print line.split() How can I split the second line in such a way that I get ['-1.000000E+00', '-1.000000E+00', '-1.000000E+00', '-1.000000E+00', '1.250000E+00', '1.250000E+00'] instead of ['-1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00', '1.250000E+00', '1.250000E+00'] Thanks in advance Nils From washakie at gmail.com Mon Jul 20 07:48:25 2009 From: washakie at gmail.com (John [H2O]) Date: Mon, 20 Jul 2009 04:48:25 -0700 (PDT) Subject: [Numpy-discussion] re carray and datetime objects Message-ID: <24568340.post@talk.nabble.com> Hello, I have a file containing mixed data types: strings, floats, datetime output(i.e. strings), and ints. Something like: #ID, name, date, value 1,sample,2008-07-10 12:34:20,344.56 And so forth. It seems using recarrays is efficient and a prefered habit to get into wrg to numpy, so I am trying the following: dtype=[('ID','int'),('name','|S10'),('date','????'),('value','float')] DATA = np.loadtxt(infile,dtype=dtype) This works if I set 'date' type to 'S20', but wouldn't it be more appropriate to use object? Also, what is the next step to then use the values as datetime objects? Do I have to take the data out of the array, convert to datetime, then put back in the array? dates = DATA['date'] dates = [dt.datetime.strptime(d,'%Y-%m-%d %H:%M:%S') for d in dates] DATA['dates'] = np.array(dates) ???? Thanks! -- View this message in context: http://www.nabble.com/recarray-and-datetime-objects-tp24568340p24568340.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From washakie at gmail.com Mon Jul 20 07:54:26 2009 From: washakie at gmail.com (John [H2O]) Date: Mon, 20 Jul 2009 04:54:26 -0700 (PDT) Subject: [Numpy-discussion] indexing recarrays Message-ID: <24568432.post@talk.nabble.com> I have a file containing mixed data types: strings, floats, datetime output(i.e. strings), and ints. Something like: #ID, name, date, value 1,sample,2008-07-10 12:34:20,344.56 Presuming I get them nicely into a recarray (see my other http://www.nabble.com/recarray-and-datetime-objects-td24568340.html post ) then I would like to work with them using the convention that I have become used to: D[:,0] #means D for all rows, column 0 But it seems now I have to use: D['columnID'] Am I missing something, or is there a way to keep referencing columns by the column integer index?? Thanks! On another note... are recarrays and structured arrays the same?? -- View this message in context: http://www.nabble.com/indexing-recarrays-tp24568432p24568432.html Sent from the Numpy-discussion mailing list archive at Nabble.com. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Mon Jul 20 11:39:50 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 20 Jul 2009 08:39:50 -0700 Subject: [Numpy-discussion] My identity Message-ID: Using a trick that Robert Kern recently posted to the list makes the identity function much faster. Current version: def identity(n, dtype=None): a = array([1]+n*[0],dtype=dtype) b = empty((n,n),dtype=dtype) b.flat = a return b Proposed version: def myidentity(n, dtype=None): a = zeros((n,n), dtype=dtype) a.flat[::n+1] = 1 return a >> timeit identity(1) 100000 loops, best of 3: 14.9 ?s per loop >> timeit identity(10) 10000 loops, best of 3: 20 ?s per loop >> timeit identity(100) 1000 loops, best of 3: 696 ?s per loop >> timeit identity(1000) 10 loops, best of 3: 73.6 ms per loop >> timeit myidentity(1) 100000 loops, best of 3: 6.57 ?s per loop >> timeit myidentity(10) 100000 loops, best of 3: 7.08 ?s per loop >> timeit myidentity(100) 100000 loops, best of 3: 16.4 ?s per loop >> timeit myidentity(1000) 100 loops, best of 3: 5.92 ms per loop It would also speed up the functions that use identity (for example np.linalg.inv). And it would make identity faster than eye. Are there any corner cases that the new version doesn't handle? Any other issues? From charlesr.harris at gmail.com Mon Jul 20 12:03:31 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 20 Jul 2009 10:03:31 -0600 Subject: [Numpy-discussion] My identity In-Reply-To: References: Message-ID: On Mon, Jul 20, 2009 at 9:39 AM, Keith Goodman wrote: > Using a trick that Robert Kern recently posted to the list makes the > identity function much faster. > > Current version: > > def identity(n, dtype=None): > a = array([1]+n*[0],dtype=dtype) > b = empty((n,n),dtype=dtype) > b.flat = a > return b > > Proposed version: > > def myidentity(n, dtype=None): > a = zeros((n,n), dtype=dtype) > a.flat[::n+1] = 1 > return a > > >> timeit identity(1) > 100000 loops, best of 3: 14.9 ?s per loop > >> timeit identity(10) > 10000 loops, best of 3: 20 ?s per loop > >> timeit identity(100) > 1000 loops, best of 3: 696 ?s per loop > >> timeit identity(1000) > 10 loops, best of 3: 73.6 ms per loop > > >> timeit myidentity(1) > 100000 loops, best of 3: 6.57 ?s per loop > >> timeit myidentity(10) > 100000 loops, best of 3: 7.08 ?s per loop > >> timeit myidentity(100) > 100000 loops, best of 3: 16.4 ?s per loop > >> timeit myidentity(1000) > 100 loops, best of 3: 5.92 ms per loop > > It would also speed up the functions that use identity (for example > np.linalg.inv). And it would make identity faster than eye. > > Are there any corner cases that the new version doesn't handle? Any > other issues? I believe r7130 made the change. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Mon Jul 20 12:05:16 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 20 Jul 2009 12:05:16 -0400 Subject: [Numpy-discussion] indexing recarrays In-Reply-To: <24568432.post@talk.nabble.com> References: <24568432.post@talk.nabble.com> Message-ID: On Jul 20, 2009, at 7:54 AM, John [H2O] wrote: > I have a file containing mixed data types: strings, floats, datetime > output(i.e. strings), and ints. Something like: > #ID, name, date, value 1,sample,2008-07-10 12:34:20,344.56 > Presuming I get them nicely into a recarray (see my other post) then > I would like to work with them using the convention that I have > become used to: > > D[:,0] #means D for all rows, column 0 > > But it seems now I have to use: > > D['columnID'] > > Am I missing something, or is there a way to keep referencing > columns by the column integer index?? > Check the shape of your array: it should be (N,), where N is the number of lines. That should let you know that your array is 1D, without any columns. What you think as columns is actually a field, and you must access it as D['columnID']. Now, it is possible to have a 2D structured array, like this one. >>> x=array([[(1, 1.0), (2, 2.0)], [(3, 3.0), (4, 4.0)]], dtype=[('f0', '>> x[:,0] array([(1, 1.0), (3, 3.0)], dtype=[('f0', ' On another note... are recarrays and structured arrays the same?? > Not quite. a recarray is a special structured array that lets you access fields as attributes (like in D.columnID) as well as items (like in D['columnID']). Because of this added functionality, recarrays are a bit slower to manipulate. From robert.kern at gmail.com Mon Jul 20 12:23:11 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 20 Jul 2009 11:23:11 -0500 Subject: [Numpy-discussion] ANN: line_profiler 1.0b2 released In-Reply-To: <954ae5aa0907192342h7222db91u5b69d71174c9f43e@mail.gmail.com> References: <3d375d730902191321m396f7ddeu8b84d9fdc562c952@mail.gmail.com> <954ae5aa0907192342h7222db91u5b69d71174c9f43e@mail.gmail.com> Message-ID: <3d375d730907200923l197d0e5ah3afe057dbd48defd@mail.gmail.com> On Mon, Jul 20, 2009 at 01:42, Nicolas Pinto wrote: > Thanks a lot Robert for this very useful tool! > > I was wondering if there is a way to make it work with Cython code (see > below) ? No, line_profiler cannot work with Cython. There is some talk on the Cython mailing list about implementing function profiling and line profiling support in Cython, but it won't be related to line_profiler for a variety of technical reasons. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwgoodman at gmail.com Mon Jul 20 12:32:03 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 20 Jul 2009 09:32:03 -0700 Subject: [Numpy-discussion] My identity In-Reply-To: References: Message-ID: On Mon, Jul 20, 2009 at 9:03 AM, Charles R Harris wrote: > > > On Mon, Jul 20, 2009 at 9:39 AM, Keith Goodman wrote: >> >> Using a trick that Robert Kern recently posted to the list makes the >> identity function much faster. >> >> Current version: >> >> def identity(n, dtype=None): >> ? ?a = array([1]+n*[0],dtype=dtype) >> ? ?b = empty((n,n),dtype=dtype) >> ? ?b.flat = a >> ? ?return b >> >> Proposed version: >> >> def myidentity(n, dtype=None): >> ? ?a = zeros((n,n), dtype=dtype) >> ? ?a.flat[::n+1] = 1 >> ? ?return a >> >> >> timeit identity(1) >> 100000 loops, best of 3: 14.9 ?s per loop >> >> timeit identity(10) >> 10000 loops, best of 3: 20 ?s per loop >> >> timeit identity(100) >> 1000 loops, best of 3: 696 ?s per loop >> >> timeit identity(1000) >> 10 loops, best of 3: 73.6 ms per loop >> >> >> timeit myidentity(1) >> 100000 loops, best of 3: 6.57 ?s per loop >> >> timeit myidentity(10) >> 100000 loops, best of 3: 7.08 ?s per loop >> >> timeit myidentity(100) >> 100000 loops, best of 3: 16.4 ?s per loop >> >> timeit myidentity(1000) >> 100 loops, best of 3: 5.92 ms per loop >> >> It would also speed up the functions that use identity (for example >> np.linalg.inv). And it would make identity faster than eye. >> >> Are there any corner cases that the new version doesn't handle? Any >> other issues? > > I believe r7130 made the change. I don't see it here (r7153): http://svn.scipy.org/svn/numpy/trunk/numpy/core/numeric.py Am I looking in the wrong place? From robert.kern at gmail.com Mon Jul 20 12:36:02 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 20 Jul 2009 11:36:02 -0500 Subject: [Numpy-discussion] Repeated Dot Operations In-Reply-To: References: Message-ID: <3d375d730907200936n5ff23cd3o21eca0d6dfe62341@mail.gmail.com> On Mon, Jul 20, 2009 at 05:27, T J wrote: > Hi, > > Is there a good way to perform dot on an arbitrary list of arrays > which avoids using a loop? ?Here is what I'd like to avoid: > > # m1, m2, m3 ?are arrays >>>> out = np.(m1.shape[0]) >>>> prod = [m1, m2, m3, m1, m2, m3, m3, m2] >>>> for m in prod: > ... out = np.dot(out, m) > ... There isn't really a need to avoid a loop in this case. The lists are almost certainly not going to be very long. Wrap it up in a function if you want the convenience of a function call. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwgoodman at gmail.com Mon Jul 20 12:45:03 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 20 Jul 2009 09:45:03 -0700 Subject: [Numpy-discussion] My identity In-Reply-To: References: Message-ID: On Mon, Jul 20, 2009 at 9:32 AM, Keith Goodman wrote: > On Mon, Jul 20, 2009 at 9:03 AM, Charles R > Harris wrote: >> >> >> On Mon, Jul 20, 2009 at 9:39 AM, Keith Goodman wrote: >>> >>> Using a trick that Robert Kern recently posted to the list makes the >>> identity function much faster. >>> >>> Current version: >>> >>> def identity(n, dtype=None): >>> ? ?a = array([1]+n*[0],dtype=dtype) >>> ? ?b = empty((n,n),dtype=dtype) >>> ? ?b.flat = a >>> ? ?return b >>> >>> Proposed version: >>> >>> def myidentity(n, dtype=None): >>> ? ?a = zeros((n,n), dtype=dtype) >>> ? ?a.flat[::n+1] = 1 >>> ? ?return a >>> >>> >> timeit identity(1) >>> 100000 loops, best of 3: 14.9 ?s per loop >>> >> timeit identity(10) >>> 10000 loops, best of 3: 20 ?s per loop >>> >> timeit identity(100) >>> 1000 loops, best of 3: 696 ?s per loop >>> >> timeit identity(1000) >>> 10 loops, best of 3: 73.6 ms per loop >>> >>> >> timeit myidentity(1) >>> 100000 loops, best of 3: 6.57 ?s per loop >>> >> timeit myidentity(10) >>> 100000 loops, best of 3: 7.08 ?s per loop >>> >> timeit myidentity(100) >>> 100000 loops, best of 3: 16.4 ?s per loop >>> >> timeit myidentity(1000) >>> 100 loops, best of 3: 5.92 ms per loop >>> >>> It would also speed up the functions that use identity (for example >>> np.linalg.inv). And it would make identity faster than eye. >>> >>> Are there any corner cases that the new version doesn't handle? Any >>> other issues? >> >> I believe r7130 made the change. > > I don't see it here (r7153): > > http://svn.scipy.org/svn/numpy/trunk/numpy/core/numeric.py > > Am I looking in the wrong place? Oh, sorry, I misunderstood. Yes, a similar change was made to eye but not to identity. From pav+sp at iki.fi Mon Jul 20 13:53:35 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Mon, 20 Jul 2009 17:53:35 +0000 (UTC) Subject: [Numpy-discussion] My identity References: Message-ID: On 2009-07-20, Keith Goodman wrote: [clip] > Oh, sorry, I misunderstood. Yes, a similar change was made to eye but > not to identity. Nasty, duplicated code there it seems... -- Pauli Virtanen From kwgoodman at gmail.com Mon Jul 20 14:17:16 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 20 Jul 2009 11:17:16 -0700 Subject: [Numpy-discussion] My identity In-Reply-To: References: Message-ID: On Mon, Jul 20, 2009 at 10:53 AM, Pauli Virtanen wrote: > On 2009-07-20, Keith Goodman wrote: > [clip] >> Oh, sorry, I misunderstood. Yes, a similar change was made to eye but >> not to identity. > > Nasty, duplicated code there it seems... So def myidentity2(n, dtype=None): return eye(n, M=n, k=0, dtype=dtype) >> timeit myidentity(10) 100000 loops, best of 3: 7.15 ?s per loop >> timeit eye(10) 100000 loops, best of 3: 8.42 ?s per loop >> timeit myidentity2(10) 100000 loops, best of 3: 9 ?s per loop There also a second identity in np.matlib that has its own code. That could be changed to def myidentity2_matlib(n, dtype=None): return asmatrix(eye(n, M=n, k=0, dtype=dtype)) or def myidentity2_matlib(n, dtype=None): return asmatrix(myidentity2(n, dtype)) From brian.lewis17 at gmail.com Mon Jul 20 15:12:52 2009 From: brian.lewis17 at gmail.com (Brian Lewis) Date: Mon, 20 Jul 2009 12:12:52 -0700 Subject: [Numpy-discussion] Update numpy.lib.ufunclike.log2 Message-ID: Should numpy.lib.ufunclike.log2 be updated to: x = nx.asanyarray(x) if y is None: y = nx.log2(x) else: nx.log2(x, y) return y -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.lewis17 at gmail.com Mon Jul 20 15:24:20 2009 From: brian.lewis17 at gmail.com (Brian Lewis) Date: Mon, 20 Jul 2009 12:24:20 -0700 Subject: [Numpy-discussion] Update numpy.lib.ufunclike.log2 In-Reply-To: References: Message-ID: On Mon, Jul 20, 2009 at 12:12 PM, Brian Lewis wrote: > Should numpy.lib.ufunclike.log2 be updated to: > > x = nx.asanyarray(x) > if y is None: > y = nx.log2(x) > else: > nx.log2(x, y) > return y > > Or perhaps removed since numpy.core.umath.log2 exists... -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Jul 20 15:44:23 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 20 Jul 2009 12:44:23 -0700 Subject: [Numpy-discussion] String manipulation In-Reply-To: References: <200905111425.46475.faltet@pytables.org> <4A083AAE.1010209@american.edu> Message-ID: <4A64C917.4090406@noaa.gov> Nils Wagner wrote: > How can I split the second line in such a way that I get > > ['-1.000000E+00', '-1.000000E+00', '-1.000000E+00', > '-1.000000E+00', '1.250000E+00', '1.250000E+00'] > > instead of > > ['-1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00', > '1.250000E+00', '1.250000E+00'] It looks like you have fixed-length fields. The naive way do do this is simple string slicing: def line2array1(line, field_len=10): nums = [] i = 0 while i < len(line): nums.append(float(line[i:i+field_len])) i += field_len return np.array(nums) Then I saw the nifty list comprehension posted by Alan(?), which led me to the one (long) liner: def line2array2(line, field_len=10): return np.array(map(float, [line[i*field_len:(i+1)*field_len] for i in range(len(line)/field_len)])) But it seems I should be able to do this using numpy arrays manipulating the data as characters. However, I had a little trouble getting a string into a numpy array as characters. This didn't work: In [55]: s Out[55]: '-1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00 1.250000E+00 1.250000E+00' In [57]: np.array(s, 'S13') Out[57]: array('-1.000000E+00', dtype='|S13') so I tried single characters: In [56]: np.array(s, 'S1') Out[56]: array('-', dtype='|S1') I still only got the first one. closer, but not quite: In [61]: np.array(tuple(s), 'S13') Out[61]: array(['-', '1', '.', '0', '0', '0', '0', '0', '0', 'E', '+', '0', '0', '-', '1', '.', '0', '0', '0', '0', '0', '0', 'E', '+', '0', '0', '-', '1', '.', '0', '0', '0', '0', '0', '0', 'E', '+', '0', '0', '-', '1', '.', '0', '0', '0', '0', '0', '0', 'E', '+', '0', '0', ' ', '1', '.', '2', '5', '0', '0', '0', '0', 'E', '+', '0', '0', ' ', '1', '.', '2', '5', '0', '0', '0', '0', 'E', '+', '0', '0'], dtype='|S13') So I ended up with this: s_array = np.array(tuple(line), dtype='S1').view(dtype='S%i'%field_len) which seems uglier than it should be, but did lead so this one-liner: np.array(tuple(line),dtype='S1').view(dtype='S%i'%field_len).astype(np.float) Is there a cleaner way to do this? (test code attached) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: test.py Type: application/x-python Size: 879 bytes Desc: not available URL: From oliphant at enthought.com Mon Jul 20 16:00:20 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Mon, 20 Jul 2009 15:00:20 -0500 Subject: [Numpy-discussion] find_common_type broken? In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDD@sernt14.essex.ac.uk> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9DDF@sernt14.essex.ac.uk> <6E65B91C-9D2B-4652-95C6-84EF33AE8F9F@enthought.com> Message-ID: <3DB0F86E-1DB6-41FA-B4BB-495F95252B0B@enthought.com> On Jul 16, 2009, at 12:59 AM, Ralf Gommers wrote: > > > This is not a problem with r_. This is correct behavior. A scalar > "float" will not cause an array "float32" to be upcast. > > This was at first counter-intuitive but I found the reason for it in > Guide to Numpy now: > > "Mixed scalar-array operations use a different set of casting rules > that ensure that > a scalar cannot upcast an array unless the scalar is of a > fundamentally different kind > of data (i.e. under a different hierarchy in the data type > hierarchy) then the array. > This rule enables you to use scalar constants in your code (which as > Python types > are interpreted accordingly in ufuncs) without worrying about > whether the precision > of the scalar constant will cause upcasting on your large (small > precision) array." > > Makes sense. > > Nonetheless, the OP did point out a flaw in find_common_type that > has been fixed in r7133. > > Great, it works for me now. > > There is still one rule I do not understand the reason for. Out of > curiosity, what is the reason for this: > In [16]: can_cast(int32, float32) > Out[16]: False > In [17]: can_cast(int64, float64) > Out[17]: True To prevent proliferation of float128 or float96 (i.e. longdouble's in a commonly used case). Not a very pretty exceptional case, but definitely useful. Thanks, -Travis -- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant at enthought.com From d_l_goldsmith at yahoo.com Mon Jul 20 16:11:01 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Mon, 20 Jul 2009 13:11:01 -0700 (PDT) Subject: [Numpy-discussion] My identity Message-ID: <387524.43679.qm@web52108.mail.re2.yahoo.com> Just to be clear, in which namespace(s) are we talking about making (or having made) the change: IIUC, the result you're talking about would be inappropriate for ufunc.identity. DG --- On Mon, 7/20/09, Keith Goodman wrote: > From: Keith Goodman > Subject: Re: [Numpy-discussion] My identity > To: "Discussion of Numerical Python" > Date: Monday, July 20, 2009, 11:17 AM > On Mon, Jul 20, 2009 at 10:53 AM, > Pauli Virtanen wrote: > > On 2009-07-20, Keith Goodman > wrote: > > [clip] > >> Oh, sorry, I misunderstood. Yes, a similar change > was made to eye but > >> not to identity. > > > > Nasty, duplicated code there it seems... > > So > > def myidentity2(n, dtype=None): > ? ? return eye(n, M=n, k=0, dtype=dtype) > > >> timeit myidentity(10) > 100000 loops, best of 3: 7.15 ?s per loop > >> timeit eye(10) > 100000 loops, best of 3: 8.42 ?s per loop > >> timeit myidentity2(10) > 100000 loops, best of 3: 9 ?s per loop > > There also a second identity in np.matlib that has its own > code. That > could be changed to > > def myidentity2_matlib(n, dtype=None): > ? ? return asmatrix(eye(n, M=n, k=0, > dtype=dtype)) > > or > > def myidentity2_matlib(n, dtype=None): > ? ? return asmatrix(myidentity2(n, dtype)) > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Mon Jul 20 16:16:06 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 20 Jul 2009 13:16:06 -0700 Subject: [Numpy-discussion] My identity In-Reply-To: <387524.43679.qm@web52108.mail.re2.yahoo.com> References: <387524.43679.qm@web52108.mail.re2.yahoo.com> Message-ID: On Mon, Jul 20, 2009 at 1:11 PM, David Goldsmith wrote: > > Just to be clear, in which namespace(s) are we talking about making (or having made) the change: IIUC, the result you're talking about would be inappropriate for ufunc.identity. np.identity np.matlib.identity From d_l_goldsmith at yahoo.com Mon Jul 20 16:43:10 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Mon, 20 Jul 2009 13:43:10 -0700 (PDT) Subject: [Numpy-discussion] My identity Message-ID: <60563.28738.qm@web52102.mail.re2.yahoo.com> OK, thanks! (Forgive my being a stickler, but I think it's good to have it in the thread, for posterity.) DG --- On Mon, 7/20/09, Keith Goodman wrote: > From: Keith Goodman > Subject: Re: [Numpy-discussion] My identity > To: "Discussion of Numerical Python" > Date: Monday, July 20, 2009, 1:16 PM > On Mon, Jul 20, 2009 at 1:11 PM, > David Goldsmith > wrote: > > > > Just to be clear, in which namespace(s) are we talking > about making (or having made) the change: IIUC, the result > you're talking about would be inappropriate for > ufunc.identity. > > np.identity > np.matlib.identity > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From lciti at essex.ac.uk Mon Jul 20 16:44:47 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Mon, 20 Jul 2009 21:44:47 +0100 Subject: [Numpy-discussion] My identity References: Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E1C@sernt14.essex.ac.uk> Just my 2 cents. It is duplicated code. But it is only 3 lines. "identity" does not need to handle rectangular matrices and non-principal diagonals, therefore it can be reasonably faster (especially for small matrices, I guess). From kwgoodman at gmail.com Mon Jul 20 17:18:39 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 20 Jul 2009 14:18:39 -0700 Subject: [Numpy-discussion] My identity In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E1C@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E1C@sernt14.essex.ac.uk> Message-ID: On Mon, Jul 20, 2009 at 1:44 PM, Citi, Luca wrote: > Just my 2 cents. > It is duplicated code. > But it is only 3 lines. > "identity" does not need to handle rectangular matrices and non-principal diagonals, > therefore it can be reasonably faster (especially for small matrices, I guess). I think that's the best trade-off too. From stefan at sun.ac.za Mon Jul 20 17:33:46 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 20 Jul 2009 23:33:46 +0200 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> Message-ID: <9457e7c80907201433i766eb9a9x47e6a449c10450d7@mail.gmail.com> Hi Chuck 2009/7/17 Charles R Harris : > PyObject*?PyTuple_GetItem(PyObject *p, Py_ssize_t pos) > Return value: Borrowed reference. > Return the object at position pos in the tuple pointed to by p. If pos is > out of bounds, return NULL and sets an IndexError exception. It's a borrowed > reference so you need to call Py_INCREF on it. I find this Python C-API > documentation useful. Have you had a look over the rest of the code? I think this would make a good addition. Travis mentioned Contexts for doing something similar, but I don't know enough about that concept to compare the two. Regards Stefan From nwagner at iam.uni-stuttgart.de Tue Jul 21 02:22:38 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 21 Jul 2009 08:22:38 +0200 Subject: [Numpy-discussion] numpy/core/code_generators/../src/multiarray/iterators.c:1778:could not find function name Message-ID: Hi all, I cannot build numpy from svn. ... adding 'build/src.linux-x86_64-2.5/numpy/core/include/numpy/numpyconfig.h' to sources. executing numpy/core/code_generators/generate_numpy_api.py numpy/core/code_generators/../src/multiarray/iterators.c 1778 Traceback (most recent call last): File "setup.py", line 186, in setup_package() File "setup.py", line 179, in setup_package configuration=configuration ) File "/data/home/nwagner/svn/numpy/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/data/home/nwagner/local/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/data/home/nwagner/local/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/data/home/nwagner/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/data/home/nwagner/svn/numpy/numpy/distutils/command/install.py", line 49, in run r = old_install.run(self) File "/data/home/nwagner/local/lib/python2.5/distutils/command/install.py", line 506, in run self.run_command('build') File "/data/home/nwagner/local/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/data/home/nwagner/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/data/home/nwagner/local/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/data/home/nwagner/local/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/data/home/nwagner/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 130, in run self.build_sources() File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 250, in build_extension_sources sources = self.generate_sources(sources, ext) File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 307, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 484, in generate_api h_file, c_file, doc_file = m.generate_api(os.path.join(build_dir, header_dir)) File "numpy/core/code_generators/generate_numpy_api.py", line 185, in generate_api do_generate_api(targets, sources) File "numpy/core/code_generators/generate_numpy_api.py", line 194, in do_generate_api numpyapi_list = genapi.get_api_functions('NUMPY_API', sources[0]) File "numpy/core/code_generators/genapi.py", line 267, in get_api_functions functions.extend(find_functions(f, tagname)) File "numpy/core/code_generators/genapi.py", line 220, in find_functions 'could not find function name') genapi.ParseError: numpy/core/code_generators/../src/multiarray/iterators.c:1778:could not find function name From david at ar.media.kyoto-u.ac.jp Tue Jul 21 02:05:02 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 21 Jul 2009 15:05:02 +0900 Subject: [Numpy-discussion] numpy/core/code_generators/../src/multiarray/iterators.c:1778:could not find function name In-Reply-To: References: Message-ID: <4A655A8E.7020604@ar.media.kyoto-u.ac.jp> Nils Wagner wrote: > Hi all, > > I cannot build numpy from svn. > Yes, I don't know why I did not caught this error on my machine. In any case, it is fixed in r7175. cheers, David From nwagner at iam.uni-stuttgart.de Tue Jul 21 02:29:03 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 21 Jul 2009 08:29:03 +0200 Subject: [Numpy-discussion] numpy/core/code_generators/../src/multiarray/iterators.c:1778:could not find function name In-Reply-To: <4A655A8E.7020604@ar.media.kyoto-u.ac.jp> References: <4A655A8E.7020604@ar.media.kyoto-u.ac.jp> Message-ID: On Tue, 21 Jul 2009 15:05:02 +0900 David Cournapeau wrote: > Nils Wagner wrote: >> Hi all, >> >> I cannot build numpy from svn. >> > > Yes, I don't know why I did not caught this error on my >machine. In any > case, it is fixed in r7175. > > cheers, > > David Great. Works for me. Nils >>>numpy.test() ====================================================================== ERROR: Failure: ImportError (No module named multiarray_tests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/core/tests/test_multiarray.py", line 7, in from numpy.core.multiarray_tests import test_neighborhood_iterator ImportError: No module named multiarray_tests ---------------------------------------------------------------------- Ran 2028 tests in 10.917s FAILED (KNOWNFAIL=1, errors=1) From nwagner at iam.uni-stuttgart.de Tue Jul 21 02:42:32 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 21 Jul 2009 08:42:32 +0200 Subject: [Numpy-discussion] String manipulation In-Reply-To: <4A64C917.4090406@noaa.gov> References: <200905111425.46475.faltet@pytables.org> <4A083AAE.1010209@american.edu> <4A64C917.4090406@noaa.gov> Message-ID: On Mon, 20 Jul 2009 12:44:23 -0700 Christopher Barker wrote: > Nils Wagner wrote: >> How can I split the second line in such a way that I get >> >> ['-1.000000E+00', '-1.000000E+00', '-1.000000E+00', >> '-1.000000E+00', '1.250000E+00', '1.250000E+00'] >> >> instead of >> >> ['-1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00', >> '1.250000E+00', '1.250000E+00'] > > It looks like you have fixed-length fields. Yes. See http://www.sdrl.uc.edu/universal-file-formats-for-modal-analysis-testing-1/file-format-storehouse/unv_0734.htm/ The naive >way do do this is simple string slicing: > > def line2array1(line, field_len=10): > nums = [] > i = 0 > while i < len(line): > nums.append(float(line[i:i+field_len])) > i += field_len > return np.array(nums) > > Then I saw the nifty list comprehension posted by >Alan(?), which led me to the one (long) liner: > > def line2array2(line, field_len=10): > return np.array(map(float, >[line[i*field_len:(i+1)*field_len] for i > in range(len(line)/field_len)])) > > But it seems I should be able to do this using numpy >arrays manipulating the data as characters. However, I >had a little trouble getting a string into a numpy array >as characters. This didn't work: > > In [55]: s > Out[55]: >'-1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00 >1.250000E+00 1.250000E+00' > > In [57]: np.array(s, 'S13') > Out[57]: > array('-1.000000E+00', > dtype='|S13') > > so I tried single characters: > > In [56]: np.array(s, 'S1') > Out[56]: > array('-', > dtype='|S1') > > I still only got the first one. > > closer, but not quite: > > In [61]: np.array(tuple(s), 'S13') > Out[61]: > array(['-', '1', '.', '0', '0', '0', '0', '0', '0', 'E', >'+', '0', '0', > '-', '1', '.', '0', '0', '0', '0', '0', '0', 'E', >'+', '0', '0', > '-', '1', '.', '0', '0', '0', '0', '0', '0', 'E', >'+', '0', '0', > '-', '1', '.', '0', '0', '0', '0', '0', '0', 'E', >'+', '0', '0', > ' ', '1', '.', '2', '5', '0', '0', '0', '0', 'E', >'+', '0', '0', > ' ', '1', '.', '2', '5', '0', '0', '0', '0', 'E', >'+', '0', '0'], > dtype='|S13') > > So I ended up with this: > s_array = np.array(tuple(line), >dtype='S1').view(dtype='S%i'%field_len) > > which seems uglier than it should be, but did lead so >this one-liner: > > np.array(tuple(line),dtype='S1').view(dtype='S%i'%field_len).astype(np.float) > > > Is there a cleaner way to do this? > > (test code attached) > > -Chris > Fixed-length fields are quite common e.g. in the area of Finite Element pre/postprocessing. Therefore It would be nice to have a function like line2array in numpy. Comments ? Nils From pgmdevlist at gmail.com Tue Jul 21 02:56:28 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 21 Jul 2009 02:56:28 -0400 Subject: [Numpy-discussion] String manipulation In-Reply-To: References: <200905111425.46475.faltet@pytables.org> <4A083AAE.1010209@american.edu> <4A64C917.4090406@noaa.gov> Message-ID: <40C94CCE-FBEF-44B5-9C2A-A1ACA973BA74@gmail.com> On Jul 21, 2009, at 2:42 AM, Nils Wagner wrote: > > Fixed-length fields are quite common e.g. in the area of > Finite Element pre/postprocessing. > Therefore It would be nice to have a function like > line2array in numpy. > Comments ? Er, there's already something like that: np.lib._iotools.LineSplitter Initialize it with either a character or an integer as delimiter, and call your instance with a string as input. When you use an integer as delimiter, it corresponds to the length of your field. eg: >>> s = '-1.000000E+00-1.000000E+00 1.000000E+00-1.000000E+00' >>> conv = np.lib._iotools.LineConverter(13) >>> np.array(conv(s)) array(['-1.000000E+00', '-1.000000E+00', '1.000000E+00', '-1.000000E +00'], dtype='|S13') >>> np.array([float(_) for _ in conv(s)]) array([-1., -1., 1., -1.]) Note that LineSplitter is already used in np.genfromtxt: >>> import StringIO >>> np.genfromtxt(StringIO.StringIO(s),delimiter=13) array([-1., -1., 1., -1.]) From pgmdevlist at gmail.com Tue Jul 21 02:57:55 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 21 Jul 2009 02:57:55 -0400 Subject: [Numpy-discussion] String manipulation In-Reply-To: <4A64C917.4090406@noaa.gov> References: <200905111425.46475.faltet@pytables.org> <4A083AAE.1010209@american.edu> <4A64C917.4090406@noaa.gov> Message-ID: On Jul 20, 2009, at 3:44 PM, Christopher Barker wrote: > ... > Is there a cleaner way to do this? > > -Chris Yes. np.lib._iotools.LineSplitter and/or np.genfromtxt From nwagner at iam.uni-stuttgart.de Tue Jul 21 03:16:55 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 21 Jul 2009 09:16:55 +0200 Subject: [Numpy-discussion] String manipulation In-Reply-To: <40C94CCE-FBEF-44B5-9C2A-A1ACA973BA74@gmail.com> References: <200905111425.46475.faltet@pytables.org> <4A083AAE.1010209@american.edu> <4A64C917.4090406@noaa.gov> <40C94CCE-FBEF-44B5-9C2A-A1ACA973BA74@gmail.com> Message-ID: On Tue, 21 Jul 2009 02:56:28 -0400 Pierre GM wrote: > > On Jul 21, 2009, at 2:42 AM, Nils Wagner wrote: >> >> Fixed-length fields are quite common e.g. in the area of >> Finite Element pre/postprocessing. >> Therefore It would be nice to have a function like >> line2array in numpy. >> Comments ? > > Er, there's already something like that: > np.lib._iotools.LineSplitter > > Initialize it with either a character or an integer as >delimiter, and > call your instance with a string as input. When you use >an integer as > delimiter, it corresponds to the length of your field. > eg: > >>> s = '-1.000000E+00-1.000000E+00 >1.000000E+00-1.000000E+00' > >>> conv = np.lib._iotools.LineConverter(13) > >>> np.array(conv(s)) > array(['-1.000000E+00', '-1.000000E+00', '1.000000E+00', >'-1.000000E > +00'], > dtype='|S13') > >>> np.array([float(_) for _ in conv(s)]) > array([-1., -1., 1., -1.]) > > Note that LineSplitter is already used in np.genfromtxt: > >>> import StringIO > >>> np.genfromtxt(StringIO.StringIO(s),delimiter=13) > array([-1., -1., 1., -1.]) > Great. I didn't know about that. Your examples are very useful. IMHO the examples should be added to http://www.scipy.org/Cookbook/InputOutput to attract interest. Nils From pgmdevlist at gmail.com Tue Jul 21 03:21:54 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 21 Jul 2009 03:21:54 -0400 Subject: [Numpy-discussion] String manipulation In-Reply-To: References: <200905111425.46475.faltet@pytables.org> <4A083AAE.1010209@american.edu> <4A64C917.4090406@noaa.gov> <40C94CCE-FBEF-44B5-9C2A-A1ACA973BA74@gmail.com> Message-ID: On Jul 21, 2009, at 3:16 AM, Nils Wagner wrote: >> >> Er, there's already something like that: >> np.lib._iotools.LineSplitter > > Great. I didn't know about that. > > Your examples are very useful. > > IMHO the examples should be added to > > http://www.scipy.org/Cookbook/InputOutput > > to attract interest. Nils, feel free to edit the corresponding entry of the Cookbook. Note that I wouldn't be surprised if a few bugs lurked in the corners, especially when dealing with a sequence of different field-lengths. Just drop me a line if you run into a problem. From dsdale24 at gmail.com Tue Jul 21 07:44:33 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 21 Jul 2009 07:44:33 -0400 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: <9457e7c80907201433i766eb9a9x47e6a449c10450d7@mail.gmail.com> References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> <9457e7c80907201433i766eb9a9x47e6a449c10450d7@mail.gmail.com> Message-ID: 2009/7/20 St?fan van der Walt : > Hi Chuck > > 2009/7/17 Charles R Harris : >> PyObject*?PyTuple_GetItem(PyObject *p, Py_ssize_t pos) >> Return value: Borrowed reference. >> Return the object at position pos in the tuple pointed to by p. If pos is >> out of bounds, return NULL and sets an IndexError exception. It's a borrowed >> reference so you need to call Py_INCREF on it. I find this Python C-API >> documentation useful. > > Have you had a look over the rest of the code? ?I think this would > make a good addition. ?Travis mentioned Contexts for doing something > similar, but I don't know enough about that concept to compare the > two. I think contexts would be very different from what is already in place. For now, it would be nice to make this one small improvement to the existing ufunc infrastructure, and maybe consider contexts (which I still don't understand) at a later time. I have improved the code slightly and added a few tests, and will post a new patch later this morning. I just need to add some documentation. Darren From dsdale24 at gmail.com Tue Jul 21 10:11:12 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 21 Jul 2009 10:11:12 -0400 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> <9457e7c80907201433i766eb9a9x47e6a449c10450d7@mail.gmail.com> Message-ID: On Tue, Jul 21, 2009 at 7:44 AM, Darren Dale wrote: > 2009/7/20 St?fan van der Walt : >> Hi Chuck >> >> 2009/7/17 Charles R Harris : >>> PyObject*?PyTuple_GetItem(PyObject *p, Py_ssize_t pos) >>> Return value: Borrowed reference. >>> Return the object at position pos in the tuple pointed to by p. If pos is >>> out of bounds, return NULL and sets an IndexError exception. It's a borrowed >>> reference so you need to call Py_INCREF on it. I find this Python C-API >>> documentation useful. >> >> Have you had a look over the rest of the code? ?I think this would >> make a good addition. ?Travis mentioned Contexts for doing something >> similar, but I don't know enough about that concept to compare the >> two. > > I think contexts would be very different from what is already in > place. For now, it would be nice to make this one small improvement to > the existing ufunc infrastructure, and maybe consider contexts (which > I still don't understand) at a later time. I have improved the code > slightly and added a few tests, and will post a new patch later this > morning. I just need to add some documentation. Here is a better patch, which includes a few additional tests and adds some documentation. It also attempts to improve the docstring and sphinx docs for __array_wrap__, which may have been a little bit misleading. There is also some whitespace cleanup in a few places. Would someone please review my work and commit the patch if it is acceptable? Pierre or Travis, would either of you have a chance to look over the implementation and the documentation changes, since you two seem to be most familiar with ufuncs and subclassing ndarray? (off topic: it would be nice if numpy had a mechanism in place for merge requests and code reviews. I've been following bzr-dev for a while now and their development model is pretty impressive.) Thank you, Darren -------------- next part -------------- A non-text attachment was scrubbed... Name: array_prepare.patch Type: application/octet-stream Size: 17364 bytes Desc: not available URL: From dsdale24 at gmail.com Tue Jul 21 10:58:56 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 21 Jul 2009 10:58:56 -0400 Subject: [Numpy-discussion] proper use of __new__ Message-ID: http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#simple-example-adding-an-extra-attribute-to-ndarray includes an example where an instance attribute is set in __new__ . However, there is a warning at http://www.scipy.org/Subclasses: ----- Some words of caution The definition in the __new__ method of default values for the subclass attributes is strongly discouraged for several reasons: * It is not thread-safe. * An array of a given subclass can be created from an array of another subclass without any call to the __new__ method at all. This situation occurs with the use of the view method. In that case, we must still initialize the attributes of the new subclass with the __array_finalize__ method. ----- In my quantities subclasses, I have implementations for __new__ like: def __new__(cls, data, units='', dtype=None, copy=True): if isinstance(data, cls): if units: data = data.rescale(units) return np.array(data, dtype=dtype, copy=copy, subok=True) ret = np.array(data, dtype=dtype, copy=copy).view(cls) ret._dimensionality.update(validate_dimensionality(units)) return ret and another definition for __new__ that is used to implement global UnitQuantity instances representing immutable things like meter and foot, and that __new__ implementation sets several instance attributes. I thought this was acceptable practice, based on the discussion in basics.subclassing.html, but now I'm not sure. How would I safely set instance attributes based on args/kwargs passed to the constructor, if not in __new__? Thanks, Darren From Chris.Barker at noaa.gov Tue Jul 21 11:56:08 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 21 Jul 2009 08:56:08 -0700 Subject: [Numpy-discussion] String manipulation In-Reply-To: References: <200905111425.46475.faltet@pytables.org> <4A083AAE.1010209@american.edu> <4A64C917.4090406@noaa.gov> Message-ID: <4A65E518.60307@noaa.gov> Pierre GM wrote: > On Jul 20, 2009, at 3:44 PM, Christopher Barker wrote: >> ... >> Is there a cleaner way to do this? > Yes. np.lib._iotools.LineSplitter and/or np.genfromtxt Great, thanks -- though the underscore in _iotools implies that this isn't supposed to be general purpose tools. Also, aside from the problem at hand, what I was getting at was whether there is a cleaner way to go from a string to a numpy array of characters or strings. I see that LineSplitter() uses a list comprehension to do the slicing, and I was looking for a way (perhaps needlessly) to use numpy instead, which requires an efficient way to get a numpy array from a string. I don't see why: np.array('a string', dtype='S1') results in a length (1,) array, for instance. Actually, I think I do -- numpy is treating the string as a single scalar, rather than a sequence of characters, and doing its best to convert that scaler to a length one string. However, I don't know if there is a compelling reason why it should do that -- in other contexts, Python generally treats strings as a sequence of characters. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From d_l_goldsmith at yahoo.com Tue Jul 21 13:16:52 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Tue, 21 Jul 2009 10:16:52 -0700 (PDT) Subject: [Numpy-discussion] String manipulation Message-ID: <752529.59185.qm@web52110.mail.re2.yahoo.com> Hi, Chris. Look at this, _I'm_ answering one of _your_ questions (correctly, I hope): --- On Tue, 7/21/09, Christopher Barker wrote: > I don't see why: > > np.array('a string', dtype='S1') > > results in a length (1,) array, for instance. > > Actually, I think I do -- numpy is treating the string as a > single > scalar, rather than a sequence of characters, and doing its > best to > convert that scaler to a length one string. However, I Well, as for why "[it's doing] its best to convert that scalar to a length one string," that's because you used dtype='S1' instead of dtype='S8': >>> np.array('a string', dtype='S1') array('a', dtype='|S1') >>> np.array('a string', dtype='S8') array('a string', dtype='|S8') but as for shape, I can't reproduce your result at all: >>> np.array('a string', dtype='S1').shape () >>> np.array('a string', dtype='S8').shape () and as for size, yes, in both cases it is treating the entire string as a single "element": >>> np.array('a string', dtype='S1').size 1 >>> np.array('a string', dtype='S8').size 1 > don't know if > there is a compelling reason why it should do that -- in > other contexts, > Python generally treats strings as a sequence of > characters. I see your point (global consistency) but personally, IMO, if you want that kind of string behavior, work in the Python namesapace: I like (though I've yet to actually use it) options (if you want to process a single string array element in the "standard" way, extract it from the array first w/ an index - it's only a few more characters of code). :-) DG > -Chris > > > > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R? ? ? ? ? ? > (206) 526-6959???voice > 7600 Sand Point Way NE???(206) > 526-6329???fax > Seattle, WA? 98115? ? > ???(206) 526-6317???main > reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pgmdevlist at gmail.com Tue Jul 21 13:26:49 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 21 Jul 2009 13:26:49 -0400 Subject: [Numpy-discussion] proper use of __new__ In-Reply-To: References: Message-ID: <1141F0CF-E999-4A1C-8C6A-7569F015F721@gmail.com> On Jul 21, 2009, at 10:58 AM, Darren Dale wrote: > http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#simple-example-adding-an-extra-attribute-to-ndarray > includes an example where an instance attribute is set in __new__ . > However, there is a warning at http://www.scipy.org/Subclasses: (Note to self: I should add the information in Subclasses to the documentation and make it consistent. ). The words of caution were intended to remind that setting the value of new array attributes in __new__ only could lead to problems, as it's possible to transform a ndarray into a subclass with a view (that is, calling __array_finalize__ without calling __new__). In the Simple Example of the doc, if .info cannot be None but must have a default DEFAULT, you should have a >>> self.info = getattr(obj, 'info', DEFAULT) in your __array_finalize__ Without access to your code, I'm likely to speak beyond myself here (could you send me a link to the latest version ?), but if you need to rescale the units, make sure it's done in __array_finalize__ as well, with something like units = getattr(obj, 'units', None) if units is not None: self.rescale(units) else: self.units = units Once again, browsing your code would let me give you some more adequate answers... From matthew.brett at gmail.com Tue Jul 21 13:37:02 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 21 Jul 2009 10:37:02 -0700 Subject: [Numpy-discussion] proper use of __new__ In-Reply-To: <1141F0CF-E999-4A1C-8C6A-7569F015F721@gmail.com> References: <1141F0CF-E999-4A1C-8C6A-7569F015F721@gmail.com> Message-ID: <1e2af89e0907211037w10af8902y1f92f5d02dfdced2@mail.gmail.com> Hi, > The words of caution were intended to remind that setting the value > of ?new array attributes in __new__ only could lead to problems, as > it's possible to transform a ndarray into a subclass with a view (that > is, calling __array_finalize__ without calling __new__). In the Simple > Example of the doc, if .info cannot be None but must have a default > DEFAULT, you should have a > ?>>> self.info = getattr(obj, 'info', DEFAULT) > in your __array_finalize__ I'm the person who removed the reference to thread-safeness in the numpy version of the subclassing docs. I remember at the time going through all the email discussions that led up to the thread-safeness question, and concluding (in discussion with Travis O and others) that there was not an increased threading problem with __new__ but understanding why it came up in the first place. I can't now reconstruct that understanding, I could have a go if y'all want some summary... Best, Matthew From robert.kern at gmail.com Tue Jul 21 14:12:22 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 21 Jul 2009 13:12:22 -0500 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> <9457e7c80907201433i766eb9a9x47e6a449c10450d7@mail.gmail.com> Message-ID: <3d375d730907211112y5b39e6dnb72f4d73057942de@mail.gmail.com> On Tue, Jul 21, 2009 at 09:11, Darren Dale wrote: > (off topic: it would be nice if numpy had a mechanism in place for > merge requests and code reviews. I've been following bzr-dev for a > while now and their development model is pretty impressive.) You can use Rietveld. numpy is already a registered repository. http://codereview.appspot.com -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dsdale24 at gmail.com Tue Jul 21 14:23:16 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 21 Jul 2009 14:23:16 -0400 Subject: [Numpy-discussion] proper use of __new__ In-Reply-To: <1141F0CF-E999-4A1C-8C6A-7569F015F721@gmail.com> References: <1141F0CF-E999-4A1C-8C6A-7569F015F721@gmail.com> Message-ID: On Tue, Jul 21, 2009 at 1:26 PM, Pierre GM wrote: > > On Jul 21, 2009, at 10:58 AM, Darren Dale wrote: > >> http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#simple-example-adding-an-extra-attribute-to-ndarray >> includes an example where an instance attribute is set in __new__ . >> However, there is a warning at http://www.scipy.org/Subclasses: > > (Note to self: I should add the information in Subclasses to the > documentation and make it consistent. ). > > The words of caution were intended to remind that setting the value > of ?new array attributes in __new__ only could lead to problems, as > it's possible to transform a ndarray into a subclass with a view (that > is, calling __array_finalize__ without calling __new__). In the Simple > Example of the doc, if .info cannot be None but must have a default > DEFAULT, you should have a > ?>>> self.info = getattr(obj, 'info', DEFAULT) > in your __array_finalize__ I think this has been addressed in quantities, but thanks for the reminder. > Without access to your code, I'm likely to speak beyond myself here > (could you send me a link to the latest version ?), but if you need to > rescale the units, make sure it's done in __array_finalize__ as well, > with something like > > units = getattr(obj, 'units', None) > if units is not None: > ? ? self.rescale(units) > else: > ? ? ?self.units = units Thanks for the suggestion, but I don't think there would be a reason to rescale when taking a view. view only takes dtype and type arguments, there is no way to specify what units you want to convert to. You either get the same units as self, or you get some kind of a view that doesnt support units. With the constructor, it is possible to pass an existing quantity (with units) and also specify the desired units, in which case we attempt to convert the old quantity into the new units. > Once again, browsing your code would let me give you some more > adequate answers... For some reason, I am not able to browse the repository at launchpad, but you can get the code with "bzr branch lp:python-quantities" or from http://pypi.python.org/pypi/quantities . The branch that works with the __array_prepare__ patch I sent can be retrieved with "bzr branch lp:dsdale24/python-quantities/quantities-array-prepare". The implementations of __new__ are in quantities/quantity.py and quantities/unitquantity.py and quantities/uncertainquantity. It is not intended to support transforming a Quantity into a UnitQuantity by using a view, there should never be a reason to do this. So I guess I am only concerned about quantity and uncertainquantity. Thanks, Darren From dsdale24 at gmail.com Tue Jul 21 14:24:40 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 21 Jul 2009 14:24:40 -0400 Subject: [Numpy-discussion] proper use of __new__ In-Reply-To: <1e2af89e0907211037w10af8902y1f92f5d02dfdced2@mail.gmail.com> References: <1141F0CF-E999-4A1C-8C6A-7569F015F721@gmail.com> <1e2af89e0907211037w10af8902y1f92f5d02dfdced2@mail.gmail.com> Message-ID: Hi Matthew, On Tue, Jul 21, 2009 at 1:37 PM, Matthew Brett wrote: > Hi, > >> The words of caution were intended to remind that setting the value >> of ?new array attributes in __new__ only could lead to problems, as >> it's possible to transform a ndarray into a subclass with a view (that >> is, calling __array_finalize__ without calling __new__). In the Simple >> Example of the doc, if .info cannot be None but must have a default >> DEFAULT, you should have a >> ?>>> self.info = getattr(obj, 'info', DEFAULT) >> in your __array_finalize__ > > I'm the person who removed the reference to thread-safeness in the > numpy version of the subclassing docs. ? I remember at the time going > through all the email discussions that led up to the thread-safeness > question, and concluding (in discussion with Travis O and others) that > there was not an increased threading problem with __new__ ?but > understanding why it came up in the first place. ?I can't now > reconstruct that understanding, I could have a go if y'all want some > summary... Did this occur on the mailing list? Could you provide a link? Thanks, Darren From matthew.brett at gmail.com Tue Jul 21 14:41:15 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 21 Jul 2009 11:41:15 -0700 Subject: [Numpy-discussion] proper use of __new__ In-Reply-To: References: <1141F0CF-E999-4A1C-8C6A-7569F015F721@gmail.com> <1e2af89e0907211037w10af8902y1f92f5d02dfdced2@mail.gmail.com> Message-ID: <1e2af89e0907211141v74eec3fbo31252eaa06cf148a@mail.gmail.com> Hi, >> I'm the person who removed the reference to thread-safeness in the >> numpy version of the subclassing docs. ? I remember at the time going >> through all the email discussions that led up to the thread-safeness >> question, and concluding (in discussion with Travis O and others) that >> there was not an increased threading problem with __new__ ?but >> understanding why it came up in the first place. ?I can't now >> reconstruct that understanding, I could have a go if y'all want some >> summary... > > Did this occur on the mailing list? Could you provide a link? Yes, here's the link: http://mail.scipy.org/pipermail/numpy-discussion/2007-February/025922.html I remember the argument now. Sturla Molden was pointing out that iff you modify attributes at the class level in __new__, this is not thread safe - as happens with any global variable. Usually of course you don't modify class attributes in the __new__ method, and in that case, expect no subclass threading problems. The key post from Travis here: http://mail.scipy.org/pipermail/numpy-discussion/2007-February/025971.html Matthew From pgmdevlist at gmail.com Tue Jul 21 14:42:03 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 21 Jul 2009 14:42:03 -0400 Subject: [Numpy-discussion] proper use of __new__ In-Reply-To: References: <1141F0CF-E999-4A1C-8C6A-7569F015F721@gmail.com> Message-ID: <90D4F6F0-9C26-4E33-B251-A4867326BE92@gmail.com> On Jul 21, 2009, at 2:23 PM, Darren Dale wrote: > >> Without access to your code, I'm likely to speak beyond myself here >> (could you send me a link to the latest version ?), but if you need >> to >> rescale the units, make sure it's done in __array_finalize__ as well, >> with something like >> >> units = getattr(obj, 'units', None) >> if units is not None: >> self.rescale(units) >> else: >> self.units = units > > Thanks for the suggestion, but I don't think there would be a reason > to rescale when taking a view. view only takes dtype and type > arguments, there is no way to specify what units you want to convert > to. I understand that very well, but could there be a case where some discrepancy between units may arise, so that obj and self no longer have the same units ? But once again, it's pure conjecture from my part at this point. > > For some reason, I am not able to browse the repository at launchpad, > but you can get the code with "bzr branch lp:python-quantities" or > from http://pypi.python.org/pypi/quantities . Cool, thx. I'll check that in the next few hours/days... From dsdale24 at gmail.com Tue Jul 21 14:48:49 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 21 Jul 2009 14:48:49 -0400 Subject: [Numpy-discussion] proper use of __new__ In-Reply-To: <1e2af89e0907211141v74eec3fbo31252eaa06cf148a@mail.gmail.com> References: <1141F0CF-E999-4A1C-8C6A-7569F015F721@gmail.com> <1e2af89e0907211037w10af8902y1f92f5d02dfdced2@mail.gmail.com> <1e2af89e0907211141v74eec3fbo31252eaa06cf148a@mail.gmail.com> Message-ID: On Tue, Jul 21, 2009 at 2:41 PM, Matthew Brett wrote: > Hi, > >>> I'm the person who removed the reference to thread-safeness in the >>> numpy version of the subclassing docs. ? I remember at the time going >>> through all the email discussions that led up to the thread-safeness >>> question, and concluding (in discussion with Travis O and others) that >>> there was not an increased threading problem with __new__ ?but >>> understanding why it came up in the first place. ?I can't now >>> reconstruct that understanding, I could have a go if y'all want some >>> summary... >> >> Did this occur on the mailing list? Could you provide a link? > > Yes, here's the link: > > http://mail.scipy.org/pipermail/numpy-discussion/2007-February/025922.html > > I remember the argument now. ?Sturla Molden was pointing out that iff > you modify attributes at the class level in __new__, this is not > thread safe - as happens with any global variable. ?Usually of course > you don't modify class attributes in the __new__ method, and in that > case, expect no subclass threading problems. ?The key post from Travis > here: > > http://mail.scipy.org/pipermail/numpy-discussion/2007-February/025971.html Thanks for the clarification. I updated the wiki to read: The definition of default values for subclass attributes (as opposed to instance attributes) in the {{{__new__}}} method is strongly discouraged for several reasons: ... From Chris.Barker at noaa.gov Tue Jul 21 16:37:37 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 21 Jul 2009 13:37:37 -0700 Subject: [Numpy-discussion] String manipulation In-Reply-To: <752529.59185.qm@web52110.mail.re2.yahoo.com> References: <752529.59185.qm@web52110.mail.re2.yahoo.com> Message-ID: <4A662711.6080603@noaa.gov> David Goldsmith wrote: > Hi, Chris. Look at this, _I'm_ answering one of _your_ questions > (correctly, I hope): maybe ;-) > --- On Tue, 7/21/09, Christopher Barker > wrote: >> I don't see why: >> >> np.array('a string', dtype='S1') >> >> results in a length (1,) array, for instance. > Well, as for why "[it's doing] its best to convert that scalar to a > length one string," that's because you used dtype='S1' instead of > dtype='S8': > >>>> np.array('a string', dtype='S1') > array('a', dtype='|S1') >>>> np.array('a string', dtype='S8') > array('a string', dtype='|S8') sure, but what I wanted was an array of characters, that's why I did 'S1'. actually, I'd like to do one more: In [27]: np.array('a string', dtype='S2') Out[27]: array('a ', dtype='|S2') would yield: array(['a ', 'st', 'ri', 'ng'], dtype='|S2') instead. > but as for shape, I can't reproduce your result at all: > >>>> np.array('a string', dtype='S1').shape > () that's 'cause I lied -- it's a numpy scalar (is that right?), not a length-1 array, just like if you do: np.array(5, dtype=np.int) > I see your point (global consistency) but personally, IMO, if you > want that kind of string behavior, work in the Python namesapace: sure, but I'd like numpy to see wider use beyond number crunching -- a homogeneous mutable n-d array is useful for a lot of things, like this. But your post gave me an idea, one could do: In [35]: line Out[35]: '-1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00 1.250000E+00 1.250000E+00' In [36]: a = np.array(line) In [37]: a Out[37]: array('-1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00 1.250000E+00 1.250000E+00', dtype='|S78') but now when I try to split it up: In [38]: a = a.view(dtype='S13') I get: ValueError: new type not compatible with array. same with 'S1': In [39]: a = a.view(dtype='S1') --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Shouldn't that work? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From d_l_goldsmith at yahoo.com Tue Jul 21 17:38:10 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Tue, 21 Jul 2009 14:38:10 -0700 (PDT) Subject: [Numpy-discussion] String manipulation Message-ID: <46650.97550.qm@web52112.mail.re2.yahoo.com> --- On Tue, 7/21/09, Christopher Barker wrote: > but now when I try to split it up: > > In [38]: a = a.view(dtype='S13') > > I get: > > ValueError: new type not compatible with array. > > Shouldn't that work? > > -Chris Submit an enhancement ticket. ;-) DG From stefan at sun.ac.za Wed Jul 22 03:49:48 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 22 Jul 2009 09:49:48 +0200 Subject: [Numpy-discussion] String manipulation In-Reply-To: <4A662711.6080603@noaa.gov> References: <752529.59185.qm@web52110.mail.re2.yahoo.com> <4A662711.6080603@noaa.gov> Message-ID: <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> 2009/7/21 Christopher Barker : > In [35]: line > Out[35]: '-1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00 > 1.250000E+00 1.250000E+00' > > In [36]: a = np.array(line) > > In [37]: a > Out[37]: > array('-1.000000E+00-1.000000E+00-1.000000E+00-1.000000E+00 1.250000E+00 > 1.250000E+00', > ? ? ? dtype='|S78') > > but now when I try to split it up: > > In [38]: a = a.view(dtype='S13') > > I get: > > ValueError: new type not compatible with array. You cannot use view on an array scalar, so you'll have to convert to an array first: x = x.reshape((1,)) x.view('S1') Regards St?fan From meine at informatik.uni-hamburg.de Fri Jul 17 10:30:38 2009 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Fri, 17 Jul 2009 16:30:38 +0200 Subject: [Numpy-discussion] How to detect ndarrays pointing to the same memory? Message-ID: <200907171630.42468.meine@informatik.uni-hamburg.de> Hi, I have a simple question: How can I detect whether two arrays share the same data? >>> a = numpy.arange(10) >>> b = a.view(numpy.ndarray) >>> >>> a is not b # False, as expected True >>> a.data is b.data # I expected this to be True False >>> >>> a.data >>> b.data # even the memory addresses differ?! >>> >>> a.data[4] = "\x55" >>> b.data[4] == "\x55" # yes, the buffers point to the same data True Ciao, Hans From gael.varoquaux at normalesup.org Wed Jul 22 05:48:29 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 22 Jul 2009 11:48:29 +0200 Subject: [Numpy-discussion] How to detect ndarrays pointing to the same memory? In-Reply-To: <200907171630.42468.meine@informatik.uni-hamburg.de> References: <200907171630.42468.meine@informatik.uni-hamburg.de> Message-ID: <20090722094829.GD27090@phare.normalesup.org> On Fri, Jul 17, 2009 at 04:30:38PM +0200, Hans Meine wrote: > Hi, > I have a simple question: How can I detect whether two arrays share the same > data? np.may_share_memory G. From stefan at sun.ac.za Wed Jul 22 06:32:00 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 22 Jul 2009 12:32:00 +0200 Subject: [Numpy-discussion] JIT ufuncs with PyPy Message-ID: <9457e7c80907220332j32d6a0e0he653849cf2ff0373@mail.gmail.com> Hi all, Pieter Holtzhausen sent me this interesting link: http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html """ I have been playing recently with the idea of NumPy and PyPy integration. My idea is to integrate PyPy's JIT with NumPy or at least a very basic subset of it. Time constraints make it impossible to hand write a JIT compiler that understands NumPy. But given PyPy's architecture we actually have a JIT generator, so we don't need to write one :-) Our JIT has shown that it can speed up small arithmetic examples significantly. What happens with something like NumPy? """ Regards St?fan From meine at informatik.uni-hamburg.de Wed Jul 22 08:06:02 2009 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 22 Jul 2009 14:06:02 +0200 Subject: [Numpy-discussion] Overloading numpy's ufuncs for better type coercion? Message-ID: <200907221406.03674.meine@informatik.uni-hamburg.de> Hi! (This mail is a reply to a personal conversation with Ullrich K?the, but is obviously of a greater concern. This is about VIGRA's new NumPy-based python bindings.) Ulli considers this behaviour of NumPy to be a bug: In [1]: a = numpy.array([200], numpy.uint8) In [2]: a + a Out[2]: array([144], dtype=uint8) However, this is well-known, often-discussed, and IMHO not really unexpected for computer programmers who ever worked with C-like languages (even Java has many such problems). Christian even said this is what he wants. OTOH, I agree that it is a missing feature that NumPy performs "coercion before the operation" (to be more precise: the temporary data type should be promoted from the operand types, and /then/ the coercion - which can also reduce the number of bits - should happen), to fix this strange behaviour: In [3]: numpy.add(a, a, numpy.empty((1, ), dtype = numpy.uint32)) Out[3]: array([144], dtype=uint32) Now, our opinions differ on how to deal with this - Ulli planned to overwrite (more or less) all ufuncs in vigranumpy in order to return float32 (which is sort of a common denominator and the type nearly all other vigranumpy functions should accept). I see two main disadvantages here: a) Choosing float32 seems to be arbitrary, and I'd like as much as possible of vigranumpy to be independent from VIGRA and its particular needs. I have seen so many requests (e.g. on the c++-sig mailing list) for *good* C++/boost- python <-> numpy bindings that it would be a pity IMO to add special cases for VIGRA by overloading __add__ etc. b) Also, I find it unexpected and undesirable to change the behaviour of such basic operations as addition on our ndarray-derived image types. IMO this brings the danger of new users being confused about the different behaviours, and even experienced vigranumpy users might eventually fall in the trap when dealing with plain ndarrays and our derived types side-by-side. Ideally, I'd like numpy to be "fixed" - I hope that the "only" obstacle is that someone needs to do it, but I am afraid of someone mentioning the term "backward-compatibility" (Ulli would surely call it "bug-compatibility" here ;-) ). But in the end, I wonder how bad this really is for the VIGRA. AFAICS, the main problem is that one needs to decide upon the pixel types for which to export algorithms (in most cases, we'll use just float32, at least for a start), and that when one loads images into arrays of the data type used in the image file, one will often end up with uint8 arrays which cannot be passed into many algorithms without an explicit conversion. However, is this really a bad problem? For example, the conversion would typically have to be performed only once (after loading), no? Then, why not simplify things further by adding a dtype= parameter to importImage()? This could even default to float32 then. Looking forward to your opinions, Hans From meine at informatik.uni-hamburg.de Wed Jul 22 08:14:42 2009 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 22 Jul 2009 14:14:42 +0200 Subject: [Numpy-discussion] Comparing the precision of dtypes? Message-ID: <200907221414.42780.meine@informatik.uni-hamburg.de> Hi, Ullrich K?the found an interesting way to compute a promoted dtype, given two arrays a and b: type = min(float32, a.dtype.type, b.dtype.type) How hackish is this? Is this likely to break on other platforms/numpy versions? Is there a better API for type promotion? Have a nice day, Hans From meine at informatik.uni-hamburg.de Wed Jul 22 09:12:37 2009 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 22 Jul 2009 15:12:37 +0200 Subject: [Numpy-discussion] VIGRA, NumPy and Fortran-order (again) In-Reply-To: References: <200907171654.22107.meine@informatik.uni-hamburg.de> Message-ID: <200907221512.38242.meine@informatik.uni-hamburg.de> On Friday 17 July 2009 22:15:31 Pauli Virtanen wrote: > On 2009-07-17, Hans Meine wrote: > > If I understood Travis' comments in the above-mentioned thread [1] > > correctly, this would already fix some of the performance issues along > > the way (since it would suddenly allow the use of special, optimized code > > paths). > > I was wondering about this too, when working on improving the > cache coherency of the reduction operations. Also these would be > more efficient if the striding of the output array could be > chosen freely. > > I wonder if it would be OK to make this change... It looks to me as if it was mostly a matter of "who's going to implement it". For example, David Cournapeau wrote [1]: "Because it is not trivial to do so [be efficient] in all cases, I guess." and then [2] "It [a transposed array] should not [make a difference], but in practise, it is not so easy to do. AFAIK, even matlab has the same problem, with less difference, though. And they have much more ressources than numpy. Not to say that we cannot do better, but this takes time." Travis simply wrote "this [the creation of c-order output arrays] would need to change in order to tak advantage of the [existing] special case" [3], without mentioning whether he'd be in favor of this change. Later he wrote there were "relatively smooth pathways for optimization" [4] which sounded promising. Anne Archibald clearly saw no reason "not to let ufuncs pick whichever order for newly-allocated arrays they want" [5] either. Finally, Travis wrote [6]: > C-order is "special" in NumPy due to the history. I agree that it doesn't > need to be and we have taken significant steps in that direction. [...] > Nobody is opposed to having faster code as long as we don't in the process > break code bases. There is also the matter of implementation. So, AFAICS, it's only a matter of how to do it and who actually does it; nobody seemed to be opposed (except that the default order in copy should stay "C" of course, which is absolutely acceptable for me). BTW: Here's pseude-code in Python which I proposed back then: [elementwise operation on array 'a'] 1) ii = [i for i, s in sorted(enumerate(ia.strides), key = lambda (i, s): -s)] 2) aprime = a.transpose(ii) # aprime will be "as C-contiguous as it gets" 3) bprime = perform_operation(aprime) # fast elementwise operation 4) jj = [ii.index(i) for i in range(bprime.ndim)] # inverse ii 5) b = bprime.transpose(jj) # let result have the same order as the input I think this could/should be added to the ufunc machinery, maybe directly after broadcasting the inputs. Hmm, actually, the above might be an unnecessary implementation trick, i.e. I proposed to transpose before and after the operation, while Travis suggested to create the output array with the right order and then try with the existing ufunc machinery. I really hope that we can improve NumPy here. Have a nice day, Hans [1] http://mail.scipy.org/pipermail/numpy-discussion/2007-November/029838.html [2] http://mail.scipy.org/pipermail/numpy-discussion/2007-November/029849.html [3] http://mail.scipy.org/pipermail/numpy-discussion/2007-November/029859.html [4] http://mail.scipy.org/pipermail/numpy-discussion/2007-November/029863.html [5] http://mail.scipy.org/pipermail/numpy-discussion/2007-November/029866.html From lciti at essex.ac.uk Wed Jul 22 09:14:32 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Wed, 22 Jul 2009 14:14:32 +0100 Subject: [Numpy-discussion] Overloading numpy's ufuncs for better typecoercion? References: <200907221406.03674.meine@informatik.uni-hamburg.de> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E21@sernt14.essex.ac.uk> Hi Hans! > Ideally, I'd like numpy to be "fixed" what do you mean by "fixed"? Are you referring to Out[2] or Out[3]? In [1]: a = numpy.array([200], numpy.uint8) In [2]: a + a Out[2]: array([144], dtype=uint8) Please do not "fix" this, that IS the correct output. What should numpy do? Promote every sum of arrays of uint8 to uint16? Or perform the operation as uint16 and cast it back to uint8 only if all elements are less than 256, therefore having the same operation on arrays of the same type return an unpredictable data type? I think the answer is simple: a = numpy.array([200]) if you want an integer a = numpy.array([200.]) if you want a float. These are pretty safe for reasonable inputs. Whenever the user writes: a = numpy.array([200], dtype=...) it means he/she knows what they are doing. If instead, you refer to In [3]: numpy.add(a, a, numpy.empty((1, ), dtype = numpy.uint32)) Out[3]: array([144], dtype=uint32) in this case I agree with you, the expected result should be 400. The inputs could be casted to the output type before performing the operation. I do not think performing the operations with the output dtype would break something. Even in borderline cases like the following: >>> b = numpy.array([400], numpy.int16) >>> c = numpy.array([200], numpy.int16) >>> numpy.subtract(b.astype(numpy.int8), c.astype(numpy.int8), numpy.empty((1, ), dtype = numpy.int8)) array([100], dtype=int8) Best, Luca From meine at informatik.uni-hamburg.de Wed Jul 22 09:34:52 2009 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 22 Jul 2009 15:34:52 +0200 Subject: [Numpy-discussion] Overloading numpy's ufuncs for better typecoercion? In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E21@sernt14.essex.ac.uk> References: <200907221406.03674.meine@informatik.uni-hamburg.de> <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E21@sernt14.essex.ac.uk> Message-ID: <200907221534.52712.meine@informatik.uni-hamburg.de> On Wednesday 22 July 2009 15:14:32 Citi, Luca wrote: > In [2]: a + a > Out[2]: array([144], dtype=uint8) > > Please do not "fix" this, that IS the correct output. No, I did not mean to fix this. (Although it should be noted that in C/C++, the result of uint8+uint8 is int.) > If instead, you refer to > In [3]: numpy.add(a, a, numpy.empty((1, ), dtype = numpy.uint32)) > Out[3]: array([144], dtype=uint32) > in this case I agree with you, the expected result should be 400. Yes, that's what I meant. > The inputs could be casted to the output type before performing the > operation. I do not think performing the operations with the output dtype > would break something. > Even in borderline cases like the following: > >>> b = numpy.array([400], numpy.int16) > >>> c = numpy.array([200], numpy.int16) > >>> numpy.subtract(b.astype(numpy.int8), c.astype(numpy.int8), > >>> numpy.empty((1, ), dtype = numpy.int8)) Indeed - I thought we had to take more care, but will this also work with int<->float conversions? No: In [22]: a, b = numpy.array([[8.6,4.9]]).T In [23]: numpy.subtract(a, b, numpy.empty((1, ), dtype = numpy.uint8)) Out[23]: array([3], dtype=uint8) In [24]: numpy.subtract(a.astype(numpy.uint8), b.astype(numpy.uint8)) Out[24]: array([4], dtype=uint8) However, I admit that this is a contrived example. ;-) Greetings, Hans From lciti at essex.ac.uk Wed Jul 22 10:43:43 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Wed, 22 Jul 2009 15:43:43 +0100 Subject: [Numpy-discussion] Overloading numpy's ufuncs for bettertypecoercion? References: <200907221406.03674.meine@informatik.uni-hamburg.de><3DA3B328CBC48B4EBB88484B8A5EA19106AF9E21@sernt14.essex.ac.uk> <200907221534.52712.meine@informatik.uni-hamburg.de> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E22@sernt14.essex.ac.uk> Hello Hans, > Although it should be noted that in C/C++, > the result of uint8+uint8 is int. But C/C++ works with scalars and often temporary results are kept in registers. On the contrary, numpy works with arrays. We cannot expect (a+b)*c to grow from uint8 to uint16 and then uint32 :-D > For example, the conversion would typically have to be > performed only once (after loading), no? Then, why not simplify things > further by adding a dtype= parameter to importImage()? This could even > default to float32 then. I think this is the cleanest, easiest, better way to go. Also, Matlab's imread reads the image in its format (e.g. uint8). An explicit conversion is needed (e.g. double()) otherwise all the operations are performed in uint8. Even i = imread('...'); i = i + 34.5 does not trigger an upcast. The only difference is that matlab saturates to 0 and 255 instead of wrapping. Best, Luca From ralf.gommers at googlemail.com Wed Jul 22 11:16:31 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 22 Jul 2009 11:16:31 -0400 Subject: [Numpy-discussion] Comparing the precision of dtypes? In-Reply-To: <200907221414.42780.meine@informatik.uni-hamburg.de> References: <200907221414.42780.meine@informatik.uni-hamburg.de> Message-ID: 2009/7/22 Hans Meine > Hi, > > Ullrich K?the found an interesting way to compute a promoted dtype, given > two > arrays a and b: > > type = min(float32, a.dtype.type, b.dtype.type) > Are you looking for the type to cast to? In that case I guess you meant max() not min(). > > How hackish is this? Is this likely to break on other platforms/numpy > versions? Is there a better API for type promotion? That could give unexpected results. Useful functions are find_common_type, mintypecode, common_type and can_cast; see http://docs.scipy.org/numpy/docs/numpy-docs/reference/routines.dtype.rst Examples where min/max probably does not do what you want, and find_common_type does: In [49]: max(float, float32) Out[49]: In [50]: find_common_type([], [float, float32]) Out[50]: dtype('float64') In [40]: max(int8, uint8) Out[40]: In [41]: find_common_type([int8, uint8], []) Out[41]: dtype('int16') Note that the second example only works with latest trunk (Travis fixed this last week) and the updated docs have not yet appeared in trunk (but see link above). Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From vel.accel at gmail.com Wed Jul 22 11:34:39 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Wed, 22 Jul 2009 11:34:39 -0400 Subject: [Numpy-discussion] branches/datetime Message-ID: <1e52e0880907220834k60c0530fmf35226eca46944b9@mail.gmail.com> Hi all, I see life in a feature I've been dreaming about for years now. :-) I'm wondering how stable this branch is and if it's ready for playing with. I ask because I'm (once again) about to write an cython extension to process vectors of epochs and records of struct tm. Naturally, I'd love to be doing this right in numpy itself. Thanks, Pete From meine at informatik.uni-hamburg.de Wed Jul 22 11:37:23 2009 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 22 Jul 2009 17:37:23 +0200 Subject: [Numpy-discussion] Comparing the precision of dtypes? In-Reply-To: References: <200907221414.42780.meine@informatik.uni-hamburg.de> Message-ID: <200907221737.23849.meine@informatik.uni-hamburg.de> On Wednesday 22 July 2009 17:16:31 Ralf Gommers wrote: > 2009/7/22 Hans Meine > > type = min(float32, a.dtype.type, b.dtype.type) > > Are you looking for the type to cast to? In that case I guess you meant > max() not min(). No, at least for integers min(..) does what one would expect max(..) to do. But... > > How hackish is this? Is this likely to break on other platforms/numpy > > versions? Is there a better API for type promotion? > > That could give unexpected results. Useful functions are find_common_type, ..find_common_type() is exactly what we were looking for; thanks a lot for the pointer! Have a nice day, Hans From ralf.gommers at googlemail.com Wed Jul 22 11:51:09 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 22 Jul 2009 11:51:09 -0400 Subject: [Numpy-discussion] Comparing the precision of dtypes? In-Reply-To: <200907221737.23849.meine@informatik.uni-hamburg.de> References: <200907221414.42780.meine@informatik.uni-hamburg.de> <200907221737.23849.meine@informatik.uni-hamburg.de> Message-ID: 2009/7/22 Hans Meine > On Wednesday 22 July 2009 17:16:31 Ralf Gommers wrote: > > 2009/7/22 Hans Meine > > > type = min(float32, a.dtype.type, b.dtype.type) > > > > Are you looking for the type to cast to? In that case I guess you meant > > max() not min(). > > No, at least for integers min(..) does what one would expect max(..) to do. > But... Hmm, not for me: In [56]: min(int16, int32) Out[56]: In [57]: max(int16, int32) Out[57]: On the other hand, even though dtype=int gives you an int32 array this does not work as expected: In [59]: min(int16, int) Out[59]: Well I guess that shows how fragile using min/max is. And I don't think there's a way to make it always work because there are types that are not simply bigger/smaller but of a different kind. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From meine at informatik.uni-hamburg.de Wed Jul 22 11:51:18 2009 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 22 Jul 2009 17:51:18 +0200 Subject: [Numpy-discussion] =?iso-8859-1?q?How_to_detect_ndarrays_pointing?= =?iso-8859-1?q?_to_the_same=09memory=3F?= In-Reply-To: <20090722094829.GD27090@phare.normalesup.org> References: <200907171630.42468.meine@informatik.uni-hamburg.de> <20090722094829.GD27090@phare.normalesup.org> Message-ID: <200907221751.21301.meine@informatik.uni-hamburg.de> On Wednesday 22 July 2009 11:48:29 Gael Varoquaux wrote: > On Fri, Jul 17, 2009 at 04:30:38PM +0200, Hans Meine wrote: > > I have a simple question: How can I detect whether two arrays share the > > same data? > > np.may_share_memory Thanks a lot, that (and it's implementation, cf. byte_bounds) does help! Best, Hans From millman at berkeley.edu Wed Jul 22 12:00:42 2009 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 22 Jul 2009 09:00:42 -0700 Subject: [Numpy-discussion] ANN: SciPy 2009 early registration ends today Message-ID: Today is the last day to register for SciPy 2009 at the early bird rates. Please register (http://conference.scipy.org/to_register ) by the end of the day to take advantage of the reduced early registration rate. The conference schedule is available here: http://conference.scipy.org/schedule The special group rate for the Marriot Hotel is no longer available. However, there are a number of closer and less expensive choices still available: http://admissions.caltech.edu/visiting/accommodations I've been staying at the Vagabond Inn for the last several years: http://www.vagabondinn-pasadena-hotel.com/ It is within easy walking distance of the conference and has just been completely renovated. Rooms at the Vagabond start at $79/night. About the conference -------------------- SciPy 2009, the 8th Python in Science conference, will be held from August 18-23, 2009 at Caltech in Pasadena, CA, USA. The conference starts with two days of tutorials to the scientific Python tools. There will be two tracks, one for introduction of the basic tools to beginners, and one for more advanced tools. The tutorials will be followed by two days of talks. Both days of talks will begin with a keynote address. The first day?s keynote will be given by Peter Norvig, the Director of Research at Google; while, the second keynote will be delivered by Jon Guyer, a Materials Scientist in the Thermodynamics and Kinetics Group at NIST. The program committee will select the remaining talks from submissions to our call for papers. All selected talks will be included in our conference proceedings edited by the program committee. After the talks each day we will provide several rooms for impromptu birds of a feather discussions. Finally, the last two days of the conference will be used for a number of coding sprints on the major software projects in our community. For the 8th consecutive year, the conference will bring together the developers and users of the open source software stack for scientific computing with Python. Attendees have the opportunity to review the available tools and how they apply to specific problems. By providing a forum for developers to share their Python expertise with the wider commercial, academic, and research communities, this conference fosters collaboration and facilitates the sharing of software components, techniques, and a vision for high level language use in scientific computing. For further information, please visit the conference homepage: http://conference.scipy.org. Important Dates --------------- * Friday, July 3: Abstracts Due * Wednesday, July 15: Announce accepted talks, post schedule * Wednesday, July 22: Early Registration ends * Tuesday-Wednesday, August 18-19: Tutorials * Thursday-Friday, August 20-21: Conference * Saturday-Sunday, August 22-23: Sprints * Friday, September 4: Papers for proceedings due Executive Committee ------------------- * Jarrod Millman, UC Berkeley, USA (Conference Chair) * Ga?l Varoquaux, INRIA Saclay, France (Program Co-Chair) * St?fan van der Walt, University of Stellenbosch, South Africa (Program Co-Chair) * Fernando P?rez, UC Berkeley, USA (Tutorial Chair) From dfnsonfsduifb at gmx.de Wed Jul 22 12:36:40 2009 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Wed, 22 Jul 2009 18:36:40 +0200 Subject: [Numpy-discussion] Getting 95%/99% margin of ndarray Message-ID: <4A674018.8050401@gmx.de> Hello list, is there some possibilty to get a p-dynamic of an array, i.e. if p=1 then the result would be (arr.min(), arr.max()), but if 0 < p < 1, then the result is so that the pth percentile of the picture is withing the range given? I cannot explain this very well, so please let me illustrate. Let's say we have an array: 0 0 0 0 1 2 3 4 5 2000 p = 0.9 -> (0, 5) This means that 90% of the pixels are within the [0; 5] range p = 0.5 -> (1, 5) p = 0.3 -> (2, 4) It it clear what I want? Is there the possibility to achieve that? If I have to implement it myself, how can I do that efficiently in Python? Kind regards, Johannes From lciti at essex.ac.uk Wed Jul 22 13:17:46 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Wed, 22 Jul 2009 18:17:46 +0100 Subject: [Numpy-discussion] Getting 95%/99% margin of ndarray References: <4A674018.8050401@gmx.de> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E28@sernt14.essex.ac.uk> You can do it "by hand" by sorting the array and taking the corresponding elements or you can use scipy.stats.scoreatpercentile that also interpolates. Best, Luca From pgmdevlist at gmail.com Wed Jul 22 13:26:42 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 22 Jul 2009 13:26:42 -0400 Subject: [Numpy-discussion] Getting 95%/99% margin of ndarray In-Reply-To: <4A674018.8050401@gmx.de> References: <4A674018.8050401@gmx.de> Message-ID: <82C47E17-59AD-469F-90B8-B8950FC14C95@gmail.com> On Jul 22, 2009, at 12:36 PM, Johannes Bauer wrote: > Hello list, > > is there some possibilty to get a p-dynamic of an array, i.e. if p=1 > then the result would be (arr.min(), arr.max()), but if 0 < p < 1, > then > the result is so that the pth percentile of the picture is withing the > range given? You could try scipy.stats.scoreatpercentile, scipy.stats.mstats.plottingposition or scipy.stats.mstats.mquantiles, which will all approximate quantiles of your distribution. To get the 90% of data you want, find the (0.05, 0.95) quantiles. More generally, to get n% data, take the (n/2)% and (1.-n/2)% quantiles... Even more approximately, you could sort your data and take the (n/2)N and (1-n/2)N ones, where n is the quantile you want and N the size of your array. From pgmdevlist at gmail.com Wed Jul 22 13:27:14 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 22 Jul 2009 13:27:14 -0400 Subject: [Numpy-discussion] branches/datetime In-Reply-To: <1e52e0880907220834k60c0530fmf35226eca46944b9@mail.gmail.com> References: <1e52e0880907220834k60c0530fmf35226eca46944b9@mail.gmail.com> Message-ID: <04E7C19E-CC6E-4834-96E6-87B08A07D46B@gmail.com> On Jul 22, 2009, at 11:34 AM, Peter Alexander wrote: > Hi all, > > I see life in a feature I've been dreaming about for years now. :-) > I'm wondering how stable this branch is and if it's ready for > playing with. I may speak out of turn here, but I don't think so. Besides Travis O. who does the bulk of the development, Marty Fuhry, a GSoC student, is also involved. We should soon have some news on this side. > I ask because I'm (once again) about to write an cython extension to > process vectors of epochs and records of struct tm. > Naturally, I'd love to be doing this right in numpy itself. Have you checked scikits.timeseries ? From lciti at essex.ac.uk Wed Jul 22 13:43:26 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Wed, 22 Jul 2009 18:43:26 +0100 Subject: [Numpy-discussion] Getting 95%/99% margin of ndarray References: <4A674018.8050401@gmx.de> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E29@sernt14.essex.ac.uk> I am afraid I misunderstand your question because I do not get the results you expected. def pdyn(a, p): a = np.sort(a) n = round((1-p) * len(a)) return a[int((n+1)/2)], a[len(a)-1-int(n/2)] # a[-int(n/2)] would not work if n<=1 >>> pdyn([0, 0, 0, 0, 1, 2, 3, 4, 5, 2000], 1) (0, 2000) >>> pdyn([0, 0, 0, 0, 1, 2, 3, 4, 5, 2000], .9) (0, 2000) >>> pdyn([0, 0, 0, 0, 1, 2, 3, 4, 5, 2000], .5) (0, 4) >>> pdyn([0, 0, 0, 0, 1, 2, 3, 4, 5, 2000], .3) (1, 3) If you have the array 0 0 0 0 1 2 3 4 5 2000 why should p = 0.5 -> (1, 5) ? I mean 10*.5 is 5, so you throw away 5 elements from the upper and lower tail (either 3+2 or 2+3) and you should end up with either (0, 4) or (0, 3), so why (1, 5) ? Best, Luca From Chris.Barker at noaa.gov Wed Jul 22 13:54:04 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 22 Jul 2009 10:54:04 -0700 Subject: [Numpy-discussion] Overloading numpy's ufuncs for better type coercion? In-Reply-To: <200907221406.03674.meine@informatik.uni-hamburg.de> References: <200907221406.03674.meine@informatik.uni-hamburg.de> Message-ID: <4A67523C.3030905@noaa.gov> Hans Meine wrote: > In [3]: numpy.add(a, a, numpy.empty((1, ), dtype = numpy.uint32)) > Out[3]: array([144], dtype=uint32) yes, it sure would be nice to fix this... > one will often end up with uint8 arrays which cannot be passed > into many algorithms without an explicit conversion. However, is this really > a bad problem? For example, the conversion would typically have to be > performed only once (after loading), no? Then, why not simplify things > further by adding a dtype= parameter to importImage()? This could even > default to float32 then. VIGRA specifically, this sounds like a fine way to go. how ever, for the broader numpy case: I want to add two unit8 arrays, and put the results into a int32 array (for instance). As pointed out this doesn't work: In [9]: a1 = np.array((200,), dtype= np.uint8) In [10]: a2 = np.array((250,), dtype= np.uint8) In [11]: a1 + a2 Out[11]: array([194], dtype=uint8) As pointed out by others -- this is the "right" behavior - we really don't want upcasting without asking for it. However, as above: In [15]: np.add(a1, a2, np.empty(a1.shape, dtype = np.int32)) Out[15]: array([194]) I am asking for upcasting, so it's too bad I don't get it. The solution is to upcast ahead of time: In [17]: np.add(a1.astype(np.int32), a2, np.empty(a1.shape, dtype = np.int32)) Out[17]: array([450]) or simply: In [18]: a1.astype(np.int32) +a2 Out[18]: array([450]) Easy enough. The one downside is that an extra temporary in the larger type is needed (but only one). That could be an issue when working with large arrays, which I suspect is the case when these issues come up -- no one use the np.add() notation unless you are trying to manage memory more carefully. Another way to write this is: In [19]: a3 = a1.astype(np.int32) In [20]: a3 += a2 In [21]: a3 Out[21]: array([450]) which I think avoids the extra temporary This is a pretty special case, ands there ways to accomplish what is needed. I suspect that's why no one has "fixed" this yet. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Jul 22 14:30:05 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 22 Jul 2009 12:30:05 -0600 Subject: [Numpy-discussion] Overloading numpy's ufuncs for better type coercion? In-Reply-To: <200907221406.03674.meine@informatik.uni-hamburg.de> References: <200907221406.03674.meine@informatik.uni-hamburg.de> Message-ID: 2009/7/22 Hans Meine > Hi! > > (This mail is a reply to a personal conversation with Ullrich K?the, but is > obviously of a greater concern. This is about VIGRA's new NumPy-based > python > bindings.) Ulli considers this behaviour of NumPy to be a bug: > > In [1]: a = numpy.array([200], numpy.uint8) > > In [2]: a + a > Out[2]: array([144], dtype=uint8) > > However, this is well-known, often-discussed, and IMHO not really > unexpected > for computer programmers who ever worked with C-like languages (even Java > has > many such problems). Christian even said this is what he wants. > > OTOH, I agree that it is a missing feature that NumPy performs "coercion > before the operation" (to be more precise: the temporary data type should > be > promoted from the operand types, and /then/ the coercion - which can also > reduce the number of bits - should happen), to fix this strange behaviour: > > In [3]: numpy.add(a, a, numpy.empty((1, ), dtype = numpy.uint32)) > Out[3]: array([144], dtype=uint32) > > Now, our opinions differ on how to deal with this - Ulli planned to > overwrite > (more or less) all ufuncs in vigranumpy in order to return float32 (which > is > sort of a common denominator and the type nearly all other vigranumpy > functions should accept). I see two main disadvantages here: > > a) Choosing float32 seems to be arbitrary, and I'd like as much as possible > of > vigranumpy to be independent from VIGRA and its particular needs. I have > seen > so many requests (e.g. on the c++-sig mailing list) for *good* C++/boost- > python <-> numpy bindings that it would be a pity IMO to add special cases > for > VIGRA by overloading __add__ etc. > > b) Also, I find it unexpected and undesirable to change the behaviour of > such > basic operations as addition on our ndarray-derived image types. IMO this > brings the danger of new users being confused about the different > behaviours, > and even experienced vigranumpy users might eventually fall in the trap > when > dealing with plain ndarrays and our derived types side-by-side. > > Ideally, I'd like numpy to be "fixed" - I hope that the "only" obstacle is > that someone needs to do it, but I am afraid of someone mentioning the term > "backward-compatibility" (Ulli would surely call it "bug-compatibility" > here > ;-) ). The rule here is that the output is computed as the common input type, then cast to the type of the output array. It will also be downcast if the output array is of lesser precision. This has been discussed on the list, my preference would have been to raise an error on mismatched types but the consensus was to keep the current casting behaviour. What you are asking for is that the ufunc upcast the arrays before the addition if the output array is of higher precision. This complicates the logic. I think it is better in this case to explicitly make a of higher precision before the call. Note that the ufunc loops don't generally handle mixed types with the exception of logical functions and, IIRC, the exponent/mantissa functions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Jul 22 14:33:56 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 22 Jul 2009 11:33:56 -0700 Subject: [Numpy-discussion] String manipulation In-Reply-To: <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> References: <752529.59185.qm@web52110.mail.re2.yahoo.com> <4A662711.6080603@noaa.gov> <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> Message-ID: <4A675B94.6000901@noaa.gov> St?fan van der Walt wrote: > You cannot use view on an array scalar, so you'll have to convert to > an array first: > > x = x.reshape((1,)) > x.view('S1') Bingo! Thanks, St?fan. I don't know why one can't uwe view on a scalar -- there is still a dat pointer isn't there?, and it can be reshaped. Butno matter, now I have a one liner that splits a string up into fixed fields, and returns a float array of the numbers in those fields: np.array((line,)).view(dtype='S%i'%field_len).astype(np.float) I suspect this is faster than the list comp method, though I haven't tested it -- I"ll leave that to someone that actually has this problem to solve ;-) I've enclosed a bit of test code with a few other approaches as well. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: test.py Type: application/x-python Size: 1041 bytes Desc: not available URL: From jonathan.taylor at utoronto.ca Wed Jul 22 14:34:32 2009 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Wed, 22 Jul 2009 14:34:32 -0400 Subject: [Numpy-discussion] performance matrix multiplication vs. matlab In-Reply-To: <954ae5aa0907192035i6b48b54rf2b4d1621720d8e6@mail.gmail.com> References: <20090604143638.xrqzx8w8v4ggckkk@www.sms.ed.ac.uk> <4A28F16C.3070607@ar.media.kyoto-u.ac.jp> <4A2955F5.9030006@hawaii.edu> <4A295A7D.1040407@ar.media.kyoto-u.ac.jp> <7f014ea60906051437p6002c6f7jb2d77374a220d2aa@mail.gmail.com> <7f014ea60906060959s6570cc32l277c5ab423f0b9ed@mail.gmail.com> <463e11f90907171257m2f5ff9d6v34924e0200145f4@mail.gmail.com> <954ae5aa0907192035i6b48b54rf2b4d1621720d8e6@mail.gmail.com> Message-ID: <463e11f90907221134m57be94f4lfb4102317489869d@mail.gmail.com> Sorry. I meant to update this thread after I had resolved my issue. This was indeed one problem. I had to set LD_LIBRARY_PATH. I also had another odd problem that I will spell out here in hopes that I save someone some trouble. Specifically, one should be very sure that the path to the blas that was compiled is correct when you configure ATLAS because it does not indicate any problems if it is not. Specifically, I tried compiling blas with make -j3 to get all my cores compiling at the same time but this actually caused a failure that I did not notice. It did create a temp_LINUX.a file in the right place so I configured ATLAS against that. Alas, many of the symbols needed were not contained in this file as BLAS had failed to compile. This was fairly hard to debug but once I got blas recompiled properly without the -j 3 switch I was able to follow the rest of the steps and everything works well. Thanks, Jonathan. On Sun, Jul 19, 2009 at 11:35 PM, Nicolas Pinto wrote: > Jonathan, > > What does "ldd /home/jtaylor/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so" > say ? > > You need to make sure that it's using the libraries in /usr/local/lib. > You can remove the ones in /usr/lib or "export > LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH". > > Hope it helps. > > Best, > > N > > On Fri, Jul 17, 2009 at 3:57 PM, Jonathan > Taylor wrote: >> Following these instructions I have the following problem when I >> import numpy. ?Does anyone know why this might be? >> >> Thanks, >> Jonathan. >> >>>>> import numpy >> Traceback (most recent call last): >> ?File "", line 1, in >> ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/__init__.py", >> line 130, in >> ? ?import add_newdocs >> ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/add_newdocs.py", >> line 9, in >> ? ?from lib import add_newdoc >> ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/lib/__init__.py", >> line 13, in >> ? ?from polynomial import * >> ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/lib/polynomial.py", >> line 18, in >> ? ?from numpy.linalg import eigvals, lstsq >> ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/__init__.py", >> line 47, in >> ? ?from linalg import * >> ?File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/linalg.py", >> line 22, in >> ? ?from numpy.linalg import lapack_lite >> ImportError: /usr/local/lib/libptcblas.so: undefined symbol: ATL_cpttrsm >> >> >> On Sat, Jun 6, 2009 at 12:59 PM, Chris Colbert wrote: >>> since there is demand, and someone already emailed me, I'll put what I >>> did in this post. It pretty much follows whats on the scipy website, >>> with a couple other things I gleaned from reading the ATLAS install >>> guide: >>> >>> and here it goes, this is valid for Ubuntu 9.04 64-bit ?(# starts a >>> comment when working in the terminal) >>> >>> >>> download lapack 3.2.1 http://www.netlib.org/lapack/lapack.tgz >>> download atlas 3.8.3 >>> http://sourceforge.net/project/downloading.php?group_id=23725&filename=atlas3.8.3.tar.bz2&a=65663372 >>> >>> create folder ?/home/your-user-name/build/atlas ? #this is where we build >>> create folder /home/your-user-name/build/lapack #atlas and lapack >>> >>> extract the folder lapack-3.2.1 to /home/your-user-name/build/lapack >>> extract the contents of atlas to /home/your-user-name/build/atlas >>> >>> >>> >>> now in the terminal: >>> >>> # remove g77 and get stuff we need >>> sudo apt-get remove g77 >>> sudo apt-get install gfortran >>> sudo apt-get install build-essential >>> sudo apt-get install python-dev >>> sudo apt-get install python-setuptools >>> sudo easy_install nose >>> >>> >>> # build lapack >>> cd /home/your-user-name/build/lapack/lapack-3.2.1 >>> cp INSTALL/make.inc.gfortran make.inc >>> >>> gedit make.inc >>> ################# >>> #in the make.inc file make sure the line ? OPTS = -O2 -fPIC -m64 >>> #and ? ?NOOPTS = -O0 -fPIC -m64 >>> #the -m64 flags build 64-bit code, if you want 32-bit, simply leave >>> #the -m64 flags out >>> ################# >>> >>> cd SRC >>> >>> #this should build lapack without error >>> make >>> >>> >>> >>> # build atlas >>> >>> cd /home/your-user-name/build/atlas >>> >>> #this is simply where we will build the atlas >>> #libs, you can name it what you want >>> mkdir Linux_X64SSE2 >>> >>> cd Linux_X64SSE2 >>> >>> #need to turn off cpu-throttling >>> sudo cpufreq-selector -g performance >>> >>> #if you don't want 64bit code remove the -b 64 flag. replace the >>> #number 2400 with your CPU frequency in MHZ >>> #i.e. my cpu is 2.53 GHZ so i put 2530 >>> ../configure -b 64 -D c -DPentiumCPS=2400 -Fa ?-alg -fPIC >>> --with-netlib-lapack=/home/your-user-name/build/lapack/lapack-3.2.1/Lapack_LINUX.a >>> >>> #the configure step takes a bit, and should end without errors >>> >>> ?#this takes a long time, go get some coffee, it should end without error >>> make build >>> >>> #this will verify the build, also long running >>> make check >>> >>> #this will test the performance of your build and give you feedback on >>> #it. your numbers should be close to the test numbers at the end >>> make time >>> >>> cd lib >>> >>> #builds single threaded .so's >>> make shared >>> >>> #builds multithreaded .so's >>> make ptshared >>> >>> #copies all of the atlas libs (and the lapack lib built with atlas) >>> #to our lib dir >>> sudo ?cp ?*.so ?/usr/local/lib/ >>> >>> >>> >>> #now we need to get and build numpy >>> >>> download numpy 1.3.0 >>> http://sourceforge.net/project/downloading.php?group_id=1369&filename=numpy-1.3.0.tar.gz&a=93506515 >>> >>> extract the folder numpy-1.3.0 to /home/your-user-name/build >>> >>> #in the terminal >>> >>> cd /home/your-user-name/build/numpy-1.3.0 >>> cp site.cfg.example site.cfg >>> >>> gedit site.cfg >>> ############################################### >>> # in site.cfg uncomment the following lines and make them look like these >>> [DEFAULT] >>> library_dirs = /usr/local/lib >>> include_dirs = /usr/local/include >>> >>> [blas_opt] >>> libraries = ptf77blas, ptcblas, atlas >>> >>> [lapack_opt] >>> libraries = lapack, ptf77blas, ptcblas, atlas >>> ################################################### >>> #if you want single threaded libs, uncomment those lines instead >>> >>> >>> #build numpy- should end without error >>> python setup.py build >>> >>> #install numpy >>> python setup.py install >>> >>> cd /home >>> >>> sudo ldconfig >>> >>> python >>>>>import numpy >>>>>numpy.test() ? #this should run with no errors (skipped tests and known-fails are ok) >>>>>a = numpy.random.randn(6000, 6000) >>>>>numpy.dot(a, a) ? ? # look at your cpu monitor and verify all cpu cores are at 100% if you built with threads >>> >>> >>> Celebrate with a beer! >>> >>> >>> Cheers! >>> >>> Chris >>> >>> >>> >>> >>> >>> On Sat, Jun 6, 2009 at 10:42 AM, Keith Goodman wrote: >>>> On Fri, Jun 5, 2009 at 2:37 PM, Chris Colbert wrote: >>>>> I'll caution anyone from using Atlas from the repos in Ubuntu 9.04? as the >>>>> package is broken: >>>>> >>>>> https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510 >>>>> >>>>> >>>>> just build Atlas yourself, you get better performance AND threading. >>>>> Building it is not the nightmare it sounds like. I think i've done it a >>>>> total of four times now, both 32-bit and 64-bit builds. >>>>> >>>>> If you need help with it,? just email me off list. >>>> >>>> That's a nice offer. I tried building ATLAS on Debian a year or two >>>> ago and got stuck. >>>> >>>> Clear out your inbox! >>>> _______________________________________________ >>>> Numpy-discussion mailing list >>>> Numpy-discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> _______________________________________________ >>> Numpy-discussion mailing list >>> Numpy-discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > Nicolas Pinto > Ph.D. Candidate, Brain & Computer Sciences > Massachusetts Institute of Technology, USA > http://web.mit.edu/pinto > From kwgoodman at gmail.com Wed Jul 22 14:35:23 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 22 Jul 2009 11:35:23 -0700 Subject: [Numpy-discussion] ValueError: cannot convert float to NaN to integer Message-ID: On Ubuntu 9.04, python 2.6.2, numpy 1.2.1 this gives a ValueError: x = np.array([1,2,3]) x[0] = np.nan ValueError: cannot convert float to NaN to integer But on Debian squeeze, python 2.5.4, numpy 1.2.1 the assignment works (well, the float nan is convert to the int 0): x[0] = np.nan x array([0,1,2]) The numpy version numbers are the same. Is this a python issue? BTW, is there a way to determine the svn revision number of numpy when installed from binary? From d_l_goldsmith at yahoo.com Wed Jul 22 14:38:45 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Wed, 22 Jul 2009 11:38:45 -0700 (PDT) Subject: [Numpy-discussion] String manipulation Message-ID: <104576.38899.qm@web52101.mail.re2.yahoo.com> By the way, Chris, what's it gonna take to get you contributing to the doc? ;-) DG --- On Wed, 7/22/09, Christopher Barker wrote: > From: Christopher Barker > Subject: Re: [Numpy-discussion] String manipulation > To: "Discussion of Numerical Python" > Date: Wednesday, July 22, 2009, 11:33 AM > St?fan van der Walt wrote: > > You cannot use view on an array scalar, so you'll have > to convert to > > an array first: > > > > x = x.reshape((1,)) > > x.view('S1') > > Bingo! > > Thanks, St?fan. > > I don't know why one can't uwe view on a scalar -- there is > still a dat pointer isn't there?, and it can be reshaped. > Butno matter, now I have a? one liner that splits a > string up into fixed fields, and returns a float array of > the numbers in those fields: > > np.array((line,)).view(dtype='S%i'%field_len).astype(np.float) > > I suspect this is faster than the list comp method, though > I haven't tested it -- I"ll leave that to someone that > actually has this problem to solve ;-) > > I've enclosed a bit of test code with a few other > approaches as well. > > -Chris > > > > > -- Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R? ? ? ? ? ? > (206) 526-6959???voice > 7600 Sand Point Way NE???(206) > 526-6329???fax > Seattle, WA? 98115? ? > ???(206) 526-6317???main > reception > > Chris.Barker at noaa.gov > > -----Inline Attachment Follows----- > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Wed Jul 22 14:51:37 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Jul 2009 13:51:37 -0500 Subject: [Numpy-discussion] ValueError: cannot convert float to NaN to integer In-Reply-To: References: Message-ID: <3d375d730907221151g12ca2715q967bc3a3473b178d@mail.gmail.com> On Wed, Jul 22, 2009 at 13:35, Keith Goodman wrote: > On Ubuntu 9.04, python 2.6.2, numpy 1.2.1 this gives a ValueError: > > x = np.array([1,2,3]) > x[0] = np.nan > ValueError: cannot convert float to NaN to integer > > But on Debian squeeze, python 2.5.4, numpy 1.2.1 the assignment works > (well, the float nan is convert to the int 0): > > x[0] = np.nan > x > ? array([0,1,2]) > > The numpy version numbers are the same. Is this a python issue? Probably. They rationalized a bunch of float behavior in Python 2.6, and this may be a consequence of that. grep the Python sources for that error message, or parts of it, to try to track down where it comes from. > BTW, is there a way to determine the svn revision number of numpy when > installed from binary? Not reliably, I don't think. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Jul 23 00:36:25 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 22 Jul 2009 22:36:25 -0600 Subject: [Numpy-discussion] sorting and nans, timings. Message-ID: Hi All, I changed the sort routines to sort nans to the end and got some timings. Sorting 100000 random doubles 100 times yields: current nan version quicksort 1.17 sec 1.29 sec mergesort 1.37 sec 1.36 sec heapsort 1.83 sec 2.12 sec Curiously, mergesort doesn't seem to suffer at all. This is using x != x for nan detection, using npy_isnan is notably slower with my compiler (gcc 4.3.0). So, what do folks want to do about sorting nans? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Jul 23 02:43:08 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 23 Jul 2009 15:43:08 +0900 Subject: [Numpy-discussion] sorting and nans, timings. In-Reply-To: References: Message-ID: <4A68067C.9070202@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > Hi All, > > I changed the sort routines to sort nans to the end and got some > timings. Sorting 100000 random doubles 100 times yields: > > current nan version > quicksort 1.17 sec 1.29 sec > mergesort 1.37 sec 1.36 sec > heapsort 1.83 sec 2.12 sec > > Curiously, mergesort doesn't seem to suffer at all. This is using x != > x for nan detection, using npy_isnan is notably slower with my > compiler (gcc 4.3.0). That's because glibc isnan is slow :) I was surprised, but our own isnan replacement is much faster than the glibc one at least (like almost twice as fast on my P4 machine). Glibc isnan avoids branching, but it seems that it tries too hard. cheers, David From david at ar.media.kyoto-u.ac.jp Thu Jul 23 03:01:05 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 23 Jul 2009 16:01:05 +0900 Subject: [Numpy-discussion] sorting and nans, timings. In-Reply-To: <4A68067C.9070202@ar.media.kyoto-u.ac.jp> References: <4A68067C.9070202@ar.media.kyoto-u.ac.jp> Message-ID: <4A680AB1.2000202@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Charles R Harris wrote: > >> Hi All, >> >> I changed the sort routines to sort nans to the end and got some >> timings. Sorting 100000 random doubles 100 times yields: >> >> current nan version >> quicksort 1.17 sec 1.29 sec >> mergesort 1.37 sec 1.36 sec >> heapsort 1.83 sec 2.12 sec >> >> Curiously, mergesort doesn't seem to suffer at all. This is using x != >> x for nan detection, using npy_isnan is notably slower with my >> compiler (gcc 4.3.0). >> > > That's because glibc isnan is slow :) I was surprised, but our own isnan > replacement is much faster than the glibc one at least (like almost > twice as fast on my P4 machine). Glibc isnan avoids branching, but it > seems that it tries too hard. > IMO, a 10-15 % increase in time for better nan handling in sort definitely worths it. cheers, David From meine at informatik.uni-hamburg.de Thu Jul 23 05:20:12 2009 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 23 Jul 2009 11:20:12 +0200 Subject: [Numpy-discussion] Comparing the precision of dtypes? In-Reply-To: References: <200907221414.42780.meine@informatik.uni-hamburg.de> Message-ID: <200907231120.13092.meine@informatik.uni-hamburg.de> On Wednesday 22 July 2009 17:16:31 Ralf Gommers wrote: > Examples where min/max probably does not do what you want, and > find_common_type does: > > In [49]: max(float, float32) > Out[49]: > In [50]: find_common_type([], [float, float32]) > Out[50]: dtype('float64') I don't understand the following behaviour: In [5]: numpy.find_common_type([float32,int32],[]) Out[5]: dtype('float32') In [6]: numpy.find_common_type([int32,float32],[]) Out[6]: dtype('int32') In both cases, I would actually expect float64 to be the answer (since int32 does not "fit into" float32). And I wonder why it depends on the order? When I actually try the same with arrays, I get float64 as expected: In [8]: a = numpy.array([1,2,3], int32) In [9]: b = numpy.array([1,2,3], float32) In [10]: a + b Out[10]: array([ 2., 4., 6.]) In [11]: b + a Out[11]: array([ 2., 4., 6.]) Is this a known bug in numpy 1.3.0, or am I missing something? Have a nice day! Hans From lciti at essex.ac.uk Thu Jul 23 05:56:50 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Thu, 23 Jul 2009 10:56:50 +0100 Subject: [Numpy-discussion] Comparing the precision of dtypes? References: <200907221414.42780.meine@informatik.uni-hamburg.de> <200907231120.13092.meine@informatik.uni-hamburg.de> Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E36@sernt14.essex.ac.uk> Hi Hans, You can follow this thread on approximately the same topic http://news.gmane.org/find-root.php?group=gmane.comp.python.numeric.general&article=31551 It should have been fixed in http://projects.scipy.org/numpy/changeset/7133 Best, Luca From scott.sinclair.za at gmail.com Thu Jul 23 06:07:46 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 23 Jul 2009 12:07:46 +0200 Subject: [Numpy-discussion] Getting 95%/99% margin of ndarray In-Reply-To: <82C47E17-59AD-469F-90B8-B8950FC14C95@gmail.com> References: <4A674018.8050401@gmx.de> <82C47E17-59AD-469F-90B8-B8950FC14C95@gmail.com> Message-ID: <6a17e9ee0907230307v68d385afl3da1c42aa4e565fb@mail.gmail.com> > 2009/7/22 Pierre GM : > You could try scipy.stats.scoreatpercentile, > scipy.stats.mstats.plottingposition or scipy.stats.mstats.mquantiles, > which will all approximate quantiles of your distribution. It seems that mquantiles doesn't do what you'd expect when the limit keyword argument is specified. There's a patch for review here: http://codereview.appspot.com/97077 Cheers, Scott From pgmdevlist at gmail.com Thu Jul 23 06:16:27 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 23 Jul 2009 06:16:27 -0400 Subject: [Numpy-discussion] Getting 95%/99% margin of ndarray In-Reply-To: <6a17e9ee0907230307v68d385afl3da1c42aa4e565fb@mail.gmail.com> References: <4A674018.8050401@gmx.de> <82C47E17-59AD-469F-90B8-B8950FC14C95@gmail.com> <6a17e9ee0907230307v68d385afl3da1c42aa4e565fb@mail.gmail.com> Message-ID: On Jul 23, 2009, at 6:07 AM, Scott Sinclair wrote: >> 2009/7/22 Pierre GM : >> You could try scipy.stats.scoreatpercentile, >> scipy.stats.mstats.plottingposition or scipy.stats.mstats.mquantiles, >> which will all approximate quantiles of your distribution. > > It seems that mquantiles doesn't do what you'd expect when the limit > keyword argument is specified. There's a patch for review here: Thx for the patch, I'll port it in the next few hours. However, I disagree with the last few lines (where the quantiles are transformed to a standard ndarray if the mask is nomask. For consistency, we should always have a MaskedArray, don't you think ? (And anyway, taking a view as a ndarray is faster than using np.asarray...) Thx again P. From slaunger at gmail.com Thu Jul 23 07:36:24 2009 From: slaunger at gmail.com (Kim Hansen) Date: Thu, 23 Jul 2009 13:36:24 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit Win XP for accumulated file size above approx. 1 GB Message-ID: OS. Win XP SP3, 32 bits Python: 2.5.4 Numpy: 1.3.0 I have am having some major problems converting a 750 MB recarray into a 850 MB recarray To save RAM I would like to use a read-only and a writeable memap for the two recarrays during the conversion. So I do something like: import os from stat import ST_SIZE import numpy as np ... records = os.stat(toconvert_path)[ST_SIZE] / toconvert_dtype.itemsize toconvert = np.memmap(toconvert_path, dtype=toconvert_dtype, mode="r").view(np.recarray) result = np.memmap(result_path, dtype = result_dtype, mode = "w+", shape=(records,)) The code manages to create the toconvert memmap (750 MB), but when trying to create the second memap object I get File "C:\Python25\Lib\site-packages\numpy\core\memmap.py", line 226, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc) WindowsError: [Error 8] Not enough storage is available to process this command By tracing before and after, I can see the file size is zero before calling mmap.mmap and has the expected 850 MB size after the WindowsError has been thrown somewhere inside mmap.mmap. There is 26 GB of free disc space, so the error message seems wrong? If I comment out the creation of the first memmap, I can successfully create the result memmap, so the error seems to be related to the accumulated size of all mmap.mmaps generated. I have other cases with somewhat smaller files to convert where the transition is OK. It seem like I begin to get these problems when the accumulated size of memmaps exceeds 1GB I am surprised by this, as http://docs.scipy.org/doc/numpy/reference/arrays.classes.html#memory-mapped-file-arrays mentions there are upper bounds to the size when using versions of python before 2.5. >From this I had the impression that there was no size limit as long as you were using ver. >=2.5 (as I am) Is it due to the 32 bit OS I am using? Is there anything I can do to resolve the problem? Best wishes, Kim From scott.sinclair.za at gmail.com Thu Jul 23 08:08:03 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 23 Jul 2009 14:08:03 +0200 Subject: [Numpy-discussion] Getting 95%/99% margin of ndarray In-Reply-To: References: <4A674018.8050401@gmx.de> <82C47E17-59AD-469F-90B8-B8950FC14C95@gmail.com> <6a17e9ee0907230307v68d385afl3da1c42aa4e565fb@mail.gmail.com> Message-ID: <6a17e9ee0907230508u54c72501h71fd3a87fd3fdcbd@mail.gmail.com> > 2009/7/23 Pierre GM : > > On Jul 23, 2009, at 6:07 AM, Scott Sinclair wrote: > >>> 2009/7/22 Pierre GM : >>> You could try scipy.stats.scoreatpercentile, >>> scipy.stats.mstats.plottingposition or scipy.stats.mstats.mquantiles, >>> which will all approximate quantiles of your distribution. >> >> It seems that mquantiles doesn't do what you'd expect when the limit >> keyword argument is specified. There's a patch for review here: > > Thx for the patch, I'll port it in the next few hours. However, I > disagree with the last few lines (where the quantiles are transformed > to a standard ndarray if the mask is nomask. For consistency, we > should always have a MaskedArray, don't you think ? (And anyway, > taking a view as a ndarray is faster than using np.asarray...) Agree it's more consistent to always return a MaskedArray. I don't remember why I chose to return an ndarray. I think that it was probably to do with the fact that an ndarray is returned when 'axis' isn't specified... >>> import numpy as np >>> import scipy as sp >>> sp.__version__ '0.8.0.dev5874' >>> from scipy.stats.mstats import mquantiles >>> a = np.array([6., 47., 49., 15., 42., 41., 7., 39., 43., 40., 36.]) >>> type(mquantiles(a)) >>> type(mquantiles(np.ma.masked_array(a))) >>> type(mquantiles(a, axis=0)) This could be fixed by forcing _quantiles1D() to always return a MaskedArray. Cheers, Scott From perfreem at gmail.com Thu Jul 23 09:14:05 2009 From: perfreem at gmail.com (per freem) Date: Thu, 23 Jul 2009 09:14:05 -0400 Subject: [Numpy-discussion] pdf for multivariate normal function? Message-ID: hi all, i'm trying to find the function for the pdf of a multivariate normal pdf. i know that the function "multivariate_normal" can be used to sample from the multivariate normal distribution, but i just want to get the pdf for a given vector of means and a covariance matrix. is there a function to do this? thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jul 23 09:16:05 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 23 Jul 2009 07:16:05 -0600 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit Win XP for accumulated file size above approx. 1 GB In-Reply-To: References: Message-ID: On Thu, Jul 23, 2009 at 5:36 AM, Kim Hansen wrote: > OS. Win XP SP3, 32 bits > Python: 2.5.4 > Numpy: 1.3.0 > > I have am having some major problems converting a 750 MB recarray into > a 850 MB recarray > > To save RAM I would like to use a read-only and a writeable memap for > the two recarrays during the conversion. > > So I do something like: > > import os > from stat import ST_SIZE > import numpy as np > ... > records = os.stat(toconvert_path)[ST_SIZE] / toconvert_dtype.itemsize > toconvert = np.memmap(toconvert_path, dtype=toconvert_dtype, > mode="r").view(np.recarray) > result = np.memmap(result_path, dtype = result_dtype, mode = "w+", > shape=(records,)) > > The code manages to create the toconvert memmap (750 MB), > but when trying to create the second memap object I get > File "C:\Python25\Lib\site-packages\numpy\core\memmap.py", line 226, > in __new__ > mm = mmap.mmap(fid.fileno(), bytes, access=acc) > WindowsError: [Error 8] Not enough storage is available to process this > command > > By tracing before and after, I can see the file size is zero before > calling mmap.mmap and has the expected 850 MB size after the > WindowsError has been thrown somewhere inside mmap.mmap. There is 26 > GB of free disc space, so the error message seems wrong? > > If I comment out the creation of the first memmap, I can successfully > create the result memmap, so the error seems to be related to the > accumulated size of all mmap.mmaps generated. I have other cases with > somewhat smaller files to convert where the transition is OK. It seem > like I begin to get these problems when the accumulated size of > memmaps exceeds 1GB > > I am surprised by this, as > > > http://docs.scipy.org/doc/numpy/reference/arrays.classes.html#memory-mapped-file-arrays > > mentions there are upper bounds to the size when using versions of > python before 2.5. > > >From this I had the impression that there was no size limit as long as > you were using ver. >=2.5 (as I am) > > Is it due to the 32 bit OS I am using? > It could be. IIRC, 32 bit windows gives user programs 2 GB of addressable memory, so your files need to fit in that space even if the data is on disk. You aren't using that much memory but you are close and it could be that other programs make up the difference. Maybe you can monitor the memory to get a better idea of the usage. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy-discussion at maubp.freeserve.co.uk Thu Jul 23 09:18:08 2009 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Thu, 23 Jul 2009 14:18:08 +0100 Subject: [Numpy-discussion] String to integer array of ASCII values Message-ID: <320fb6e00907230618p1f2bcba6qc3483098a4986203@mail.gmail.com> Dear all, I've looked over some of the documentation for creating an array, e.g. http://docs.scipy.org/doc/numpy/user/basics.creation.html#arrays-creation http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html However, I don't see an example quite like what I want to do. I want to be able to take a python string (e.g. "ABCDEF") and turn it into an array of the ASCII values (i.e. [65, 66, 67, 68, 69, 70] for this example). >>> import numpy >>> numpy.__version__ '1.1.1' I can get the result I want like this, but I would like a faster way: >>> numpy.array([ord(letter) for letter in "ABCDEF"]) array([65, 66, 67, 68, 69, 70]) I know in C that a string can been regarded as an array of unsigned integers - so I'd like to get NumPy to do that for me. I'm guessing there is a magic data type I can specify. Using "c" appears to mean characters which is close but isn't what I want: >>> numpy.array("ABCDEF", "c") array(['A', 'B', 'C', 'D', 'E', 'F'], dtype='|S1') I eventually found this works: >>> numpy.frombuffer("ABCDEF", numpy.byte) array([65, 66, 67, 68, 69, 70], dtype=int8) But why don't these work too? >>> numpy.array("ABCDEF", numpy.byte) Traceback (most recent call last): ... ValueError: setting an array element with a sequence. >>> numpy.fromiter("ABCDEF", numpy.byte, count=6) Traceback (most recent call last): ... ValueError: setting an array element with a sequence. So, is using frombuffer the only or best option? Thanks, Peter From slaunger at gmail.com Thu Jul 23 09:48:14 2009 From: slaunger at gmail.com (Kim Hansen) Date: Thu, 23 Jul 2009 15:48:14 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit Win XP for accumulated file size above approx. 1 GB In-Reply-To: References: Message-ID: 2009/7/23 Charles R Harris : > > >> >> Is it due to the 32 bit OS I am using? > > It could be. IIRC, 32 bit windows gives user programs 2 GB of addressable > memory, so your files need to fit in that space even if the data is on disk. > You aren't using that much memory but you are close and it could be that > other programs make up the difference. Maybe you can monitor the memory to > get a better idea of the usage. > > Chuck Hi Chuck, If I use the Windows task manager to see how much memory is used by the Python application when running the memmap test it says Before loading first memmap: 8.588 MB After loading first memmap: 8.596 MB i.e. only an additional 8 kB for having the 750 MB recarray available Maybe I am measuring memory usage wrong? Kim Kim From emmanuelle.gouillart at normalesup.org Thu Jul 23 10:17:37 2009 From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart) Date: Thu, 23 Jul 2009 16:17:37 +0200 Subject: [Numpy-discussion] Numpy/Scipy and the Python African Tour Message-ID: <20090723141737.GF10321@phare.normalesup.org> Dear users of Numpy and Scipy, here is an informal report on the last event of the Python African Tour, which took place in Dakar (Senegal) on July 6-10th. It might interest only a fraction of the lists, so I apologize for the spamming. What is the Python African Tour? ---------------------------------- It is a recent initiative to promote the use of Python in African countries, that we owe to Kamon Ayeva (http://kayeva.wordpress.com/). Born a Togolese, Kamon now lives in Paris (France) where he works as a web developer and a trainer in web technologies. It occurred to him that but a few African developers (Hi, Stefan! :D) participate in Python-related development or conferences. IT-related technologies and web development can contribute to economic growth in developing countries, and, because of the clarity and flexibility of the language, Python-trained developers may have a competitive advantage to develop successful businesses. The Python African Tour sends a few volunteers to places in Africa where a small core of Python users or developers already exists. These deliver a training over a few days, about Python and some of its applications. The goal is to create a small but dynamical community of users that will keep on growing after the volunteers have left. A first event was held in Morocco last December. Whereas the emphasis had been put on an introduction to the Python language and to the Django web framework, I proposed to give this time an additional course about the use of Python in scientific computing, which was nicknamed "Scientific Python". Numpy/Scipy is an opportunity to present the language as attractive for not only geeks, but also academic staff -- the people who may precisely teach the language in the universities and engineering departments. Organization of the Dakar event ---------------------------------- This year, three European trainers flew to Dakar for the event: Kamon Ayeva, Daniel Nouri and myself. For all courses we worked in pairs with three local organizers and Python experts: Thomas No?l, Ousmane Wilane and Sergiu Mihai. The course was organized as follows: * 1.5 day for a general introduction to Python * 1 day of specialization, either on the Django web framework (Kamon and Daniel) or on Python for scientific computing (Emmanuelle) * 1 day of barcamp, or informal talks showing how we would use Python in our daily work. * 1.5 day of "sprint", I should rather say practical work. The hosting organizations were the AUF (Agence universitaire de la Francophonie), the Dakar Linux User Group (DakarLUG) and the ESP (Ecole Sup?rieure Polytechnique, a master-level engineer school). The local team did a great job on announcing the event, rounding up potential attendees (and sponsors!), and selecting people. The course was free of charge, but there were only a limited number of positions. Finally, about 50 people showed up at the course (20 having chosen the Scientific Python course). Facilities at AUF were excellent: three rooms with a total of 70 PCs running the latest Ubuntu. The course on "Scientific Python" ---------------------------------- Course material ............... Whereas we reused the slides of the Moroccan event on the introduction to Python, I wrote the slides for the 3-4 hour course about Scientific Python. The slides were written in French, English being an issue for a part of the students. They can be found on http://www.dakarlug.org/pat/scientifique/html/ for French-reading people (my apologies to the others! I'm considering translating the slides to English but I will do it more rapidly if I'm given the incentive to do so, so tell me if you may be interested by an English version!). Given the duration of the course and the fact that the trainees had already been introduced to Python, I chose to concentrate mostly on Numpy, and on Scipy to a lesser extent. However, I tried to explain as soon as possible how to use numpy and scipy in a "real life" scientific workflow: plotting data with matplotlib or mayavi, opening data files, finding documentation, etc. How it went ............ Together with Thomas No?l, we followed the same group of 20 people during the training week. The group was composed of both students (master and PhD), and a few senior academic staff, professors and researchers. I'm glad to say that the course was a real success, insofar as everybody was convinced he or she could use Python with some benefit for his or her research. Most people were already using either Matlab (with some license issues...), R or Java for scientific computing. They were attracted by the following features of Python: * Python is a free software, so there are no license problems * with its wide range of scientific modules, almost *everything* can be done with Python Also, people were encouraged when they saw at the end of the week that they could work on their own data using their usual algorithms in Python, without too many difficulties. Speaking about difficulties, one may say that there was some disparity in the level of experience inside the group, some being very well at ease with array programming while others were obviously discovering programming. It was therefore difficult to meet everybody's needs, but well, I hope everybody learned something! Some pictures of the event are on http://dakarlug.org/pat/galerie/. What's next: outlook on the PAT -------------------------------- The Dakar event was a very encouraging one, and some other countries (Togo or Zambia) are already considered for the next stop of the PAT. Of course, before keeping moving towards the next event, we first would like to see what happens in Dakar after the PAT has left. Some pedagogical staff decided there would be a course on Python next academic year, and this is a very positive decision: the one-week course was definitely too short for the students. Now the big question is who is going to teach the course, as all professors were beginners in Python. A new mailing-list has also been created to gather the trainees together after the event. If you are interested by the PAT and you want to know more about past and future events, you may write to the dedicated mailing-list python-african-tour-discussion at lists.openplans.org, or take a look at the website http://www.coactivate.org/projects/python-african-tour. Thanks a lot for your patience if you have read so far! Cheers, Emmanuelle From charlesr.harris at gmail.com Thu Jul 23 10:28:28 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 23 Jul 2009 08:28:28 -0600 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit Win XP for accumulated file size above approx. 1 GB In-Reply-To: References: Message-ID: On Thu, Jul 23, 2009 at 7:48 AM, Kim Hansen wrote: > 2009/7/23 Charles R Harris : > > > > > >> > >> Is it due to the 32 bit OS I am using? > > > > It could be. IIRC, 32 bit windows gives user programs 2 GB of addressable > > memory, so your files need to fit in that space even if the data is on > disk. > > You aren't using that much memory but you are close and it could be that > > other programs make up the difference. Maybe you can monitor the memory > to > > get a better idea of the usage. > > > > Chuck > Hi Chuck, > > If I use the Windows task manager to see how much memory is used by > the Python application when running the memmap test it says > > Before loading first memmap: 8.588 MB > After loading first memmap: 8.596 MB > > i.e. only an additional 8 kB for having the 750 MB recarray available > > Maybe I am measuring memory usage wrong? > Hmm, I don't know what you should be looking at in XP. Memmapped files are sort of like virtual memory and exist in the address space even if they aren't in physical memory. When you address an element that isn't in physical memory there is a page fault and the OS reads in the needed page from disk. If you read through the file physical memory will probably fill up because the OS will keep try to keep as many pages in physical memory as possible in case they are referenced again. But I am not sure how windows does it's memory accounting or how it is displayed, someone here more familiar with windows may be able to tell you what to look for. Or you could try running on a 64 bit system if there is one available. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jul 23 10:43:06 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 23 Jul 2009 08:43:06 -0600 Subject: [Numpy-discussion] pdf for multivariate normal function? In-Reply-To: References: Message-ID: On Thu, Jul 23, 2009 at 7:14 AM, per freem wrote: > hi all, > > i'm trying to find the function for the pdf of a multivariate normal pdf. i > know that the function "multivariate_normal" can be used to sample from the > multivariate normal distribution, but i just want to get the pdf for a given > vector of means and a covariance matrix. is there a function to do this? > Well, what does a pdf mean in the multidimensional case? One way to convert the density function into a Stieltjes type measure is to plot the integral over a polytope with one corner at [-inf, -inf,....] and the diagonally opposite corner at the plotting point, but the multidimensional display of the result might not be very informative. What do you actually want here? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jul 23 10:54:16 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 23 Jul 2009 08:54:16 -0600 Subject: [Numpy-discussion] String to integer array of ASCII values In-Reply-To: <320fb6e00907230618p1f2bcba6qc3483098a4986203@mail.gmail.com> References: <320fb6e00907230618p1f2bcba6qc3483098a4986203@mail.gmail.com> Message-ID: On Thu, Jul 23, 2009 at 7:18 AM, Peter < numpy-discussion at maubp.freeserve.co.uk> wrote: > Dear all, > > I've looked over some of the documentation for creating an array, e.g. > http://docs.scipy.org/doc/numpy/user/basics.creation.html#arrays-creation > http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html > > However, I don't see an example quite like what I want to do. I want > to be able to take a python string (e.g. "ABCDEF") and turn it into > an array of the ASCII values (i.e. [65, 66, 67, 68, 69, 70] for this > example). > > >>> import numpy > >>> numpy.__version__ > '1.1.1' > > I can get the result I want like this, but I would like a faster way: > > >>> numpy.array([ord(letter) for letter in "ABCDEF"]) > array([65, 66, 67, 68, 69, 70]) > > I know in C that a string can been regarded as an array of unsigned > integers - so I'd like to get NumPy to do that for me. I'm guessing > there is a magic data type I can specify. Using "c" appears to mean > characters which is close but isn't what I want: > > >>> numpy.array("ABCDEF", "c") > array(['A', 'B', 'C', 'D', 'E', 'F'], > dtype='|S1') > > I eventually found this works: > > >>> numpy.frombuffer("ABCDEF", numpy.byte) > array([65, 66, 67, 68, 69, 70], dtype=int8) > > But why don't these work too? > > >>> numpy.array("ABCDEF", numpy.byte) > Traceback (most recent call last): > ... > ValueError: setting an array element with a sequence. > >>> numpy.fromiter("ABCDEF", numpy.byte, count=6) > Traceback (most recent call last): > ... > ValueError: setting an array element with a sequence. > > So, is using frombuffer the only or best option? > Would something like In [2]: array("ABCDEF", 'c').view(uint8) Out[2]: array([65, 66, 67, 68, 69, 70], dtype=uint8) work for you? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.h.jaffe at gmail.com Thu Jul 23 11:42:48 2009 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu, 23 Jul 2009 16:42:48 +0100 Subject: [Numpy-discussion] pdf for multivariate normal function? In-Reply-To: References: Message-ID: <4A6884F8.7090309@gmail.com> Hi, Charles R Harris wrote: > > On Thu, Jul 23, 2009 at 7:14 AM, per freem > wrote: > > i'm trying to find the function for the pdf of a multivariate normal > pdf. i know that the function "multivariate_normal" can be used to > sample from the multivariate normal distribution, but i just want to > get the pdf for a given vector of means and a covariance matrix. is > there a function to do this? > > Well, what does a pdf mean in the multidimensional case? One way to > convert the density function into a Stieltjes type measure is to plot > the integral over a polytope with one corner at [-inf, -inf,....] and > the diagonally opposite corner at the plotting point, but the > multidimensional display of the result might not be very informative. > What do you actually want here? You are confusing PDF (Probability Density Functions) with CDF (Cumulative Density Function), I think. The PDF is well-defined for multivariate distributions. It is defined so that P(x) dx is the probability to be in the infinitesimal range (x,x+dx). For a multivariate gaussian, it's P(x|m, C) = [1/det(2 pi C)] exp{ -1/2 (x-m)^T C^{-1} (x-m) } in matrix notation, where m is the mean and C is the covariance matrix. Andrew From a.h.jaffe at gmail.com Thu Jul 23 11:42:48 2009 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu, 23 Jul 2009 16:42:48 +0100 Subject: [Numpy-discussion] pdf for multivariate normal function? In-Reply-To: References: Message-ID: <4A6884F8.7090309@gmail.com> Hi, Charles R Harris wrote: > > On Thu, Jul 23, 2009 at 7:14 AM, per freem > wrote: > > i'm trying to find the function for the pdf of a multivariate normal > pdf. i know that the function "multivariate_normal" can be used to > sample from the multivariate normal distribution, but i just want to > get the pdf for a given vector of means and a covariance matrix. is > there a function to do this? > > Well, what does a pdf mean in the multidimensional case? One way to > convert the density function into a Stieltjes type measure is to plot > the integral over a polytope with one corner at [-inf, -inf,....] and > the diagonally opposite corner at the plotting point, but the > multidimensional display of the result might not be very informative. > What do you actually want here? You are confusing PDF (Probability Density Functions) with CDF (Cumulative Density Function), I think. The PDF is well-defined for multivariate distributions. It is defined so that P(x) dx is the probability to be in the infinitesimal range (x,x+dx). For a multivariate gaussian, it's P(x|m, C) = [1/det(2 pi C)] exp{ -1/2 (x-m)^T C^{-1} (x-m) } in matrix notation, where m is the mean and C is the covariance matrix. Andrew From numpy-discussion at maubp.freeserve.co.uk Thu Jul 23 11:50:06 2009 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Thu, 23 Jul 2009 16:50:06 +0100 Subject: [Numpy-discussion] String to integer array of ASCII values In-Reply-To: References: <320fb6e00907230618p1f2bcba6qc3483098a4986203@mail.gmail.com> Message-ID: <320fb6e00907230850p4778f0baj3f74ea63dc4c6475@mail.gmail.com> On Thu, Jul 23, 2009 at 3:54 PM, Charles R Harris wrote: > > Would something like > > In [2]: array("ABCDEF", 'c').view(uint8) > Out[2]: array([65, 66, 67, 68, 69, 70], dtype=uint8) > > work for you? Yes, that also looks good. I guess I have a couple of options to benchmark now :) Thank you, Peter From charlesr.harris at gmail.com Thu Jul 23 12:07:19 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 23 Jul 2009 10:07:19 -0600 Subject: [Numpy-discussion] String to integer array of ASCII values In-Reply-To: <320fb6e00907230850p4778f0baj3f74ea63dc4c6475@mail.gmail.com> References: <320fb6e00907230618p1f2bcba6qc3483098a4986203@mail.gmail.com> <320fb6e00907230850p4778f0baj3f74ea63dc4c6475@mail.gmail.com> Message-ID: On Thu, Jul 23, 2009 at 9:50 AM, Peter < numpy-discussion at maubp.freeserve.co.uk> wrote: > On Thu, Jul 23, 2009 at 3:54 PM, Charles R > Harris wrote: > > > > Would something like > > > > In [2]: array("ABCDEF", 'c').view(uint8) > > Out[2]: array([65, 66, 67, 68, 69, 70], dtype=uint8) > > > > work for you? > > Yes, that also looks good. I guess I have a couple of options to > benchmark now :) > One more: In [1]: fromstring("ABCDEF", dtype=uint8) Out[1]: array([65, 66, 67, 68, 69, 70], dtype=uint8) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy-discussion at maubp.freeserve.co.uk Thu Jul 23 12:24:56 2009 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Thu, 23 Jul 2009 17:24:56 +0100 Subject: [Numpy-discussion] String to integer array of ASCII values In-Reply-To: References: <320fb6e00907230618p1f2bcba6qc3483098a4986203@mail.gmail.com> <320fb6e00907230850p4778f0baj3f74ea63dc4c6475@mail.gmail.com> Message-ID: <320fb6e00907230924w477552bmce2c2da6ed8ced1f@mail.gmail.com> On Thu, Jul 23, 2009 at 5:07 PM, Charles R Harris wrote: > > One more: > > ?In [1]: fromstring("ABCDEF", dtype=uint8) > Out[1]: array([65, 66, 67, 68, 69, 70], dtype=uint8) > > Chuck I should have guessed that one. Why isn't numpy.fromstring listed with the other entries in the "From existing data" section here? http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html This looks like a simple improvement to the documentation... Peter From dsdale24 at gmail.com Thu Jul 23 12:54:45 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Thu, 23 Jul 2009 12:54:45 -0400 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> <9457e7c80907201433i766eb9a9x47e6a449c10450d7@mail.gmail.com> Message-ID: On Tue, Jul 21, 2009 at 10:11 AM, Darren Dale wrote: > On Tue, Jul 21, 2009 at 7:44 AM, Darren Dale wrote: >> 2009/7/20 St?fan van der Walt : >>> Hi Chuck >>> >>> 2009/7/17 Charles R Harris : >>>> PyObject*?PyTuple_GetItem(PyObject *p, Py_ssize_t pos) >>>> Return value: Borrowed reference. >>>> Return the object at position pos in the tuple pointed to by p. If pos is >>>> out of bounds, return NULL and sets an IndexError exception. It's a borrowed >>>> reference so you need to call Py_INCREF on it. I find this Python C-API >>>> documentation useful. >>> >>> Have you had a look over the rest of the code? ?I think this would >>> make a good addition. ?Travis mentioned Contexts for doing something >>> similar, but I don't know enough about that concept to compare the >>> two. >> >> I think contexts would be very different from what is already in >> place. For now, it would be nice to make this one small improvement to >> the existing ufunc infrastructure, and maybe consider contexts (which >> I still don't understand) at a later time. I have improved the code >> slightly and added a few tests, and will post a new patch later this >> morning. I just need to add some documentation. > > Here is a better patch, which includes a few additional tests and adds > some documentation. It also attempts to improve the docstring and > sphinx docs for __array_wrap__, which may have been a little bit > misleading. There is also some whitespace cleanup in a few places. > Would someone please review my work and commit the patch if it is > acceptable? Pierre or Travis, would either of you have a chance to > look over the implementation and the documentation changes, since you > two seem to be most familiar with ufuncs and subclassing ndarray? It looks like part of my patch has been clobbered by changes introduced in svn 7184-7191. What else should I be doing so a patch like this can be committed relatively quickly? Darren From d_l_goldsmith at yahoo.com Thu Jul 23 12:58:17 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Thu, 23 Jul 2009 09:58:17 -0700 (PDT) Subject: [Numpy-discussion] String to integer array of ASCII values Message-ID: <617398.84444.qm@web52107.mail.re2.yahoo.com> --- On Thu, 7/23/09, Peter wrote: > I should have guessed that one. Why isn't numpy.fromstring > listed with the > other entries in the "From existing data" section here? Maybe because it got listed in the I/O category and wasn't to be double listed? > http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html > > This looks like a simple improvement to the > documentation... Yup, that it is; corrected here: http://docs.scipy.org/numpy/docs/numpy-docs/reference/routines.array-creation.rst/ But not yet showing up at your link. :-( DG From jacopo.pecci at gmail.com Thu Jul 23 13:13:26 2009 From: jacopo.pecci at gmail.com (Jacopo) Date: Thu, 23 Jul 2009 17:13:26 +0000 (UTC) Subject: [Numpy-discussion] numpy subclass Message-ID: I am facing the opposite problem described here: http://docs.scipy.org/doc/numpy/user/basics.subclassing.html In short, I want to create a class Dummy, inherited from np.ndarray, which returns a plain array whenever an instance is sliced or viewed. I cannot figure out how. I would really appreciate your help, here is a chunk of code, what is commented was an attempt which didn?t work out. Thanks a lot, Jacopo p.s. I can be more wordy in explain the problem i just dont want to overwhelm you. import numpy as np class Dummy(np.ndarray): def __new__(cls, array): obj=array.view(cls) return obj def __array_finalize__(self, obj): #self=self.view(np.ndarray) #self=np.asarray(self) d=Dummy(np.ones(5)) d2=d[:3] print type(d2) # I wish this was np.array From pgmdevlist at gmail.com Thu Jul 23 13:30:23 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 23 Jul 2009 13:30:23 -0400 Subject: [Numpy-discussion] numpy subclass In-Reply-To: References: Message-ID: <6560A000-C7CF-40F8-B450-E4E4C83AEF42@gmail.com> On Jul 23, 2009, at 1:13 PM, Jacopo wrote: > > In short, I want to create a class Dummy, inherited from np.ndarray, > which > returns a plain array whenever an instance is sliced or viewed. I > cannot figure > out how. Easy enough for slices: class Dummy(np.ndarray): def __new__(cls, array): obj=array.view(cls) return obj def __getslice__(self, i, j): return np.ndarray.__getslice__(self, i, j).view(np.ndarray) For views, the problem needs to be better described: why would you want to get just a basic ndarray ? From jacopo.pecci at gmail.com Thu Jul 23 16:03:52 2009 From: jacopo.pecci at gmail.com (Jacopo) Date: Thu, 23 Jul 2009 20:03:52 +0000 (UTC) Subject: [Numpy-discussion] numpy subclass References: <6560A000-C7CF-40F8-B450-E4E4C83AEF42@gmail.com> Message-ID: Pierre GM gmail.com> writes: > > > On Jul 23, 2009, at 1:13 PM, Jacopo wrote: > > > > In short, I want to create a class Dummy, inherited from np.ndarray, > > which > > returns a plain array whenever an instance is sliced or viewed. I > > cannot figure > > out how. > > Easy enough for slices: > class Dummy(np.ndarray): > def __new__(cls, array): > obj=array.view(cls) > return obj > def __getslice__(self, i, j): > return np.ndarray.__getslice__(self, i, j).view(np.ndarray) > > For views, the problem needs to be better described: why would you > want to get just a basic ndarray ? > Pierre, Dummy(np.ndarray) is designed to contain a 2d-array and 2 vectors, one for the x-Axis and the other for the y-Axis. I use this coordinates to select and order row and columns of the 2d-array. i.e if I sum two Dummys I may have to reorder the column of one term to be consistent with the other. After the rearrangement I want to perform some liner algebra with the matrices. So let's I want to compute the transpose, I dont want to keep track also of the changes in the x axis and y axis, at this stage I am happy to lose this information and not to have the overhead. (e.g, if i compute the inverse the axis start to have no meaning anymore.) Thanks, Jacopo From matthew.brett at gmail.com Thu Jul 23 16:33:53 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 23 Jul 2009 13:33:53 -0700 Subject: [Numpy-discussion] numpy subclass In-Reply-To: References: Message-ID: <1e2af89e0907231333r2209802dwf2ff04345e97c172@mail.gmail.com> Hi, > In short, I want to create a class Dummy, inherited from np.ndarray, which > returns a plain array whenever an instance is sliced or viewed. I cannot figure > out how. > class Dummy(np.ndarray): > > def __new__(cls, array): > ? obj=array.view(cls) > ? return obj > def __array_finalize__(self, obj): > ? #self=self.view(np.ndarray) > ? #self=np.asarray(self) I think the problem here is that self.view(np.ndarray) returns another object, which is of type ndarray, pointing to the data in self. The problem being that you have not changed the object pointed to by ``self``, you've only made the local variable ``self`` point to something different that will then be thrown away when your new ``self`` variable goes out of scope. What you want to do here, is to modify the class of ``self`` inplace; I don't know how to do that... Best, Matthew From matthew.brett at gmail.com Thu Jul 23 18:56:18 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 23 Jul 2009 15:56:18 -0700 Subject: [Numpy-discussion] Odd nosetest error in test_umath.py Message-ID: <1e2af89e0907231556o427ec5d4x22f27ffa05e45b21@mail.gmail.com> Hi, I just got an email about a broken buildbot build. Looking at the logs, I see this: ====================================================================== FAIL: Test bug in reduceat with structured arrays copied for speed. ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/lib/python-support/python2.5/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/numpybb/Buildbot/numpy/b14/numpy-install/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 818, in test_reduceat assert_array_almost_equal(h1, h2) File "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line 537, in assert_array_almost_equal File "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line 395, in assert_array_compare AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 701.1171875, 801.1171875, 1001.1171875, 7501.1171875], dtype=float32) y: array([ 700., 800., 1000., 7500.], dtype=float32) I'm posting because I saw this bug before on one of my own machines. It is an extremely strange error that seemed to occur in the following circumstances: 1) Running via nosetests 2) With one or more other tests in the same file as the test above, where the test was a TestCase class 3) With nose 0.10.3, 0.10.4, but not 0.11 The error given seemed to be reliable, but differed in the numbers given for different versions of the test file (but stayed the same for any given version when the test was repeated) Consider this example on my own machine: mb312 at fedora11:~$ nosetests sp/numpy/core/tests/test_umath.py ... (mismatch 100.0%) x: array([ 800., 900., 1100., 7600.], dtype=float32) y: array([ 700., 800., 1000., 7500.], dtype=float32) ---------------------------------------------------------------------- Ran 71 tests in 0.064s FAILED (errors=1, failures=1) (where the error is a SkipTest). Now, just running the failing test via nose: mb312 at fedora11:~$ nosetests sp/numpy/core/tests/test_umath.py:test_reduceat . ---------------------------------------------------------------------- Ran 1 test in 0.001s OK I'm flagging it in case y'all run into it, and in case someone knows what it means, and whether it is a problem. It does not occur, as I mentioned, for nose 0.11 Matthew From charlesr.harris at gmail.com Thu Jul 23 19:49:14 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 23 Jul 2009 17:49:14 -0600 Subject: [Numpy-discussion] Odd nosetest error in test_umath.py In-Reply-To: <1e2af89e0907231556o427ec5d4x22f27ffa05e45b21@mail.gmail.com> References: <1e2af89e0907231556o427ec5d4x22f27ffa05e45b21@mail.gmail.com> Message-ID: On Thu, Jul 23, 2009 at 4:56 PM, Matthew Brett wrote: > Hi, > > I just got an email about a broken buildbot build. Looking at the > logs, I see this: > > ====================================================================== > FAIL: Test bug in reduceat with structured arrays copied for speed. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/var/lib/python-support/python2.5/nose/case.py", line 182, in > runTest > self.test(*self.arg) > File > "/home/numpybb/Buildbot/numpy/b14/numpy-install/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", > line 818, in test_reduceat > assert_array_almost_equal(h1, h2) > File > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", > line 537, in assert_array_almost_equal > File > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", > line 395, in assert_array_compare > AssertionError: > Arrays are not almost equal > > (mismatch 100.0%) > x: array([ 701.1171875, 801.1171875, 1001.1171875, > 7501.1171875], dtype=float32) > y: array([ 700., 800., 1000., 7500.], dtype=float32) > > I'm posting because I saw this bug before on one of my own machines. > It is an extremely strange error that seemed to occur in the > following circumstances: > > 1) Running via nosetests > 2) With one or more other tests in the same file as the test above, > where the test was a TestCase class > 3) With nose 0.10.3, 0.10.4, but not 0.11 > > The error given seemed to be reliable, but differed in the numbers > given for different versions of the test file (but stayed the same for > any given version when the test was repeated) > > Consider this example on my own machine: > > mb312 at fedora11:~$ nosetests sp/numpy/core/tests/test_umath.py > ... > (mismatch 100.0%) > x: array([ 800., 900., 1100., 7600.], dtype=float32) > y: array([ 700., 800., 1000., 7500.], dtype=float32) > > ---------------------------------------------------------------------- > Ran 71 tests in 0.064s > > FAILED (errors=1, failures=1) > > (where the error is a SkipTest). > > Now, just running the failing test via nose: > > mb312 at fedora11:~$ nosetests > sp/numpy/core/tests/test_umath.py:test_reduceat > . > ---------------------------------------------------------------------- > Ran 1 test in 0.001s > > OK > It's been there ever since Travis added the test, but it is sporadic. It's probably a reference counting error somewhere but no one has tracked it down yet. There is a ticket open on it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 23 20:30:39 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 23 Jul 2009 17:30:39 -0700 Subject: [Numpy-discussion] Odd nosetest error in test_umath.py In-Reply-To: References: <1e2af89e0907231556o427ec5d4x22f27ffa05e45b21@mail.gmail.com> Message-ID: <1e2af89e0907231730i5168a036s98299cf1c92b0d83@mail.gmail.com> Hi, > It's been there ever since Travis added the test, but it is sporadic. It's > probably a reference counting error somewhere but no one has tracked it down > yet. There is a ticket open on it. As a demonstration that this is not fully predictable; here's me copying the test file, running it once to get the error, and then again in the same terminal and not getting the error: mb312 at fedora11:~/tmp$ cp ../sp/numpy/core/tests/test_umath.py diagnose_test.py mb312 at fedora11:~/tmp$ nosetests diagnose_test.py ....................E.................................................F ====================================================================== ERROR: diagnose_test.TestComplexFunctions.test_branch_cuts_failing ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/mb312/usr/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", line 167, in knownfailer raise KnownFailureTest, msg KnownFailureTest: These branch cuts are known to fail ====================================================================== FAIL: Test bug in reduceat with structured arrays copied for speed. ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/mb312/tmp/diagnose_test.py", line 730, in test_reduceat assert_array_almost_equal(h1, h2) File "/home/mb312/usr/local/lib64/python2.6/site-packages/numpy/testing/utils.py", line 537, in assert_array_almost_equal header='Arrays are not almost equal') File "/home/mb312/usr/local/lib64/python2.6/site-packages/numpy/testing/utils.py", line 395, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 7.07971090e+31, 7.07971090e+31, 7.07971090e+31, 7.07971090e+31], dtype=float32) y: array([ 700., 800., 1000., 7500.], dtype=float32) ---------------------------------------------------------------------- Ran 71 tests in 0.063s FAILED (errors=1, failures=1) mb312 at fedora11:~/tmp$ nosetests diagnose_test.py ....................E.................................................. ====================================================================== ERROR: diagnose_test.TestComplexFunctions.test_branch_cuts_failing ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/mb312/usr/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", line 167, in knownfailer raise KnownFailureTest, msg KnownFailureTest: These branch cuts are known to fail ---------------------------------------------------------------------- Ran 71 tests in 0.062s FAILED (errors=1) .... From david at ar.media.kyoto-u.ac.jp Thu Jul 23 22:54:55 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 24 Jul 2009 11:54:55 +0900 Subject: [Numpy-discussion] pdf for multivariate normal function? In-Reply-To: References: Message-ID: <4A69227F.50002@ar.media.kyoto-u.ac.jp> per freem wrote: > hi all, > > i'm trying to find the function for the pdf of a multivariate normal > pdf. i know that the function "multivariate_normal" can be used to > sample from the multivariate normal distribution, but i just want to > get the pdf for a given vector of means and a covariance matrix. is > there a function to do this? AFAIK, there is no such function in scipy. There is one in the em subpackage of the learn scikits. The file which implements it is self-contained, so you could use this in the meantime: http://projects.scipy.org/scikits/browser/trunk/learn/scikits/learn/machine/em/densities.py I wrote this code some time ago, so it is not optimal in various ways, but it should get the job done. cheers, David From scott.sinclair.za at gmail.com Fri Jul 24 01:41:19 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Fri, 24 Jul 2009 07:41:19 +0200 Subject: [Numpy-discussion] String to integer array of ASCII values In-Reply-To: <617398.84444.qm@web52107.mail.re2.yahoo.com> References: <617398.84444.qm@web52107.mail.re2.yahoo.com> Message-ID: <6a17e9ee0907232241y3cf784cbk6180f3ded7c56086@mail.gmail.com> > 2009/7/23 David Goldsmith : > > --- On Thu, 7/23/09, Peter wrote: > >> I should have guessed that one. Why isn't numpy.fromstring >> listed with the >> other entries in the "From existing data" section here? > >> This looks like a simple improvement to the >> documentation... > > Yup, that it is; corrected here: > > http://docs.scipy.org/numpy/docs/numpy-docs/reference/routines.array-creation.rst/ > > But not yet showing up at your link. :-( That's because "somebody" needs to make a doc patch from the Wiki, apply it to SVN (or a local copy of) Numpy, rebuild the docs and post them on the web :) Cheers, Scott From slaunger at gmail.com Fri Jul 24 03:45:36 2009 From: slaunger at gmail.com (Kim Hansen) Date: Fri, 24 Jul 2009 09:45:36 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit Win XP for accumulated file size above approx. 1 GB In-Reply-To: References: Message-ID: 2009/7/23 Charles R Harris : >> Maybe I am measuring memory usage wrong? > > Hmm, I don't know what you should be looking at in XP. Memmapped files are > sort of like virtual memory and exist in the address space even if they > aren't in physical memory.? When you address an element that isn't in > physical memory there is a page fault and the OS reads in the needed page > from disk. If you read through the file physical memory will probably fill > up because the OS will keep try to keep as many pages in physical memory as > possible in case they are referenced again. But I am not sure how windows > does it's memory accounting or how it is displayed, someone here more > familiar with windows may be able to tell you what to look for. Or you could > try running on a 64 bit system if there is one available. > > Chuck Yes, it is indeed my general experience with memmaps, that as you start to access them, there are bursts of high memory usage, and somehow as you indicate, there must be happening some allocation of addresses which then hit a high wall. I tried to write a small test scripts, which gradually created more an more Python mmap.mmaps (here in chunks of 100 MB, but size per mmap does not matter): import itertools import mmap import os files = [] mmaps = [] file_names= [] mmap_cap=0 bytes_per_mmap = 100 * 1024 ** 2 try: for i in itertools.count(1): file_name = "d:/%d.tst" % i file_names.append(file_name) f = open(file_name, "w+b") files.append(f) mm = mmap.mmap(f.fileno(), bytes_per_mmap) mmaps.append(mm) mmap_cap += bytes_per_mmap print "Created %d writeable mmaps containing %d MB" % (i, mmap_cap/(1024**2)) #Clean up finally: print "Removing mmaps..." for mm, f, file_name in zip(mmaps, files, file_names): mm.close() f.close() os.remove(file_name) print "Done..." Here is the output: Created 1 writeable mmaps containing 100 MB Created 2 writeable mmaps containing 200 MB Created 3 writeable mmaps containing 300 MB Created 4 writeable mmaps containing 400 MB Created 5 writeable mmaps containing 500 MB Created 6 writeable mmaps containing 600 MB Created 7 writeable mmaps containing 700 MB Created 8 writeable mmaps containing 800 MB Created 9 writeable mmaps containing 900 MB Created 10 writeable mmaps containing 1000 MB Created 11 writeable mmaps containing 1100 MB Created 12 writeable mmaps containing 1200 MB Created 13 writeable mmaps containing 1300 MB Created 14 writeable mmaps containing 1400 MB Created 15 writeable mmaps containing 1500 MB Created 16 writeable mmaps containing 1600 MB Created 17 writeable mmaps containing 1700 MB Created 18 writeable mmaps containing 1800 MB Removing mmaps... Done... Traceback (most recent call last): File "C:\svn-sandbox\research\scipy\scipy\src\com\terma\kha\mmaptest.py", line 16, in mm = mmap.mmap(f.fileno(), bytes_per_mmap) WindowsError: [Error 8] Not enough storage is available to process this command although there is 26 GB free storage on the drive. Such a <2 GB limit is not mentioned in the documentation for Python 2.5.4 - at least not in the mmap documentation, so I am surprised this is the case. I think I will make a post about it on python.org Unfortunately, I do not have a 64 bit system on which I can test this. Cheers, Kim From lciti at essex.ac.uk Fri Jul 24 05:29:34 2009 From: lciti at essex.ac.uk (Citi, Luca) Date: Fri, 24 Jul 2009 10:29:34 +0100 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB References: Message-ID: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> Hello! I have access to both a 32bit and a 64bit linux machine. I had to change your code (appended) because I got an error about not being able to create a mmap larger than the file. Here are the results... On the 32bit machine: lciti at xps2:~$ python /tmp/ppp.py Created 1 writeable mmaps containing 100 MB Created 2 writeable mmaps containing 200 MB Created 3 writeable mmaps containing 300 MB Created 4 writeable mmaps containing 400 MB Created 5 writeable mmaps containing 500 MB [......] Created 24 writeable mmaps containing 2400 MB Created 25 writeable mmaps containing 2500 MB Created 26 writeable mmaps containing 2600 MB Created 27 writeable mmaps containing 2700 MB Created 28 writeable mmaps containing 2800 MB Created 29 writeable mmaps containing 2900 MB Created 30 writeable mmaps containing 3000 MB Removing mmaps... Done... Traceback (most recent call last): File "/tmp/ppp.py", line 19, in mm = mmap.mmap(f.fileno(), 0) mmap.error: [Errno 12] Cannot allocate memory On the 64bit machine I can create 510 mmaps both with bytes_per_mmap at 100MiB and 1GiB: Created 1 writeable mmaps containing 1000 MB Created 2 writeable mmaps containing 2000 MB Created 3 writeable mmaps containing 3000 MB Created 4 writeable mmaps containing 4000 MB Created 5 writeable mmaps containing 5000 MB Created 6 writeable mmaps containing 6000 MB [......] Created 501 writeable mmaps containing 501000 MB Created 502 writeable mmaps containing 502000 MB Created 503 writeable mmaps containing 503000 MB Created 504 writeable mmaps containing 504000 MB Created 505 writeable mmaps containing 505000 MB Created 506 writeable mmaps containing 506000 MB Created 507 writeable mmaps containing 507000 MB Created 508 writeable mmaps containing 508000 MB Created 509 writeable mmaps containing 509000 MB Created 510 writeable mmaps containing 510000 MB Removing mmaps... Done... Traceback (most recent call last): File "/tmp/ppp.py", line 19, in mm = mmap.mmap(f.fileno(), 0) mmap.error: [Errno 24] Too many open files I do not even have 510GiB free in the disk. But I think that is because the ext3 filesystem allows sparse files. I think this shows that the maximum mapped space cannot be more than the maximum address space which is 2**64 for 64bit machines, 2GiB for windows32 and 3GiB for linux32. Under WindowsXP, you can try to increase it from 2GiB to 3GiB using the /3GB switch in the boot.ini Best, Luca ### CODE ### import itertools import mmap import os files = [] mmaps = [] file_names= [] mmap_cap=0 bytes_per_mmap = 100 * 1024 ** 2 try: for i in itertools.count(1): file_name = "/home/lciti/%d.tst" % i file_names.append(file_name) f = open(file_name, "w+b") files.append(f) f.seek(bytes_per_mmap) f.write('a') f.seek(0) mm = mmap.mmap(f.fileno(), 0) mmaps.append(mm) mmap_cap += bytes_per_mmap print "Created %d writeable mmaps containing %d MB" % (i,mmap_cap/(1024**2)) #Clean up finally: print "Removing mmaps..." for mm, f, file_name in zip(mmaps, files, file_names): mm.close() f.close() os.remove(file_name) print "Done..." From slaunger at gmail.com Fri Jul 24 06:39:01 2009 From: slaunger at gmail.com (Kim Hansen) Date: Fri, 24 Jul 2009 12:39:01 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> Message-ID: 2009/7/24 Citi, Luca : > Hello! > I have access to both a 32bit and a 64bit linux machine. > > I had to change your code (appended) because I got an error about > not being able to create a mmap larger than the file. > Here are the results... > > On the 32bit machine: > > lciti at xps2:~$ python /tmp/ppp.py > Created 1 writeable mmaps containing 100 MB > Created 2 writeable mmaps containing 200 MB > Created 3 writeable mmaps containing 300 MB > Created 4 writeable mmaps containing 400 MB > Created 5 writeable mmaps containing 500 MB > [......] > Created 24 writeable mmaps containing 2400 MB > Created 25 writeable mmaps containing 2500 MB > Created 26 writeable mmaps containing 2600 MB > Created 27 writeable mmaps containing 2700 MB > Created 28 writeable mmaps containing 2800 MB > Created 29 writeable mmaps containing 2900 MB > Created 30 writeable mmaps containing 3000 MB > Removing mmaps... > Done... > Traceback (most recent call last): > ?File "/tmp/ppp.py", line 19, in > ? ?mm = mmap.mmap(f.fileno(), 0) > mmap.error: [Errno 12] Cannot allocate memory > > > > > On the 64bit machine I can create 510 mmaps > both with bytes_per_mmap at 100MiB and 1GiB: > > Created 1 writeable mmaps containing 1000 MB > Created 2 writeable mmaps containing 2000 MB > Created 3 writeable mmaps containing 3000 MB > Created 4 writeable mmaps containing 4000 MB > Created 5 writeable mmaps containing 5000 MB > Created 6 writeable mmaps containing 6000 MB > [......] > Created 501 writeable mmaps containing 501000 MB > Created 502 writeable mmaps containing 502000 MB > Created 503 writeable mmaps containing 503000 MB > Created 504 writeable mmaps containing 504000 MB > Created 505 writeable mmaps containing 505000 MB > Created 506 writeable mmaps containing 506000 MB > Created 507 writeable mmaps containing 507000 MB > Created 508 writeable mmaps containing 508000 MB > Created 509 writeable mmaps containing 509000 MB > Created 510 writeable mmaps containing 510000 MB > Removing mmaps... > Done... > Traceback (most recent call last): > ?File "/tmp/ppp.py", line 19, in > ? ?mm = mmap.mmap(f.fileno(), 0) > mmap.error: [Errno 24] Too many open files > > > > I do not even have 510GiB free in the disk. But I > think that is because the ext3 filesystem allows > sparse files. > > I think this shows that the maximum mapped space > cannot be more than the maximum address space > which is 2**64 for 64bit machines, 2GiB for windows32 > and 3GiB for linux32. > > Under WindowsXP, you can try to increase it from 2GiB to > 3GiB using the /3GB switch in the boot.ini > > Best, > Luca > Hi Luca, thx for trying. Your test clearly shows that 32 bits imposes a severe limitation. I tried adding the /3GB switch to boot.ini as you suggested: multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect /3GB and rebooted the system. Unfortunately that did not change anything for me. I still hit a hard deck around 1.9 GB. Strange. Best wishes, Kim From david at ar.media.kyoto-u.ac.jp Fri Jul 24 06:24:54 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 24 Jul 2009 19:24:54 +0900 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> Message-ID: <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> Kim Hansen wrote: > 2009/7/24 Citi, Luca : > >> Hello! >> I have access to both a 32bit and a 64bit linux machine. >> >> I had to change your code (appended) because I got an error about >> not being able to create a mmap larger than the file. >> Here are the results... >> >> On the 32bit machine: >> >> lciti at xps2:~$ python /tmp/ppp.py >> Created 1 writeable mmaps containing 100 MB >> Created 2 writeable mmaps containing 200 MB >> Created 3 writeable mmaps containing 300 MB >> Created 4 writeable mmaps containing 400 MB >> Created 5 writeable mmaps containing 500 MB >> [......] >> Created 24 writeable mmaps containing 2400 MB >> Created 25 writeable mmaps containing 2500 MB >> Created 26 writeable mmaps containing 2600 MB >> Created 27 writeable mmaps containing 2700 MB >> Created 28 writeable mmaps containing 2800 MB >> Created 29 writeable mmaps containing 2900 MB >> Created 30 writeable mmaps containing 3000 MB >> Removing mmaps... >> Done... >> Traceback (most recent call last): >> File "/tmp/ppp.py", line 19, in >> mm = mmap.mmap(f.fileno(), 0) >> mmap.error: [Errno 12] Cannot allocate memory >> >> >> >> >> On the 64bit machine I can create 510 mmaps >> both with bytes_per_mmap at 100MiB and 1GiB: >> >> Created 1 writeable mmaps containing 1000 MB >> Created 2 writeable mmaps containing 2000 MB >> Created 3 writeable mmaps containing 3000 MB >> Created 4 writeable mmaps containing 4000 MB >> Created 5 writeable mmaps containing 5000 MB >> Created 6 writeable mmaps containing 6000 MB >> [......] >> Created 501 writeable mmaps containing 501000 MB >> Created 502 writeable mmaps containing 502000 MB >> Created 503 writeable mmaps containing 503000 MB >> Created 504 writeable mmaps containing 504000 MB >> Created 505 writeable mmaps containing 505000 MB >> Created 506 writeable mmaps containing 506000 MB >> Created 507 writeable mmaps containing 507000 MB >> Created 508 writeable mmaps containing 508000 MB >> Created 509 writeable mmaps containing 509000 MB >> Created 510 writeable mmaps containing 510000 MB >> Removing mmaps... >> Done... >> Traceback (most recent call last): >> File "/tmp/ppp.py", line 19, in >> mm = mmap.mmap(f.fileno(), 0) >> mmap.error: [Errno 24] Too many open files >> >> >> >> I do not even have 510GiB free in the disk. But I >> think that is because the ext3 filesystem allows >> sparse files. >> >> I think this shows that the maximum mapped space >> cannot be more than the maximum address space >> which is 2**64 for 64bit machines, 2GiB for windows32 >> and 3GiB for linux32. >> >> Under WindowsXP, you can try to increase it from 2GiB to >> 3GiB using the /3GB switch in the boot.ini >> >> Best, >> Luca >> >> > Hi Luca, thx for trying. Your test clearly shows that 32 bits imposes > a severe limitation. > > I tried adding the /3GB switch to boot.ini as you suggested: > multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP > Professional" /noexecute=optin /fastdetect /3GB > and rebooted the system. > > Unfortunately that did not change anything for me. I still hit a hard > deck around 1.9 GB. Strange. > The 3Gb thing only works for application specifically compiled for it: http://blogs.msdn.com/oldnewthing/archive/2004/08/12/213468.aspx I somewhat doubt python is built with this, but you could check this in python sources to be sure, cheers, David From slaunger at gmail.com Fri Jul 24 06:55:36 2009 From: slaunger at gmail.com (Kim Hansen) Date: Fri, 24 Jul 2009 12:55:36 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> Message-ID: >> >> I tried adding the /3GB switch to boot.ini as you suggested: >> multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP >> Professional" /noexecute=optin /fastdetect /3GB >> and rebooted the system. >> >> Unfortunately that did not change anything for me. I still hit a hard >> deck around 1.9 GB. Strange. >> > > The 3Gb thing only works for application specifically compiled for it: > > http://blogs.msdn.com/oldnewthing/archive/2004/08/12/213468.aspx > > I somewhat doubt python is built with this, but you could check this in > python sources to be sure, > > cheers, > > David Ahh, that explains it. Thank you for that enlightening link. Anyway would it not be worth mentioning in the memmap documentation that there is this 32 bit limitation, or is it so straightforwardly obvious (it was not for me) that his is the case? The reason it isn't obvious for me is because I can read and manipulate files >200 GB in Python with no problems (yes I process that large files), so I thought why should it not be capable of handling quite large memmaps as well... Cheers, Kim From david at ar.media.kyoto-u.ac.jp Fri Jul 24 06:52:51 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 24 Jul 2009 19:52:51 +0900 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> Message-ID: <4A699283.6010201@ar.media.kyoto-u.ac.jp> Kim Hansen wrote: >>> I tried adding the /3GB switch to boot.ini as you suggested: >>> multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP >>> Professional" /noexecute=optin /fastdetect /3GB >>> and rebooted the system. >>> >>> Unfortunately that did not change anything for me. I still hit a hard >>> deck around 1.9 GB. Strange. >>> >>> >> The 3Gb thing only works for application specifically compiled for it: >> >> http://blogs.msdn.com/oldnewthing/archive/2004/08/12/213468.aspx >> >> I somewhat doubt python is built with this, but you could check this in >> python sources to be sure, >> >> cheers, >> >> David >> > Ahh, that explains it. Thank you for that enlightening link. Anyway > would it not be worth mentioning in the memmap documentation that > there is this 32 bit limitation, or is it so straightforwardly obvious > (it was not for me) that his is the case? > Well, the questions has popped up a few times already, so I guess this is not so obvious :) 32 bits architecture fundamentally means that a pointer is 32 bits, so you can only address 2^32 different memory locations. The 2Gb instead of 4Gb is a consequence on how windows and linux kernels work. You can mmap a file which is bigger than 4Gb (as you can allocate more than 4Gb, at least in theory, on a 32 bits system), but you cannot 'see' more than 4Gb at the same time because the pointer is too small. Raymond Chen gives an example on windows: http://blogs.msdn.com/oldnewthing/archive/2004/08/10/211890.aspx I don't know if it is possible to do so in python, though. > The reason it isn't obvious for me is because I can read and > manipulate files >200 GB in Python with no problems (yes I process > that large files), so I thought why should it not be capable of > handling quite large memmaps as well... > Handling large files is no problem on 32 bits: it is just a matter of API (and kernel/fs support). You move the file location using a 64 bits integer and so on. Handling more than 4 Gb of memory at the same time is much more difficult. To address more than 4Gb, you would need a segmented architecture in your memory handling (with a first address for a segment, and a second address for the location within one segment). cheers, David From kwgoodman at gmail.com Sat Jul 25 11:24:23 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 25 Jul 2009 08:24:23 -0700 Subject: [Numpy-discussion] ValueError: cannot convert float to NaN to integer In-Reply-To: <3d375d730907221151g12ca2715q967bc3a3473b178d@mail.gmail.com> References: <3d375d730907221151g12ca2715q967bc3a3473b178d@mail.gmail.com> Message-ID: On Wed, Jul 22, 2009 at 11:51 AM, Robert Kern wrote: > On Wed, Jul 22, 2009 at 13:35, Keith Goodman wrote: >> On Ubuntu 9.04, python 2.6.2, numpy 1.2.1 this gives a ValueError: >> >> x = np.array([1,2,3]) >> x[0] = np.nan >> ValueError: cannot convert float to NaN to integer >> >> But on Debian squeeze, python 2.5.4, numpy 1.2.1 the assignment works >> (well, the float nan is convert to the int 0): >> >> x[0] = np.nan >> x >> ? array([0,1,2]) >> >> The numpy version numbers are the same. Is this a python issue? > > Probably. They rationalized a bunch of float behavior in Python 2.6, > and this may be a consequence of that. grep the Python sources for > that error message, or parts of it, to try to track down where it > comes from. The same error shows up when running the numpy unit tests with python 2.6 (but not with python 2.5): ====================================================================== ERROR: Tests the min/max functions with explicit outputs ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/dist-packages/numpy/ma/tests/test_core.py", line 699, in test_minmax_funcs_with_output result = npfunc(xm,axis=0,out=nout) File "/usr/lib/python2.6/dist-packages/numpy/core/fromnumeric.py", line 1569, in amin return amin(axis, out) File "/usr/lib/python2.6/dist-packages/numpy/ma/core.py", line 3119, in min np.putmask(out, newmask, np.nan) ValueError: cannot convert float NaN to integer From dsdale24 at gmail.com Sat Jul 25 20:33:58 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Sat, 25 Jul 2009 20:33:58 -0400 Subject: [Numpy-discussion] suggestion for generalizing numpy functions In-Reply-To: References: <9457e7c80907131509s2b4a4a60jfe7db7b64e7520e1@mail.gmail.com> <9457e7c80907201433i766eb9a9x47e6a449c10450d7@mail.gmail.com> Message-ID: On Thu, Jul 23, 2009 at 12:54 PM, Darren Dale wrote: > On Tue, Jul 21, 2009 at 10:11 AM, Darren Dale wrote: >> On Tue, Jul 21, 2009 at 7:44 AM, Darren Dale wrote: >>> 2009/7/20 St?fan van der Walt : >>>> Hi Chuck >>>> >>>> 2009/7/17 Charles R Harris : >>>>> PyObject*?PyTuple_GetItem(PyObject *p, Py_ssize_t pos) >>>>> Return value: Borrowed reference. >>>>> Return the object at position pos in the tuple pointed to by p. If pos is >>>>> out of bounds, return NULL and sets an IndexError exception. It's a borrowed >>>>> reference so you need to call Py_INCREF on it. I find this Python C-API >>>>> documentation useful. >>>> >>>> Have you had a look over the rest of the code? ?I think this would >>>> make a good addition. ?Travis mentioned Contexts for doing something >>>> similar, but I don't know enough about that concept to compare the >>>> two. >>> >>> I think contexts would be very different from what is already in >>> place. For now, it would be nice to make this one small improvement to >>> the existing ufunc infrastructure, and maybe consider contexts (which >>> I still don't understand) at a later time. I have improved the code >>> slightly and added a few tests, and will post a new patch later this >>> morning. I just need to add some documentation. >> >> Here is a better patch, which includes a few additional tests and adds >> some documentation. It also attempts to improve the docstring and >> sphinx docs for __array_wrap__, which may have been a little bit >> misleading. There is also some whitespace cleanup in a few places. >> Would someone please review my work and commit the patch if it is >> acceptable? Pierre or Travis, would either of you have a chance to >> look over the implementation and the documentation changes, since you >> two seem to be most familiar with ufuncs and subclassing ndarray? > > It looks like part of my patch has been clobbered by changes > introduced in svn 7184-7191. What else should I be doing so a patch > like this can be committed relatively quickly? Could I please obtain commit privileges so I can commit this feature to svn myself? Darren From ralf.gommers at googlemail.com Sun Jul 26 14:34:08 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 26 Jul 2009 14:34:08 -0400 Subject: [Numpy-discussion] fft issues Message-ID: Hi, I came across a few issues in the fft module: 1. fftshift and ifftshift do not accept an integer for the `axes` argument. I opened a ticket, #1182, and attached a patch with test that fixes this. 2. The functions refft, irefft, refft2, irefft2, refftn and irefftn have been deprecated for at least three years. Isn't it time to remove them? 3. For the hfft/ihfft functions, I am unsure what this means: "a signal whose spectrum has Hermitian symmetry". Can someone give a one-liner for checking this condition, and give an example where signals like this may occur? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jul 27 00:06:37 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 26 Jul 2009 22:06:37 -0600 Subject: [Numpy-discussion] Commit privileges for Darren Dale Message-ID: Hi All, In case it got buried in the thread, Darren is asking for commit privileges. I think it's a good idea. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d_l_goldsmith at yahoo.com Mon Jul 27 00:15:04 2009 From: d_l_goldsmith at yahoo.com (David Goldsmith) Date: Sun, 26 Jul 2009 21:15:04 -0700 (PDT) Subject: [Numpy-discussion] fft issues Message-ID: <153347.824.qm@web52110.mail.re2.yahoo.com> --- On Sun, 7/26/09, Ralf Gommers wrote: > 3. For the hfft/ihfft functions, I am unsure what this > means: "a signal whose spectrum has Hermitian > symmetry". Can someone give a one-liner for checking > this condition, and give an example where signals like this > may occur? I think I can handle this one too. DG From david at ar.media.kyoto-u.ac.jp Mon Jul 27 03:00:34 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 27 Jul 2009 16:00:34 +0900 Subject: [Numpy-discussion] Why does assert_array_almost_equal sometimes raise ValueError instead of AssertionError ? Message-ID: <4A6D5092.5040809@ar.media.kyoto-u.ac.jp> Hi, In some cases, some of the testing functions assert_array_* raise a ValueError instead of AssertionError: >>> np.testing.assert_array_almost_equal(np.array([1, 2, np.nan]), np.array([1, 2, 3])) # raises ValueError >>> np.testing.assert_array_almost_equal(np.array([1, 2, np.inf]), np.array([1, 2, 3])) # raises AssertionError This seems at least inconsistent - is there a rationale or should this be considered as a bug ? cheers, David From slaunger at gmail.com Mon Jul 27 06:11:42 2009 From: slaunger at gmail.com (Kim Hansen) Date: Mon, 27 Jul 2009 12:11:42 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: <4A699283.6010201@ar.media.kyoto-u.ac.jp> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> <4A699283.6010201@ar.media.kyoto-u.ac.jp> Message-ID: 2009/7/24 David Cournapeau : > > Well, the questions has popped up a few times already, so I guess this > is not so obvious :) 32 bits architecture fundamentally means that a > pointer is 32 bits, so you can only address 2^32 different memory > locations. The 2Gb instead of 4Gb is a consequence on how windows and > linux kernels work. You can mmap a file which is bigger than 4Gb (as you > can allocate more than 4Gb, at least in theory, on a 32 bits system), > but you cannot 'see' more than 4Gb at the same time because the pointer > is too small. > > Raymond Chen gives an example on windows: > > http://blogs.msdn.com/oldnewthing/archive/2004/08/10/211890.aspx > > I don't know if it is possible to do so in python, though. > >> The reason it isn't obvious for me is because I can read and >> manipulate files >200 GB in Python with no problems (yes I process >> that large files), so I thought why should it not be capable of >> handling quite large memmaps as well... >> > > Handling large files is no problem on 32 bits: it is just a matter of > API (and kernel/fs support). You move the file location using a 64 bits > integer and so on. Handling more than 4 Gb of memory at the same time is > much more difficult. To address more than 4Gb, you would need a > segmented architecture in your memory handling (with a first address for > a segment, and a second address for the location within one segment). > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > OK, I understand what you are saying. However, in my application it would really be nice to have the ability to "typecast" recarrays with an accumulated size in ecxess of 2GB onto files such that I could have the convenient slicing notation available for accessing the data. >From my (admittedly ignorant) point of view it seems like an implementation detail for me, that there is a problem with some intermediate memory address space. My typical use case would be to access and process the large filemapped, readonly recarray in chunks of up to 1,000,000 records 100 bytes each, or for instance pick every 1000th element of a specific field. That is data structures, which I can easily have in RAM while working at it. I think it would be cool to have an alternative (possible readonly) memmap implementation (filearray?), which is not just a wrapper around mmap.mmap (with its 32 bit address space limitation), but which (simply?) operates directly on the files with seek and read. I think that could be very usefull (well for me at least, that is). In my specific case, I will probably now proceed and make some poor mans wrapping convenience methods implementing just the specific featuires I need as I do not have the insight to subclass an ndarray myself and override the needed methods. In that manner I can go to >2GB still with low memory usage, but it will not be pretty. Kim From david at ar.media.kyoto-u.ac.jp Mon Jul 27 06:04:56 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 27 Jul 2009 19:04:56 +0900 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> <4A699283.6010201@ar.media.kyoto-u.ac.jp> Message-ID: <4A6D7BC8.9000400@ar.media.kyoto-u.ac.jp> Kim Hansen wrote: > >From my (admittedly ignorant) point of view it seems like an > implementation detail for me, that there is a problem with some > intermediate memory address space. > Yes, it is an implementation detail, but as is 32 vs 64 bits :) > My typical use case would be to access and process the large > filemapped, readonly recarray in chunks of up to 1,000,000 records 100 > bytes each, or for instance pick every 1000th element of a specific > field. That is data structures, which I can easily have in RAM while > working at it. > > I think it would be cool to have an alternative (possible readonly) > memmap implementation (filearray?), which is not just a wrapper around > mmap.mmap (with its 32 bit address space limitation), but which > (simply?) operates directly on the files with seek and read. I think > that could be very usefull (well for me at least, that is). In my > specific case, I will probably now proceed and make some poor mans > wrapping convenience methods implementing just the specific featuires > I need as I do not have the insight to subclass an ndarray myself and > override the needed methods. In that manner I can go to >2GB still > with low memory usage, but it will not be pretty. > I think it would be quite complicated. One fundamental "limitation" of numpy is that it views a contiguous chunk of memory. You can't have one numpy array which is the union of two memory blocks with a hole in between, so if you slice every 1000 items, the underlying memory of the array still needs to 'view' the whole thing. I think it is not possible to support what you want with one numpy array. I think the simple solution really is to go 64 bits, that's exactly the kind of things it is used for. If your machine is relatively recent, it supports 64 bits addressing. cheers, David From slaunger at gmail.com Mon Jul 27 06:37:53 2009 From: slaunger at gmail.com (Kim Hansen) Date: Mon, 27 Jul 2009 12:37:53 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: <4A6D7BC8.9000400@ar.media.kyoto-u.ac.jp> References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> <4A699283.6010201@ar.media.kyoto-u.ac.jp> <4A6D7BC8.9000400@ar.media.kyoto-u.ac.jp> Message-ID: > > I think it would be quite complicated. One fundamental "limitation" of > numpy is that it views a contiguous chunk of memory. You can't have one > numpy array which is the union of two memory blocks with a hole in > between, so if you slice every 1000 items, the underlying memory of the > array still needs to 'view' the whole thing. I think it is not possible > to support what you want with one numpy array. Yes, I see the problem in getting the same kind of reuse of objects using simple indexing. For my specific case, I will just allocate a new array as containing a copy of every 100th element and return this array. It will basically give me the same result as the original recarray is for read-only purposes only. This will be very simple implement for the specific cases I have > > I think the simple solution really is to go 64 bits, that's exactly the > kind of things it is used for. If your machine is relatively recent, it > supports 64 bits addressing. > The machine is new and shiny with loads of processing power and many TB of HDD storage. I am however bound to 32 bits Win XP OS as there are some other costum made third-party and very expensive applications running on that machine (which generate the large files I analyze), which can only run on 32 bits, oh well.... Cheers, Kim > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From seb.haase at gmail.com Mon Jul 27 06:44:51 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Mon, 27 Jul 2009 12:44:51 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> <4A699283.6010201@ar.media.kyoto-u.ac.jp> <4A6D7BC8.9000400@ar.media.kyoto-u.ac.jp> Message-ID: Is PyTables any option for you ? -- Sebastian Haase On Mon, Jul 27, 2009 at 12:37 PM, Kim Hansen wrote: >> >> I think it would be quite complicated. One fundamental "limitation" of >> numpy is that it views a contiguous chunk of memory. You can't have one >> numpy array which is the union of two memory blocks with a hole in >> between, so if you slice every 1000 items, the underlying memory of the >> array still needs to 'view' the whole thing. I think it is not possible >> to support what you want with one numpy array. > > Yes, I see the problem in getting the same kind of reuse of objects > using simple indexing. For my specific case, I will just allocate a > new array as containing a copy of every 100th element and return this > array. It will basically give me the same result as the original > recarray is for read-only purposes only. This will be very simple > implement for the specific cases I have > >> >> I think the simple solution really is to go 64 bits, that's exactly the >> kind of things it is used for. If your machine is relatively recent, it >> supports 64 bits addressing. >> > The machine is new and shiny with loads of processing power and many > TB of HDD storage. I am however bound to 32 bits Win XP OS as there > are some other costum made third-party and very expensive applications > running on that machine (which generate the large files I analyze), > which can only run on 32 bits, oh well.... > > Cheers, > > Kim > > >> cheers, >> >> David >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robince at gmail.com Mon Jul 27 06:46:08 2009 From: robince at gmail.com (Robin) Date: Mon, 27 Jul 2009 11:46:08 +0100 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> <4A699283.6010201@ar.media.kyoto-u.ac.jp> <4A6D7BC8.9000400@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Jul 27, 2009 at 11:37 AM, Kim Hansen wrote: > The machine is new and shiny with loads of processing power and many > TB of HDD storage. I am however bound to 32 bits Win XP OS as there > are some other costum made third-party and very expensive applications > running on that machine (which generate the large files I analyze), > which can only run on 32 bits, oh well.... You could think about using some kind of virtualisation - this is exactly the sort of situation where I find it really useful. You can run a 64 bit host OS, then have 32 bit XP as a 'guest' in VMware or Virtualbox or some other virtualisation software. With recent CPU's there is very little performance penalty (running 32bit on a 64bit host) and it can be very convenient (it is easy to map network drives between guest and host which perform very well since the 'network' is virtual) Cheers Robin From david at ar.media.kyoto-u.ac.jp Mon Jul 27 06:28:21 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 27 Jul 2009 19:28:21 +0900 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> <4A699283.6010201@ar.media.kyoto-u.ac.jp> <4A6D7BC8.9000400@ar.media.kyoto-u.ac.jp> Message-ID: <4A6D8145.2050107@ar.media.kyoto-u.ac.jp> Kim Hansen wrote: > The machine is new and shiny with loads of processing power and many > TB of HDD storage. I am however bound to 32 bits Win XP OS as there > are some other costum made third-party and very expensive applications > running on that machine (which generate the large files I analyze), > which can only run on 32 bits, oh well.... > Windows 64 bits can run 32 bits applications - very few applications are 64 bits to this day (For example, even the most recent visual studio (2008) do not run in 64 bits AFAIK). But scipy does not work on windows 64 bits ATM - although numpy does without problem if you build it by yourself. David From slaunger at gmail.com Mon Jul 27 07:18:04 2009 From: slaunger at gmail.com (Kim Hansen) Date: Mon, 27 Jul 2009 13:18:04 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> <4A699283.6010201@ar.media.kyoto-u.ac.jp> <4A6D7BC8.9000400@ar.media.kyoto-u.ac.jp> Message-ID: > > You could think about using some kind of virtualisation - this is > exactly the sort of situation where I find it really useful. > > You can run a 64 bit host OS, then have 32 bit XP as a 'guest' in > VMware or Virtualbox or some other virtualisation software. With > recent CPU's there is very little performance penalty (running 32bit > on a 64bit host) and it can be very convenient (it is easy to map > network drives between guest and host which perform very well since > the 'network' is virtual) > > Cheers > > Robin That is actually a very good idea. I had never thought of that myself. I will give it some consideration once I am sure I have exhausted options which do not require installing a new OS on the machine. Somewhat reluctant concerning risks, like, will the 32 bit drivers still work on the HW the propriatary 32 bit program is using, on a 64 bit host OS etc....also because I haven't tried using VMware etc - I've heard it should be quite easy though.... Cheers, Kim From slaunger at gmail.com Mon Jul 27 07:41:36 2009 From: slaunger at gmail.com (Kim Hansen) Date: Mon, 27 Jul 2009 13:41:36 +0200 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> <4A699283.6010201@ar.media.kyoto-u.ac.jp> <4A6D7BC8.9000400@ar.media.kyoto-u.ac.jp> Message-ID: 2009/7/27 Sebastian Haase : > Is PyTables any option for you ? > > -- > Sebastian Haase > That may indeed be something for me! I had heard the name before but I never realized exactly what it was. However, i have just seen their first tutorial video, and it seems like a very, very useful package and easy to use package, which could meet my needs. Thanks for telling me! I will install it and start toying around with it right away. Cheers, Kim From charlesr.harris at gmail.com Mon Jul 27 10:52:01 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Jul 2009 08:52:01 -0600 Subject: [Numpy-discussion] Why does assert_array_almost_equal sometimes raise ValueError instead of AssertionError ? In-Reply-To: <4A6D5092.5040809@ar.media.kyoto-u.ac.jp> References: <4A6D5092.5040809@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Jul 27, 2009 at 1:00 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi, > > In some cases, some of the testing functions assert_array_* raise a > ValueError instead of AssertionError: > > >>> np.testing.assert_array_almost_equal(np.array([1, 2, np.nan]), > np.array([1, 2, 3])) # raises ValueError > >>> np.testing.assert_array_almost_equal(np.array([1, 2, np.inf]), > np.array([1, 2, 3])) # raises AssertionError > > This seems at least inconsistent - is there a rationale or should this > be considered as a bug ? > I think that should be considered a bug. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Jul 27 11:41:52 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 27 Jul 2009 08:41:52 -0700 Subject: [Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB In-Reply-To: References: <3DA3B328CBC48B4EBB88484B8A5EA19106AF9E39@sernt14.essex.ac.uk> <4A698BF6.8010602@ar.media.kyoto-u.ac.jp> <4A699283.6010201@ar.media.kyoto-u.ac.jp> <4A6D7BC8.9000400@ar.media.kyoto-u.ac.jp> Message-ID: <4A6DCAC0.2020201@noaa.gov> Kim Hansen wrote: > Yes, I see the problem in getting the same kind of reuse of objects > using simple indexing. For my specific case, I will just allocate a > new array as containing a copy of every 100th element and return this > array. It will basically give me the same result as the original > recarray is for read-only purposes only. This will be very simple > implement for the specific cases I have It does sound like PyTables may do just what you want, but if not: You may be able to get this simple use case handles by writing your own simple memory mapped array implementation, perhaps as a subclass of ndarray, or perhaps from scratch. Python's "duck typing" could allow you to simply plop your implementation in where you need it. This does require that you really do only need a few simple features... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cournape at gmail.com Mon Jul 27 11:56:41 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 28 Jul 2009 00:56:41 +0900 Subject: [Numpy-discussion] Installed C libraries and using core math library Message-ID: <5b8d13220907270856t42b806ben29c47886cad9f702@mail.gmail.com> Hi, I have recently integrated my work on numpy.distutils to build so called "installable" C libraries, that is pure C libraries which can be installed and reused by 3rd parties. The general documentation is in the distutils section: http://docs.scipy.org/doc/numpy/reference/distutils.html#building-installable-c-libraries Example to reuse the math library is given in the core math library section in the C api chapter: http://docs.scipy.org/doc/numpy/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension cheers, David From Chris.Barker at noaa.gov Mon Jul 27 19:29:16 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 27 Jul 2009 16:29:16 -0700 Subject: [Numpy-discussion] String manipulation summary In-Reply-To: <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> References: <752529.59185.qm@web52110.mail.re2.yahoo.com> <4A662711.6080603@noaa.gov> <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> Message-ID: <4A6E384C.1030703@noaa.gov> Hi all, When I first saws this problem: reading in a fixed-width text file as numbers, it struck me that you really should be able to do it, and do it well, with numpy by slicing character arrays. I got carried away, and worked out a number of ways to do it. Lastly was a method inspired by a recent thread: "String to integer array of ASCII values", which did indeed inspire the fastest way. Here's what I have : # my naive first attempt: def line2array0(line, field_len): nums = [] i = 0 while i < len(line): nums.append(float(line[i:i+field_len])) i += field_len return np.array(nums) # list comprehension def line2array1(line, field_len): return np.array(map(float,[line[i*field_len:(i+1)*field_len] for i in range(len(line)/field_len)])) # convert to a tuple, then to an 'S1' array -- no real reason to do # this, as I figured out the next way. def line2array2(line, field_len): return np.array(tuple(line), dtype = 'S1').view(dtype='S%i'%field_len).astype(np.float) # convert directly to a string array, then break into fields. def line2array3(line, field_len): return np.array((line,)).view(dtype='S%i'%field_len).astype(np.float) # use dtype-'c' instead of 'S1' -- better. def line2array4(line, field_len): return np.array(line, dtype='c').view(dtype='S%i'%field_len).astype(np.float) # and the winner is: use fromstring to go straight to a 'c' array: def line2array5(line, field_len): return np.fromstring(line, dtype='c').view(dtype='S%i'%field_len).astype(np.float) Here are some timings: Timing with a 10 number string: List comp: 36.8073430061 convert to tuple: 57.9741871357 auto convert: 43.4103589058 char type: 46.0047719479 fromstring: 23.998103857 without float conversion: 11.4827179909 So list comprehension is pretty fast, but using fromstring, and then slicing is much better. The last one is the same thing, but without the convertion from strings to float, showing that that's a big chunk of time no matter how you slice it. Timing with a 100 number string: List comp: 163.281736135 convert to tuple: 333.081432104 auto convert: 138.934411049 char type: 279.897207975 fromstring: 121.395509005 without float conversion: 12.8342208862 Interesting -- I thought a longer string would give greater advantage to fromstring approach -- but I was wrong, now the time to parse strings into floats is really washing everything else out -- so it doesn't matter much how you do it, though I'd go with either list comprehension (which is what I think is used in np.genfromtxt), or the fromstring method, which I kind of like 'cause it's numpy. test and timing code attached. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: test.py Type: application/x-python Size: 3584 bytes Desc: not available URL: From charlesr.harris at gmail.com Mon Jul 27 19:44:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Jul 2009 17:44:39 -0600 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot Message-ID: ====================================================================== ERROR: test_nan_array (test_utils.TestApproxEqual) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", line 236, in test_nan_array self._assert_func(anan, anan) File "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line 433, in assert_approx_equal ValueError: math domain error ====================================================================== ERROR: test_nan_items (test_utils.TestApproxEqual) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", line 248, in test_nan_items self._assert_func(anan, anan) File "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line 433, in assert_approx_equal ValueError: math domain error ---------------------------------------------------------------------- Ran 2177 tests in 30.179s I expect there is a problem with the FreeBSD math libraries. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Mon Jul 27 19:47:28 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 27 Jul 2009 19:47:28 -0400 Subject: [Numpy-discussion] String manipulation summary In-Reply-To: <4A6E384C.1030703@noaa.gov> References: <752529.59185.qm@web52110.mail.re2.yahoo.com> <4A662711.6080603@noaa.gov> <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> <4A6E384C.1030703@noaa.gov> Message-ID: <7f014ea60907271647w56be6175ga61fd92fdb73eb14@mail.gmail.com> what machine spec are you using? Using your last function line2array5 WITH float conversion, i get the following timing on a mobile quad core extreme: In [24]: a = np.arange(100).astype(str).tostring() In [25]: a Out[25]: '0123456789111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999' In [26]: %timeit line2array(a, 1) 10000 loops, best of 3: 37.1 ?s per loop In [27]: a = np.arange(1000).astype(str).tostring() In [28]: %timeit line2array(a, 10) 10000 loops, best of 3: 45.2 ?s per loop Cheers, Chris On Mon, Jul 27, 2009 at 7:29 PM, Christopher Barker wrote: > Hi all, > > When I first saws this problem: reading in a fixed-width text file as > numbers, it struck me that you really should be able to do it, and do it > well, with numpy by slicing character arrays. > > I got carried away, and worked out a number of ways to do it. Lastly was a > method inspired by a recent thread: "String to integer array of ASCII > values", which did indeed inspire the fastest way. Here's what I have : > > # my naive first attempt: > def line2array0(line, field_len): > ? ?nums = [] > ? ?i = 0 > ? ?while i < len(line): > ? ? ? ?nums.append(float(line[i:i+field_len])) > ? ? ? ?i += field_len > ? ?return np.array(nums) > > # list comprehension > def line2array1(line, field_len): > ? ?return np.array(map(float,[line[i*field_len:(i+1)*field_len] for i in > range(len(line)/field_len)])) > > # convert to a tuple, then to an 'S1' array -- no real reason to do > # this, as I figured out the next way. > def line2array2(line, field_len): > ? ?return np.array(tuple(line), dtype = > 'S1').view(dtype='S%i'%field_len).astype(np.float) > > # convert directly to a string array, then break into fields. > def line2array3(line, field_len): > ? ?return np.array((line,)).view(dtype='S%i'%field_len).astype(np.float) > > # use dtype-'c' instead of 'S1' -- better. > def line2array4(line, field_len): > ? ?return np.array(line, > dtype='c').view(dtype='S%i'%field_len).astype(np.float) > > # and the winner is: use fromstring to go straight to a 'c' array: > def line2array5(line, field_len): > ? ?return np.fromstring(line, > dtype='c').view(dtype='S%i'%field_len).astype(np.float) > > Here are some timings: > > Timing with a 10 number string: > List comp: 36.8073430061 > convert to tuple: 57.9741871357 > auto convert: 43.4103589058 > char type: 46.0047719479 > fromstring: 23.998103857 > without float conversion: 11.4827179909 > > So list comprehension is pretty fast, but using fromstring, and then slicing > is much better. The last one is the same thing, but without the convertion > from strings to float, showing that that's a big chunk of time no matter how > you slice it. > > Timing with a 100 number string: > List comp: 163.281736135 > convert to tuple: 333.081432104 > auto convert: 138.934411049 > char type: 279.897207975 > fromstring: 121.395509005 > without float conversion: 12.8342208862 > > > Interesting -- I thought a longer string would give greater advantage to > fromstring approach -- but I was wrong, now the time to parse strings into > floats is really washing everything else out -- so it doesn't matter much > how you do it, though I'd go with either list comprehension (which is what I > think is used in np.genfromtxt), or the fromstring method, which I kind of > like 'cause it's numpy. > > test and timing code attached. > > -Chris > > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From Chris.Barker at noaa.gov Mon Jul 27 20:00:38 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 27 Jul 2009 17:00:38 -0700 Subject: [Numpy-discussion] String manipulation summary In-Reply-To: <7f014ea60907271647w56be6175ga61fd92fdb73eb14@mail.gmail.com> References: <752529.59185.qm@web52110.mail.re2.yahoo.com> <4A662711.6080603@noaa.gov> <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> <4A6E384C.1030703@noaa.gov> <7f014ea60907271647w56be6175ga61fd92fdb73eb14@mail.gmail.com> Message-ID: <4A6E3FA6.9090503@noaa.gov> Chris Colbert wrote: > what machine spec are you using? Dual 2Ghz PPC OS-X 10.4. Python2.5, numpy 1.3.0rc1 (hmm -- I should upgrade that!) > Using your last function line2array5 WITH float conversion, i get the > following timing on a mobile quad core extreme: > > In [24]: a = np.arange(100).astype(str).tostring() > > In [25]: a > Out[25]: '0123456789111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999' > > In [26]: %timeit line2array(a, 1) > 10000 loops, best of 3: 37.1 ?s per loop > > In [27]: a = np.arange(1000).astype(str).tostring() > > In [28]: %timeit line2array(a, 10) > 10000 loops, best of 3: 45.2 ?s per loop and I get, timing it the same way (I forgot about ipython's timeit!): In [83]: %timeit line2array5(a, 1) 10000 loops, best of 3: 125 ?s per loop In [84]: a = np.arange(1000).astype(str).tostring() In [85]: %timeit line2array5(a, 1) 1000 loops, best of 3: 1.09 ms per loop similar to yours, but longer with an older machine. I wonder why my other timings came out the way they did? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cournape at gmail.com Mon Jul 27 23:37:04 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 28 Jul 2009 12:37:04 +0900 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: References: Message-ID: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> On Tue, Jul 28, 2009 at 8:44 AM, Charles R Harris wrote: > ERROR: test_nan_items (test_utils.TestApproxEqual) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", > line 248, in test_nan_items > > self._assert_func(anan, anan) > File > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line > 433, in assert_approx_equal > ValueError: math domain error > > ---------------------------------------------------------------------- > > Ran 2177 tests in 30.179s > > > I expect there is a problem with the FreeBSD math libraries. Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) Reading the code for assert_approx_equal, using log/pow functions on nan should produce this error. I will work on a fix, cheers, David From martyfuhry at gmail.com Tue Jul 28 00:03:29 2009 From: martyfuhry at gmail.com (Marty Fuhry) Date: Tue, 28 Jul 2009 00:03:29 -0400 Subject: [Numpy-discussion] datetime Parsing Functions Message-ID: I'm a Summer of Code student working on the datetime implementation. The past few weeks I've been writing code to parse between dates and longs with a frequency. I'd like some feedback on my code before it gets integrated into the NumPy datetime branch: Clone URL: git://github.com/martyfuhry/npy_datetime.git Blog: http://fuhrysoc.blogspot.com The main file in question is in the parsing directory called c_parse.c. I have been working on the setitem and getitem routines for datetime64 objects (current functions in c_parse.c are long_to_date(time / string) and date(time / string)_to_long). The frequencies in the NEP are all supported, but I haven't yet implemented derived units for custom frequencies. The tests (located in the parsing/tests directory) are thorough, but I'd appreciate feedback on those, too. Is there anything I should do differently? Are there sections that need more documentation? Am I forgetting anything? Thanks in advance for any help. -Marty Fuhry From charlesr.harris at gmail.com Tue Jul 28 00:22:02 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Jul 2009 22:22:02 -0600 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> Message-ID: On Mon, Jul 27, 2009 at 9:37 PM, David Cournapeau wrote: > On Tue, Jul 28, 2009 at 8:44 AM, Charles R > Harris wrote: > > > ERROR: test_nan_items (test_utils.TestApproxEqual) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > > "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", > > line 248, in test_nan_items > > > > self._assert_func(anan, anan) > > File > > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", > line > > 433, in assert_approx_equal > > ValueError: math domain error > > > > ---------------------------------------------------------------------- > > > > Ran 2177 tests in 30.179s > > > > > > I expect there is a problem with the FreeBSD math libraries. > > Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) > Reading the code for assert_approx_equal, using log/pow functions > nan should produce this error. I will work on a fix, > Yeah, the function is kind of a mess. I think most of the math can be avoided. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jul 28 00:16:22 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 28 Jul 2009 13:16:22 +0900 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> Message-ID: <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Mon, Jul 27, 2009 at 9:37 PM, David Cournapeau > wrote: > > On Tue, Jul 28, 2009 at 8:44 AM, Charles R > Harris > wrote: > > > ERROR: test_nan_items (test_utils.TestApproxEqual) > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > > "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", > > line 248, in test_nan_items > > > > self._assert_func(anan, anan) > > File > > > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", > line > > 433, in assert_approx_equal > > ValueError: math domain error > > > > > ---------------------------------------------------------------------- > > > > Ran 2177 tests in 30.179s > > > > > > I expect there is a problem with the FreeBSD math libraries. > > Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) > Reading the code for assert_approx_equal, using log/pow functions > nan should produce this error. I will work on a fix, > > > Yeah, the function is kind of a mess. Yes - I am afraid I have added to the mess with my nan/inf handling :( I tried reimplementing from a clean state, but quite a few unit tests depend on some 'obscure' features. For example, do we really want assert_array_equal(np.array([1, 1]), 1) to be true ? I understand it is handy, but it can hide bugs, and it makes writing the function more complicated (as you can't check that both arrays have the same shape, and covering every corner case with handling scalar/scalar arrays/arrays is difficult). I am also concerned with speed issues (they are likely to be a bottleneck in many unit tests). > I think most of the math can be avoided. Do you know of a way to avoid log/pow for approx equal ? cheers, David From charlesr.harris at gmail.com Tue Jul 28 00:58:30 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Jul 2009 22:58:30 -0600 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Jul 27, 2009 at 10:16 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > > On Mon, Jul 27, 2009 at 9:37 PM, David Cournapeau > > wrote: > > > > On Tue, Jul 28, 2009 at 8:44 AM, Charles R > > Harris > > wrote: > > > > > ERROR: test_nan_items (test_utils.TestApproxEqual) > > > > > > ---------------------------------------------------------------------- > > > Traceback (most recent call last): > > > File > > > > > > "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", > > > line 248, in test_nan_items > > > > > > self._assert_func(anan, anan) > > > File > > > > > > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", > > line > > > 433, in assert_approx_equal > > > ValueError: math domain error > > > > > > > > > ---------------------------------------------------------------------- > > > > > > Ran 2177 tests in 30.179s > > > > > > > > > I expect there is a problem with the FreeBSD math libraries. > > > > Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) > > Reading the code for assert_approx_equal, using log/pow functions > > nan should produce this error. I will work on a fix, > > > > > > Yeah, the function is kind of a mess. > > Yes - I am afraid I have added to the mess with my nan/inf handling :( I > tried reimplementing from a clean state, but quite a few unit tests > depend on some 'obscure' features. For example, do we really want > assert_array_equal(np.array([1, 1]), 1) to be true ? I understand it is > handy, but it can hide bugs, and it makes writing the function more > complicated (as you can't check that both arrays have the same shape, > and covering every corner case with handling scalar/scalar arrays/arrays > is difficult). > > I am also concerned with speed issues (they are likely to be a > bottleneck in many unit tests). > > > I think most of the math can be avoided. > > Do you know of a way to avoid log/pow for approx equal ? > I'd just look at the difference and see if it exceeded some fraction of the expected value. There is the problem of zero, which could be handled in the usual way as diff < abserr + relerr. I think abserr would need to be a new keyword with a default value. Since the relerr is specified by digits you probably need 1/10**digits, but there should be no problem with that. Really, only a multiplication should be needed to determine the precision to x digits. Nan and +/- inf need to be special cased but I haven't thought about that. I agree that the broadcasting is both a convenience and a trap here, but its probably a bit late to change that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jul 28 00:48:27 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 28 Jul 2009 13:48:27 +0900 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> Message-ID: <4A6E831B.1020200@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > I'd just look at the difference and see if it exceeded some fraction > of the expected value. There is the problem of zero, which could be > handled in the usual way as diff < abserr + relerr. I think abserr > would need to be a new keyword with a default value. Since the relerr > is specified by digits you probably need 1/10**digits, but there > should be no problem with that. Really, only a multiplication should > be needed to determine the precision to x digits. Ok, I will have a shot at this. > Nan and +/- inf need to be special cased but I haven't thought about that. They are a constant source of frustration. It is like it defies my brain, and I end up doing very stupid things after working for too long with nan and inf, especially when complex numbers are involved :) > > I agree that the broadcasting is both a convenience and a trap here, > but its probably a bit late to change that. I thought about having an additional assert_exactly_equal: both arguments will need to have the same shape and type (dtype for arrays), negative zero and positive zero will be different as well. I also have another function which may be useful in one of my scipy branch for scipy.special: having the max error in term of ULPS. cheers, David From charlesr.harris at gmail.com Tue Jul 28 01:51:02 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Jul 2009 23:51:02 -0600 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: <4A6E831B.1020200@ar.media.kyoto-u.ac.jp> References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> <4A6E831B.1020200@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Jul 27, 2009 at 10:48 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > I'd just look at the difference and see if it exceeded some fraction > > of the expected value. There is the problem of zero, which could be > > handled in the usual way as diff < abserr + relerr. I think abserr > > would need to be a new keyword with a default value. Since the relerr > > is specified by digits you probably need 1/10**digits, but there > > should be no problem with that. Really, only a multiplication should > > be needed to determine the precision to x digits. > > Ok, I will have a shot at this. > > > Nan and +/- inf need to be special cased but I haven't thought about > that. > > They are a constant source of frustration. It is like it defies my > brain, and I end up doing very stupid things after working for too long > with nan and inf, especially when complex numbers are involved :) > Yeah, they leave me boggled too. O_o That's why I haven't thought about them... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Tue Jul 28 02:44:06 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 28 Jul 2009 08:44:06 +0200 Subject: [Numpy-discussion] DumbArray, LoadArray, Numeric -- numpy Message-ID: Hi all, how can I import arrays in numpy which are stored by DumpArray in the old Numeric package ? Nils From nwagner at iam.uni-stuttgart.de Tue Jul 28 02:47:40 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 28 Jul 2009 08:47:40 +0200 Subject: [Numpy-discussion] New test failures Message-ID: >>> numpy.__version__ '1.4.0.dev7270' Python 2.5.1 on 64-bit box ====================================================================== FAIL: test_umath.TestComplexFunctions.test_precisions_consistent ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 545, in test_precisions_consistent assert_almost_equal(fcf, fcd, decimal=6, err_msg='fch-fcd %s'%f) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 383, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: (0.666239+1.06128j) DESIRED: (0.666239432493+1.06127506191j) ====================================================================== FAIL: test_csingle (test_linalg.TestEig) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 39, in test_csingle self.do(a, b) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 94, in do assert_almost_equal(dot(a, evectors), multiply(evectors, evalues)) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 383, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [[-0.28785625-0.21230127j 2.13664055+3.62986112j] [ 0.20296739+0.16528448j 4.73750353+6.42351294j]] DESIRED: [[-0.28785625-0.21230127j 2.13664031+3.62986112j] [ 0.20296741+0.16528448j 4.73750353+6.42351341j]] ====================================================================== FAIL: test_csingle (test_linalg.TestPinv) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 39, in test_csingle self.do(a, b) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 124, in do assert_almost_equal(dot(a, a_ginv), identity(asarray(a).shape[0])) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 383, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [[ 1.00000024e+00 +2.38418579e-07j -5.96046448e-08 +0.00000000e+00j] [ 5.36441803e-07 +3.57627869e-07j 9.99999821e-01 +0.00000000e+00j]] DESIRED: [[ 1. 0.] [ 0. 1.]] ====================================================================== FAIL: test_csingle (test_linalg.TestSVD) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 39, in test_csingle self.do(a, b) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 100, in do assert_almost_equal(a, dot(multiply(u, s), vt)) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 383, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [[ 1.+2.j 2.+3.j] [ 3.+4.j 4.+5.j]] DESIRED: [[ 1.00000012+2.00000024j 2.00000024+3.00000024j] [ 3.00000048+4.00000048j 4.00000048+5.j ]] ---------------------------------------------------------------------- Ran 2179 tests in 67.754s FAILED (KNOWNFAIL=1, failures=4) From david at ar.media.kyoto-u.ac.jp Tue Jul 28 02:58:39 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 28 Jul 2009 15:58:39 +0900 Subject: [Numpy-discussion] New test failures In-Reply-To: References: Message-ID: <4A6EA19F.50300@ar.media.kyoto-u.ac.jp> Hi Nils, Nils Wagner wrote: >>>> numpy.__version__ >>>> > '1.4.0.dev7270' > Python 2.5.1 on 64-bit box > > > AssertionError: Items are not equal: > ACTUAL: [[-0.28785625-0.21230127j 2.13664055+3.62986112j] > [ 0.20296739+0.16528448j 4.73750353+6.42351294j]] > DESIRED: [[-0.28785625-0.21230127j > 2.13664031+3.62986112j] > [ 0.20296741+0.16528448j 4.73750353+6.42351341j]] I am working on it - I think some of those errors are buggy unit tests, though - for a single precision complex number, actual and desired only differ in on ULPS in the above case, for example. cheers, David From seb.haase at gmail.com Tue Jul 28 08:15:18 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 28 Jul 2009 14:15:18 +0200 Subject: [Numpy-discussion] distutils -- compiling extension twice, two sets of compiler settings Message-ID: Hi, maybe someone here could help me by using distutils: my problem is that I have some C++ files that I need to get compiled twice, first to make a "singleFloat"-module using gcc -DUSE_DOUBLE=0 and then to make the "doubleFloat"-module using gcc -DUSE_DOUBLE=1. Apparently distutils recognises (like "make") that the object files are already there and skips the gcc for the second run. My setup.py so far is something like this: ext_ccg_s = Extension('_CCG_smodule', [ 'Priithon_25_C/ccg/CCG_caller.cpp', 'Priithon_25_C/ccg/ccg_funct.cpp', 'Priithon_25_C/ccg/ccg.cpp', 'Priithon_25_C/ccg/CCG_s_wrap.cxx', ], define_macros=[('USE_DOUBLE', '0'),], ) ext_ccg_d = Extension('_CCG_dmodule', [ 'Priithon_25_C/ccg/CCG_caller.cpp', 'Priithon_25_C/ccg/ccg_funct.cpp', 'Priithon_25_C/ccg/ccg.cpp', 'Priithon_25_C/ccg/CCG_d_wrap.cxx', ], define_macros=[('USE_DOUBLE', '1'),], ) setup(... ext_package='Priithon_bin', ext_modules=[ext_ccg_s, ext_ccg_d], ) Thanks for any help, -- Sebastian Haase From dsdale24 at gmail.com Tue Jul 28 08:47:42 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 28 Jul 2009 08:47:42 -0400 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: References: Message-ID: On Mon, Jul 27, 2009 at 12:06 AM, Charles R Harris wrote: > Hi All, > > In case it got buried in the thread, Darren is asking for commit privileges. > I think it's a good idea. Thank you for saying so. What can I do to help move this to the point where I can actually start committing? I would like to get my array_prepare patch into svn soon so I can bundle a new beta of Quantities in time for scipy-09. I'm going to be on vacation July 31-August 9, should I wait until I get back before checking it in (assuming I'm granted commit rights)? Darren From stefan at sun.ac.za Tue Jul 28 08:57:39 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 28 Jul 2009 14:57:39 +0200 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: References: Message-ID: <9457e7c80907280557j22474d25w592b020be5cab53c@mail.gmail.com> 2009/7/27 Charles R Harris : > In case it got buried in the thread, Darren is asking for commit privileges. > I think it's a good idea. I agree. Unfortunately, I can't modify the svn-password file due to permissions, so I'll CC Peter Wang who has root access. Peter: we'd like to give SVN access to Darren Dale. Could you assist? Regards St?fan From charlesr.harris at gmail.com Tue Jul 28 10:54:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Jul 2009 08:54:39 -0600 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: References: Message-ID: On Tue, Jul 28, 2009 at 6:47 AM, Darren Dale wrote: > On Mon, Jul 27, 2009 at 12:06 AM, Charles R > Harris wrote: > > Hi All, > > > > In case it got buried in the thread, Darren is asking for commit > privileges. > > I think it's a good idea. > > Thank you for saying so. What can I do to help move this to the point > where I can actually start committing? I would like to get my > array_prepare patch into svn soon so I can bundle a new beta of > Quantities in time for scipy-09. > > I'm going to be on vacation July 31-August 9, should I wait until I > get back before checking it in (assuming I'm granted commit rights)? > I think you are the best judge of the timing. The only reason I can see for delay is if a problem turns up and you aren't there to fix it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Jul 28 10:59:17 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 28 Jul 2009 23:59:17 +0900 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: References: Message-ID: <5b8d13220907280759g7dd4ba14n19f02197c8915e7f@mail.gmail.com> On Tue, Jul 28, 2009 at 9:47 PM, Darren Dale wrote: > On Mon, Jul 27, 2009 at 12:06 AM, Charles R > Harris wrote: >> Hi All, >> >> In case it got buried in the thread, Darren is asking for commit privileges. >> I think it's a good idea. > > Thank you for saying so. What can I do to help move this to the point > where I can actually start committing? I would like to get my > array_prepare patch into svn soon so I can bundle a new beta of > Quantities in time for scipy-09. > > I'm going to be on vacation July 31-August 9, should I wait until I > get back before checking it in (assuming I'm granted commit rights)? Why not putting your code under a svn branch or a bzr/git/whatever import of the trunk, take care of it, and then committing it after you come back ? I am unfortunately not in position to comment on your code, I am not familiar on this part of numpy, but I would like someone else to take 'responsibility' that your code is OK if possible. cheers, David From dsdale24 at gmail.com Tue Jul 28 11:24:42 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 28 Jul 2009 11:24:42 -0400 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: <5b8d13220907280759g7dd4ba14n19f02197c8915e7f@mail.gmail.com> References: <5b8d13220907280759g7dd4ba14n19f02197c8915e7f@mail.gmail.com> Message-ID: On Tue, Jul 28, 2009 at 10:59 AM, David Cournapeau wrote: > On Tue, Jul 28, 2009 at 9:47 PM, Darren Dale wrote: >> On Mon, Jul 27, 2009 at 12:06 AM, Charles R >> Harris wrote: >>> Hi All, >>> >>> In case it got buried in the thread, Darren is asking for commit privileges. >>> I think it's a good idea. >> >> Thank you for saying so. What can I do to help move this to the point >> where I can actually start committing? I would like to get my >> array_prepare patch into svn soon so I can bundle a new beta of >> Quantities in time for scipy-09. >> >> I'm going to be on vacation July 31-August 9, should I wait until I >> get back before checking it in (assuming I'm granted commit rights)? > > Why not putting your code under a svn branch or a bzr/git/whatever > import of the trunk, take care of it, and then committing it after you > come back ? > > I am unfortunately not in position to comment on your code, I am not > familiar on this part of numpy, but I would like someone else to take > 'responsibility' that your code is OK if possible. I would also feel more comfortable if someone knowledgeable would have a look, since I don't have a lot of experience with the c api. But I'm having difficulty soliciting a response to my requests for review. Darren From daniel.wheeler2 at gmail.com Tue Jul 28 11:38:47 2009 From: daniel.wheeler2 at gmail.com (Daniel Wheeler) Date: Tue, 28 Jul 2009 11:38:47 -0400 Subject: [Numpy-discussion] installation issue with a non-standard location of a python build Message-ID: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> Hi, I'm having an installation issue building numpy 1.3.0 using python 2.6.2 which has been built in my local area in a non-standard place and was configured with --prefix=/users/wd15/usr/i686/4.0. When I run numpy's config ("python setup.py config") everything seems to be fine. However, when I run "python setup.py build", it gives the following error: $ python --version Python 2.6.2 $ uname -a Linux xxx.xxx.xxx 2.6.18-4-k7 #1 SMP Mon Mar 26 17:57:15 UTC 2007 i686 GNU/Linux $ python setup.py build Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 ... File "numpy/core/setup.py", line 186, in check_types "Cannot compiler 'Python.h'. Perhaps you need to "\ SystemError: Cannot compiler 'Python.h'. Perhaps you need to install python-dev|python-devel. $ Now, as I compiled python from source, "Python.h" is sitting neatly in "/users/wd15/usr/i686/4.0/include/python2.6". The output from build (not included above) even claims that the compile options point there: C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/users/wd15/usr/i686/4.0/include/python2.6 -c' so the build seems to be aware of the include location. Any ideas? Cheers -- Daniel Wheeler From cournape at gmail.com Tue Jul 28 12:02:01 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Jul 2009 01:02:01 +0900 Subject: [Numpy-discussion] installation issue with a non-standard location of a python build In-Reply-To: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> References: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> Message-ID: <5b8d13220907280902s18c5aec1x8665a7292f1312b0@mail.gmail.com> On Wed, Jul 29, 2009 at 12:38 AM, Daniel Wheeler wrote: > Hi, > > I'm having an installation issue building numpy 1.3.0 using python > 2.6.2 which has been built in my local area in a non-standard place > and was configured with --prefix=/users/wd15/usr/i686/4.0. When I run > numpy's config ("python setup.py config") everything seems to be fine. > However, when I run "python setup.py build", it gives the following > error: > Can you post the full output of python setup.py build ? cheers, David From Chris.Barker at noaa.gov Tue Jul 28 12:38:34 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 28 Jul 2009 09:38:34 -0700 Subject: [Numpy-discussion] Naming of OS-X binaries Message-ID: <4A6F298A.803@noaa.gov> Hi folks, I just went to get the latest stable build, and found that there was this for Python 2.5: numpy-1.3.0-py2.5-macosx10.5.dmg However, as far as I can tell, that is really built for the python.org framework build of python, which works on 10.3.9 and above, not just 10.5. I do know that it works on a 10.4 PPC machine. In fact, if I see "10.5" in the name, I'm likely to assume that it is built for the Apple-supplied python2.5, which came with 10.5 It really should be re-named. Maybe something like: numpy-1.3.0-py2.5-macosx10-py.org-univ.dmg or numpy-1.3.0-py2.5-macosx10.3.dmg If you want to be compact. As long as someone is taking the trouble to build a binary, people might as well be able to know they can use it! I know, the world of multiple builds for OS-X is still a mess. SIGH. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From daniel.wheeler2 at gmail.com Tue Jul 28 13:38:44 2009 From: daniel.wheeler2 at gmail.com (Daniel Wheeler) Date: Tue, 28 Jul 2009 13:38:44 -0400 Subject: [Numpy-discussion] installation issue with a non-standard location of a python build In-Reply-To: <5b8d13220907280902s18c5aec1x8665a7292f1312b0@mail.gmail.com> References: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> <5b8d13220907280902s18c5aec1x8665a7292f1312b0@mail.gmail.com> Message-ID: <80b160a0907281038n55930fd7s159421d89f3390f0@mail.gmail.com> On Tue, Jul 28, 2009 at 12:02 PM, David Cournapeau wrote: > On Wed, Jul 29, 2009 at 12:38 AM, Daniel > Can you post the full output of python setup.py build ? Here it is: Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /users/wd15/usr/i686/4.0/lib libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /users/wd15/usr/i686/4.0/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/atlas libraries ptf77blas,ptcblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in /users/wd15/usr/i686/4.0/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib/atlas FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c include_dirs = ['/usr/include'] /users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/config.py:361: DeprecationWarning: +++++++++++++++++++++++++++++++++++++++++++++++++ Usage of get_output is deprecated: please do not use it anymore, and avoid configuration checks involving running executable on the target machine. +++++++++++++++++++++++++++++++++++++++++++++++++ DeprecationWarning) customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc: error trying to exec 'cc1': execvp: No such file or directory gcc: error trying to exec 'cc1': execvp: No such file or directory failure. removing: _configtest.c _configtest.o Status: 255 Output: FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/usr/include'] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /users/wd15/usr/i686/4.0/lib libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /users/wd15/usr/i686/4.0/lib libraries lapack_atlas not found in /users/wd15/usr/i686/4.0/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/atlas libraries lapack_atlas not found in /usr/lib/atlas libraries ptf77blas,ptcblas,atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /users/wd15/usr/i686/4.0/lib libraries lapack_atlas not found in /users/wd15/usr/i686/4.0/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib/atlas libraries lapack_atlas not found in /usr/lib/atlas numpy.distutils.system_info.atlas_info FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = f77 include_dirs = ['/usr/include'] customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc: error trying to exec 'cc1': execvp: No such file or directory gcc: error trying to exec 'cc1': execvp: No such file or directory failure. removing: _configtest.c _configtest.o Status: 255 Output: FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/usr/include'] running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src building py_modules sources building library "npymath" sources building extension "numpy.core._sort" sources Generating build/src.linux-i686-2.6/numpy/core/include/numpy/config.h customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/users/wd15/usr/i686/4.0/include/python2.6 -c' gcc: _configtest.c gcc: error trying to exec 'cc1': execvp: No such file or directory gcc: error trying to exec 'cc1': execvp: No such file or directory failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 172, in setup_package() File "setup.py", line 165, in setup_package configuration=configuration ) File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build_src.py", line 130, in run self.build_sources() File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build_src.py", line 250, in build_extension_sources sources = self.generate_sources(sources, ext) File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build_src.py", line 307, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 286, in generate_config_h moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir) File "numpy/core/setup.py", line 30, in check_types out = check_types(*a, **kw) File "numpy/core/setup.py", line 186, in check_types "Cannot compiler 'Python.h'. Perhaps you need to "\ SystemError: Cannot compiler 'Python.h'. Perhaps you need to install python-dev|python-devel. -- Daniel Wheeler From fperez.net at gmail.com Tue Jul 28 13:39:44 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 28 Jul 2009 10:39:44 -0700 Subject: [Numpy-discussion] [SciPy-user] Numpy/Scipy and the Python African Tour In-Reply-To: <20090723141737.GF10321@phare.normalesup.org> References: <20090723141737.GF10321@phare.normalesup.org> Message-ID: Hi Emanuelle, On Thu, Jul 23, 2009 at 7:17 AM, Emmanuelle Gouillart wrote: > Dear users of Numpy and Scipy, > > here is an informal report on the last event of the Python African Tour, > which took place in Dakar (Senegal) on July 6-10th. It might interest Many thanks for this great post, I am very happy to see this kind of effort sprouting organically in different places. I had the chance to do something similar with Stefan, his advisor and colleagues a few years ago in Stellenbosch, South Africa, and it was a phenomenal experience. The audience was extremely engaged and the facilities first-rate, and I enjoyed it enormously. I'm glad to hear that you've set up a mailing list for your students to continue growing, but I'd also encourage them to subscribe to the 'normal' numpy/scipy lists. Even if they initially feel a bit shy about posting, simply 'lurking' on the lists for a while and reading the flow of information can be very useful both to learn technical ideas and to get familiar with the discussion patterns of a community. This will ease them into naturally engaging the upstream projects when they feel comfortable, which I think in the long term is both useful and necessary. > Speaking about difficulties, one may say that there was some disparity in > the level of experience inside the group, some being very well at ease > with array programming while others were obviously discovering > programming. It was therefore difficult to meet everybody's needs, but > well, I hope everybody learned something! Audience heterogeneity: that's always the hardest part in these things, and we face it everywhere. It's the main source of concern even at the main Scipy conference intro tutorial, since the audience tends to be all over the place in terms of background and experience. Mmh, I just had a thought. A number of us have by now taught reasonably organized workshops of this kind in different places and institutions. I wonder if it wouldn't be a good idea to have a page on the scipy site collecting this information, with a short paragraph about the event and a link to any additional external materials that may exist. This could encourage others to create similar things in their own communities, whether with the help of one of the previous presenters or fully independently. Does that sound useful? In any case, thanks again for the great post! Best, f From charlesr.harris at gmail.com Tue Jul 28 13:41:50 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Jul 2009 11:41:50 -0600 Subject: [Numpy-discussion] nans and sort orders. Message-ID: Hi All, I've changed the sort routines so that they sort nans to the end. The sort order is now Real types: [R, nan] Complex types: [R + Rj, R + nanj, nan + Rj, nan + nanj] Where R stands for the usual extended range of reals, i.e. [-inf, finite, +inf] and when two complex numbers have the same pattern of nans then they are sorted on the non-nan part, if any. So the sorts now work with nans but that implies some other fixups on which I would like some feedback. 1) Search sorted -- Search sorted depends on the compare function of the ndarray, I think that compare function should be changed to correspond to the sort order. 2) Methods argmax/argmin/ -- should be compatible with max/min 3) max/min ufuncs -- Problematic, min would fail to propagate nans, should probably stay the same. 4) fmax/fmin ufuncs -- Should stay with c definition 5) Logical comparison ufuncs -- I think these should remain compatible with the c operators. Note that there is a mix of implementations, max/min on void types use the type comparison functions for instance. In other words, changing the type comparisons could mean rewriting some of the other functions. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Jul 28 19:57:42 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Jul 2009 08:57:42 +0900 Subject: [Numpy-discussion] installation issue with a non-standard location of a python build In-Reply-To: <80b160a0907281038n55930fd7s159421d89f3390f0@mail.gmail.com> References: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> <5b8d13220907280902s18c5aec1x8665a7292f1312b0@mail.gmail.com> <80b160a0907281038n55930fd7s159421d89f3390f0@mail.gmail.com> Message-ID: <5b8d13220907281657m30595802k95b91de5fa4bd5de@mail.gmail.com> On Wed, Jul 29, 2009 at 2:38 AM, Daniel Wheeler wrote: > -Wall -Wstrict-prototypes -fPIC > > compile options: '-c' > gcc: _configtest.c > gcc: error trying to exec 'cc1': execvp: No such file or directory > gcc: error trying to exec 'cc1': execvp: No such file or directory That's your problem right there: your compiler is broken. What happens if you call gcc by yourself on a trivial C program ? David From dalcinl at gmail.com Tue Jul 28 22:44:19 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 28 Jul 2009 23:44:19 -0300 Subject: [Numpy-discussion] distutils -- compiling extension twice, two sets of compiler settings In-Reply-To: References: Message-ID: Would two brand new .cpp files abusing of #include do the trick? For example, for '_CCG_dmodule': //file: CCG_dmodule.cxx #define USE_DOUBLE 1 #include "CCG_caller.cpp" #include "ccg_funct.cpp" #include "ccg.cpp" #include "CCG_d_wrap.cxx" Then you can use: ext_ccg_d = Extension('_CCG_dmodule', ['Priithon_25_C/ccg/CCG_dmodule.cxx'], depends=[ ] An similarly for '_CCG_smodule', you got the idea... I know, this is a vile workaround, but trust me... you do not want to learn to hack distutils internals :-)... On Tue, Jul 28, 2009 at 9:15 AM, Sebastian Haase wrote: > Hi, > > maybe someone here could help me by using distutils: > my problem is that I have some C++ files that I need to get compiled twice, > first to make a "singleFloat"-module using ?gcc -DUSE_DOUBLE=0 > and then > to make the "doubleFloat"-module using ?gcc -DUSE_DOUBLE=1. > > Apparently distutils recognises (like "make") that the object files > are already there and skips the gcc for the second run. > My setup.py so far is something like this: > ext_ccg_s = Extension('_CCG_smodule', [ > ? ? ? ?'Priithon_25_C/ccg/CCG_caller.cpp', > ? ? ? ?'Priithon_25_C/ccg/ccg_funct.cpp', > ? ? ? ?'Priithon_25_C/ccg/ccg.cpp', > ? ? ? ?'Priithon_25_C/ccg/CCG_s_wrap.cxx', > ? ? ? ?], > ? ? ? ? ? ? ? ? ? ? ?define_macros=[('USE_DOUBLE', '0'),], > ) > ext_ccg_d = Extension('_CCG_dmodule', [ > ? ? ? ?'Priithon_25_C/ccg/CCG_caller.cpp', > ? ? ? ?'Priithon_25_C/ccg/ccg_funct.cpp', > ? ? ? ?'Priithon_25_C/ccg/ccg.cpp', > ? ? ? ?'Priithon_25_C/ccg/CCG_d_wrap.cxx', > ? ? ? ?], > ? ? ? ? ? ? ? ? ? ? ?define_macros=[('USE_DOUBLE', '1'),], > ) > setup(... > ? ? ?ext_package='Priithon_bin', > ? ? ?ext_modules=[ext_ccg_s, ext_ccg_d], > ? ? ) > > Thanks for any help, > > -- > Sebastian Haase > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ndbecker2 at gmail.com Wed Jul 29 07:30:51 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 29 Jul 2009 07:30:51 -0400 Subject: [Numpy-discussion] Functions to pack/unpack bytes? Message-ID: Does numpy have functions to convert between e.g. an array of uint32 and uint8, where the uint32 array is a packed version of the uint8 array (selecting little/big endian)? From ndbecker2 at gmail.com Wed Jul 29 08:57:02 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 29 Jul 2009 08:57:02 -0400 Subject: [Numpy-discussion] Functions to pack/unpack bytes? References: Message-ID: Neal Becker wrote: > Does numpy have functions to convert between e.g. an array of uint32 and > uint8, where the uint32 array is a packed version of the uint8 array > (selecting little/big endian)? I've been thinking about making a ufunc (in python) to do this, but have no idea how to make a ufunc for a function that consumes 4 consecutive values of it's input and produces an output array of 1/4 the size of the input. Is that possible? From zachary.pincus at yale.edu Wed Jul 29 09:35:11 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 29 Jul 2009 09:35:11 -0400 Subject: [Numpy-discussion] Functions to pack/unpack bytes? In-Reply-To: References: Message-ID: <972A7CCD-D421-4422-AF89-7D5CFAC8E09B@yale.edu> > Does numpy have functions to convert between e.g. an array of uint32 > and > uint8, where the uint32 array is a packed version of the uint8 array > (selecting little/big endian)? You could use the ndarray constructor to look at the memory differently: In : a = numpy.arange(240, 260, dtype=numpy.uint32) In : a Out: array([240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259], dtype=uint32) In : b = numpy.ndarray(shape=(len(a)*4,), dtype=numpy.uint8, buffer=a) In : b Out: array([240, 0, 0, 0, 241, 0, 0, 0, 242, 0, 0, 0, 243, 0, 0, 0, 244, 0, 0, 0, 245, 0, 0, 0, 246, 0, 0, 0, 247, 0, 0, 0, 248, 0, 0, 0, 249, 0, 0, 0, 250, 0, 0, 0, 251, 0, 0, 0, 252, 0, 0, 0, 253, 0, 0, 0, 254, 0, 0, 0, 255, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 2, 1, 0, 0, 3, 1, 0, 0], dtype=uint8) I assume for selecting little/big-endian going the other way, you could use the other methods of specifying dtypes that allow for byte- order descriptors. (Like dtype objects or the format strings.) Zach From stefan at sun.ac.za Wed Jul 29 10:00:32 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 29 Jul 2009 16:00:32 +0200 Subject: [Numpy-discussion] Functions to pack/unpack bytes? In-Reply-To: <972A7CCD-D421-4422-AF89-7D5CFAC8E09B@yale.edu> References: <972A7CCD-D421-4422-AF89-7D5CFAC8E09B@yale.edu> Message-ID: <9457e7c80907290700j4c14144x7701a0a739819894@mail.gmail.com> 2009/7/29 Zachary Pincus : >> Does numpy have functions to convert between e.g. an array of uint32 >> and >> uint8, where the uint32 array is a packed version of the uint8 array >> (selecting little/big endian)? > > > You could use the ndarray constructor to look at the memory differently: > > In : a = numpy.arange(240, 260, dtype=numpy.uint32) > In : a > Out: > array([240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, > ? ? ? ?253, 254, 255, 256, 257, 258, 259], dtype=uint32) > > In : b = numpy.ndarray(shape=(len(a)*4,), dtype=numpy.uint8, buffer=a) > > In : b > Out: > array([240, ? 0, ? 0, ? 0, 241, ? 0, ? 0, ? 0, 242, ? 0, ? 0, ? 0, 243, > ? ? ? ? ?0, ? 0, ? 0, 244, ? 0, ? 0, ? 0, 245, ? 0, ? 0, ? 0, 246, ? 0, > ? ? ? ? ?0, ? 0, 247, ? 0, ? 0, ? 0, 248, ? 0, ? 0, ? 0, 249, ? 0, ? 0, > ? ? ? ? ?0, 250, ? 0, ? 0, ? 0, 251, ? 0, ? 0, ? 0, 252, ? 0, ? 0, ? 0, > ? ? ? ?253, ? 0, ? 0, ? 0, 254, ? 0, ? 0, ? 0, 255, ? 0, ? 0, ? 0, ? 0, > ? ? ? ? ?1, ? 0, ? 0, ? 1, ? 1, ? 0, ? 0, ? 2, ? 1, ? 0, ? 0, ? 3, ? 1, > ? ? ? ? ?0, ? 0], dtype=uint8) > > > I assume for selecting little/big-endian going the other way, you > could use the other methods of specifying dtypes that allow for byte- > order descriptors. (Like dtype objects or the format strings.) Something like: In [17]: np.dtype(np.int32).newbyteorder('>') Out[17]: dtype('>i4') In [18]: dt = np.dtype(np.int32).newbyteorder('>') In [19]: x = np.array([123, 456, 789], dtype=dt) In [20]: x.view(np.uint8) Out[20]: array([ 0, 0, 0, 123, 0, 0, 1, 200, 0, 0, 3, 21], dtype=uint8) Regards St?fan From ndbecker2 at gmail.com Wed Jul 29 10:20:19 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 29 Jul 2009 10:20:19 -0400 Subject: [Numpy-discussion] Functions to pack/unpack bytes? References: <972A7CCD-D421-4422-AF89-7D5CFAC8E09B@yale.edu> <9457e7c80907290700j4c14144x7701a0a739819894@mail.gmail.com> Message-ID: St?fan van der Walt wrote: > 2009/7/29 Zachary Pincus : >>> Does numpy have functions to convert between e.g. an array of uint32 >>> and >>> uint8, where the uint32 array is a packed version of the uint8 array >>> (selecting little/big endian)? >> >> >> You could use the ndarray constructor to look at the memory differently: >> >> In : a = numpy.arange(240, 260, dtype=numpy.uint32) >> In : a >> Out: >> array([240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, >> 253, 254, 255, 256, 257, 258, 259], dtype=uint32) >> >> In : b = numpy.ndarray(shape=(len(a)*4,), dtype=numpy.uint8, buffer=a) >> >> In : b >> Out: >> array([240, 0, 0, 0, 241, 0, 0, 0, 242, 0, 0, 0, 243, >> 0, 0, 0, 244, 0, 0, 0, 245, 0, 0, 0, 246, 0, >> 0, 0, 247, 0, 0, 0, 248, 0, 0, 0, 249, 0, 0, >> 0, 250, 0, 0, 0, 251, 0, 0, 0, 252, 0, 0, 0, >> 253, 0, 0, 0, 254, 0, 0, 0, 255, 0, 0, 0, 0, >> 1, 0, 0, 1, 1, 0, 0, 2, 1, 0, 0, 3, 1, >> 0, 0], dtype=uint8) >> >> >> I assume for selecting little/big-endian going the other way, you >> could use the other methods of specifying dtypes that allow for byte- >> order descriptors. (Like dtype objects or the format strings.) > > Something like: > > In [17]: np.dtype(np.int32).newbyteorder('>') > Out[17]: dtype('>i4') > > In [18]: dt = np.dtype(np.int32).newbyteorder('>') > > In [19]: x = np.array([123, 456, 789], dtype=dt) > > In [20]: x.view(np.uint8) > Out[20]: array([ 0, 0, 0, 123, 0, 0, 1, 200, 0, 0, 3, > 21], dtype=uint8) > > Regards > St?fan Can 'view' switch byteorder? Doesn't seem to work: import numpy as np a = np.arange(10, dtype=np.uint32) b1 = a.view (np.dtype(np.uint32).newbyteorder('<')) c1 = b1.view(np.uint8) b2 = a.view (np.dtype(np.uint32).newbyteorder('>')) c2 = b2.view(np.uint8) print c1 print c2 [0 0 0 0 1 0 0 0 2 0 0 0 3 0 0 0 4 0 0 0 5 0 0 0 6 0 0 0 7 0 0 0 8 0 0 0 9 0 0 0] [0 0 0 0 1 0 0 0 2 0 0 0 3 0 0 0 4 0 0 0 5 0 0 0 6 0 0 0 7 0 0 0 8 0 0 0 9 0 0 0] From pav at iki.fi Wed Jul 29 10:44:21 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 29 Jul 2009 14:44:21 +0000 (UTC) Subject: [Numpy-discussion] Functions to pack/unpack bytes? References: <972A7CCD-D421-4422-AF89-7D5CFAC8E09B@yale.edu> <9457e7c80907290700j4c14144x7701a0a739819894@mail.gmail.com> Message-ID: Wed, 29 Jul 2009 10:20:19 -0400, Neal Becker kirjoitti: [clip] > Can 'view' switch byteorder? Doesn't seem to work: [clip] I think not: view only alters the dtype, ie., the interpretation of the block of memory. It does not physically reorder the data in memory, which is necessary for unpacking to a different byte order. The reliable way to unpack seems to be np.asarray(a, dtype=' References: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> <5b8d13220907280902s18c5aec1x8665a7292f1312b0@mail.gmail.com> <80b160a0907281038n55930fd7s159421d89f3390f0@mail.gmail.com> <5b8d13220907281657m30595802k95b91de5fa4bd5de@mail.gmail.com> Message-ID: <80b160a0907291133p198c010bxf6baae6dc63a6aea@mail.gmail.com> Oh dear. I'd changed some environment variables when I switched to using python 2.6 and was accidentally using a local version of gcc, which evidently doesn't work correctly. Everything builds fine now. Thanks and sorry for the wasted time. On Tue, Jul 28, 2009 at 7:57 PM, David Cournapeau wrote: > On Wed, Jul 29, 2009 at 2:38 AM, Daniel > Wheeler wrote: >> -Wall -Wstrict-prototypes -fPIC >> >> compile options: '-c' >> gcc: _configtest.c >> gcc: error trying to exec 'cc1': execvp: No such file or directory >> gcc: error trying to exec 'cc1': execvp: No such file or directory > > That's your problem right there: your compiler is broken. What happens > if you call gcc by yourself on a trivial C program ? > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Daniel Wheeler From eadrogue at gmx.net Wed Jul 29 17:36:39 2009 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Wed, 29 Jul 2009 23:36:39 +0200 Subject: [Numpy-discussion] question about a data structure Message-ID: <20090729213639.GA10538@doriath.local> Hello, Suppose I want to store some data (a non-negative integer) related to an "event" involving two entities. It occurs to me that the way to do it is using a 2-d array, like a table: For example: a b c a 1 b 2 0 c 5 in the event 'b-a' the data is 2, and in the event 'c-a' is 5, etc. Now, I need to be able to differentiate between 0 and 'no data'. I know I could use a number like -1, but this would affect arithmetic operations on the array, and also I need that indexing one such location with no data returns an empty array. Is it possible to do this with the standard array class? Thanks. Ernest From robert.kern at gmail.com Wed Jul 29 17:39:01 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 29 Jul 2009 16:39:01 -0500 Subject: [Numpy-discussion] question about a data structure In-Reply-To: <20090729213639.GA10538@doriath.local> References: <20090729213639.GA10538@doriath.local> Message-ID: <3d375d730907291439q7442d92am7fd1824a6fc63681@mail.gmail.com> 2009/7/29 Ernest Adrogu? : > Hello, > > Suppose I want to store some data (a non-negative integer) > related to an "event" involving two entities. It occurs to me that > the way to do it is using a 2-d array, like a table: > > For example: > > ?a ?b ?c > a ? ?1 > b 2 ? ? 0 > c 5 > > in the event 'b-a' the data is 2, and in the event 'c-a' is 5, etc. > > Now, I need to be able to differentiate between 0 and 'no data'. > I know I could use a number like -1, but this would affect arithmetic > operations on the array, and also I need that indexing one such > location with no data returns an empty array. > > Is it possible to do this with the standard array class? Not really. Use masked arrays. Or use a dictionary. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Wed Jul 29 17:54:24 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 29 Jul 2009 14:54:24 -0700 Subject: [Numpy-discussion] question about a data structure In-Reply-To: <3d375d730907291439q7442d92am7fd1824a6fc63681@mail.gmail.com> References: <20090729213639.GA10538@doriath.local> <3d375d730907291439q7442d92am7fd1824a6fc63681@mail.gmail.com> Message-ID: <4A70C510.4070407@noaa.gov> Robert Kern wrote: > 2009/7/29 Ernest Adrogu? : >> Now, I need to be able to differentiate between 0 and 'no data'. >> Is it possible to do this with the standard array class? > > Not really. Use masked arrays. Or use a dictionary. masked arrays are made precisely for this. However, if you can use floating point numbers, you could use NaN for the no data entries. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From liu.ian0571 at gmail.com Wed Jul 29 21:02:49 2009 From: liu.ian0571 at gmail.com (ian) Date: Thu, 30 Jul 2009 09:02:49 +0800 Subject: [Numpy-discussion] how to configure f2py on windows XP? Message-ID: <4a70f12f.15bb720a.6509.ffffbce0@mx.google.com> Hi,everyone? I installed MSVS2005 + Intel visual fortran 9.1 + Python 2.5 + numpy-1.3.0-win32-superpack-python2.5.exe on my computer. The numpy works well: Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer's internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 1.2 >>> import numpy >>> but when I use : python c:\python25\scripts\f2py.py --fcompiler=intel -c hello.f -m hello , the result is: running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src building extension "hello" sources f2py options: [] f2py:> c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5\hellomodule.c creating c:\docume~1\admini~1\locals~1\temp\tmpwktrfc creating c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5 Reading fortran codes... Reading file 'hello.f' (format:fix,strict) Post-processing... Block: hello Block: foo Post-processing (stage 2)... Building modules... Building module "hello"... Constructing wrapper function "foo"... foo(a) Wrote C/API module "hello" to file "c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5/hellomodule.c" adding 'c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5\fortranobject.c' to sources. adding 'c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5' to include_dirs. copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.c -> c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5 copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.h -> c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5 running build_ext No module named msvccompiler in numpy.distutils; trying from distutils error: python was built with Visula Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. who could give me some suggestions on how to configure f2py? Thanks in advance. ian liu.ian0571 at gmail.com 2009-07-30 From sccolbert at gmail.com Wed Jul 29 21:51:09 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Wed, 29 Jul 2009 21:51:09 -0400 Subject: [Numpy-discussion] how to configure f2py on windows XP? In-Reply-To: <4a70f12f.15bb720a.6509.ffffbce0@mx.google.com> References: <4a70f12f.15bb720a.6509.ffffbce0@mx.google.com> Message-ID: <7f014ea60907291851q6b72111dyd5be13855f576117@mail.gmail.com> unless you have a visual studio 2003 compiler, you may have to use python 2.6. 2009/7/29 ian : > Hi,everyone? > > I installed MSVS2005 + Intel visual fortran 9.1 + Python 2.5 + numpy-1.3.0-win32-superpack-python2.5.exe on my computer. The numpy works well: > > Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 > Type "copyright", "credits" or "license()" for more information. > > **************************************************************** > Personal firewall software may warn about the connection IDLE > makes to its subprocess using this computer's internal loopback > interface. This connection is not visible on any external > interface and no data is sent to or received from the Internet. > **************************************************************** > > IDLE 1.2 >>>> import numpy >>>> > > but when I use : python c:\python25\scripts\f2py.py --fcompiler=intel -c hello.f -m hello , the result is: > > running build > running config_cc > unifing config_cc, config, build_clib, build_ext, build commands --compiler options > running config_fc > unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options > running build_src > building extension "hello" sources > f2py options: [] > f2py:> c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5\hellomodule.c > creating c:\docume~1\admini~1\locals~1\temp\tmpwktrfc > creating c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5 > Reading fortran codes... > Reading file 'hello.f' (format:fix,strict) > Post-processing... > Block: hello > Block: foo > Post-processing (stage 2)... > Building modules... > Building module "hello"... > Constructing wrapper function "foo"... > foo(a) > Wrote C/API module "hello" to file "c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5/hellomodule.c" > adding 'c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5\fortranobject.c' to sources. > adding 'c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5' to include_dirs. > copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.c -> c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5 > copying C:\Python25\lib\site-packages\numpy\f2py\src\fortranobject.h -> c:\docume~1\admini~1\locals~1\temp\tmpwktrfc\src.win32-2.5 > running build_ext > No module named msvccompiler in numpy.distutils; trying from distutils > error: python was built with Visula Studio 2003; > extensions must be built with a compiler than can generate compatible binaries. > Visual Studio 2003 was not found on this system. If you have Cygwin > installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. > > who could give me some suggestions on how to configure f2py? > > > Thanks in advance. > > > ian > liu.ian0571 at gmail.com > 2009-07-30 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From oliphant at enthought.com Wed Jul 29 21:51:38 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 29 Jul 2009 19:51:38 -0600 Subject: [Numpy-discussion] Functions to pack/unpack bytes? In-Reply-To: References: Message-ID: Doesn't .view work for that? Travis Sent from my iPhone On Jul 29, 2009, at 6:57 AM, Neal Becker wrote: > Neal Becker wrote: > >> Does numpy have functions to convert between e.g. an array of >> uint32 and >> uint8, where the uint32 array is a packed version of the uint8 array >> (selecting little/big endian)? > > I've been thinking about making a ufunc (in python) to do this, but > have no > idea how to make a ufunc for a function that consumes 4 consecutive > values > of it's input and produces an output array of 1/4 the size of the > input. Is > that possible? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From david at ar.media.kyoto-u.ac.jp Thu Jul 30 02:17:47 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 30 Jul 2009 15:17:47 +0900 Subject: [Numpy-discussion] how to configure f2py on windows XP? In-Reply-To: <7f014ea60907291851q6b72111dyd5be13855f576117@mail.gmail.com> References: <4a70f12f.15bb720a.6509.ffffbce0@mx.google.com> <7f014ea60907291851q6b72111dyd5be13855f576117@mail.gmail.com> Message-ID: <4A713B0B.2030606@ar.media.kyoto-u.ac.jp> Chris Colbert wrote: > unless you have a visual studio 2003 compiler, you may have to use python 2.6. > > > Mingw works, as well. This way, you don't have to care about which VS version to use for which python interpreter on which platform, cheers, David From reedev at zonnet.nl Thu Jul 30 02:41:44 2009 From: reedev at zonnet.nl (Raymond de Vries) Date: Thu, 30 Jul 2009 08:41:44 +0200 Subject: [Numpy-discussion] (newbie) How can I use NumPy to wrap my C++ class with 2-dimensional arrays? Message-ID: <4A7140A8.2040305@zonnet.nl> Hi everyone, (I sent this message yesterday as well but somehow it didn't come through...) I would like to ask your advice how I can use NumPy to wrap my existing C++ library with 2-dimensional arrays. I am wrapping the library with swig and I have the typemap function declaration. But now I am struggling a lot to find an implementation for getting access to the 2-dimensional arrays in my C++ classes. I have wrapped most of the library but the multi-dimensional arrays are problematic. From what I have read, I can use NumPy for this.. There is so much information about NumPy, Numeric etc that it is not clear anymore which example I can use best. It's clear that NumPy is probably the way to go. This is a small example of the class that I would like to wrap: struct MyClass { float array1[2][2]; std::vector array2[2]; }; I've also read that I need to call import_array() but where should I put this? Is there a small example with NumPy c-api to wrap/convert my 2-dimensional arrays to python? Thanks a lot already Raymond From sccolbert at gmail.com Thu Jul 30 02:44:11 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 30 Jul 2009 02:44:11 -0400 Subject: [Numpy-discussion] how to configure f2py on windows XP? In-Reply-To: <4A713B0B.2030606@ar.media.kyoto-u.ac.jp> References: <4a70f12f.15bb720a.6509.ffffbce0@mx.google.com> <7f014ea60907291851q6b72111dyd5be13855f576117@mail.gmail.com> <4A713B0B.2030606@ar.media.kyoto-u.ac.jp> Message-ID: <7f014ea60907292344p25f7e226ta0f3c7dd8409ddad@mail.gmail.com> true, I saw intel visual fortran, and made the wrong association thinking it was some sort of visual studio plugin. (stupid, i know...) On Thu, Jul 30, 2009 at 2:17 AM, David Cournapeau wrote: > Chris Colbert wrote: >> unless you have a visual studio 2003 compiler, you may have to use python 2.6. >> >> >> > > Mingw works, as well. This way, you don't have to care about which VS > version to use for which python interpreter on which platform, > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From matthieu.brucher at gmail.com Thu Jul 30 02:52:07 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 30 Jul 2009 08:52:07 +0200 Subject: [Numpy-discussion] (newbie) How can I use NumPy to wrap my C++ class with 2-dimensional arrays? In-Reply-To: <4A7140A8.2040305@zonnet.nl> References: <4A7140A8.2040305@zonnet.nl> Message-ID: Hi, In fact, it's not that complicated. You may know the way how copying a vector, and this is all you need (http://matt.eifelle.com/2008/01/04/transforming-a-c-vector-into-a-numpy-array/). You will have to copy your data, it is the safest way to ensure that the data is always valid. For the std::vector<>[], I suggest you convert it to a single vector, as the data inside this "array" is not contiguous and it can thus be cumbersome to create a Numpy array from that. Once the structure is prepared, you have to allocate the dimensions, but this may be available online. In case you know that the C++ data won't go away before the Python array, you can always wrap the container (http://matt.eifelle.com/2008/11/04/exposing-an-array-interface-with-swig-for-a-cc-structure/) with SWIG. Matthieu 2009/7/30 Raymond de Vries : > Hi everyone, > > (I sent this message yesterday as well but somehow it didn't come > through...) > > I would like to ask your advice how I can use NumPy to wrap my existing > C++ library with 2-dimensional arrays. I am wrapping the library with > swig and I have the typemap function declaration. But now I am > struggling a lot to find an implementation for getting access to the > 2-dimensional arrays in my C++ classes. I have wrapped most of the > library but the multi-dimensional arrays are problematic. From what I > have read, I can use NumPy for this.. > > There is so much information about NumPy, Numeric etc that it is not > clear anymore which example I can use best. It's clear that NumPy is > probably the way to go. > > This is a small example of the class that I would like to wrap: > struct MyClass { > ?float array1[2][2]; > ?std::vector array2[2]; > }; > > I've also read that I need to call import_array() but where should I put > this? > > Is there a small example with NumPy c-api to wrap/convert my > 2-dimensional arrays to python? > > Thanks a lot already > Raymond > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From fperez.net at gmail.com Thu Jul 30 03:34:53 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 30 Jul 2009 00:34:53 -0700 Subject: [Numpy-discussion] Reductions and binary ops on recarrays... Message-ID: Howdy, we recently had a discussion about being able to do some common things like reductions and binary operations on recarrays, and there didn't seem to be much consensus on it being needed in the core of numpy. Since we do actually need this quite pressingly for everyday tasks, we wrote a very simple version of this today, I'm attaching it here in case it proves useful to others. Basically it lets you do reductions and binary operations on record arrays whose dtype is a simple composite of native ones (example at the end). For our needs it's quite useful, so it may also be to others. Cheers, f Example: >>> import recarrutil as ru >>> dt = np.dtype(dict(names=['x','y'],formats=[float,float])) >>> x = np.arange(6,dtype=float).reshape(2,3) >>> y = np.arange(10,16,dtype=float).reshape(2,3) >>> z = np.empty( (2,3), dt).view(np.recarray) >>> z.x = x >>> z.y = y >>> z rec.array([[(0.0, 10.0), (1.0, 11.0), (2.0, 12.0)], [(3.0, 13.0), (4.0, 14.0), (5.0, 15.0)]], dtype=[('x', '>> ru.mean(z) rec.array((2.5, 12.5), dtype=[('x', '>> ru.mean(z,0) rec.array([(1.5, 11.5), (2.5, 12.5), (3.5, 13.5)], dtype=[('x', '>> ru.mean(z,1) rec.array([(1.0, 11.0), (4.0, 14.0)], dtype=[('x', '>> ru.add(z,z) rec.array([[(0.0, 20.0), (2.0, 22.0), (4.0, 24.0)], [(6.0, 26.0), (8.0, 28.0), (10.0, 30.0)]], dtype=[('x', '>> ru.subtract(z,z) rec.array([[(0.0, 0.0), (0.0, 0.0), (0.0, 0.0)], [(0.0, 0.0), (0.0, 0.0), (0.0, 0.0)]], dtype=[('x', ' From stefan at sun.ac.za Thu Jul 30 03:47:14 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 30 Jul 2009 09:47:14 +0200 Subject: [Numpy-discussion] Reductions and binary ops on recarrays... In-Reply-To: References: Message-ID: <9457e7c80907300047y10948e03x116b8f1a35ca9e3a@mail.gmail.com> 2009/7/30 Fernando Perez : > we recently had a discussion about being able to do some common things > like reductions and binary operations on recarrays, and there didn't > seem to be much consensus on it being needed in the core of numpy. > > Since ?we do actually need this quite pressingly for everyday tasks, > we wrote a very simple version of this today, I'm attaching it here in > case it proves useful to others. I'm in favour of such a patch, but I'd like to see whether we can't do it at the C level for structured arrays in general. Regards St?fan From reedev at zonnet.nl Thu Jul 30 04:52:16 2009 From: reedev at zonnet.nl (Raymond de Vries) Date: Thu, 30 Jul 2009 10:52:16 +0200 Subject: [Numpy-discussion] (newbie) How can I use NumPy to wrap my C++ class with 2-dimensional arrays? In-Reply-To: References: <4A7140A8.2040305@zonnet.nl> Message-ID: <4A715F40.1060903@zonnet.nl> Hi Matthieu, Thanks for your quick reply! I did not find this page earlier. I have studied the first page and now I realize again that all my efforts so far (last few days) have crashed on the lack of calling import_array(). Do you have a suggestion for calling import_array() as well? Everywhere I see *that* import_array() needs to be called, but not how... I will take a look at the conversion to a single vector. I hope this can be avoided because I cannot simply change the library. regards Raymond Matthieu Brucher wrote: > Hi, > > In fact, it's not that complicated. You may know the way how copying a > vector, and this is all you need > (http://matt.eifelle.com/2008/01/04/transforming-a-c-vector-into-a-numpy-array/). > You will have to copy your data, it is the safest way to ensure that > the data is always valid. > For the std::vector<>[], I suggest you convert it to a single vector, > as the data inside this "array" is not contiguous and it can thus be > cumbersome to create a Numpy array from that. > > Once the structure is prepared, you have to allocate the dimensions, > but this may be available online. > > In case you know that the C++ data won't go away before the Python > array, you can always wrap the container > (http://matt.eifelle.com/2008/11/04/exposing-an-array-interface-with-swig-for-a-cc-structure/) > with SWIG. > > Matthieu > > 2009/7/30 Raymond de Vries : > >> Hi everyone, >> >> (I sent this message yesterday as well but somehow it didn't come >> through...) >> >> I would like to ask your advice how I can use NumPy to wrap my existing >> C++ library with 2-dimensional arrays. I am wrapping the library with >> swig and I have the typemap function declaration. But now I am >> struggling a lot to find an implementation for getting access to the >> 2-dimensional arrays in my C++ classes. I have wrapped most of the >> library but the multi-dimensional arrays are problematic. From what I >> have read, I can use NumPy for this.. >> >> There is so much information about NumPy, Numeric etc that it is not >> clear anymore which example I can use best. It's clear that NumPy is >> probably the way to go. >> >> This is a small example of the class that I would like to wrap: >> struct MyClass { >> float array1[2][2]; >> std::vector array2[2]; >> }; >> >> I've also read that I need to call import_array() but where should I put >> this? >> >> Is there a small example with NumPy c-api to wrap/convert my >> 2-dimensional arrays to python? >> >> Thanks a lot already >> Raymond >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > > From reedev at zonnet.nl Thu Jul 30 05:07:16 2009 From: reedev at zonnet.nl (Raymond de Vries) Date: Thu, 30 Jul 2009 11:07:16 +0200 Subject: [Numpy-discussion] (newbie) How can I use NumPy to wrap my C++ class with 2-dimensional arrays? In-Reply-To: <4A715F40.1060903@zonnet.nl> References: <4A7140A8.2040305@zonnet.nl> <4A715F40.1060903@zonnet.nl> Message-ID: <4A7162C4.8030605@zonnet.nl> Hi, I'm sorry, I guess I did not search properly before.... For the record, I solved my import_array() question: just need to add %init %{ import_array(); %} and the typemap for the std::vector works ok. Thanks for that! Now the rest... Thanks Raymond Raymond de Vries wrote: > Hi Matthieu, > > Thanks for your quick reply! I did not find this page earlier. I have > studied the first page and now I realize again that all my efforts so > far (last few days) have crashed on the lack of calling import_array(). > Do you have a suggestion for calling import_array() as well? Everywhere > I see *that* import_array() needs to be called, but not how... > > I will take a look at the conversion to a single vector. I hope this can > be avoided because I cannot simply change the library. > > regards > Raymond > > > Matthieu Brucher wrote: > >> Hi, >> >> In fact, it's not that complicated. You may know the way how copying a >> vector, and this is all you need >> (http://matt.eifelle.com/2008/01/04/transforming-a-c-vector-into-a-numpy-array/). >> You will have to copy your data, it is the safest way to ensure that >> the data is always valid. >> For the std::vector<>[], I suggest you convert it to a single vector, >> as the data inside this "array" is not contiguous and it can thus be >> cumbersome to create a Numpy array from that. >> >> Once the structure is prepared, you have to allocate the dimensions, >> but this may be available online. >> >> In case you know that the C++ data won't go away before the Python >> array, you can always wrap the container >> (http://matt.eifelle.com/2008/11/04/exposing-an-array-interface-with-swig-for-a-cc-structure/) >> with SWIG. >> >> Matthieu >> >> 2009/7/30 Raymond de Vries : >> >> >>> Hi everyone, >>> >>> (I sent this message yesterday as well but somehow it didn't come >>> through...) >>> >>> I would like to ask your advice how I can use NumPy to wrap my existing >>> C++ library with 2-dimensional arrays. I am wrapping the library with >>> swig and I have the typemap function declaration. But now I am >>> struggling a lot to find an implementation for getting access to the >>> 2-dimensional arrays in my C++ classes. I have wrapped most of the >>> library but the multi-dimensional arrays are problematic. From what I >>> have read, I can use NumPy for this.. >>> >>> There is so much information about NumPy, Numeric etc that it is not >>> clear anymore which example I can use best. It's clear that NumPy is >>> probably the way to go. >>> >>> This is a small example of the class that I would like to wrap: >>> struct MyClass { >>> float array1[2][2]; >>> std::vector array2[2]; >>> }; >>> >>> I've also read that I need to call import_array() but where should I put >>> this? >>> >>> Is there a small example with NumPy c-api to wrap/convert my >>> 2-dimensional arrays to python? >>> >>> Thanks a lot already >>> Raymond >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >>> >> >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From stefan at sun.ac.za Thu Jul 30 05:09:08 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 30 Jul 2009 11:09:08 +0200 Subject: [Numpy-discussion] Agenda items for steering committee meeting, 2009/08 Message-ID: <9457e7c80907300209g7043c1a8ue10e0e9ad3142e59@mail.gmail.com> Hi all, We are soliciting agenda items for the SciPy Steering Committee meeting taking place during the SciPy09 conference. Please view the current list at http://www.scipy.org/SteeringCommittee_08_2009 and post new items to this thread. Kind regards St?fan From matthieu.brucher at gmail.com Thu Jul 30 05:54:09 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 30 Jul 2009 11:54:09 +0200 Subject: [Numpy-discussion] (newbie) How can I use NumPy to wrap my C++ class with 2-dimensional arrays? In-Reply-To: <4A7162C4.8030605@zonnet.nl> References: <4A7140A8.2040305@zonnet.nl> <4A715F40.1060903@zonnet.nl> <4A7162C4.8030605@zonnet.nl> Message-ID: 2009/7/30 Raymond de Vries : > Hi, > > I'm sorry, I guess I did not search properly before.... For the record, > I solved my import_array() question: just need to add > %init %{ > import_array(); > %} Indeed, the solution is as simple as this ;) The trouble is to find the information! > and the typemap for the std::vector works ok. Thanks for that! You're welcome. > Now the rest... > > Thanks > Raymond > > > Raymond de Vries wrote: >> Hi Matthieu, >> >> Thanks for your quick reply! I did not find this page earlier. I have >> studied the first page and now I realize again that all my efforts so >> far (last few days) have crashed on the lack of calling import_array(). >> Do you have a suggestion for calling import_array() as well? Everywhere >> I see *that* import_array() needs to be called, but not how... >> >> I will take a look at the conversion to a single vector. I hope this can >> be avoided because I cannot simply change the library. >> >> regards >> Raymond >> >> >> Matthieu Brucher wrote: >> >>> Hi, >>> >>> In fact, it's not that complicated. You may know the way how copying a >>> vector, and this is all you need >>> (http://matt.eifelle.com/2008/01/04/transforming-a-c-vector-into-a-numpy-array/). >>> You will have to copy your data, it is the safest way to ensure that >>> the data is always valid. >>> For the std::vector<>[], I suggest you convert it to a single vector, >>> as the data inside this "array" is not contiguous and it can thus be >>> cumbersome to create a Numpy array from that. >>> >>> Once the structure is prepared, you have to allocate the dimensions, >>> but this may be available online. >>> >>> In case you know that the C++ data won't go away before the Python >>> array, you can always wrap the container >>> (http://matt.eifelle.com/2008/11/04/exposing-an-array-interface-with-swig-for-a-cc-structure/) >>> with SWIG. >>> >>> Matthieu >>> >>> 2009/7/30 Raymond de Vries : >>> >>> >>>> Hi everyone, >>>> >>>> (I sent this message yesterday as well but somehow it didn't come >>>> through...) >>>> >>>> I would like to ask your advice how I can use NumPy to wrap my existing >>>> C++ library with 2-dimensional arrays. I am wrapping the library with >>>> swig and I have the typemap function declaration. But now I am >>>> struggling a lot to find an implementation for getting access to the >>>> 2-dimensional arrays in my C++ classes. I have wrapped most of the >>>> library but the multi-dimensional arrays are problematic. From what I >>>> have read, I can use NumPy for this.. >>>> >>>> There is so much information about NumPy, Numeric etc that it is not >>>> clear anymore which example I can use best. It's clear that NumPy is >>>> probably the way to go. >>>> >>>> This is a small example of the class that I would like to wrap: >>>> struct MyClass { >>>> ?float array1[2][2]; >>>> ?std::vector array2[2]; >>>> }; >>>> >>>> I've also read that I need to call import_array() but where should I put >>>> this? >>>> >>>> Is there a small example with NumPy c-api to wrap/convert my >>>> 2-dimensional arrays to python? >>>> >>>> Thanks a lot already >>>> Raymond >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>>> >>> >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From reedev at zonnet.nl Thu Jul 30 06:19:16 2009 From: reedev at zonnet.nl (Raymond de Vries) Date: Thu, 30 Jul 2009 12:19:16 +0200 Subject: [Numpy-discussion] (newbie) How can I use NumPy to wrap my C++ class with 2-dimensional arrays? In-Reply-To: References: <4A7140A8.2040305@zonnet.nl> <4A715F40.1060903@zonnet.nl> <4A7162C4.8030605@zonnet.nl> Message-ID: <4A7173A4.6070608@zonnet.nl> Hi > Indeed, the solution is as simple as this ;) The trouble is to find > the information! > Yes, there is so much information everywhere. And it's hard to make the first steps. >>>> For the std::vector<>[], I suggest you convert it to a single vector, >>>> as the data inside this "array" is not contiguous and it can thus be >>>> cumbersome to create a Numpy array from that. >>>> I am now ready to do this. To be certain, 'contiguous' means that the std::vector<>'s are not the same length, right? Would that mean that I'd better use a tuple of lists or so? (or list of lists or so). thanks for your time Raymond >>>> Once the structure is prepared, you have to allocate the dimensions, >>>> but this may be available online. >>>> >>>> In case you know that the C++ data won't go away before the Python >>>> array, you can always wrap the container >>>> (http://matt.eifelle.com/2008/11/04/exposing-an-array-interface-with-swig-for-a-cc-structure/) >>>> with SWIG. >>>> >>>> Matthieu >>>> >>>> 2009/7/30 Raymond de Vries : >>>> >>>> >>>> >>>>> Hi everyone, >>>>> >>>>> (I sent this message yesterday as well but somehow it didn't come >>>>> through...) >>>>> >>>>> I would like to ask your advice how I can use NumPy to wrap my existing >>>>> C++ library with 2-dimensional arrays. I am wrapping the library with >>>>> swig and I have the typemap function declaration. But now I am >>>>> struggling a lot to find an implementation for getting access to the >>>>> 2-dimensional arrays in my C++ classes. I have wrapped most of the >>>>> library but the multi-dimensional arrays are problematic. From what I >>>>> have read, I can use NumPy for this.. >>>>> >>>>> There is so much information about NumPy, Numeric etc that it is not >>>>> clear anymore which example I can use best. It's clear that NumPy is >>>>> probably the way to go. >>>>> >>>>> This is a small example of the class that I would like to wrap: >>>>> struct MyClass { >>>>> float array1[2][2]; >>>>> std::vector array2[2]; >>>>> }; >>>>> >>>>> I've also read that I need to call import_array() but where should I put >>>>> this? >>>>> >>>>> Is there a small example with NumPy c-api to wrap/convert my >>>>> 2-dimensional arrays to python? >>>>> >>>>> Thanks a lot already >>>>> Raymond >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > > From matthieu.brucher at gmail.com Thu Jul 30 06:57:46 2009 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 30 Jul 2009 12:57:46 +0200 Subject: [Numpy-discussion] (newbie) How can I use NumPy to wrap my C++ class with 2-dimensional arrays? In-Reply-To: <4A7173A4.6070608@zonnet.nl> References: <4A7140A8.2040305@zonnet.nl> <4A715F40.1060903@zonnet.nl> <4A7162C4.8030605@zonnet.nl> <4A7173A4.6070608@zonnet.nl> Message-ID: 2009/7/30 Raymond de Vries : > Hi > >> Indeed, the solution is as simple as this ;) The trouble is to find >> the information! >> > Yes, there is so much information everywhere. And it's hard to make the > first steps. >>>>> For the std::vector<>[], I suggest you convert it to a single vector, >>>>> as the data inside this "array" is not contiguous and it can thus be >>>>> cumbersome to create a Numpy array from that. >>>>> > I am now ready to do this. To be certain, 'contiguous' means that the > std::vector<>'s are not the same length, right? Would that mean that I'd > better use a tuple of lists or so? (or list of lists or so). > > thanks for your time > Raymond Contiguous means that the whole data is in one big chunk. If it is not, you have to rely on other Numpy functions (I don't know all of them, perhaps you will find one that satisfies your need), and the data may then be copied (not sure though). Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher From josef.pktd at gmail.com Thu Jul 30 10:55:03 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 30 Jul 2009 10:55:03 -0400 Subject: [Numpy-discussion] Reductions and binary ops on recarrays... In-Reply-To: <9457e7c80907300047y10948e03x116b8f1a35ca9e3a@mail.gmail.com> References: <9457e7c80907300047y10948e03x116b8f1a35ca9e3a@mail.gmail.com> Message-ID: <1cd32cbb0907300755t37836328l2ddb904331e6c5c5@mail.gmail.com> 2009/7/30 St?fan van der Walt : > 2009/7/30 Fernando Perez : >> we recently had a discussion about being able to do some common things >> like reductions and binary operations on recarrays, and there didn't >> seem to be much consensus on it being needed in the core of numpy. >> >> Since ?we do actually need this quite pressingly for everyday tasks, >> we wrote a very simple version of this today, I'm attaching it here in >> case it proves useful to others. > > I'm in favour of such a patch, but I'd like to see whether we can't do > it at the C level for structured arrays in general. > > Regards > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Are these functions really for a relevant use case of structured arrays. I haven't seen any examples of multidimensional structured arrays, but from a quick reading it doesn't seem to handle mixed types (raises error) or nested structured arrays (I'm not sure), for which I have seen a lot more examples. I was looking for or writing something similar but only for 1d structured arrays, i.e. a 2d dataset. for homogenous dtypes it is relatively easy to create a view on which standard array operations can be applied. mixed dtypes however, I wanted a version that can handle mixed dtypes, in my case integer and floats, that upcasts all numerical types to the highest dtype, to floats in my examples. integers where categorical data that I want as integers for e.g. np.bincount . temporary/ conversion array reuse When many array operations have to be applied to the data of the structured array, it is better to keep a converted copy of the structured array around, instead of doing the conversion each time. Although, since it's a copy, I used it read only. For example calculating in sequence mean, variance and correlation, deviations from mean and so on, requires only one conversion. For me it would have been more useful to have better documentation and helper functions to convert structured arrays to standard arrays for the calculations. I looked at this mostly for the statistical use, where I didn't want the result to be structured arrays, so these recarrutil might not be of much use in this case, and consist to a large part of functionality that won't be needed, e.g. the multidimensional overhead. The recarray helper functions are useful and build in support as Stefan proposes would be nice. However, since I started only recently to us, I'm not sure what the relevant structure (dimensionality and dtypes) of structured/rec arrays are. But nested and mixed dtypes seem to be more important than multidimensionality in the examples I have seen. For example when we don't have a balanced panel so the structured array cannot be reshaped into a rectangular shape according to some variables, then reductions and operations like groupby are more useful for data analysis http://matplotlib.sourceforge.net/api/mlab_api.html#matplotlib.mlab.rec_groupby my 2c, after a brief look at the code Josef From Nicolas.Rougier at loria.fr Thu Jul 30 11:53:01 2009 From: Nicolas.Rougier at loria.fr (Nicolas Rougier) Date: Thu, 30 Jul 2009 17:53:01 +0200 Subject: [Numpy-discussion] Memory layout of record arrays Message-ID: Hello, I've been using record arrays to create arrays with different types and since I'm doing a lot of computation on each of the different fields, the default memory layout does not serve my computations. Ideally, I would like to have record arrays where each field is a contiguous block of memory. I searched the dtype documentation but did not find anything about it. In the meantime, I wrote a class that emulates this behavior, but it does not inherit from nd.array and I need inheritance. To sum up, I would like to have: >>> Z = np.zeros(3, dtype=[('x',np.float32), ('y',np.int)]) >>> print Z['x'].flags[['C_CONTIGUOUS'] False <= should be True >>> print Z['y'].flags[['C_CONTIGUOUS'] False <= should be True >>> Z.shape == Z['x'].shape == Z['y'].shape True Is there any obvious solution that I missed ? Nicolas From cournape at gmail.com Thu Jul 30 12:52:46 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 31 Jul 2009 01:52:46 +0900 Subject: [Numpy-discussion] Memory layout of record arrays In-Reply-To: References: Message-ID: <5b8d13220907300952w4e29b668v40972061e0a0db3d@mail.gmail.com> On Fri, Jul 31, 2009 at 12:53 AM, Nicolas Rougier wrote: > > > Hello, > > I've been using record arrays to create arrays with different types > and since I'm doing a lot of computation on each of the different > fields, the default memory layout does not serve my computations. > Ideally, I would like to have record arrays where each field is a > contiguous block of memory. I don't think you can do it with record arrays: one of the fundamental design choice of numpy array layout is that the data pointer points to one block of N items, where each item is described by the dtype. To have contiguous layout for each member of the structured dtype, you need two arrays with the corresponding dtype. cheers, David From npuloski at gmail.com Thu Jul 30 13:08:07 2009 From: npuloski at gmail.com (Nanime Puloski) Date: Thu, 30 Jul 2009 13:08:07 -0400 Subject: [Numpy-discussion] Internal Math Library of Numpy Message-ID: Does Numpy use Python's standard math library when calculating elementary functions such as exp(x) and acos(x)? Also, what is the internal library of Numpy and Python's standard math library? Are they platform independent? -------------- next part -------------- An HTML attachment was scrubbed... URL: From npuloski at gmail.com Thu Jul 30 13:14:48 2009 From: npuloski at gmail.com (Nanime Puloski) Date: Thu, 30 Jul 2009 13:14:48 -0400 Subject: [Numpy-discussion] Differences Between Arrays and Matrices in Numpy Message-ID: What are some differences between arrays and matrices using the Numpy library? When would one want to use arrays instead of matrices and vice versa? -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Jul 30 13:20:53 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 30 Jul 2009 12:20:53 -0500 Subject: [Numpy-discussion] Differences Between Arrays and Matrices in Numpy In-Reply-To: References: Message-ID: <3d375d730907301020m3e397ad0xd0adf4f0ccc7cdd8@mail.gmail.com> On Thu, Jul 30, 2009 at 12:14, Nanime Puloski wrote: > What are some differences between arrays and matrices using the Numpy > library? I have already pointed you to the relevant documentation on python-list. There really isn't a better overview. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nadavh at visionsense.com Thu Jul 30 13:26:51 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 30 Jul 2009 20:26:51 +0300 Subject: [Numpy-discussion] Differences Between Arrays and Matrices in Numpy References: Message-ID: <710F2847B0018641891D9A21602763605AD102@ex3.envision.co.il> Matrices are 2D array with some additional properties that make them more handy for linear algebra. For instance mat1*mat2 is a matrix multiplication, while arr1*arr2 is an element by element multiplication. However np.dot(arr1, arr2) is a matrix multiplication between two arrays. As a rule, you can replicate any matrix operation with arrays. If you do not do linear-algebra only calculation, you better use arrays. Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Nanime Puloski ????: ? 30-????-09 20:14 ??: numpy-discussion at scipy.org ????: [Numpy-discussion] Differences Between Arrays and Matrices in Numpy What are some differences between arrays and matrices using the Numpy library? When would one want to use arrays instead of matrices and vice versa? From Nicolas.Rougier at loria.fr Thu Jul 30 14:01:11 2009 From: Nicolas.Rougier at loria.fr (Nicolas Rougier) Date: Thu, 30 Jul 2009 20:01:11 +0200 Subject: [Numpy-discussion] Memory layout of record arrays In-Reply-To: <5b8d13220907300952w4e29b668v40972061e0a0db3d@mail.gmail.com> References: <5b8d13220907300952w4e29b668v40972061e0a0db3d@mail.gmail.com> Message-ID: <22B1DC1D-C934-465B-AB50-E8C5EADF127B@loria.fr> Thanks for the quick answer. It makes sense. I will have to find some other way to do it then. Nicolas On 30 Jul, 2009, at 18:52 , David Cournapeau wrote: > On Fri, Jul 31, 2009 at 12:53 AM, Nicolas > Rougier wrote: >> >> >> Hello, >> >> I've been using record arrays to create arrays with different types >> and since I'm doing a lot of computation on each of the different >> fields, the default memory layout does not serve my computations. >> Ideally, I would like to have record arrays where each field is a >> contiguous block of memory. > > I don't think you can do it with record arrays: one of the fundamental > design choice of numpy array layout is that the data pointer points to > one block of N items, where each item is described by the dtype. To > have contiguous layout for each member of the structured dtype, you > need two arrays with the corresponding dtype. > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From stevec at couttsnet.com Thu Jul 30 10:51:24 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Thu, 30 Jul 2009 14:51:24 +0000 (UTC) Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value Message-ID: I am getting this error when trying to run a script using scipy. Python 2.5 atlas-3.9.0 lapack-3.2 numpy-1.3.0 scipy-0.7.1 Anyone any ideas how I can fix this? Regards From fperez.net at gmail.com Thu Jul 30 14:32:28 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 30 Jul 2009 11:32:28 -0700 Subject: [Numpy-discussion] Reductions and binary ops on recarrays... In-Reply-To: <9457e7c80907300047y10948e03x116b8f1a35ca9e3a@mail.gmail.com> References: <9457e7c80907300047y10948e03x116b8f1a35ca9e3a@mail.gmail.com> Message-ID: 2009/7/30 St?fan van der Walt : > I'm in favour of such a patch, but I'd like to see whether we can't do > it at the C level for structured arrays in general. > That would indeed be ideal. But I should add that I was not proposing it as a patch, rather as a utility others might find useful to keep around. I simply don't have the bandwidth right now to develop this idea to the level where it can be pushed upstream, yet we find these tools very useful for our daily work, so I figured they could also be useful to someone else. Cheers, f From fperez.net at gmail.com Thu Jul 30 14:41:10 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 30 Jul 2009 11:41:10 -0700 Subject: [Numpy-discussion] Reductions and binary ops on recarrays... In-Reply-To: <1cd32cbb0907300755t37836328l2ddb904331e6c5c5@mail.gmail.com> References: <9457e7c80907300047y10948e03x116b8f1a35ca9e3a@mail.gmail.com> <1cd32cbb0907300755t37836328l2ddb904331e6c5c5@mail.gmail.com> Message-ID: On Thu, Jul 30, 2009 at 7:55 AM, wrote: > Are these functions really for a relevant use case of structured > arrays. I haven't seen any examples of multidimensional structured > arrays, but from a quick reading it doesn't seem to handle mixed types > (raises error) or nested structured arrays (I'm not sure), for which I > have seen a lot more examples. In our work, multidimensional record arrays with homogeneous types are a regular occurrence. This code was written for *our* problems, not to be completely general, and it was posted as "if it's useful to you, feel free to use it". I don't have the time/bandwidth to work on this idea for core numpy inclusion. Your other comments are all equally valid ideas, and all those would be necessary considerations for someone who wants to develop something like this to have full generality. Other things that would need to be done: - Proper support for broadcasting - mixed binary ops with scalars or normal arrays - masked array support Perhaps some enterprising soul will come back later with a robust implementation of all this... But I'm afraid that won't be me :) This discussion could serve as a good starting point of simple code and points to keep in mind for such a task, so thanks for the feedback. Cheers, f From josef.pktd at gmail.com Thu Jul 30 14:56:58 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 30 Jul 2009 14:56:58 -0400 Subject: [Numpy-discussion] Reductions and binary ops on recarrays... In-Reply-To: References: <9457e7c80907300047y10948e03x116b8f1a35ca9e3a@mail.gmail.com> <1cd32cbb0907300755t37836328l2ddb904331e6c5c5@mail.gmail.com> Message-ID: <1cd32cbb0907301156u7d7bee88jf390d9b8180edf46@mail.gmail.com> On Thu, Jul 30, 2009 at 2:41 PM, Fernando Perez wrote: > On Thu, Jul 30, 2009 at 7:55 AM, wrote: >> Are these functions really for a relevant use case of structured >> arrays. I haven't seen any examples of multidimensional structured >> arrays, but from a quick reading it doesn't seem to handle mixed types >> (raises error) or nested structured arrays (I'm not sure), for which I >> have seen a lot more examples. > > In our work, multidimensional record arrays with homogeneous types are > a regular occurrence. ?This code was written for *our* problems, not > to be completely general, and it was posted as "if it's useful to you, > feel free to use it". ?I don't have the time/bandwidth to work on this > idea for core numpy inclusion. > > Your other comments are all equally valid ideas, and all those would > be necessary considerations for someone who wants to develop something > like this to have full generality. ?Other things that would need to be > done: > > - Proper support for broadcasting > - mixed binary ops with scalars or normal arrays > - masked array support > > Perhaps some enterprising soul will come back later with a robust > implementation of all this... ?But I'm afraid that won't be me :) > This discussion could serve as a good starting point of simple code > and points to keep in mind for such a task, so thanks for the > feedback. > > Cheers, > > f Thanks for the example. I tried to read through the code of the different recarray utility functions to see how you actually use structured arrays for data analysis, instead of just as a storage dictionary. Code examples are the most useful documentation that I have found for this. Josef From bbands at gmail.com Thu Jul 30 15:13:23 2009 From: bbands at gmail.com (BBands) Date: Thu, 30 Jul 2009 12:13:23 -0700 Subject: [Numpy-discussion] Python 3.n and Scipy Numpy Message-ID: <6e8360ad0907301213p7ee36e58oee573eab21544cdc@mail.gmail.com> Could someone point me toward some information on Scipy/Numpy and Python 3.1? I'd like to upgrade, but can't seem to find the path. jab From nmb at wartburg.edu Thu Jul 30 15:33:24 2009 From: nmb at wartburg.edu (Neil Martinsen-Burrell) Date: Thu, 30 Jul 2009 14:33:24 -0500 Subject: [Numpy-discussion] Python 3.n and Scipy Numpy In-Reply-To: <6e8360ad0907301213p7ee36e58oee573eab21544cdc@mail.gmail.com> References: <6e8360ad0907301213p7ee36e58oee573eab21544cdc@mail.gmail.com> Message-ID: <4A71F584.8030108@wartburg.edu> On 2009-07-30 14:13 , BBands wrote: > Could someone point me toward some information on Scipy/Numpy and > Python 3.1? I'd like to upgrade, but can't seem to find the path. There is no such path right now. Numpy and Scipy are only available for Python 2.x (with x >= 4). As very large C extensions, the work of porting Numpy and Scipy to Python 3.x hasn't been undertaken, although it will be in time. If you have a particular situation in which you need to upgrade, please let us know more about it, so that the NumPy developers can target their porting efforts appropriately. -Neil From rmay31 at gmail.com Thu Jul 30 15:34:07 2009 From: rmay31 at gmail.com (Ryan May) Date: Thu, 30 Jul 2009 14:34:07 -0500 Subject: [Numpy-discussion] Python 3.n and Scipy Numpy In-Reply-To: <6e8360ad0907301213p7ee36e58oee573eab21544cdc@mail.gmail.com> References: <6e8360ad0907301213p7ee36e58oee573eab21544cdc@mail.gmail.com> Message-ID: On Thu, Jul 30, 2009 at 2:13 PM, BBands wrote: > Could someone point me toward some information on Scipy/Numpy and > Python 3.1? I'd like to upgrade, but can't seem to find the path. Scipy/Numpy are have not yet been ported to python 3.x Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma Sent from Norman, Oklahoma, United States -------------- next part -------------- An HTML attachment was scrubbed... URL: From Klaus.Noekel at gmx.de Thu Jul 30 15:34:44 2009 From: Klaus.Noekel at gmx.de (Klaus Noekel) Date: Thu, 30 Jul 2009 21:34:44 +0200 Subject: [Numpy-discussion] Timeline for 1.4.0 and installer for Windows 64bit ? Message-ID: <4A71F5D4.10601@gmx.de> Dear folks, just over a month ago there was a thread about plans for numpy, and IIRC somebody had volunteered to try and put together a working AMD64 version with an installer. Since then I have not heard about the issue again - but I may have missed part of the discussion. Have the plans firmed up by now? Is there a tentative date for a beta or RC? In particular: how much hope is there for a reasonably usable AMD64 numpy under Windows? Thanks for any update! Klaus N?kel From eadrogue at gmx.net Thu Jul 30 15:40:03 2009 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Thu, 30 Jul 2009 21:40:03 +0200 Subject: [Numpy-discussion] question about a data structure In-Reply-To: <4A70C510.4070407@noaa.gov> References: <20090729213639.GA10538@doriath.local> <3d375d730907291439q7442d92am7fd1824a6fc63681@mail.gmail.com> <4A70C510.4070407@noaa.gov> Message-ID: <20090730194003.GA16709@doriath.local> 29/07/09 @ 14:54 (-0700), thus spake Christopher Barker: > Robert Kern wrote: > > 2009/7/29 Ernest Adrogu? : > >> Now, I need to be able to differentiate between 0 and 'no data'. > > >> Is it possible to do this with the standard array class? > > > > Not really. Use masked arrays. Or use a dictionary. > > masked arrays are made precisely for this. However, if you can use > floating point numbers, you could use NaN for the no data entries. Thanks for the advice. Actually, I think I'll use a combination of a dictionary and a masked array. I have been playing around a bit and it doesn't seem too difficult to do. It works by adding an extra property to the array objects, which is a tuple with the "keys", and then use the __getitem__ and __setitem__ methods to translate the keys into numbers. This way it's possible to use strings in subscripts and slices. Ernest From bbands at gmail.com Thu Jul 30 15:50:12 2009 From: bbands at gmail.com (BBands) Date: Thu, 30 Jul 2009 12:50:12 -0700 Subject: [Numpy-discussion] Python 3.n and Scipy Numpy In-Reply-To: <4A71F584.8030108@wartburg.edu> References: <6e8360ad0907301213p7ee36e58oee573eab21544cdc@mail.gmail.com> <4A71F584.8030108@wartburg.edu> Message-ID: <6e8360ad0907301250k2f97c14fvb3a1d43314fb0a37@mail.gmail.com> On Thu, Jul 30, 2009 at 12:33 PM, Neil Martinsen-Burrell wrote: > On 2009-07-30 14:13 , BBands wrote: >> Could someone point me toward some information on Scipy/Numpy and >> Python 3.1? I'd like to upgrade, but can't seem to find the path. > > There is no such path right now. ?Numpy and Scipy are only available for > Python 2.x (with x >= 4). ?As very large C extensions, the work of > porting Numpy and Scipy to Python 3.x hasn't been undertaken, although > it will be in time. ?If you have a particular situation in which you > need to upgrade, please let us know more about it, so that the NumPy > developers can target their porting efforts appropriately. Thanks, I'll step forward to 2.6.2 and stop there for now. (No particular needs now. I'll have to see how this upgrade goes first.) Best regards, jab From pav+sp at iki.fi Thu Jul 30 15:56:20 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Thu, 30 Jul 2009 19:56:20 +0000 (UTC) Subject: [Numpy-discussion] Internal Math Library of Numpy References: Message-ID: On 2009-07-30, Nanime Puloski wrote: > Does Numpy use Python's standard math library when calculating > elementary functions such as exp(x) and acos(x)? No. > Also, what is the internal library of Numpy and Python's > standard math library? The standard C library, plus custom implementations for functions that are missing in it. > Are they platform independent? In practice, yes. There are some possible obscure corner cases currently in complex-valued inf/nan handling, though. -- Pauli Virtanen From wright at esrf.fr Thu Jul 30 16:54:22 2009 From: wright at esrf.fr (Jon Wright) Date: Thu, 30 Jul 2009 22:54:22 +0200 Subject: [Numpy-discussion] Timeline for 1.4.0 and installer for Windows 64bit ? In-Reply-To: <4A71F5D4.10601@gmx.de> References: <4A71F5D4.10601@gmx.de> Message-ID: <4A72087E.4090200@esrf.fr> This one works for me with python2.6 for amd64 from python.org: http://sourceforge.net/projects/numpy/files/NumPy/1.3.0/numpy-1.3.0.win-amd64-py2.6.msi/download Thanks to David C. I believe Klaus Noekel wrote: > Dear folks, > > just over a month ago there was a thread about plans for numpy, and IIRC > somebody had volunteered to try and put together a working AMD64 version > with an installer. > > Since then I have not heard about the issue again - but I may have > missed part of the discussion. Have the plans firmed up by now? Is there > a tentative date for a beta or RC? In particular: how much hope is there > for a reasonably usable AMD64 numpy under Windows? > > Thanks for any update! > > Klaus N?kel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From amenity at enthought.com Thu Jul 30 19:48:18 2009 From: amenity at enthought.com (Amenity Applewhite) Date: Thu, 30 Jul 2009 18:48:18 -0500 Subject: [Numpy-discussion] EPD Webinar August 7: Speed up Python (& wait list for non-subscribers!) Message-ID: Friday, August 7th 1pm CDT How do I...speed up Python? Exploration of Weave and Cython We wanted to let you know that next week we'll host another installment of our popular EPD webinar series. Although only EPD Basic or above subscribers are guaranteed seats at EPD webinars, we invite non-subscribers to add their names to the wating list for each event. If there are available seats, you will be notified by next Thursday and given access to the webinar. Links to the waiting lists and upcoming topics are posted here: http://enthought.com/training/webinars.php These events feature detailed demonstrations of powerful Python techniques that Enthought developers use to enhance our applications or development process. Participants are often invited to participate in the demonstration, and are welcome to join the interactive VOIP discussion later in the session. This is a great opportunity to learn new methods and interact with our expert developers. If you have topics you'd like to see addressed during the webinar, feel free to let us know at media at enthought.com. How do I...speed up Python: Exploration of Weave and Cython Python excels as a language for rapid development and interactive technical computing. Sometimes, however, the full speed of the machine is needed. Fortunately, Python is also an excellent "glue" language that allows simple integration to compiled languages. In this webinar, we will present a few of the methods useful for speeding up sections of code. In particular we will focus on weave and Cython. Weave is a sub-package of SciPy (scipy.weave) and allows seamless integration with C/C++. Sections of code that need to be sped up can be re-coded in C and seamlessly integrated into the remaining Python program using weave. Cython is a newer Python-like language which is compiled to an extension module that can be dynamically loaded into Python just like any other Python module. Cython also supports writing new built-in Python types and tries to compile loops to C-speeds when variable types are indicated. Both weave and Cython integrate well with NumPy which makes them ideal whenever you need to iterate and calculate with large amounts of data and can't find a built-in (vectorized) approach. Once again, to add your name to the wait-list: http://enthought.com/training/webinars.php We hope to see you there! Thanks, Enthought Media Quick Links... Enthought.com Enthought Python Distribution (EPD) Enthought Webinars @Facebook @Twitter -- Amenity Applewhite Enthought, Inc. Scientific Computing Solutions www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Jul 30 23:15:34 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 31 Jul 2009 12:15:34 +0900 Subject: [Numpy-discussion] Timeline for 1.4.0 and installer for Windows 64bit ? In-Reply-To: <4A71F5D4.10601@gmx.de> References: <4A71F5D4.10601@gmx.de> Message-ID: <4A7261D6.5070409@ar.media.kyoto-u.ac.jp> Hi Klaus, Klaus Noekel wrote: > Dear folks, > > just over a month ago there was a thread about plans for numpy, and IIRC > somebody had volunteered to try and put together a working AMD64 version > with an installer. > > Since then I have not heard about the issue again - but I may have > missed part of the discussion. Have the plans firmed up by now? Is there > a tentative date for a beta or RC? In particular: how much hope is there > for a reasonably usable AMD64 numpy under Windows? > There were some discussion about pushing 1.4.0 'early', but instead, I think we let it slipped - one consequence is that there will be enough time for 1.4.0 to be released with proper AMD64 support on windows. The real issue is not numpy per-se, but making scipy work on top of numpy in 64 bits mode. It is hard to give an exact date as to when those issues will be fixed, but it is being worked on. cheers, David From stevec at couttsnet.com Fri Jul 31 04:05:32 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 08:05:32 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?**_On_entry_to_ILAENV_parameter_numb?= =?utf-8?q?er_2_had_an=09illegal_value?= References: Message-ID: Steven Coutts couttsnet.com> writes: > > I am getting this error when trying to run a script using scipy. > > Python 2.5 > > atlas-3.9.0 > lapack-3.2 > numpy-1.3.0 > scipy-0.7.1 > > Anyone any ideas how I can fix this? > > Regards > Also tried Atlas 3.9.11 and 3.8.3 with no luck. Anyone got any ideas? Regards From david at ar.media.kyoto-u.ac.jp Fri Jul 31 04:07:50 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 31 Jul 2009 17:07:50 +0900 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value In-Reply-To: References: Message-ID: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> Steven Coutts wrote: > Steven Coutts couttsnet.com> writes: > > >> I am getting this error when trying to run a script using scipy. >> >> Python 2.5 >> >> atlas-3.9.0 >> lapack-3.2 >> numpy-1.3.0 >> scipy-0.7.1 >> >> Anyone any ideas how I can fix this? >> Lapack 3.2 is problematic, so I would try to downgrade to 3.1.1 first. Which OS are you on ? The exact core you are running, as well as the output of the test suite (numpy.test()) would be helpful, cheers, David From stevec at couttsnet.com Fri Jul 31 04:54:19 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 09:54:19 +0100 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> Message-ID: <1249030460.336856@nntpgw.ncl.ac.uk> David Cournapeau wrote: > > Lapack 3.2 is problematic, so I would try to downgrade to 3.1.1 first. > Which OS are you on ? The exact core you are running, as well as the > output of the test suite (numpy.test()) would be helpful, > > cheers, > > David Ok, I have downgraded to lapack 3.1.1 and re-compiled and re-installed everything. Numpy test seg faults :( http://pastebin.com/d56fd6e44 This is on Linux (CentOS) and a Quad-Core AMD Opteron(tm) Processor 2350 HE Regards From david at ar.media.kyoto-u.ac.jp Fri Jul 31 05:06:21 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 31 Jul 2009 18:06:21 +0900 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value In-Reply-To: <1249030460.336856@nntpgw.ncl.ac.uk> References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> Message-ID: <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> Steven Coutts wrote: > David Cournapeau wrote: > >> Lapack 3.2 is problematic, so I would try to downgrade to 3.1.1 first. >> Which OS are you on ? The exact core you are running, as well as the >> output of the test suite (numpy.test()) would be helpful, >> >> cheers, >> >> David >> > > Ok, I have downgraded to lapack 3.1.1 and re-compiled and re-installed > everything. > > Numpy test seg faults :( http://pastebin.com/d56fd6e44 > Not sure it can be considered as progress :) Could you run the test suite with verbosity set to ten (numpy.test(verbose=10)) ? Also, which fortran compiler are you using ? g77 or gfortran ? cheers, David From stevec at couttsnet.com Fri Jul 31 05:39:58 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 10:39:58 +0100 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> Message-ID: <1249033198.686107@nntpgw.ncl.ac.uk> David Cournapeau wrote: > > Not sure it can be considered as progress :) Could you run the test > suite with verbosity set to ten (numpy.test(verbose=10)) ? > > Also, which fortran compiler are you using ? g77 or gfortran ? > > cheers, > > David Ok, http://pastebin.com/ddaae526 Seg faults here -: test_cdouble (test_linalg.TestEigh) I am using gfortran. Regards From stevec at couttsnet.com Fri Jul 31 06:45:11 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 11:45:11 +0100 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> Message-ID: <1249037111.723816@nntpgw.ncl.ac.uk> Steven Coutts wrote: > > Ok, http://pastebin.com/ddaae526 > > Seg faults here -: > > test_cdouble (test_linalg.TestEigh) > > I am using gfortran. > > Regards Ok, downgraded numpy to 1.2.1 and all the tests pass now! Regards From stevec at couttsnet.com Fri Jul 31 07:03:13 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 11:03:13 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?**_On_entry_to_ILAENV_parameter_numb?= =?utf-8?q?er_2_had=09an_illegal_value?= References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> Message-ID: Steven Coutts couttsnet.com> writes: > > Ok, downgraded numpy to 1.2.1 and all the tests pass now! > > Regards > But now scipy seg faults straight away http://pastebin.com/de13dd62 Downgraded scipy to 0.7.0 and still the same seg fault :( From stevec at couttsnet.com Fri Jul 31 07:10:29 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 11:10:29 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?**_On_entry_to_ILAENV_parameter_numb?= =?utf-8?q?er_2_had=09an_illegal_value?= References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> Message-ID: Steven Coutts couttsnet.com> writes: > > But now scipy seg faults straight away > > http://pastebin.com/de13dd62 > > Downgraded scipy to 0.7.0 and still the same seg fault :( > Downgrading all the way back to scipy-0.6.0 makes scipy work. Any ideas why no newer version work? Regards From david at ar.media.kyoto-u.ac.jp Fri Jul 31 06:54:06 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 31 Jul 2009 19:54:06 +0900 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value In-Reply-To: <1249037111.723816@nntpgw.ncl.ac.uk> References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> Message-ID: <4A72CD4E.7000606@ar.media.kyoto-u.ac.jp> Steven Coutts wrote: > > Ok, downgraded numpy to 1.2.1 and all the tests pass now! > That's really strange - Linux is the most tested configuration, numpy 1.3.0 should run without problems. There is something unusual with your configuration that I am missing. Could you build numpy 1.3.0 from scratch and paste the output (rm -rf build && python setup.py build &> build.log) ? If you are willing to do it, I would also be interested whether numpy works ok if linked against BLAS/LAPACK instead of atlas (i.e. build numpy, again from scratch, with ATLAS=None python setup.py build, and then run the test suite). cheers, David From david at ar.media.kyoto-u.ac.jp Fri Jul 31 06:55:24 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 31 Jul 2009 19:55:24 +0900 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value In-Reply-To: References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> Message-ID: <4A72CD9C.1040803@ar.media.kyoto-u.ac.jp> Steven Coutts wrote: > Steven Coutts couttsnet.com> writes: > > >> Ok, downgraded numpy to 1.2.1 and all the tests pass now! >> >> Regards >> >> > > But now scipy seg faults straight away > > http://pastebin.com/de13dd62 > > Downgraded scipy to 0.7.0 and still the same seg fault :( > Make sure your scipy is built against numpy 1.2.1. You cannot build scipy against numpy 1.3.0, and then use scipy with numpy 1.2.1 (but you can build against numpy 1.2.1 and use scipy with numpy 1.3.0). cheers, David > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > From stevec at couttsnet.com Fri Jul 31 07:26:41 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 11:26:41 +0000 (UTC) Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> <4A72CD9C.1040803@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: > > Make sure your scipy is built against numpy 1.2.1. You cannot build > scipy against numpy 1.3.0, and then use scipy with numpy 1.2.1 (but you > can build against numpy 1.2.1 and use scipy with numpy 1.3.0). > > cheers, > > David Yes it was built against the correct version of Numpy (I triple checked this!) Regards From dineshbvadhia at hotmail.com Fri Jul 31 07:29:57 2009 From: dineshbvadhia at hotmail.com (Dinesh B Vadhia) Date: Fri, 31 Jul 2009 04:29:57 -0700 Subject: [Numpy-discussion] Timeline for 1.4.0 and installer for Windows64bit ? In-Reply-To: <4A7261D6.5070409@ar.media.kyoto-u.ac.jp> References: <4A71F5D4.10601@gmx.de> <4A7261D6.5070409@ar.media.kyoto-u.ac.jp> Message-ID: A suggestion: How about releasing Numpy for the AMD64 version first (without Scipy) and then follow up with a later release with Scipy support? This would satisfy Numpy-only users which can't be a bad thing rather than having a version that is not usable (I believe) by either Numpy or Scipy users. Dinesh From: David Cournapeau Sent: Thursday, July 30, 2009 8:15 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Timeline for 1.4.0 and installer for Windows64bit ? Hi Klaus, Klaus Noekel wrote: > Dear folks, > > just over a month ago there was a thread about plans for numpy, and IIRC > somebody had volunteered to try and put together a working AMD64 version > with an installer. > > Since then I have not heard about the issue again - but I may have > missed part of the discussion. Have the plans firmed up by now? Is there > a tentative date for a beta or RC? In particular: how much hope is there > for a reasonably usable AMD64 numpy under Windows? > There were some discussion about pushing 1.4.0 'early', but instead, I think we let it slipped - one consequence is that there will be enough time for 1.4.0 to be released with proper AMD64 support on windows. The real issue is not numpy per-se, but making scipy work on top of numpy in 64 bits mode. It is hard to give an exact date as to when those issues will be fixed, but it is being worked on. cheers, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From stevec at couttsnet.com Fri Jul 31 07:32:05 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 11:32:05 +0000 (UTC) Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> <4A72CD4E.7000606@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: > > That's really strange - Linux is the most tested configuration, numpy > 1.3.0 should run without problems. There is something unusual with your > configuration that I am missing. > > Could you build numpy 1.3.0 from scratch and paste the output (rm -rf > build && python setup.py build &> build.log) ? If you are willing to do > it, I would also be interested whether numpy works ok if linked against > BLAS/LAPACK instead of atlas (i.e. build numpy, again from scratch, with > ATLAS=None python setup.py build, and then run the test suite). > Ok build.log is here http://pastebin.com/d5eb5dc20 I'll try building against BLAS/LAPACK after lunch. Regards From david at ar.media.kyoto-u.ac.jp Fri Jul 31 07:12:55 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 31 Jul 2009 20:12:55 +0900 Subject: [Numpy-discussion] Timeline for 1.4.0 and installer for Windows64bit ? In-Reply-To: References: <4A71F5D4.10601@gmx.de> <4A7261D6.5070409@ar.media.kyoto-u.ac.jp> Message-ID: <4A72D1B7.70309@ar.media.kyoto-u.ac.jp> Dinesh B Vadhia wrote: > A suggestion: > > How about releasing Numpy for the AMD64 version first (without Scipy) > and then follow up with a later release with Scipy support? This > would satisfy Numpy-only users which can't be a bad thing rather than > having a version that is not usable (I believe) by either Numpy or > Scipy users. Because scipy needs code I would pull out from numpy. You can build numpy for amd64 by yourself at the moment, BTW. I am afraid that distributing one numpy binary now would only makes things more confusing when scipy for amd64 will be out, as it will not work with this numpy, but only with a later one. cheers, David From stevec at couttsnet.com Fri Jul 31 07:37:49 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 11:37:49 +0000 (UTC) Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> <4A72CD4E.7000606@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: If you are willing to do > it, I would also be interested whether numpy works ok if linked against > BLAS/LAPACK instead of atlas (i.e. build numpy, again from scratch, with > ATLAS=None python setup.py build, and then run the test suite). > Yes that appears to work fine, all tests run. Regards From david at ar.media.kyoto-u.ac.jp Fri Jul 31 07:20:03 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 31 Jul 2009 20:20:03 +0900 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value In-Reply-To: References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> <4A72CD4E.7000606@ar.media.kyoto-u.ac.jp> Message-ID: <4A72D363.8090603@ar.media.kyoto-u.ac.jp> Steven Coutts wrote: > David Cournapeau ar.media.kyoto-u.ac.jp> writes: > > If you are willing to do > >> it, I would also be interested whether numpy works ok if linked against >> BLAS/LAPACK instead of atlas (i.e. build numpy, again from scratch, with >> ATLAS=None python setup.py build, and then run the test suite). >> >> > > Yes that appears to work fine, all tests run. > So that's a problem with ATLAS. Maybe a gcc bug ? Another user contacted me privately for my rpm repository, and got exactly the same problem with the rpms, on CENTOS 5.3 as well. I will try to look at it on a centos VM if I have time this WE, cheers, David From stevec at couttsnet.com Fri Jul 31 07:48:15 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 11:48:15 +0000 (UTC) Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> <4A72CD4E.7000606@ar.media.kyoto-u.ac.jp> <4A72D363.8090603@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: > > So that's a problem with ATLAS. Maybe a gcc bug ? Another user contacted > me privately for my rpm repository, and got exactly the same problem > with the rpms, on CENTOS 5.3 as well. I will try to look at it on a > centos VM if I have time this WE, > I think the user who contacted you is the person I am installing this for, he said he had mailed you:) As I said I'm back on numpy 1.2.1 and scipy-0.6.0, anything above that just seg faults or gets the ILAENV error. Think I must have tried nearly every possible combination of versions of packages (including atlas and lapack)! But it's running so it's not a biggy for now. If this were my box I'd give you access to have a bit of a look at, but unfortunately it's not mine. :) Regards From reedev at zonnet.nl Fri Jul 31 07:51:44 2009 From: reedev at zonnet.nl (Raymond de Vries) Date: Fri, 31 Jul 2009 13:51:44 +0200 Subject: [Numpy-discussion] (newbie) How can I use NumPy to wrap my C++ class with 2-dimensional arrays? In-Reply-To: References: <4A7140A8.2040305@zonnet.nl> <4A715F40.1060903@zonnet.nl> <4A7162C4.8030605@zonnet.nl> <4A7173A4.6070608@zonnet.nl> Message-ID: <4A72DAD0.7040601@zonnet.nl> Hi Matthieu, Thanks for the explanation. After having looked at the documentation, I decided to do my own plain Python c-api implementation. Thanks for your time Raymond Matthieu Brucher wrote: > 2009/7/30 Raymond de Vries : > >> Hi >> >> >>> Indeed, the solution is as simple as this ;) The trouble is to find >>> the information! >>> >>> >> Yes, there is so much information everywhere. And it's hard to make the >> first steps. >> >>>>>> For the std::vector<>[], I suggest you convert it to a single vector, >>>>>> as the data inside this "array" is not contiguous and it can thus be >>>>>> cumbersome to create a Numpy array from that. >>>>>> >>>>>> >> I am now ready to do this. To be certain, 'contiguous' means that the >> std::vector<>'s are not the same length, right? Would that mean that I'd >> better use a tuple of lists or so? (or list of lists or so). >> >> thanks for your time >> Raymond >> > > Contiguous means that the whole data is in one big chunk. If it is > not, you have to rely on other Numpy functions (I don't know all of > them, perhaps you will find one that satisfies your need), and the > data may then be copied (not sure though). > > Matthieu > From Nicolas.Rougier at loria.fr Fri Jul 31 10:04:29 2009 From: Nicolas.Rougier at loria.fr (Nicolas Rougier) Date: Fri, 31 Jul 2009 16:04:29 +0200 Subject: [Numpy-discussion] Memory layout of record arrays In-Reply-To: <22B1DC1D-C934-465B-AB50-E8C5EADF127B@loria.fr> References: <5b8d13220907300952w4e29b668v40972061e0a0db3d@mail.gmail.com> <22B1DC1D-C934-465B-AB50-E8C5EADF127B@loria.fr> Message-ID: <1249049069.7783.10.camel@sulfur.loria.fr> I've cooked a very rudimentary implementation of what I would like to have, however I've got a small problem concerning array shape. The idea is to have a parent array (a group) that can be instantiated like a regular array and that build internally all the relevant contiguous child arrays. I would like those child arrays to not be reshapable (at least not without telling their parent first) and I naively overloaded the shape property but got a lot of problems with infinite recursion. Is there a way to do that ? Nicolas I put the code below (around 100 lines): class array(np.ndarray): def __new__(subtype, shape=(1,1), dtype=np.float32, order=None, group=None): obj = np.ndarray.__new__(subtype, shape=shape, dtype=dtype, order=order) obj._group = group return obj def _get_group(self): return self._group or self def _set_group(self, group): if self.size == group.size: self.shape = group.shape self._group = group else: raise ValueError, \ 'shape mismatch: objects cannot be broadcast to a single shape' group = property(_get_group, _set_group, doc = '''Group to which this array belongs to''') def _get_shape(self): return self.ctypes.shape def _set_shape(self, shape): if self.group == None: self.ctypes.shape = shape else: raise AttributeError, \ '''Cannot reshape a child array (''group'' is not None)''' # shape = property(_get_shape, _set_shape, doc='''c-types shape''') class group(object): def __init__(self, shape, dtype=None, order=None): object.__setattr__(self,'_arrays', {}) self._shape = shape self._dtype = dtype if len(dtype) == 0: self._dtype = np.dtype('f0',dtype) for i in range(len(self._dtype)): name, dtype = self._dtype[i] self._arrays[name] = array(shape=shape,dtype=dtype,order=order) def __getattr__(self, key): if key in self._arrays.keys(): return self._arrays[key] else: return object.__getattribute__(self, key) def __setattr__(self, key, value): if key in self._arrays.keys(): self._arrays[key][...] = value else: object.__setattr__(self, key, value) def __getitem__(self, key): return self._arrays[key] def __setitem__(self, key, value): self._arrays[key][...] = value def __len__(self): return len(self._arrays[self._arrays.keys()[0]]) def _get_shape(self): return self._shape def _set_shape(self, shape): for key in self._arrays.keys(): self._arrays[key].shape = shape self._shape = shape shape = property(_get_shape, _set_shape) def _get_dtype(self): return self._dtype def _set_dtype(self): raise AttributeError, \ '''attribute 'dtype' of 'group' objects is not writable''' dtype = property(_get_dtype, _set_dtype) def _get_size(self): return self._arrays[self._arrays.keys()[0]].size def _set_size(self): raise AttributeError, \ '''attribute 'size' of 'group' objects is not writable''' size = property(_get_size, _set_size) def __repr__(self): s = 'group(\n' for i in range(len(self._dtype)): name,dtype = self._dtype[i] t = "'%s': " % name a = repr(self._arrays[name]).replace('\n', '\n' +' '*len(t)) s += t+a+'\n' s += ')' return s if __name__ == '__main__': G = group((3,3), dtype = [('r',np.float32),('g',np.int32),('b',np.bool)]) G['r'] = G.g = G.b = 0 print G print G.r.shape = (9,1) print G.r.shape print G.shape On Thu, 2009-07-30 at 20:01 +0200, Nicolas Rougier wrote: > > Thanks for the quick answer. It makes sense. > I will have to find some other way to do it then. > > Nicolas > > > On 30 Jul, 2009, at 18:52 , David Cournapeau wrote: > > > On Fri, Jul 31, 2009 at 12:53 AM, Nicolas > > Rougier wrote: > >> > >> > >> Hello, > >> > >> I've been using record arrays to create arrays with different types > >> and since I'm doing a lot of computation on each of the different > >> fields, the default memory layout does not serve my computations. > >> Ideally, I would like to have record arrays where each field is a > >> contiguous block of memory. > > > > I don't think you can do it with record arrays: one of the fundamental > > design choice of numpy array layout is that the data pointer points to > > one block of N items, where each item is described by the dtype. To > > have contiguous layout for each member of the structured dtype, you > > need two arrays with the corresponding dtype. > > > > cheers, > > > > David > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Fri Jul 31 10:55:20 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 31 Jul 2009 08:55:20 -0600 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value In-Reply-To: References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> <4A72CD4E.7000606@ar.media.kyoto-u.ac.jp> <4A72D363.8090603@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Jul 31, 2009 at 5:48 AM, Steven Coutts wrote: > David Cournapeau ar.media.kyoto-u.ac.jp> writes: > > > > So that's a problem with ATLAS. Maybe a gcc bug ? Another user contacted > > me privately for my rpm repository, and got exactly the same problem > > with the rpms, on CENTOS 5.3 as well. I will try to look at it on a > > centos VM if I have time this WE, > > > > I think the user who contacted you is the person I am installing this for, > he > said he had mailed you:) > > As I said I'm back on numpy 1.2.1 and scipy-0.6.0, anything above that just > seg > faults or gets the ILAENV error. Think I must have tried nearly every > possible > combination of versions of packages (including atlas and lapack)! But it's > running so it's not a biggy for now. > > If this were my box I'd give you access to have a bit of a look at, but > unfortunately it's not mine. :) > Sounds like this problem might depend on the specific combination of compiler, hardware, and Atlas. Could you tell us which amd chip and compiler versions you are using? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stevec at couttsnet.com Fri Jul 31 11:06:08 2009 From: stevec at couttsnet.com (Steven Coutts) Date: Fri, 31 Jul 2009 16:06:08 +0100 Subject: [Numpy-discussion] ** On entry to ILAENV parameter number 2 had an illegal value References: <4A72A656.7040802@ar.media.kyoto-u.ac.jp> <1249030460.336856@nntpgw.ncl.ac.uk> <4A72B40D.7080004@ar.media.kyoto-u.ac.jp> <1249033198.686107@nntpgw.ncl.ac.uk> <1249037111.723816@nntpgw.ncl.ac.uk> <4A72CD4E.7000606@ar.media.kyoto-u.ac.jp> <4A72D363.8090603@ar.media.kyoto-u.ac.jp> Message-ID: <1249052768.732591@nntpgw.ncl.ac.uk> Charles R Harris wrote: > > Sounds like this problem might depend on the specific combination of > compiler, hardware, and Atlas. Could you tell us which amd chip and > compiler versions you are using? > > Chuck processor : 0 vendor_id : AuthenticAMD cpu family : 16 model : 2 model name : Quad-Core AMD Opteron(tm) Processor 2350 HE stepping : 3 cpu MHz : 1994.999 cache size : 512 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu de tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good pni cx16 popcnt lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch bogomips : 3999.28 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ts ttp tm stc 100mhzsteps hwpstate gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-44) GNU Fortran (GCC) 4.1.2 20080704 (Red Hat 4.1.2-44) From bbands at gmail.com Fri Jul 31 12:49:58 2009 From: bbands at gmail.com (BBands) Date: Fri, 31 Jul 2009 09:49:58 -0700 Subject: [Numpy-discussion] Python 3.n and Scipy Numpy In-Reply-To: <6e8360ad0907301250k2f97c14fvb3a1d43314fb0a37@mail.gmail.com> References: <6e8360ad0907301213p7ee36e58oee573eab21544cdc@mail.gmail.com> <4A71F584.8030108@wartburg.edu> <6e8360ad0907301250k2f97c14fvb3a1d43314fb0a37@mail.gmail.com> Message-ID: <6e8360ad0907310949j57cc4e47s6bd352a34e8b4e8c@mail.gmail.com> A short field report. Python 2.6.2 installed and working with the latest versions of: numpy scipy MySQLdb gnuplot.py and gnuplot rpy and R 2.9.1 PyScripter PyWin32 Rpyc 2.6 The little bit of 3.n syntax I have tried works. So far, this seems like a stable and trouble-free stack. Thanks for the fine work, jab -- John Bollinger, CFA, CMT www.BollingerBands.com If you advance far enough, you arrive at the beginning. From jh at physics.ucf.edu Fri Jul 31 13:06:37 2009 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 31 Jul 2009 13:06:37 -0400 Subject: [Numpy-discussion] SciPy Foundation Message-ID: About sixteen months ago, I launched the SciPy Documentation Project and its Marathon. Dozens pitched in and now numpy docs are rapidly approaching a professional level. The "pink wave" ("Needs Review" status) is at 56% today! There is consensus among doc writers that much of the rest can be labeled in the "unimportant" category, so we're close to starting the review push (hold your fire, there is a web site mod to be done first). We're also nearing the end of the summer, and it's time to look ahead. The path for docs is clear, but the path for SciPy is not. I think our weakest area right now is organization of the project. There is no consensus-based plan for improvement of the whole toward a stated goal, no centralized coordination of work, and no funded work focused on many of our weaknesses, notwithstanding my doc effort and what Enthought does for code. I define success as popular adoption in preference to commercial packages. I believe in vote-with-your-feet: this goal will not be reached until all aspects of the package and its presentation to the world exceed those of our commercial competition. Scipy is now a grass roots effort, but that takes it only so far. Other projects, such as OpenOffice and Sage, don't follow this model and do produce quality products that compete with commercial offerings, at least on open-source platforms. Before we can even hope for that, we have to do the following: - Docs - Rest of numpy reference pages reviewed and proofed or marked unimportant - Scipy reference pages - User manual for the whole toolstack - Multiple commercial books - Packaging - Personal Package Archive or equivalent for every release of every OS for the full toolstack (There are tools that do this but we don't use them. NSF requires Metronome - http://nmi.cs.wisc.edu/ - for funding most development grants, so right now we're not even on NSF's radar.) - Track record of having the whole toolstack installation "just work" in a few command lines or clicks for *everyone* - Regular, scheduled releases of numpy and scipy - Coordinated releases of numpy, scipy, and stable scikits into PPA system - Public communication - A real marketing plan - Executing on that plan - Web site geared toward multiple audiences, run by experts at that kind of communication - More webinars, conference booths, training, aimed at all levels - Demos, testimonials, topical forums, all showcased - Code - A full design review for numpy 2.0 - No more inconsistencies like median(), lacking "out", degrees option for angle functions? - Trimming of financial functions, maybe others, from numpy? - Package structure review (eliminate "fromnumeric"?) - Goal that this be the last breakage for numpy API (the real 1.0) - Scipy - Is it maintainable? should it be broken up? - Clear code addition path (or decide never to add more) - Docs (see above) - Add-on packages - Both existence of and good indexing/integration/support for field-specific packages - Clearer development path for new packages - Central hosting system for packages (svn, mailing lists, web, build integration, etc.) - Simultaneous releases of stable packages along with numpy/scipy I posted a basic improvement plan some years back. The core ideas have not changed; it is linked from the bottom of http://scipy.org/Developer_Zone. I chose our major weakness to begin with and started the doc project, using some money I could justify spending simply for the utility of docs for my own research. I funded the work of two doc coordinators, one each this summer and last. Looking at http://docs.scipy.org/numpy/stats/, you can see that when a doc coordinator was being paid (summers), work got done. When not, then not. Without publicly announcing what these guys made, I'll be the first to admit that it wasn't a lot. Yet, those small sums bought a huge contribution to numpy through the work of several dozen volunteers and the major contributions of a few. My conclusion is that active and constant coordination is central to motivating volunteer work, and that without a salary we cannot depend on coordination remaining active. On the other hand, I have heard Enthought's leaders bemoan the high cost of devoting employee time to this project, and the low returns available from selling support to universities and non-profit research institutes. Their leadership has moved us forward, particularly in the area of code, but has not provided the momentum necessary to carry us forward on all fronts. It is time for the public and education sectors to kick in some resources and organizational leadership. We are, after all, benefitting immensely. Since the cost of employee time is not so high for us in the public and education sectors, I propose to continue hiring people like Stefan and David as UCF employees or contractors, and to expand to hiring others in areas like packaging and marketing, provided that funding for those hires can be found. However, my grant situation is no longer as rich as it has been the past two years, and the needs going forward are greater than in the past if we're now to tackle all the points above. So, I will not be hiring another doc guru from my research grants next year. I am confident that others are willing to pitch in financially, but few will pitch in a full FTE, and we need several. We can (and will) set up a donations site, but donation sites tend to receive pizza money unless a sugar daddy comes along. Those benefitting most from the software, notably education, non-profit research, and government institutions, are *forbidden* from making donations by the terms of their grants. NSF doesn't give you money so you can give it away. We need to provide services they can buy on subcontract and a means for handling payments from them. Selling support does not solve the problem, as that requires spending most of the income on servicing that particular client. Rather, we need to sell a chunk of documentation or the packaging of a particular release, and then provide the product not just to that client but to everyone. We can also propose directly for federal and corporate grant funds. I have spoken with several NASA and NSF program managers and with Google's Federal Accounts Representative, and the possibilities for funding are good. But, I am not going to do this alone. We need a strong proposal team to be credible. So, I am seeking a group that is willing to work with me to put up the infrastructure of a funded project, to write grant proposals, and to coordinate a financial effort. Members of this group must have a track record of funded grants, business success, foundation support, etc. We might call it the SciPy Foundation. It could be based at UCF, which has a low overhead rate and has infrastructure (like an HR staff), or it might be independent if we can find a good director willing to devote significant time for relatively low pay compared to what they can likely make elsewhere. I would envision hiring permanent coordinators for docs, packaging, and marketing communications. Enthought appears to have code covered by virtue of having hired Travis, Robert, etc.; how to integrate that with this effort is an open question but not a difficult one, I think, as code is our strongest asset at this point. I invite discussion of this approach and the task list above on the scipy-dev at scipy.org mailing list. If you are seeing this post elsewhere, please reply only on scipy-dev at scipy.org. If you are eligible to lead funding proposals and are interested in participating in grant writing and management activities related to work in our weak areas, please contact me directly. Thanks, --jh-- Prof. Joseph Harrington Planetary Sciences Group Department of Physics MAP 414 4000 Central Florida Blvd. University of Central Florida Orlando, FL 32816-2385 jh at physics.ucf.edu planets.ucf.edu From ndbecker2 at gmail.com Fri Jul 31 13:47:50 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 31 Jul 2009 13:47:50 -0400 Subject: [Numpy-discussion] Functions to pack/unpack bytes? References: Message-ID: Travis Oliphant wrote: > Doesn't .view work for that? > > Travis > > > Sent from my iPhone > > On Jul 29, 2009, at 6:57 AM, Neal Becker wrote: > >> Neal Becker wrote: >> >>> Does numpy have functions to convert between e.g. an array of >>> uint32 and >>> uint8, where the uint32 array is a packed version of the uint8 array >>> (selecting little/big endian)? >> >> I've been thinking about making a ufunc (in python) to do this, but >> have no >> idea how to make a ufunc for a function that consumes 4 consecutive >> values >> of it's input and produces an output array of 1/4 the size of the >> input. Is >> that possible? >> Can 'view' switch byteorder? Doesn't seem to work: import numpy as np a = np.arange(10, dtype=np.uint32) b1 = a.view (np.dtype(np.uint32).newbyteorder('<')) c1 = b1.view(np.uint8) b2 = a.view (np.dtype(np.uint32).newbyteorder('>')) c2 = b2.view(np.uint8) print c1 print c2 [0 0 0 0 1 0 0 0 2 0 0 0 3 0 0 0 4 0 0 0 5 0 0 0 6 0 0 0 7 0 0 0 8 0 0 0 9 0 0 0] [0 0 0 0 1 0 0 0 2 0 0 0 3 0 0 0 4 0 0 0 5 0 0 0 6 0 0 0 7 0 0 0 8 0 0 0 9 0 0 0] From sccolbert at gmail.com Fri Jul 31 18:15:25 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 31 Jul 2009 18:15:25 -0400 Subject: [Numpy-discussion] Is this a bug? Message-ID: <7f014ea60907311515v20a30cbdh73413aca81d9fcec@mail.gmail.com> Numpy 1.3 In [1]: import numpy as np In [2]: a = np.zeros(5).fill(5) In [3]: a In [4]: type(a) Out[4]: In [5]: a = np.zeros(5) In [6]: a.fill(5) In [7]: a Out[7]: array([ 5., 5., 5., 5., 5.]) What i'm trying to do may not be the best way, but I think it should still have worked. Thoughts? Cheers, Chris From robert.kern at gmail.com Fri Jul 31 18:20:06 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 31 Jul 2009 17:20:06 -0500 Subject: [Numpy-discussion] Is this a bug? In-Reply-To: <7f014ea60907311515v20a30cbdh73413aca81d9fcec@mail.gmail.com> References: <7f014ea60907311515v20a30cbdh73413aca81d9fcec@mail.gmail.com> Message-ID: <3d375d730907311520p18263685lf15d1b97a3f49c84@mail.gmail.com> On Fri, Jul 31, 2009 at 17:15, Chris Colbert wrote: > Numpy 1.3 > > In [1]: import numpy as np > > In [2]: a = np.zeros(5).fill(5) > > In [3]: a > > In [4]: type(a) > Out[4]: > > In [5]: a = np.zeros(5) > > In [6]: a.fill(5) > > In [7]: a > Out[7]: array([ 5., ?5., ?5., ?5., ?5.]) > > > What i'm trying to do may not be the best way, but I think it should > still have worked. > > Thoughts? Not a bug. As you can see from In[6], .fill() does not return anything (except the obligatory None). This is just like how list.sort() returns None. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sccolbert at gmail.com Fri Jul 31 18:30:30 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 31 Jul 2009 18:30:30 -0400 Subject: [Numpy-discussion] Is this a bug? In-Reply-To: <3d375d730907311520p18263685lf15d1b97a3f49c84@mail.gmail.com> References: <7f014ea60907311515v20a30cbdh73413aca81d9fcec@mail.gmail.com> <3d375d730907311520p18263685lf15d1b97a3f49c84@mail.gmail.com> Message-ID: <7f014ea60907311530j7b1b647au6043e79d709d08ef@mail.gmail.com> ahh, yeah I see now. Thanks! nothing like making myself look the fool on a friday! Cheers! On Fri, Jul 31, 2009 at 6:20 PM, Robert Kern wrote: > On Fri, Jul 31, 2009 at 17:15, Chris Colbert wrote: >> Numpy 1.3 >> >> In [1]: import numpy as np >> >> In [2]: a = np.zeros(5).fill(5) >> >> In [3]: a >> >> In [4]: type(a) >> Out[4]: >> >> In [5]: a = np.zeros(5) >> >> In [6]: a.fill(5) >> >> In [7]: a >> Out[7]: array([ 5., ?5., ?5., ?5., ?5.]) >> >> >> What i'm trying to do may not be the best way, but I think it should >> still have worked. >> >> Thoughts? > > Not a bug. As you can see from In[6], .fill() does not return anything > (except the obligatory None). This is just like how list.sort() > returns None. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Fri Jul 31 18:59:35 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 31 Jul 2009 16:59:35 -0600 Subject: [Numpy-discussion] Is this a bug? In-Reply-To: <7f014ea60907311530j7b1b647au6043e79d709d08ef@mail.gmail.com> References: <7f014ea60907311515v20a30cbdh73413aca81d9fcec@mail.gmail.com> <3d375d730907311520p18263685lf15d1b97a3f49c84@mail.gmail.com> <7f014ea60907311530j7b1b647au6043e79d709d08ef@mail.gmail.com> Message-ID: On Fri, Jul 31, 2009 at 4:30 PM, Chris Colbert wrote: > ahh, yeah I see now. Thanks! > > nothing like making myself look the fool on a friday! > If you have to choose a day, Friday is the day of choice. Or a least it supposedly works that way for bad news and politics. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Fri Jul 31 19:01:15 2009 From: geometrian at gmail.com (Ian Mallett) Date: Fri, 31 Jul 2009 16:01:15 -0700 Subject: [Numpy-discussion] Is this a bug? In-Reply-To: References: <7f014ea60907311515v20a30cbdh73413aca81d9fcec@mail.gmail.com> <3d375d730907311520p18263685lf15d1b97a3f49c84@mail.gmail.com> <7f014ea60907311530j7b1b647au6043e79d709d08ef@mail.gmail.com> Message-ID: Awww, it's fun to be foolish on Fridays! -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Jul 27 19:29:16 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 27 Jul 2009 16:29:16 -0700 Subject: [Numpy-discussion] String manipulation summary In-Reply-To: <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> References: <752529.59185.qm@web52110.mail.re2.yahoo.com> <4A662711.6080603@noaa.gov> <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> Message-ID: <4A6E384C.1030703@noaa.gov> Hi all, When I first saws this problem: reading in a fixed-width text file as numbers, it struck me that you really should be able to do it, and do it well, with numpy by slicing character arrays. I got carried away, and worked out a number of ways to do it. Lastly was a method inspired by a recent thread: "String to integer array of ASCII values", which did indeed inspire the fastest way. Here's what I have : # my naive first attempt: def line2array0(line, field_len): nums = [] i = 0 while i < len(line): nums.append(float(line[i:i+field_len])) i += field_len return np.array(nums) # list comprehension def line2array1(line, field_len): return np.array(map(float,[line[i*field_len:(i+1)*field_len] for i in range(len(line)/field_len)])) # convert to a tuple, then to an 'S1' array -- no real reason to do # this, as I figured out the next way. def line2array2(line, field_len): return np.array(tuple(line), dtype = 'S1').view(dtype='S%i'%field_len).astype(np.float) # convert directly to a string array, then break into fields. def line2array3(line, field_len): return np.array((line,)).view(dtype='S%i'%field_len).astype(np.float) # use dtype-'c' instead of 'S1' -- better. def line2array4(line, field_len): return np.array(line, dtype='c').view(dtype='S%i'%field_len).astype(np.float) # and the winner is: use fromstring to go straight to a 'c' array: def line2array5(line, field_len): return np.fromstring(line, dtype='c').view(dtype='S%i'%field_len).astype(np.float) Here are some timings: Timing with a 10 number string: List comp: 36.8073430061 convert to tuple: 57.9741871357 auto convert: 43.4103589058 char type: 46.0047719479 fromstring: 23.998103857 without float conversion: 11.4827179909 So list comprehension is pretty fast, but using fromstring, and then slicing is much better. The last one is the same thing, but without the convertion from strings to float, showing that that's a big chunk of time no matter how you slice it. Timing with a 100 number string: List comp: 163.281736135 convert to tuple: 333.081432104 auto convert: 138.934411049 char type: 279.897207975 fromstring: 121.395509005 without float conversion: 12.8342208862 Interesting -- I thought a longer string would give greater advantage to fromstring approach -- but I was wrong, now the time to parse strings into floats is really washing everything else out -- so it doesn't matter much how you do it, though I'd go with either list comprehension (which is what I think is used in np.genfromtxt), or the fromstring method, which I kind of like 'cause it's numpy. test and timing code attached. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: test.py Type: application/x-python Size: 3584 bytes Desc: not available URL: From charlesr.harris at gmail.com Mon Jul 27 19:44:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Jul 2009 17:44:39 -0600 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot Message-ID: ====================================================================== ERROR: test_nan_array (test_utils.TestApproxEqual) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", line 236, in test_nan_array self._assert_func(anan, anan) File "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line 433, in assert_approx_equal ValueError: math domain error ====================================================================== ERROR: test_nan_items (test_utils.TestApproxEqual) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", line 248, in test_nan_items self._assert_func(anan, anan) File "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line 433, in assert_approx_equal ValueError: math domain error ---------------------------------------------------------------------- Ran 2177 tests in 30.179s I expect there is a problem with the FreeBSD math libraries. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Mon Jul 27 19:47:28 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 27 Jul 2009 19:47:28 -0400 Subject: [Numpy-discussion] String manipulation summary In-Reply-To: <4A6E384C.1030703@noaa.gov> References: <752529.59185.qm@web52110.mail.re2.yahoo.com> <4A662711.6080603@noaa.gov> <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> <4A6E384C.1030703@noaa.gov> Message-ID: <7f014ea60907271647w56be6175ga61fd92fdb73eb14@mail.gmail.com> what machine spec are you using? Using your last function line2array5 WITH float conversion, i get the following timing on a mobile quad core extreme: In [24]: a = np.arange(100).astype(str).tostring() In [25]: a Out[25]: '0123456789111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999' In [26]: %timeit line2array(a, 1) 10000 loops, best of 3: 37.1 ?s per loop In [27]: a = np.arange(1000).astype(str).tostring() In [28]: %timeit line2array(a, 10) 10000 loops, best of 3: 45.2 ?s per loop Cheers, Chris On Mon, Jul 27, 2009 at 7:29 PM, Christopher Barker wrote: > Hi all, > > When I first saws this problem: reading in a fixed-width text file as > numbers, it struck me that you really should be able to do it, and do it > well, with numpy by slicing character arrays. > > I got carried away, and worked out a number of ways to do it. Lastly was a > method inspired by a recent thread: "String to integer array of ASCII > values", which did indeed inspire the fastest way. Here's what I have : > > # my naive first attempt: > def line2array0(line, field_len): > ? ?nums = [] > ? ?i = 0 > ? ?while i < len(line): > ? ? ? ?nums.append(float(line[i:i+field_len])) > ? ? ? ?i += field_len > ? ?return np.array(nums) > > # list comprehension > def line2array1(line, field_len): > ? ?return np.array(map(float,[line[i*field_len:(i+1)*field_len] for i in > range(len(line)/field_len)])) > > # convert to a tuple, then to an 'S1' array -- no real reason to do > # this, as I figured out the next way. > def line2array2(line, field_len): > ? ?return np.array(tuple(line), dtype = > 'S1').view(dtype='S%i'%field_len).astype(np.float) > > # convert directly to a string array, then break into fields. > def line2array3(line, field_len): > ? ?return np.array((line,)).view(dtype='S%i'%field_len).astype(np.float) > > # use dtype-'c' instead of 'S1' -- better. > def line2array4(line, field_len): > ? ?return np.array(line, > dtype='c').view(dtype='S%i'%field_len).astype(np.float) > > # and the winner is: use fromstring to go straight to a 'c' array: > def line2array5(line, field_len): > ? ?return np.fromstring(line, > dtype='c').view(dtype='S%i'%field_len).astype(np.float) > > Here are some timings: > > Timing with a 10 number string: > List comp: 36.8073430061 > convert to tuple: 57.9741871357 > auto convert: 43.4103589058 > char type: 46.0047719479 > fromstring: 23.998103857 > without float conversion: 11.4827179909 > > So list comprehension is pretty fast, but using fromstring, and then slicing > is much better. The last one is the same thing, but without the convertion > from strings to float, showing that that's a big chunk of time no matter how > you slice it. > > Timing with a 100 number string: > List comp: 163.281736135 > convert to tuple: 333.081432104 > auto convert: 138.934411049 > char type: 279.897207975 > fromstring: 121.395509005 > without float conversion: 12.8342208862 > > > Interesting -- I thought a longer string would give greater advantage to > fromstring approach -- but I was wrong, now the time to parse strings into > floats is really washing everything else out -- so it doesn't matter much > how you do it, though I'd go with either list comprehension (which is what I > think is used in np.genfromtxt), or the fromstring method, which I kind of > like 'cause it's numpy. > > test and timing code attached. > > -Chris > > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From Chris.Barker at noaa.gov Mon Jul 27 20:00:38 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 27 Jul 2009 17:00:38 -0700 Subject: [Numpy-discussion] String manipulation summary In-Reply-To: <7f014ea60907271647w56be6175ga61fd92fdb73eb14@mail.gmail.com> References: <752529.59185.qm@web52110.mail.re2.yahoo.com> <4A662711.6080603@noaa.gov> <9457e7c80907220049s15438d12j5cc2c71581a06b4a@mail.gmail.com> <4A6E384C.1030703@noaa.gov> <7f014ea60907271647w56be6175ga61fd92fdb73eb14@mail.gmail.com> Message-ID: <4A6E3FA6.9090503@noaa.gov> Chris Colbert wrote: > what machine spec are you using? Dual 2Ghz PPC OS-X 10.4. Python2.5, numpy 1.3.0rc1 (hmm -- I should upgrade that!) > Using your last function line2array5 WITH float conversion, i get the > following timing on a mobile quad core extreme: > > In [24]: a = np.arange(100).astype(str).tostring() > > In [25]: a > Out[25]: '0123456789111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999' > > In [26]: %timeit line2array(a, 1) > 10000 loops, best of 3: 37.1 ?s per loop > > In [27]: a = np.arange(1000).astype(str).tostring() > > In [28]: %timeit line2array(a, 10) > 10000 loops, best of 3: 45.2 ?s per loop and I get, timing it the same way (I forgot about ipython's timeit!): In [83]: %timeit line2array5(a, 1) 10000 loops, best of 3: 125 ?s per loop In [84]: a = np.arange(1000).astype(str).tostring() In [85]: %timeit line2array5(a, 1) 1000 loops, best of 3: 1.09 ms per loop similar to yours, but longer with an older machine. I wonder why my other timings came out the way they did? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cournape at gmail.com Mon Jul 27 23:37:04 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 28 Jul 2009 12:37:04 +0900 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: References: Message-ID: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> On Tue, Jul 28, 2009 at 8:44 AM, Charles R Harris wrote: > ERROR: test_nan_items (test_utils.TestApproxEqual) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", > line 248, in test_nan_items > > self._assert_func(anan, anan) > File > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", line > 433, in assert_approx_equal > ValueError: math domain error > > ---------------------------------------------------------------------- > > Ran 2177 tests in 30.179s > > > I expect there is a problem with the FreeBSD math libraries. Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) Reading the code for assert_approx_equal, using log/pow functions on nan should produce this error. I will work on a fix, cheers, David From martyfuhry at gmail.com Tue Jul 28 00:03:29 2009 From: martyfuhry at gmail.com (Marty Fuhry) Date: Tue, 28 Jul 2009 00:03:29 -0400 Subject: [Numpy-discussion] datetime Parsing Functions Message-ID: I'm a Summer of Code student working on the datetime implementation. The past few weeks I've been writing code to parse between dates and longs with a frequency. I'd like some feedback on my code before it gets integrated into the NumPy datetime branch: Clone URL: git://github.com/martyfuhry/npy_datetime.git Blog: http://fuhrysoc.blogspot.com The main file in question is in the parsing directory called c_parse.c. I have been working on the setitem and getitem routines for datetime64 objects (current functions in c_parse.c are long_to_date(time / string) and date(time / string)_to_long). The frequencies in the NEP are all supported, but I haven't yet implemented derived units for custom frequencies. The tests (located in the parsing/tests directory) are thorough, but I'd appreciate feedback on those, too. Is there anything I should do differently? Are there sections that need more documentation? Am I forgetting anything? Thanks in advance for any help. -Marty Fuhry From charlesr.harris at gmail.com Tue Jul 28 00:22:02 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Jul 2009 22:22:02 -0600 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> Message-ID: On Mon, Jul 27, 2009 at 9:37 PM, David Cournapeau wrote: > On Tue, Jul 28, 2009 at 8:44 AM, Charles R > Harris wrote: > > > ERROR: test_nan_items (test_utils.TestApproxEqual) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > > "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", > > line 248, in test_nan_items > > > > self._assert_func(anan, anan) > > File > > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", > line > > 433, in assert_approx_equal > > ValueError: math domain error > > > > ---------------------------------------------------------------------- > > > > Ran 2177 tests in 30.179s > > > > > > I expect there is a problem with the FreeBSD math libraries. > > Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) > Reading the code for assert_approx_equal, using log/pow functions > nan should produce this error. I will work on a fix, > Yeah, the function is kind of a mess. I think most of the math can be avoided. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jul 28 00:16:22 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 28 Jul 2009 13:16:22 +0900 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> Message-ID: <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Mon, Jul 27, 2009 at 9:37 PM, David Cournapeau > wrote: > > On Tue, Jul 28, 2009 at 8:44 AM, Charles R > Harris > wrote: > > > ERROR: test_nan_items (test_utils.TestApproxEqual) > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > > "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", > > line 248, in test_nan_items > > > > self._assert_func(anan, anan) > > File > > > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", > line > > 433, in assert_approx_equal > > ValueError: math domain error > > > > > ---------------------------------------------------------------------- > > > > Ran 2177 tests in 30.179s > > > > > > I expect there is a problem with the FreeBSD math libraries. > > Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) > Reading the code for assert_approx_equal, using log/pow functions > nan should produce this error. I will work on a fix, > > > Yeah, the function is kind of a mess. Yes - I am afraid I have added to the mess with my nan/inf handling :( I tried reimplementing from a clean state, but quite a few unit tests depend on some 'obscure' features. For example, do we really want assert_array_equal(np.array([1, 1]), 1) to be true ? I understand it is handy, but it can hide bugs, and it makes writing the function more complicated (as you can't check that both arrays have the same shape, and covering every corner case with handling scalar/scalar arrays/arrays is difficult). I am also concerned with speed issues (they are likely to be a bottleneck in many unit tests). > I think most of the math can be avoided. Do you know of a way to avoid log/pow for approx equal ? cheers, David From charlesr.harris at gmail.com Tue Jul 28 00:58:30 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Jul 2009 22:58:30 -0600 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Jul 27, 2009 at 10:16 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > > On Mon, Jul 27, 2009 at 9:37 PM, David Cournapeau > > wrote: > > > > On Tue, Jul 28, 2009 at 8:44 AM, Charles R > > Harris > > wrote: > > > > > ERROR: test_nan_items (test_utils.TestApproxEqual) > > > > > > ---------------------------------------------------------------------- > > > Traceback (most recent call last): > > > File > > > > > > "/tmp/numpy-buildbot/b12/numpy-install/lib/python2.5/site-packages/numpy/testing/tests/test_utils.py", > > > line 248, in test_nan_items > > > > > > self._assert_func(anan, anan) > > > File > > > > > > "../numpy-install/lib/python2.5/site-packages/numpy/testing/utils.py", > > line > > > 433, in assert_approx_equal > > > ValueError: math domain error > > > > > > > > > ---------------------------------------------------------------------- > > > > > > Ran 2177 tests in 30.179s > > > > > > > > > I expect there is a problem with the FreeBSD math libraries. > > > > Actually, I am surprised it did not fail on Linux/Windows/Mac OS X :) > > Reading the code for assert_approx_equal, using log/pow functions > > nan should produce this error. I will work on a fix, > > > > > > Yeah, the function is kind of a mess. > > Yes - I am afraid I have added to the mess with my nan/inf handling :( I > tried reimplementing from a clean state, but quite a few unit tests > depend on some 'obscure' features. For example, do we really want > assert_array_equal(np.array([1, 1]), 1) to be true ? I understand it is > handy, but it can hide bugs, and it makes writing the function more > complicated (as you can't check that both arrays have the same shape, > and covering every corner case with handling scalar/scalar arrays/arrays > is difficult). > > I am also concerned with speed issues (they are likely to be a > bottleneck in many unit tests). > > > I think most of the math can be avoided. > > Do you know of a way to avoid log/pow for approx equal ? > I'd just look at the difference and see if it exceeded some fraction of the expected value. There is the problem of zero, which could be handled in the usual way as diff < abserr + relerr. I think abserr would need to be a new keyword with a default value. Since the relerr is specified by digits you probably need 1/10**digits, but there should be no problem with that. Really, only a multiplication should be needed to determine the precision to x digits. Nan and +/- inf need to be special cased but I haven't thought about that. I agree that the broadcasting is both a convenience and a trap here, but its probably a bit late to change that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Jul 28 00:48:27 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 28 Jul 2009 13:48:27 +0900 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> Message-ID: <4A6E831B.1020200@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > I'd just look at the difference and see if it exceeded some fraction > of the expected value. There is the problem of zero, which could be > handled in the usual way as diff < abserr + relerr. I think abserr > would need to be a new keyword with a default value. Since the relerr > is specified by digits you probably need 1/10**digits, but there > should be no problem with that. Really, only a multiplication should > be needed to determine the precision to x digits. Ok, I will have a shot at this. > Nan and +/- inf need to be special cased but I haven't thought about that. They are a constant source of frustration. It is like it defies my brain, and I end up doing very stupid things after working for too long with nan and inf, especially when complex numbers are involved :) > > I agree that the broadcasting is both a convenience and a trap here, > but its probably a bit late to change that. I thought about having an additional assert_exactly_equal: both arguments will need to have the same shape and type (dtype for arrays), negative zero and positive zero will be different as well. I also have another function which may be useful in one of my scipy branch for scipy.special: having the max error in term of ULPS. cheers, David From charlesr.harris at gmail.com Tue Jul 28 01:51:02 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Jul 2009 23:51:02 -0600 Subject: [Numpy-discussion] Test failures on FreeBSD buildbot In-Reply-To: <4A6E831B.1020200@ar.media.kyoto-u.ac.jp> References: <5b8d13220907272037g10bb2805xf2d6c71972c3f398@mail.gmail.com> <4A6E7B96.7070703@ar.media.kyoto-u.ac.jp> <4A6E831B.1020200@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Jul 27, 2009 at 10:48 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > I'd just look at the difference and see if it exceeded some fraction > > of the expected value. There is the problem of zero, which could be > > handled in the usual way as diff < abserr + relerr. I think abserr > > would need to be a new keyword with a default value. Since the relerr > > is specified by digits you probably need 1/10**digits, but there > > should be no problem with that. Really, only a multiplication should > > be needed to determine the precision to x digits. > > Ok, I will have a shot at this. > > > Nan and +/- inf need to be special cased but I haven't thought about > that. > > They are a constant source of frustration. It is like it defies my > brain, and I end up doing very stupid things after working for too long > with nan and inf, especially when complex numbers are involved :) > Yeah, they leave me boggled too. O_o That's why I haven't thought about them... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Tue Jul 28 02:44:06 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 28 Jul 2009 08:44:06 +0200 Subject: [Numpy-discussion] DumbArray, LoadArray, Numeric -- numpy Message-ID: Hi all, how can I import arrays in numpy which are stored by DumpArray in the old Numeric package ? Nils From nwagner at iam.uni-stuttgart.de Tue Jul 28 02:47:40 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 28 Jul 2009 08:47:40 +0200 Subject: [Numpy-discussion] New test failures Message-ID: >>> numpy.__version__ '1.4.0.dev7270' Python 2.5.1 on 64-bit box ====================================================================== FAIL: test_umath.TestComplexFunctions.test_precisions_consistent ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 545, in test_precisions_consistent assert_almost_equal(fcf, fcd, decimal=6, err_msg='fch-fcd %s'%f) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 383, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: (0.666239+1.06128j) DESIRED: (0.666239432493+1.06127506191j) ====================================================================== FAIL: test_csingle (test_linalg.TestEig) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 39, in test_csingle self.do(a, b) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 94, in do assert_almost_equal(dot(a, evectors), multiply(evectors, evalues)) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 383, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [[-0.28785625-0.21230127j 2.13664055+3.62986112j] [ 0.20296739+0.16528448j 4.73750353+6.42351294j]] DESIRED: [[-0.28785625-0.21230127j 2.13664031+3.62986112j] [ 0.20296741+0.16528448j 4.73750353+6.42351341j]] ====================================================================== FAIL: test_csingle (test_linalg.TestPinv) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 39, in test_csingle self.do(a, b) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 124, in do assert_almost_equal(dot(a, a_ginv), identity(asarray(a).shape[0])) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 383, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [[ 1.00000024e+00 +2.38418579e-07j -5.96046448e-08 +0.00000000e+00j] [ 5.36441803e-07 +3.57627869e-07j 9.99999821e-01 +0.00000000e+00j]] DESIRED: [[ 1. 0.] [ 0. 1.]] ====================================================================== FAIL: test_csingle (test_linalg.TestSVD) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 39, in test_csingle self.do(a, b) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 100, in do assert_almost_equal(a, dot(multiply(u, s), vt)) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 383, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [[ 1.+2.j 2.+3.j] [ 3.+4.j 4.+5.j]] DESIRED: [[ 1.00000012+2.00000024j 2.00000024+3.00000024j] [ 3.00000048+4.00000048j 4.00000048+5.j ]] ---------------------------------------------------------------------- Ran 2179 tests in 67.754s FAILED (KNOWNFAIL=1, failures=4) From david at ar.media.kyoto-u.ac.jp Tue Jul 28 02:58:39 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 28 Jul 2009 15:58:39 +0900 Subject: [Numpy-discussion] New test failures In-Reply-To: References: Message-ID: <4A6EA19F.50300@ar.media.kyoto-u.ac.jp> Hi Nils, Nils Wagner wrote: >>>> numpy.__version__ >>>> > '1.4.0.dev7270' > Python 2.5.1 on 64-bit box > > > AssertionError: Items are not equal: > ACTUAL: [[-0.28785625-0.21230127j 2.13664055+3.62986112j] > [ 0.20296739+0.16528448j 4.73750353+6.42351294j]] > DESIRED: [[-0.28785625-0.21230127j > 2.13664031+3.62986112j] > [ 0.20296741+0.16528448j 4.73750353+6.42351341j]] I am working on it - I think some of those errors are buggy unit tests, though - for a single precision complex number, actual and desired only differ in on ULPS in the above case, for example. cheers, David From seb.haase at gmail.com Tue Jul 28 08:15:18 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 28 Jul 2009 14:15:18 +0200 Subject: [Numpy-discussion] distutils -- compiling extension twice, two sets of compiler settings Message-ID: Hi, maybe someone here could help me by using distutils: my problem is that I have some C++ files that I need to get compiled twice, first to make a "singleFloat"-module using gcc -DUSE_DOUBLE=0 and then to make the "doubleFloat"-module using gcc -DUSE_DOUBLE=1. Apparently distutils recognises (like "make") that the object files are already there and skips the gcc for the second run. My setup.py so far is something like this: ext_ccg_s = Extension('_CCG_smodule', [ 'Priithon_25_C/ccg/CCG_caller.cpp', 'Priithon_25_C/ccg/ccg_funct.cpp', 'Priithon_25_C/ccg/ccg.cpp', 'Priithon_25_C/ccg/CCG_s_wrap.cxx', ], define_macros=[('USE_DOUBLE', '0'),], ) ext_ccg_d = Extension('_CCG_dmodule', [ 'Priithon_25_C/ccg/CCG_caller.cpp', 'Priithon_25_C/ccg/ccg_funct.cpp', 'Priithon_25_C/ccg/ccg.cpp', 'Priithon_25_C/ccg/CCG_d_wrap.cxx', ], define_macros=[('USE_DOUBLE', '1'),], ) setup(... ext_package='Priithon_bin', ext_modules=[ext_ccg_s, ext_ccg_d], ) Thanks for any help, -- Sebastian Haase From dsdale24 at gmail.com Tue Jul 28 08:47:42 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 28 Jul 2009 08:47:42 -0400 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: References: Message-ID: On Mon, Jul 27, 2009 at 12:06 AM, Charles R Harris wrote: > Hi All, > > In case it got buried in the thread, Darren is asking for commit privileges. > I think it's a good idea. Thank you for saying so. What can I do to help move this to the point where I can actually start committing? I would like to get my array_prepare patch into svn soon so I can bundle a new beta of Quantities in time for scipy-09. I'm going to be on vacation July 31-August 9, should I wait until I get back before checking it in (assuming I'm granted commit rights)? Darren From stefan at sun.ac.za Tue Jul 28 08:57:39 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 28 Jul 2009 14:57:39 +0200 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: References: Message-ID: <9457e7c80907280557j22474d25w592b020be5cab53c@mail.gmail.com> 2009/7/27 Charles R Harris : > In case it got buried in the thread, Darren is asking for commit privileges. > I think it's a good idea. I agree. Unfortunately, I can't modify the svn-password file due to permissions, so I'll CC Peter Wang who has root access. Peter: we'd like to give SVN access to Darren Dale. Could you assist? Regards St?fan From charlesr.harris at gmail.com Tue Jul 28 10:54:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Jul 2009 08:54:39 -0600 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: References: Message-ID: On Tue, Jul 28, 2009 at 6:47 AM, Darren Dale wrote: > On Mon, Jul 27, 2009 at 12:06 AM, Charles R > Harris wrote: > > Hi All, > > > > In case it got buried in the thread, Darren is asking for commit > privileges. > > I think it's a good idea. > > Thank you for saying so. What can I do to help move this to the point > where I can actually start committing? I would like to get my > array_prepare patch into svn soon so I can bundle a new beta of > Quantities in time for scipy-09. > > I'm going to be on vacation July 31-August 9, should I wait until I > get back before checking it in (assuming I'm granted commit rights)? > I think you are the best judge of the timing. The only reason I can see for delay is if a problem turns up and you aren't there to fix it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Jul 28 10:59:17 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 28 Jul 2009 23:59:17 +0900 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: References: Message-ID: <5b8d13220907280759g7dd4ba14n19f02197c8915e7f@mail.gmail.com> On Tue, Jul 28, 2009 at 9:47 PM, Darren Dale wrote: > On Mon, Jul 27, 2009 at 12:06 AM, Charles R > Harris wrote: >> Hi All, >> >> In case it got buried in the thread, Darren is asking for commit privileges. >> I think it's a good idea. > > Thank you for saying so. What can I do to help move this to the point > where I can actually start committing? I would like to get my > array_prepare patch into svn soon so I can bundle a new beta of > Quantities in time for scipy-09. > > I'm going to be on vacation July 31-August 9, should I wait until I > get back before checking it in (assuming I'm granted commit rights)? Why not putting your code under a svn branch or a bzr/git/whatever import of the trunk, take care of it, and then committing it after you come back ? I am unfortunately not in position to comment on your code, I am not familiar on this part of numpy, but I would like someone else to take 'responsibility' that your code is OK if possible. cheers, David From dsdale24 at gmail.com Tue Jul 28 11:24:42 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 28 Jul 2009 11:24:42 -0400 Subject: [Numpy-discussion] Commit privileges for Darren Dale In-Reply-To: <5b8d13220907280759g7dd4ba14n19f02197c8915e7f@mail.gmail.com> References: <5b8d13220907280759g7dd4ba14n19f02197c8915e7f@mail.gmail.com> Message-ID: On Tue, Jul 28, 2009 at 10:59 AM, David Cournapeau wrote: > On Tue, Jul 28, 2009 at 9:47 PM, Darren Dale wrote: >> On Mon, Jul 27, 2009 at 12:06 AM, Charles R >> Harris wrote: >>> Hi All, >>> >>> In case it got buried in the thread, Darren is asking for commit privileges. >>> I think it's a good idea. >> >> Thank you for saying so. What can I do to help move this to the point >> where I can actually start committing? I would like to get my >> array_prepare patch into svn soon so I can bundle a new beta of >> Quantities in time for scipy-09. >> >> I'm going to be on vacation July 31-August 9, should I wait until I >> get back before checking it in (assuming I'm granted commit rights)? > > Why not putting your code under a svn branch or a bzr/git/whatever > import of the trunk, take care of it, and then committing it after you > come back ? > > I am unfortunately not in position to comment on your code, I am not > familiar on this part of numpy, but I would like someone else to take > 'responsibility' that your code is OK if possible. I would also feel more comfortable if someone knowledgeable would have a look, since I don't have a lot of experience with the c api. But I'm having difficulty soliciting a response to my requests for review. Darren From daniel.wheeler2 at gmail.com Tue Jul 28 11:38:47 2009 From: daniel.wheeler2 at gmail.com (Daniel Wheeler) Date: Tue, 28 Jul 2009 11:38:47 -0400 Subject: [Numpy-discussion] installation issue with a non-standard location of a python build Message-ID: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> Hi, I'm having an installation issue building numpy 1.3.0 using python 2.6.2 which has been built in my local area in a non-standard place and was configured with --prefix=/users/wd15/usr/i686/4.0. When I run numpy's config ("python setup.py config") everything seems to be fine. However, when I run "python setup.py build", it gives the following error: $ python --version Python 2.6.2 $ uname -a Linux xxx.xxx.xxx 2.6.18-4-k7 #1 SMP Mon Mar 26 17:57:15 UTC 2007 i686 GNU/Linux $ python setup.py build Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 ... File "numpy/core/setup.py", line 186, in check_types "Cannot compiler 'Python.h'. Perhaps you need to "\ SystemError: Cannot compiler 'Python.h'. Perhaps you need to install python-dev|python-devel. $ Now, as I compiled python from source, "Python.h" is sitting neatly in "/users/wd15/usr/i686/4.0/include/python2.6". The output from build (not included above) even claims that the compile options point there: C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/users/wd15/usr/i686/4.0/include/python2.6 -c' so the build seems to be aware of the include location. Any ideas? Cheers -- Daniel Wheeler From cournape at gmail.com Tue Jul 28 12:02:01 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Jul 2009 01:02:01 +0900 Subject: [Numpy-discussion] installation issue with a non-standard location of a python build In-Reply-To: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> References: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> Message-ID: <5b8d13220907280902s18c5aec1x8665a7292f1312b0@mail.gmail.com> On Wed, Jul 29, 2009 at 12:38 AM, Daniel Wheeler wrote: > Hi, > > I'm having an installation issue building numpy 1.3.0 using python > 2.6.2 which has been built in my local area in a non-standard place > and was configured with --prefix=/users/wd15/usr/i686/4.0. When I run > numpy's config ("python setup.py config") everything seems to be fine. > However, when I run "python setup.py build", it gives the following > error: > Can you post the full output of python setup.py build ? cheers, David From Chris.Barker at noaa.gov Tue Jul 28 12:38:34 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 28 Jul 2009 09:38:34 -0700 Subject: [Numpy-discussion] Naming of OS-X binaries Message-ID: <4A6F298A.803@noaa.gov> Hi folks, I just went to get the latest stable build, and found that there was this for Python 2.5: numpy-1.3.0-py2.5-macosx10.5.dmg However, as far as I can tell, that is really built for the python.org framework build of python, which works on 10.3.9 and above, not just 10.5. I do know that it works on a 10.4 PPC machine. In fact, if I see "10.5" in the name, I'm likely to assume that it is built for the Apple-supplied python2.5, which came with 10.5 It really should be re-named. Maybe something like: numpy-1.3.0-py2.5-macosx10-py.org-univ.dmg or numpy-1.3.0-py2.5-macosx10.3.dmg If you want to be compact. As long as someone is taking the trouble to build a binary, people might as well be able to know they can use it! I know, the world of multiple builds for OS-X is still a mess. SIGH. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From daniel.wheeler2 at gmail.com Tue Jul 28 13:38:44 2009 From: daniel.wheeler2 at gmail.com (Daniel Wheeler) Date: Tue, 28 Jul 2009 13:38:44 -0400 Subject: [Numpy-discussion] installation issue with a non-standard location of a python build In-Reply-To: <5b8d13220907280902s18c5aec1x8665a7292f1312b0@mail.gmail.com> References: <80b160a0907280838k2a551dc7h9864b2617232bc5@mail.gmail.com> <5b8d13220907280902s18c5aec1x8665a7292f1312b0@mail.gmail.com> Message-ID: <80b160a0907281038n55930fd7s159421d89f3390f0@mail.gmail.com> On Tue, Jul 28, 2009 at 12:02 PM, David Cournapeau wrote: > On Wed, Jul 29, 2009 at 12:38 AM, Daniel > Can you post the full output of python setup.py build ? Here it is: Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /users/wd15/usr/i686/4.0/lib libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /users/wd15/usr/i686/4.0/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/atlas libraries ptf77blas,ptcblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in /users/wd15/usr/i686/4.0/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib/atlas FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c include_dirs = ['/usr/include'] /users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/config.py:361: DeprecationWarning: +++++++++++++++++++++++++++++++++++++++++++++++++ Usage of get_output is deprecated: please do not use it anymore, and avoid configuration checks involving running executable on the target machine. +++++++++++++++++++++++++++++++++++++++++++++++++ DeprecationWarning) customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc: error trying to exec 'cc1': execvp: No such file or directory gcc: error trying to exec 'cc1': execvp: No such file or directory failure. removing: _configtest.c _configtest.o Status: 255 Output: FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/usr/include'] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /users/wd15/usr/i686/4.0/lib libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /users/wd15/usr/i686/4.0/lib libraries lapack_atlas not found in /users/wd15/usr/i686/4.0/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/atlas libraries lapack_atlas not found in /usr/lib/atlas libraries ptf77blas,ptcblas,atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /users/wd15/usr/i686/4.0/lib libraries lapack_atlas not found in /users/wd15/usr/i686/4.0/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib/atlas libraries lapack_atlas not found in /usr/lib/atlas numpy.distutils.system_info.atlas_info FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = f77 include_dirs = ['/usr/include'] customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc: error trying to exec 'cc1': execvp: No such file or directory gcc: error trying to exec 'cc1': execvp: No such file or directory failure. removing: _configtest.c _configtest.o Status: 255 Output: FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/usr/include'] running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src building py_modules sources building library "npymath" sources building extension "numpy.core._sort" sources Generating build/src.linux-i686-2.6/numpy/core/include/numpy/config.h customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/users/wd15/usr/i686/4.0/include/python2.6 -c' gcc: _configtest.c gcc: error trying to exec 'cc1': execvp: No such file or directory gcc: error trying to exec 'cc1': execvp: No such file or directory failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 172, in setup_package() File "setup.py", line 165, in setup_package configuration=configuration ) File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/users/wd15/usr/i686/4.0/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build_src.py", line 130, in run self.build_sources() File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build_src.py", line 250, in build_extension_sources sources = self.generate_sources(sources, ext) File "/users/wd15/Documents/python/numpy-1.3.0/numpy/distutils/command/build_src.py", line 307, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 286, in generate_config_h moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir) File "numpy/core/setup.py", line 30, in check_types out = check_types(*a, **kw) File "numpy/core/setup.py", line 186, in check_types "Cannot compiler 'Python.h'. Perhaps you need to "\ SystemError: Cannot compiler 'Python.h'. Perhaps you need to install python-dev|python-devel. -- Daniel Wheeler From fperez.net at gmail.com Tue Jul 28 13:39:44 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 28 Jul 2009 10:39:44 -0700 Subject: [Numpy-discussion] [SciPy-user] Numpy/Scipy and the Python African Tour In-Reply-To: <20090723141737.GF10321@phare.normalesup.org> References: <20090723141737.GF10321@phare.normalesup.org> Message-ID: Hi Emanuelle, On Thu, Jul 23, 2009 at 7:17 AM, Emmanuelle Gouillart wrote: > Dear users of Numpy and Scipy, > > here is an informal report on the last event of the Python African Tour, > which took place in Dakar (Senegal) on July 6-10th. It might interest Many thanks for this great post, I am very happy to see this kind of effort sprouting organically in different places. I had the chance to do something similar with Stefan, his advisor and colleagues a few years ago in Stellenbosch, South Africa, and it was a phenomenal experience. The audience was extremely engaged and the facilities first-rate, and I enjoyed it enormously. I'm glad to hear that you've set up a mailing list for your students to continue growing, but I'd also encourage them to subscribe to the 'normal' numpy/scipy lists. Even if they initially feel a bit shy about posting, simply 'lurking' on the lists for a while and reading the flow of information can be very useful both to learn technical ideas and to get familiar with the discussion patterns of a community. This will ease them into naturally engaging the upstream projects when they feel comfortable, which I think in the long term is both useful and necessary. > Speaking about difficulties, one may say that there was some disparity in > the level of experience inside the group, some being very well at ease > with array programming while others were obviously discovering > programming. It was therefore difficult to meet everybody's needs, but > well, I hope everybody learned something! Audience heterogeneity: that's always the hardest part in these things, and we face it everywhere. It's the main source of concern even at the main Scipy conference intro tutorial, since the audience tends to be all over the place in terms of background and experience. Mmh, I just had a thought. A number of us have by now taught reasonably organized workshops of this kind in different places and institutions. I wonder if it wouldn't be a good idea to have a page on the scipy site collecting this information, with a short paragraph about the event and a link to any additional external materials that may exist. This could encourage others to create similar things in their own communities, whether with the help of one of the previous presenters or fully independently. Does that sound useful? In any case, thanks again for the great post! Best, f From charlesr.harris at gmail.com Tue Jul 28 13:41:50 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Jul 2009 11:41:50 -0600 Subject: [Numpy-discussion] nans and sort orders. Message-ID: Hi All, I've changed the sort routines so that they sort nans to the end. The sort order is now Real types: [R, nan] Complex types: [R + Rj, R + nanj, nan + Rj, nan + nanj] Where R stands for the usual extended range of reals, i.e. [-inf, finite, +inf] and when two complex numbers have the same pattern of nans then they are sorted on the non-nan part, if any. So the sorts now work with nans but that implies some other fixups on which I would like some feedback. 1) Search sorted -- Search sorted depends on the compare function of the ndarray, I think that compare function should be changed to correspond to the sort order. 2) Methods argmax/argmin/ -- should be compatible with max/min 3) max/min ufuncs -- Problematic, min would fail to propagate nans, should probably stay the same. 4) fmax/fmin ufuncs -- Should stay with c definition 5) Logical comparison ufuncs -- I think these should remain compatible with the c operators. Note that there is a mix of implementations, max/min on void types use the type comparison functions for instance. In other words, changing the type comparisons could mean rewriting some of the other functions. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: