From devnew at gmail.com Thu Nov 1 02:22:52 2007 From: devnew at gmail.com (dev new) Date: Thu, 1 Nov 2007 11:52:52 +0530 Subject: [Numpy-discussion] how to get/set column Message-ID: <21af559c0710312322u28102b00xa1ec3cc056232733@mail.gmail.com> is there a method for numpy arrays and matrices to get/set a particular column i know that a row can be fetched by mymat[1,:] etc can this be done for a column dn -------------- next part -------------- An HTML attachment was scrubbed... URL: From schut at sarvision.nl Thu Nov 1 03:54:43 2007 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 01 Nov 2007 08:54:43 +0100 Subject: [Numpy-discussion] strange problem compiling against ndarrayobject.h Message-ID: Hmm, it seems my original message did not come through? Not in gmane, at least... Well, here's again: Hi numpy and/or gdal guru's, I'm suddenly getting into trouble compiling gdal's python extension, when it includes ndarrayobject.h from numpy. First it complains about my python not being unicode, though I am sure it is unicode, and then it bails out with all kinds of errors (see snippet below). Numpy people, please read on even though this might seem a gdal problem, because it clearly is numpy related, see the snippet... I've tried everything I could think of, but to no avail. I hope one of you can give me a hint, because I'm kind of clueless, and even don't know in what direction I should search... Here is a paste from the output from building (python setup.py build) in the gdal/swig/python dir. make[2]: Entering directory `/usr/local/src/gdal/swig/python' python setup.py build numpy include /usr/lib64/python2.5/site-packages/numpy/core/include running build running build_py creating build creating build/lib.linux-x86_64-2.5 copying gdal.py -> build/lib.linux-x86_64-2.5 copying ogr.py -> build/lib.linux-x86_64-2.5 copying osr.py -> build/lib.linux-x86_64-2.5 copying gdalconst.py -> build/lib.linux-x86_64-2.5 creating build/lib.linux-x86_64-2.5/osgeo copying osgeo/gdalnumeric.py -> build/lib.linux-x86_64-2.5/osgeo copying osgeo/gdalconst.py -> build/lib.linux-x86_64-2.5/osgeo copying osgeo/__init__.py -> build/lib.linux-x86_64-2.5/osgeo copying osgeo/osr.py -> build/lib.linux-x86_64-2.5/osgeo copying osgeo/gdal_array.py -> build/lib.linux-x86_64-2.5/osgeo copying osgeo/gdal.py -> build/lib.linux-x86_64-2.5/osgeo copying osgeo/ogr.py -> build/lib.linux-x86_64-2.5/osgeo running build_ext building 'osgeo._gdal' extension C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC creating build/temp.linux-x86_64-2.5 creating build/temp.linux-x86_64-2.5/extensions compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr -I/usr/lib64/python2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c' x86_64-pc-linux-gnu-gcc: extensions/gdal_wrap.cpp extensions/gdal_wrap.cpp: In function ?PyObject* _wrap_MajorObject_SetMetadata__SWIG_0(PyObject*, PyObject*)?: extensions/gdal_wrap.cpp:4570: warning: deprecated conversion from string constant to ?char*? x86_64-pc-linux-gnu-g++ -pthread -shared build/temp.linux-x86_64-2.5/extensions/gdal_wrap.o -L../../.libs -L../.. -L/usr/lib64 -lgdal -lpython2.5 -o build/lib.linux-x86_64-2.5/osgeo/_gdal.so building 'osgeo._gdalconst' extension C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr -I/usr/lib64/python2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c' x86_64-pc-linux-gnu-gcc: extensions/gdalconst_wrap.c x86_64-pc-linux-gnu-gcc -pthread -shared build/temp.linux-x86_64-2.5/extensions/gdalconst_wrap.o -L../../.libs -L../.. -L/usr/lib64 -lgdal -lpython2.5 -o build/lib.linux-x86_64-2.5/osgeo/_gdalconst.so building 'osgeo._osr' extension C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr -I/usr/lib64/python2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c' x86_64-pc-linux-gnu-gcc: extensions/osr_wrap.cpp x86_64-pc-linux-gnu-g++ -pthread -shared build/temp.linux-x86_64-2.5/extensions/osr_wrap.o -L../../.libs -L../.. -L/usr/lib64 -lgdal -lpython2.5 -o build/lib.linux-x86_64-2.5/osgeo/_osr.so building 'osgeo._ogr' extension C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr -I/usr/lib64/python2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c' x86_64-pc-linux-gnu-gcc: extensions/ogr_wrap.cpp x86_64-pc-linux-gnu-g++ -pthread -shared build/temp.linux-x86_64-2.5/extensions/ogr_wrap.o -L../../.libs -L../.. -L/usr/lib64 -lgdal -lpython2.5 -o build/lib.linux-x86_64-2.5/osgeo/_ogr.so building 'osgeo._gdal_array' extension C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr -I/usr/lib64/python2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c' x86_64-pc-linux-gnu-gcc: extensions/_gdal_array.cpp In file included from /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, from extensions/gdal_array.h:10, from extensions/_gdal_array.cpp:33: /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:92:2: error: #error Must use Python with unicode enabled. In file included from /usr/include/python2.5/Python.h:62, from extensions/gdal_array.h:16, from extensions/_gdal_array.cpp:33: /usr/include/python2.5/pyport.h:105:1: warning: "PY_SSIZE_T_MAX" redefined In file included from /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, from extensions/gdal_array.h:10, from extensions/_gdal_array.cpp:33: /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:931:1: warning: this is the location of the previous definition In file included from /usr/include/python2.5/Python.h:62, from extensions/gdal_array.h:16, from extensions/_gdal_array.cpp:33: /usr/include/python2.5/pyport.h:107:1: warning: "PY_SSIZE_T_MIN" redefined In file included from /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, from extensions/gdal_array.h:10, from extensions/_gdal_array.cpp:33: /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:932:1: warning: this is the location of the previous definition In file included from /usr/include/python2.5/Python.h:131, from extensions/gdal_array.h:16, from extensions/_gdal_array.cpp:33: /usr/include/python2.5/abstract.h:762:1: warning: "PyIndex_Check" redefined In file included from /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, from extensions/gdal_array.h:10, from extensions/_gdal_array.cpp:33: /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:937:1: warning: this is the location of the previous definition In file included from /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, from extensions/gdal_array.h:10, from extensions/_gdal_array.cpp:33: /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:919: error: ?Py_intptr_t? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:920: error: ?Py_uintptr_t? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1012: error: expected initializer before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: error: typedef ?PyArray_SetItemFunc? is initialized (use __typeof__ instead) /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: error: ?PyObject? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: error: expected primary-expression before ?,? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: error: expected primary-expression before ?void? /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: error: expected primary-expression before ?void? /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1015: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1015: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1016: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1026: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1026: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1028: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1028: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1029: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1031: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1041: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1043: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1044: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1044: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1046: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1050: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1052: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1053: error: ?npy_intp? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1056: error: ISO C++ forbids declaration of ?npy_intp? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1056: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1069: error: ISO C++ forbids declaration of ?PyArray_GetItemFunc? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1069: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1070: error: ISO C++ forbids declaration of ?PyArray_SetItemFunc? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1070: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1121: error: ISO C++ forbids declaration of ?PyObject? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1121: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1171: error: ?PyObject_HEAD? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1190: error: ISO C++ forbids declaration of ?PyObject? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1190: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1194: error: ISO C++ forbids declaration of ?PyObject? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1194: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1203: error: ISO C++ forbids declaration of ?PyObject? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1203: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1213: error: ISO C++ forbids declaration of ?PyObject_HEAD? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1214: error: expected ?;? before ?char? /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1216: error: ISO C++ forbids declaration of ?npy_intp? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1216: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1217: error: ISO C++ forbids declaration of ?npy_intp? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1217: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1219: error: ISO C++ forbids declaration of ?PyObject? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1219: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1229: error: ISO C++ forbids declaration of ?PyObject? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1229: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1238: error: ?PyObject_HEAD? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1246: error: ?PyObject_HEAD? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1249: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1253: error: ?PyObject? has not been declared /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1395: error: ISO C++ forbids declaration of ?PyObject_HEAD? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1396: error: expected ?;? before ?int? /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1397: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1398: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1399: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1400: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1401: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1402: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1542: error: ISO C++ forbids declaration of ?PyObject_HEAD? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1543: error: expected ?;? before ?int? /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1544: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1545: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1547: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1596: error: ISO C++ forbids declaration of ?PyObject_HEAD? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1600: error: expected ?;? before ?int? /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1602: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1604: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1606: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1622: error: ?npy_intp? does not name a type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1624: error: ISO C++ forbids declaration of ?PyObject? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1624: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1799: error: ISO C++ forbids declaration of ?npy_intp? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1799: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1800: error: ISO C++ forbids declaration of ?npy_intp? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1800: error: expected ?;? before ?*? token /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1802: error: ISO C++ forbids declaration of ?PyObject? with no type /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1802: error: expected ?;? before ?*? token In file included from /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1814, from /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, from extensions/gdal_array.h:10, from extensions/_gdal_array.cpp:33: /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h: In function ?int _import_array()?: /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:945: error: ?PyObject? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:945: error: ?numpy? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:945: error: ?PyImport_ImportModule? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:946: error: ?c_api? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:948: error: ?PyObject_GetAttrString? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:949: error: ?Py_DECREF? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:950: error: ?PyCObject_Check? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:951: error: ?PyCObject_AsVoidPtr? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:953: error: ?Py_DECREF? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:958: error: ?PyExc_RuntimeError? was not declared in this scope /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:958: error: ?PyErr_Format? was not declared in this scope In file included from /usr/include/python2.5/Python.h:62, from extensions/gdal_array.h:16, from extensions/_gdal_array.cpp:33: /usr/include/python2.5/pyport.h: At global scope: /usr/include/python2.5/pyport.h:97: error: conflicting declaration ?typedef ssize_t Py_ssize_t? /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:930: error: ?Py_ssize_t? has a previous declaration as ?typedef int Py_ssize_t? extensions/_gdal_array.cpp: In function ?void init_gdal_array()?: extensions/_gdal_array.cpp:118: warning: deprecated conversion from string constant to ?char*? extensions/_gdal_array.cpp: In destructor ?virtual NUMPYDataset::~NUMPYDataset()?: extensions/_gdal_array.cpp:162: error: ?struct PyArrayObject? has no member named ?ob_refcnt? extensions/_gdal_array.cpp: In static member function ?static GDALDataset* NUMPYDataset::Open(GDALOpenInfo*)?: extensions/_gdal_array.cpp:382: error: ?struct PyArrayObject? has no member named ?ob_refcnt? extensions/_gdal_array.cpp:393: error: ?struct PyArrayObject? has no member named ?dimensions? extensions/_gdal_array.cpp:394: error: ?struct PyArrayObject? has no member named ?strides? extensions/_gdal_array.cpp:395: error: ?struct PyArrayObject? has no member named ?dimensions? extensions/_gdal_array.cpp:396: error: ?struct PyArrayObject? has no member named ?strides? extensions/_gdal_array.cpp:397: error: ?struct PyArrayObject? has no member named ?dimensions? extensions/_gdal_array.cpp:398: error: ?struct PyArrayObject? has no member named ?strides? extensions/_gdal_array.cpp:404: error: ?struct PyArrayObject? has no member named ?dimensions? extensions/_gdal_array.cpp:405: error: ?struct PyArrayObject? has no member named ?strides? extensions/_gdal_array.cpp:406: error: ?struct PyArrayObject? has no member named ?dimensions? extensions/_gdal_array.cpp:407: error: ?struct PyArrayObject? has no member named ?strides? extensions/_gdal_array.cpp:418: error: ?struct PyArrayObject? has no member named ?data? going on like this for some more, but I hope this suffices... TIA, Vincent. From schut at sarvision.nl Thu Nov 1 04:08:55 2007 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 01 Nov 2007 09:08:55 +0100 Subject: [Numpy-discussion] strange problem compiling against ndarrayobject.h In-Reply-To: References: Message-ID: Hi all, It appeared to be a gdal issue after all: the arrayobject header file was being included before the python headers... Glad it wasn't something like me having borked my numpy build :) Cheers, Vincent. Vincent Schut wrote: > Hmm, it seems my original message did not come through? Not in gmane, at > least... Well, here's again: > > Hi numpy and/or gdal guru's, > > I'm suddenly getting into trouble compiling gdal's python extension, > when it includes ndarrayobject.h from numpy. First it complains about my > python not being unicode, though I am sure it is unicode, and then it > bails out with all kinds of errors (see snippet below). > Numpy people, please read on even though this might seem a gdal problem, > because it clearly is numpy related, see the snippet... > > I've tried everything I could think of, but to no avail. I hope one of > you can give me a hint, because I'm kind of clueless, and even don't > know in what direction I should search... > > Here is a paste from the output from building (python setup.py build) in > the gdal/swig/python dir. > > make[2]: Entering directory `/usr/local/src/gdal/swig/python' > python setup.py build > numpy include /usr/lib64/python2.5/site-packages/numpy/core/include > running build > running build_py > creating build > creating build/lib.linux-x86_64-2.5 > copying gdal.py -> build/lib.linux-x86_64-2.5 > copying ogr.py -> build/lib.linux-x86_64-2.5 > copying osr.py -> build/lib.linux-x86_64-2.5 > copying gdalconst.py -> build/lib.linux-x86_64-2.5 > creating build/lib.linux-x86_64-2.5/osgeo > copying osgeo/gdalnumeric.py -> build/lib.linux-x86_64-2.5/osgeo > copying osgeo/gdalconst.py -> build/lib.linux-x86_64-2.5/osgeo > copying osgeo/__init__.py -> build/lib.linux-x86_64-2.5/osgeo > copying osgeo/osr.py -> build/lib.linux-x86_64-2.5/osgeo > copying osgeo/gdal_array.py -> build/lib.linux-x86_64-2.5/osgeo > copying osgeo/gdal.py -> build/lib.linux-x86_64-2.5/osgeo > copying osgeo/ogr.py -> build/lib.linux-x86_64-2.5/osgeo > running build_ext > building 'osgeo._gdal' extension > C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing > -DNDEBUG -fPIC > > creating build/temp.linux-x86_64-2.5 > creating build/temp.linux-x86_64-2.5/extensions > compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr > -I/usr/lib64/python2.5/site-packages/numpy/core/include > -I/usr/include/python2.5 -c' > x86_64-pc-linux-gnu-gcc: extensions/gdal_wrap.cpp > extensions/gdal_wrap.cpp: In function ?PyObject* > _wrap_MajorObject_SetMetadata__SWIG_0(PyObject*, PyObject*)?: > extensions/gdal_wrap.cpp:4570: warning: deprecated conversion from > string constant to ?char*? > x86_64-pc-linux-gnu-g++ -pthread -shared > build/temp.linux-x86_64-2.5/extensions/gdal_wrap.o -L../../.libs -L../.. > -L/usr/lib64 -lgdal -lpython2.5 -o build/lib.linux-x86_64-2.5/osgeo/_gdal.so > building 'osgeo._gdalconst' extension > C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing > -DNDEBUG -fPIC > > compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr > -I/usr/lib64/python2.5/site-packages/numpy/core/include > -I/usr/include/python2.5 -c' > x86_64-pc-linux-gnu-gcc: extensions/gdalconst_wrap.c > x86_64-pc-linux-gnu-gcc -pthread -shared > build/temp.linux-x86_64-2.5/extensions/gdalconst_wrap.o -L../../.libs > -L../.. -L/usr/lib64 -lgdal -lpython2.5 -o > build/lib.linux-x86_64-2.5/osgeo/_gdalconst.so > building 'osgeo._osr' extension > C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing > -DNDEBUG -fPIC > > compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr > -I/usr/lib64/python2.5/site-packages/numpy/core/include > -I/usr/include/python2.5 -c' > x86_64-pc-linux-gnu-gcc: extensions/osr_wrap.cpp > x86_64-pc-linux-gnu-g++ -pthread -shared > build/temp.linux-x86_64-2.5/extensions/osr_wrap.o -L../../.libs -L../.. > -L/usr/lib64 -lgdal -lpython2.5 -o build/lib.linux-x86_64-2.5/osgeo/_osr.so > building 'osgeo._ogr' extension > C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing > -DNDEBUG -fPIC > > compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr > -I/usr/lib64/python2.5/site-packages/numpy/core/include > -I/usr/include/python2.5 -c' > x86_64-pc-linux-gnu-gcc: extensions/ogr_wrap.cpp > x86_64-pc-linux-gnu-g++ -pthread -shared > build/temp.linux-x86_64-2.5/extensions/ogr_wrap.o -L../../.libs -L../.. > -L/usr/lib64 -lgdal -lpython2.5 -o build/lib.linux-x86_64-2.5/osgeo/_ogr.so > building 'osgeo._gdal_array' extension > C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing > -DNDEBUG -fPIC > > compile options: '-I../../port -I../../gcore -I../../alg -I../../ogr > -I/usr/lib64/python2.5/site-packages/numpy/core/include > -I/usr/include/python2.5 -c' > x86_64-pc-linux-gnu-gcc: extensions/_gdal_array.cpp > In file included from > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, > from extensions/gdal_array.h:10, > from extensions/_gdal_array.cpp:33: > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:92:2: > error: #error Must use Python with unicode enabled. > In file included from /usr/include/python2.5/Python.h:62, > from extensions/gdal_array.h:16, > from extensions/_gdal_array.cpp:33: > /usr/include/python2.5/pyport.h:105:1: warning: "PY_SSIZE_T_MAX" redefined > In file included from > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, > from extensions/gdal_array.h:10, > from extensions/_gdal_array.cpp:33: > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:931:1: > warning: this is the location of the previous definition > In file included from /usr/include/python2.5/Python.h:62, > from extensions/gdal_array.h:16, > from extensions/_gdal_array.cpp:33: > /usr/include/python2.5/pyport.h:107:1: warning: "PY_SSIZE_T_MIN" redefined > In file included from > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, > from extensions/gdal_array.h:10, > from extensions/_gdal_array.cpp:33: > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:932:1: > warning: this is the location of the previous definition > In file included from /usr/include/python2.5/Python.h:131, > from extensions/gdal_array.h:16, > from extensions/_gdal_array.cpp:33: > /usr/include/python2.5/abstract.h:762:1: warning: "PyIndex_Check" redefined > In file included from > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, > from extensions/gdal_array.h:10, > from extensions/_gdal_array.cpp:33: > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:937:1: > warning: this is the location of the previous definition > In file included from > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, > from extensions/gdal_array.h:10, > from extensions/_gdal_array.cpp:33: > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:919: > error: ?Py_intptr_t? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:920: > error: ?Py_uintptr_t? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1012: > error: expected initializer before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: > error: typedef ?PyArray_SetItemFunc? is initialized (use __typeof__ instead) > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: > error: ?PyObject? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: > error: expected primary-expression before ?,? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: > error: expected primary-expression before ?void? > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1013: > error: expected primary-expression before ?void? > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1015: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1015: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1016: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1026: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1026: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1028: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1028: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1029: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1031: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1041: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1043: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1044: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1044: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1046: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1050: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1052: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1053: > error: ?npy_intp? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1056: > error: ISO C++ forbids declaration of ?npy_intp? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1056: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1069: > error: ISO C++ forbids declaration of ?PyArray_GetItemFunc? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1069: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1070: > error: ISO C++ forbids declaration of ?PyArray_SetItemFunc? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1070: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1121: > error: ISO C++ forbids declaration of ?PyObject? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1121: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1171: > error: ?PyObject_HEAD? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1190: > error: ISO C++ forbids declaration of ?PyObject? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1190: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1194: > error: ISO C++ forbids declaration of ?PyObject? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1194: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1203: > error: ISO C++ forbids declaration of ?PyObject? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1203: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1213: > error: ISO C++ forbids declaration of ?PyObject_HEAD? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1214: > error: expected ?;? before ?char? > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1216: > error: ISO C++ forbids declaration of ?npy_intp? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1216: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1217: > error: ISO C++ forbids declaration of ?npy_intp? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1217: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1219: > error: ISO C++ forbids declaration of ?PyObject? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1219: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1229: > error: ISO C++ forbids declaration of ?PyObject? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1229: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1238: > error: ?PyObject_HEAD? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1246: > error: ?PyObject_HEAD? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1249: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1253: > error: ?PyObject? has not been declared > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1395: > error: ISO C++ forbids declaration of ?PyObject_HEAD? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1396: > error: expected ?;? before ?int? > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1397: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1398: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1399: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1400: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1401: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1402: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1542: > error: ISO C++ forbids declaration of ?PyObject_HEAD? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1543: > error: expected ?;? before ?int? > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1544: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1545: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1547: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1596: > error: ISO C++ forbids declaration of ?PyObject_HEAD? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1600: > error: expected ?;? before ?int? > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1602: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1604: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1606: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1622: > error: ?npy_intp? does not name a type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1624: > error: ISO C++ forbids declaration of ?PyObject? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1624: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1799: > error: ISO C++ forbids declaration of ?npy_intp? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1799: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1800: > error: ISO C++ forbids declaration of ?npy_intp? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1800: > error: expected ?;? before ?*? token > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1802: > error: ISO C++ forbids declaration of ?PyObject? with no type > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1802: > error: expected ?;? before ?*? token > In file included from > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:1814, > from > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/arrayobject.h:14, > from extensions/gdal_array.h:10, > from extensions/_gdal_array.cpp:33: > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h: > In function ?int _import_array()?: > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:945: > error: ?PyObject? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:945: > error: ?numpy? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:945: > error: ?PyImport_ImportModule? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:946: > error: ?c_api? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:948: > error: ?PyObject_GetAttrString? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:949: > error: ?Py_DECREF? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:950: > error: ?PyCObject_Check? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:951: > error: ?PyCObject_AsVoidPtr? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:953: > error: ?Py_DECREF? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:958: > error: ?PyExc_RuntimeError? was not declared in this scope > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/__multiarray_api.h:958: > error: ?PyErr_Format? was not declared in this scope > In file included from /usr/include/python2.5/Python.h:62, > from extensions/gdal_array.h:16, > from extensions/_gdal_array.cpp:33: > /usr/include/python2.5/pyport.h: At global scope: > /usr/include/python2.5/pyport.h:97: error: conflicting declaration > ?typedef ssize_t Py_ssize_t? > /usr/lib64/python2.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:930: > error: ?Py_ssize_t? has a previous declaration as ?typedef int Py_ssize_t? > extensions/_gdal_array.cpp: In function ?void init_gdal_array()?: > extensions/_gdal_array.cpp:118: warning: deprecated conversion from > string constant to ?char*? > extensions/_gdal_array.cpp: In destructor ?virtual > NUMPYDataset::~NUMPYDataset()?: > extensions/_gdal_array.cpp:162: error: ?struct PyArrayObject? has no > member named ?ob_refcnt? > extensions/_gdal_array.cpp: In static member function ?static > GDALDataset* NUMPYDataset::Open(GDALOpenInfo*)?: > extensions/_gdal_array.cpp:382: error: ?struct PyArrayObject? has no > member named ?ob_refcnt? > extensions/_gdal_array.cpp:393: error: ?struct PyArrayObject? has no > member named ?dimensions? > extensions/_gdal_array.cpp:394: error: ?struct PyArrayObject? has no > member named ?strides? > extensions/_gdal_array.cpp:395: error: ?struct PyArrayObject? has no > member named ?dimensions? > extensions/_gdal_array.cpp:396: error: ?struct PyArrayObject? has no > member named ?strides? > extensions/_gdal_array.cpp:397: error: ?struct PyArrayObject? has no > member named ?dimensions? > extensions/_gdal_array.cpp:398: error: ?struct PyArrayObject? has no > member named ?strides? > extensions/_gdal_array.cpp:404: error: ?struct PyArrayObject? has no > member named ?dimensions? > extensions/_gdal_array.cpp:405: error: ?struct PyArrayObject? has no > member named ?strides? > extensions/_gdal_array.cpp:406: error: ?struct PyArrayObject? has no > member named ?dimensions? > extensions/_gdal_array.cpp:407: error: ?struct PyArrayObject? has no > member named ?strides? > extensions/_gdal_array.cpp:418: error: ?struct PyArrayObject? has no > member named ?data? > > going on like this for some more, but I hope this suffices... > > TIA, > Vincent. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From bernhard.voigt at gmail.com Thu Nov 1 05:09:53 2007 From: bernhard.voigt at gmail.com (bernhard.voigt at gmail.com) Date: Thu, 01 Nov 2007 09:09:53 -0000 Subject: [Numpy-discussion] how to get/set column In-Reply-To: <21af559c0710312322u28102b00xa1ec3cc056232733@mail.gmail.com> References: <21af559c0710312322u28102b00xa1ec3cc056232733@mail.gmail.com> Message-ID: <1193908193.982644.53020@d55g2000hsg.googlegroups.com> It's just the other way around: mymat[:,0] # first column mymat[:,1] # second column Take a look at the tutorial: http://scipy.org/Tentative_NumPy_Tutorial#head-864862d3f2bb4c32f04260fac61eb4ef34788c4c best! bernhard On Nov 1, 7:22 am, "dev new" wrote: > is there a method for numpy arrays and matrices to get/set a particular > column > > i know that a row can be fetched by mymat[1,:] etc > can this be done for a column > dn > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From faltet at carabos.com Thu Nov 1 08:56:37 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 1 Nov 2007 13:56:37 +0100 Subject: [Numpy-discussion] vectorizing loops In-Reply-To: References: <472123B6.1050609@jpl.nasa.gov> <200710311118.15239.faltet@carabos.com> Message-ID: <200711011356.38449.faltet@carabos.com> A Wednesday 31 October 2007, Timothy Hochberg escrigu?: > On Oct 31, 2007 3:18 AM, Francesc Altet wrote: > > [SNIP] > > > Incidentally, all the improvements of the PyTables flavor of > > numexpr have been reported to the original authors, but, for the > > sake of keeping numexpr simple, they decided to implement only some > > of them. However, people is encouraged to try out the Pytables > > flavor from: > > My original concern was that we'd start overflowing the code cache > and slow things down. Some experiments seemed to confirm this on some > processors, but it then turned out those were in error. At least > that's my recollection. Because of that, and because PyTables is, as > far as I know, the major user of numexpr, I suspect I'd be fine > putting those changes in now. I say "suspect" since it's been a long > time since I looked at the relevant patches, and I should probably > look at those again before I commit myself. I just haven't had the > free time and motivation to go back and look at the patches. I can't > speak for David though. Yes, I remember that you were mainly concerned with overflowing the code cache, and yes, I think you can be confident this is not the case, as my benchmarking in many CPU's (ranging from 7 year old AMD Duron, Pentiums 4 and up to relatively recents AMD Opterons), doesn't show any slow down with the PyTables version of numexpr, but rather the contrary, it speeds things up, specially in contexts where booleans and/or strided/unaligned arrays are used, where the improvement can be up to 2x on recent CPU's (as shown in my previous post). If I remember correctly, another point where you (specially David) were not very keen to include was the support for strings, arguing that numexpr is meant mainly to deal with numerical data, not textual. However, our experience is that adding this support was not too complicated (mainly due to NumPy flexibility), and can be helpful in some instances. As for one, we use them for evaluating expressions like 'array_string == "some_string"', and this can be very convenient to use when you are in the middle of potentially complex boolean expressions that you want to evaluate *fast*. At any rate, we would be glad if you would like to integrate our patches in the main numexpr, as there is not much sense to have different implementations of numexpr (most specially when it seems that there are not much users out there). So, count on us for any question you may have in this regard. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From cookedm at physics.mcmaster.ca Thu Nov 1 10:14:45 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 1 Nov 2007 10:14:45 -0400 Subject: [Numpy-discussion] vectorizing loops In-Reply-To: <200711011356.38449.faltet@carabos.com> References: <472123B6.1050609@jpl.nasa.gov> <200710311118.15239.faltet@carabos.com> <200711011356.38449.faltet@carabos.com> Message-ID: <73A6DD54-4D2C-4D95-9C1F-AE50AF424A8A@physics.mcmaster.ca> On Nov 1, 2007, at 08:56 , Francesc Altet wrote: > A Wednesday 31 October 2007, Timothy Hochberg escrigu?: >> On Oct 31, 2007 3:18 AM, Francesc Altet wrote: >> >> [SNIP] >> >>> Incidentally, all the improvements of the PyTables flavor of >>> numexpr have been reported to the original authors, but, for the >>> sake of keeping numexpr simple, they decided to implement only some >>> of them. However, people is encouraged to try out the Pytables >>> flavor from: >> >> My original concern was that we'd start overflowing the code cache >> and slow things down. Some experiments seemed to confirm this on some >> processors, but it then turned out those were in error. At least >> that's my recollection. Because of that, and because PyTables is, as >> far as I know, the major user of numexpr, I suspect I'd be fine >> putting those changes in now. I say "suspect" since it's been a long >> time since I looked at the relevant patches, and I should probably >> look at those again before I commit myself. I just haven't had the >> free time and motivation to go back and look at the patches. I can't >> speak for David though. > > If I remember correctly, another point where you (specially David) > were > not very keen to include was the support for strings, arguing that > numexpr is meant mainly to deal with numerical data, not textual. > However, our experience is that adding this support was not too > complicated (mainly due to NumPy flexibility), and can be helpful in > some instances. As for one, we use them for evaluating expressions > like 'array_string == "some_string"', and this can be very convenient > to use when you are in the middle of potentially complex boolean > expressions that you want to evaluate *fast*. > > At any rate, we would be glad if you would like to integrate our > patches > in the main numexpr, as there is not much sense to have different > implementations of numexpr (most specially when it seems that there > are > not much users out there). So, count on us for any question you may > have in this regard. Well, I don't have much time to work on it, but if you make sure your patches on the scipy Trac apply clean, I'll have a quick look at them and apply them. Since you've had them working in production code, they should be good ;-) Another issue is that numexpr is still in the scipy sandbox, so only those who enable it will use it (or use it through PyTables). One problem with moving it out is that Tim reports the compile times on Windows are ridiculous (20 mins!). Maybe numexpr should become a scikit? It certainly doesn't need the rest of scipy. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From subscriber100 at rjs.org Thu Nov 1 10:56:30 2007 From: subscriber100 at rjs.org (Ray Schumacher) Date: Thu, 01 Nov 2007 06:56:30 -0800 Subject: [Numpy-discussion] numpy FFT memory accumulation In-Reply-To: References: Message-ID: <6.2.3.4.2.20071101061852.034fe7c0@rjs.org> At 11:55 PM 10/31/2007, Travis wrote: >Ray S wrote: > > I am using > > fftRes = abs(fft.rfft(data_array[end-2**15:end])) > > >At first glance, I would say that I don't expect memory to be growing >here, so it looks like a problem with rfft that deserves looking into. I saw that Numeric did also (I still use Numeric for smaller array speed) but much more slowly. I will try to repeat with a small demo and post. >Does data_array keep growing? no, it is a 64k circular buffer Which reminds me, I've been wanting to try to build a Numeric circular buffer object; one that, when sliced or assigned to, auto-magically wraps as needed, exposing only an extra pointer attribute for the "end". I currently always do it with Python if-s and pointer vars for these products that only do "real time" data analysis. >From: "Anne Archibald" >If the range is *really* small, you can try using a DFT - sometimes >that is fast enough, and gives you just the bins you're curious about. I've considered weave'ing a simple sine transform with specified range, but until I do and test I won't know if my own implementation is any faster than just the FFTPACK. >If the range is bigger than that, but still a small fraction of the >FFT size, you can do some tricks where you band-pass filter the data >... >There are also "zoom fft" and "chirp-z" techniques which are supposed >to give you only part of the FFT, but the wisdom is that unless you >want less than a few percent of the data points you're better just >FFTing and throwing lots of data away. I've tried zoom before; the issue was just that - 2 FFTs and a shift or convolution eats a lot of CPU cycles and falls behind the real time data. The range of interest in the Fourrier domain is small, 3kHz-7kHz. The sample rate is high for precise phase information. I've got some more testing to do, it seems. Thanks, Ray -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.15.15/1101 - Release Date: 10/31/2007 10:06 AM From charlesr.harris at gmail.com Thu Nov 1 11:18:50 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Nov 2007 09:18:50 -0600 Subject: [Numpy-discussion] numpy FFT memory accumulation In-Reply-To: <6.2.3.4.2.20071101061852.034fe7c0@rjs.org> References: <6.2.3.4.2.20071101061852.034fe7c0@rjs.org> Message-ID: On 11/1/07, Ray Schumacher wrote: > At 11:55 PM 10/31/2007, Travis wrote: > >Ray S wrote: > > > I am using > > > fftRes = abs(fft.rfft(data_array[end-2**15:end])) > > > > >At first glance, I would say that I don't expect memory to be growing > >here, so it looks like a problem with rfft that deserves looking into. > > I saw that Numeric did also (I still use Numeric for smaller array > speed) but much more slowly. > I will try to repeat with a small demo and post. > > >Does data_array keep growing? > > no, it is a 64k circular buffer > Which reminds me, I've been wanting to try to build a Numeric > circular buffer object; one that, when sliced or assigned to, > auto-magically wraps as needed, exposing only an extra pointer > attribute for the "end". I currently always do it with Python if-s > and pointer vars for these products that only do "real time" data analysis. > In Python, collections.deque makes a pretty good circular buffer. Numpy will make an array out of it, which involves a copy, but it might be better than what you are doing now. In [14]: from collections import deque In [15]: a = deque([1,2,3,4]) In [16]: fft(a) Out[16]: array([ 10.+0.j, -2.+2.j, -2.+0.j, -2.-2.j]) In [17]: a.append(5) In [18]: a.popleft() Out[18]: 1 In [19]: fft(a) Out[19]: array([ 14.+0.j, -2.+2.j, -2.+0.j, -2.-2.j]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From subscriber100 at rjs.org Thu Nov 1 15:16:03 2007 From: subscriber100 at rjs.org (Ray S) Date: Thu, 01 Nov 2007 11:16:03 -0800 Subject: [Numpy-discussion] numpy FFT memory accumulation In-Reply-To: References: Message-ID: <6.2.3.4.2.20071101105313.058fbf88@rjs.org> At 09:00 AM 11/1/2007, Chuck wrote: In Python, collections.deque makes a pretty good circular buffer. Numpy will make an array out of it, which involves a copy, but it might be better than what you are doing now. hmmm, I'll think more about that - and the copy is only at program start, it seems the fft can always be fft(d[:-N]) >>> from collections import deque >>> import numpy as N >>> d = deque(N.zeros(10,)) >>> d.extend(N.ones(4,)) >>> d deque([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0]) >>> [d.pop() for i in range(4)] [1.0, 1.0, 1.0, 1.0] >>> d deque([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]) An additional complication is that I pass the numpy (or Numeric) array address to the ctypes library call so that the data is placed directly into the array from the call. I use the if/else end wrap logic to determine whether I need to do a split and copy if the new data wraps. Thanks, Ray From subscriber100 at rjs.org Thu Nov 1 15:38:52 2007 From: subscriber100 at rjs.org (Ray S) Date: Thu, 01 Nov 2007 11:38:52 -0800 Subject: [Numpy-discussion] numpy FFT memory accumulation In-Reply-To: References: Message-ID: <6.2.3.4.2.20071101111616.06d30b28@rjs.org> At 09:00 AM 11/1/2007, you wrote: I saw that Numeric did also (I still use Numeric for smaller array speed) but much more slowly. I will try to repeat with a small demo and post. It turns out to be some aspect of mixing numpy and Numeric; the attached *Stable.py files allocate memory that stays exactly the same mixedGrows.py grows by about 50kB/s while changing slightly, randomly; note that the array is assigned as numpy and operated on with Numeric functions. My main app also uses mmap/numpy to assign the arrays to shared memory for multiple processes, and does other operations as well, possibly accounting for the faster memory growth. I'll re-factor to an all-numpy solution and test. I used Numeric functions for the ~40% speed increase, but I don't have a compiled version of arrayfrombuffer() for Numeric http://www.canonical.org/~kragen/sw/arrayfrombuffer/ (compiler version woes) so, I must use numpy.frombuffer() with mmap to create the shared arrays... Ray -------------- next part -------------- import numpy import numpy.fft as fft CIRC_BUFFER_SIZE = 2**16 LARGE_FFT_SIZE = 2048*8 numpy.random.seed() dataArray = numpy.zeros((CIRC_BUFFER_SIZE*2,), numpy.int32) while 1: fftArray = abs(fft.rfft(dataArray[:LARGE_FFT_SIZE])) fftArray[:-1] = fftArray[1:] ## shift the bins lose the DC maxBin = numpy.argmax(fftArray[1000:5000])+1000 print "\r%d" % (maxBin), dataArray[0:CIRC_BUFFER_SIZE] = numpy.random.binomial(5000000, .5, (CIRC_BUFFER_SIZE,)) -------------- next part -------------- import RandomArray import Numeric import FFT CIRC_BUFFER_SIZE = 2**16 LARGE_FFT_SIZE = 2048*8 RandomArray.seed() def freqAnalysis(): global infoArray,dataArray fftArray = abs(FFT.real_fft(dataArray[:LARGE_FFT_SIZE])) maxBin = Numeric.argmax(fftArray[1000:5000])+1000 return maxBin def getMoreData(): global infoArray,dataArray dataArray[0:CIRC_BUFFER_SIZE] = RandomArray.binomial(5000000, .5, (CIRC_BUFFER_SIZE,)) ## small array to hold shared variables ## 0 -> bufferInsertPos ## 1 -> dataTotSamples ## 2 -> sampPerRot ## 3 -> run signal infoArray = Numeric.zeros((4,), Numeric.Float32) dataArray = Numeric.zeros((CIRC_BUFFER_SIZE*2,), Numeric.Int32) while 1: infoArray[2] = freqAnalysis() ## sampPerRot print "\r%d" % (infoArray[2]), getMoreData() -------------- next part -------------- import RandomArray import Numeric import FFT import mmap import numpy CIRC_BUFFER_SIZE = 2**16 LARGE_FFT_SIZE = 2048*8 RandomArray.seed() dataArray = numpy.zeros((CIRC_BUFFER_SIZE*2,), numpy.int32) while 1: fftArray = abs(FFT.real_fft(dataArray[:LARGE_FFT_SIZE])) fftArray[:-1] = fftArray[1:] ## shift the bins lose the DC maxBin = Numeric.argmax(fftArray[1000:5000])+1000 print "\r%d" % (maxBin), dataArray[0:CIRC_BUFFER_SIZE] = RandomArray.normal(5000000, 2000000, (CIRC_BUFFER_SIZE,)) From aisaac at american.edu Thu Nov 1 15:20:11 2007 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 1 Nov 2007 15:20:11 -0400 Subject: [Numpy-discussion] matrix problem: float to matrix power In-Reply-To: References: Message-ID: On Wed, 31 Oct 2007, Timothy Hochberg apparently wrote: > because M**n results in the matrix power of n. It would be > confusing if n**M did a broadcast element wise power. In an attempt to summarize: scalar to a matrix power 1. may have been overlooked, or may have been omitted as confusing 2. if overlooked there are two possible answers as to what it could reasonably be a. a**m = exp(ln(a)*m) natural definition b. a**m = a**m.A broadcasting In languages that provide an answer, I have the impression that the answer is usually (b). (E.g., GAUSS.) However perhaps in NumPy this is the least interesting way to go? But it is what I was expecting. Cheers, Alan Isaac From tim.hochberg at ieee.org Thu Nov 1 15:41:42 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 1 Nov 2007 12:41:42 -0700 Subject: [Numpy-discussion] matrix problem: float to matrix power In-Reply-To: References: Message-ID: On Nov 1, 2007 12:20 PM, Alan G Isaac wrote: > On Wed, 31 Oct 2007, Timothy Hochberg apparently wrote: > > because M**n results in the matrix power of n. It would be > > confusing if n**M did a broadcast element wise power. > > > > In an attempt to summarize: > scalar to a matrix power > > 1. may have been overlooked, or may have been omitted as > confusing > 2. if overlooked there are two possible answers as to what > it could reasonably be > a. a**m = exp(ln(a)*m) natural definition For some definition of natural: it makes sense once you see it, but is seem like most people expected something else (in our four person sample, only one person came up with this definition -- not that that's good statistics.). > > b. a**m = a**m.A broadcasting > > In languages that provide an answer, I have the impression > that the answer is usually (b). (E.g., GAUSS.) However > perhaps in NumPy this is the least interesting way to go? > But it is what I was expecting. I'd inclined to leave it unimplemented. It seems like it's likely to cause confusion and my impression is that it's not a common enough construction that having to use a function should be a big burden for most if not all people. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg.net at gmail.com Thu Nov 1 15:50:10 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 1 Nov 2007 13:50:10 -0600 Subject: [Numpy-discussion] Problem with numpy on Leopard Message-ID: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> Hi, It turns out that Leopard includes numpy. But it is an older version that won't detect the version string of gfortran correctly (thus preventing scipy from being installed). But, when I downloaded the numpy svn and did python setup.py python was still finding the older version of numpy. The problem is a trivial but unfortunate ordering of things in the default sys.path. Here is the message that I sent to the pythonmac-sig about this: ************************************************************************ I have been playing around with python on Leopard today. Overall, I am very pleased, but I just ran into a problem that will affect a large number of users. In Leopard, Apple includes a number of python packages in: ls /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python CoreGraphics easy_install.pyc setuptools-0.6c7-py2.5.egg-info OpenSSL fetchmailconf.py site.py PyObjC fetchmailconf.pyc site.pyc PyObjC.pth fetchmailconf.pyo site.pyo PyRSS2Gen-1.0.0-py2.5.egg-info libsvn svn PyRSS2Gen.py macholib twisted PyRSS2Gen.pyc macholib-1.2.1.dev-py2.5.egg-info wx Twisted-2.4.0-py2.5.egg-info modulegraph wx-2.8-mac-unicode Twisted_Words-0.4.0-py2.5.egg-info modulegraph-0.7.2.dev-py2.5.egg-info wxPython Twisted_Xish-0.4.0-py2.5.egg-info numpy wxPython_common-2.8.4.0-py2.5.egg-info altgraph numpy-1.0.1-py2.5.egg-info wxaddons altgraph-0.6.8.dev-py2.5.egg-info pkg_resources.py wxaddons-2.8.4.0-py2.5.egg-info bdist_mpkg pkg_resources.pyc wxversion.py bdist_mpkg-0.4.3.dev-py2.5.egg-info py2app wxversion.pyc bonjour py2app-0.4.1.dev-py2.5.egg-info xattr bonjour_py-0.2-py2.5.egg-info pyOpenSSL-0.6-py2.5.egg-info xattr-0.5-py2.5.egg-info dateutil python_dateutil-1.2-py2.5.egg-info zope easy_install.py setuptools zope.interface-3.3.0-py2.5.egg-info At first I was very excited to see that Apple is including things like setuptools, numpy, twisted...... But then I saw that some of the versions are quite old. Naturally, I grabbed the latest numpy version from svn and did a python setup.py install. Everything built just fine, but when I did an import numpy, I got Apples version: >>> import numpy >>> print numpy.__file__ /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/numpy/__init__.pyc After looking at sys.path, it became clear what the problem is: Running python setup.py install on Leopard causes packages to be installed in the usual: /Library/Python/2.5/site-packages But, Apple put this directory _after_ /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python in sys.path. This, even if a user installs a newer version of one of these packages, the builtin python will always use Apple's older version. Obviously, I can set PYTHONPATH to get /Library/Python/2.5/site-packages earlier in the sys.path list, but this is going to bite many people. Is this a bug that I should report to Apple, or is there a better work around? Also, this information needs to be posted somewhere so we don't see this question a billion times on various lists over the lifetime of Leopard. Cheers, Brian From matthieu.brucher at gmail.com Thu Nov 1 15:57:43 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 1 Nov 2007 20:57:43 +0100 Subject: [Numpy-discussion] Problem with numpy on Leopard In-Reply-To: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> References: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> Message-ID: Hi, The problem will arise for every package, not only numpy, so Apple fixing this is the best solution IMHO. Matthieu 2007/11/1, Brian Granger : > > Hi, > > It turns out that Leopard includes numpy. But it is an older version > that won't detect the version string of gfortran correctly (thus > preventing scipy from being installed). But, when I downloaded the > numpy svn and did python setup.py python was still finding the older > version of numpy. > > The problem is a trivial but unfortunate ordering of things in the > default sys.path. > > Here is the message that I sent to the pythonmac-sig about this: > ************************************************************************ > > > I have been playing around with python on Leopard today. Overall, I > am very pleased, but I just ran into a problem that will affect a > large number of users. > > In Leopard, Apple includes a number of python packages in: > > ls > /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python > > CoreGraphics easy_install.pyc > setuptools-0.6c7-py2.5.egg-info > OpenSSL fetchmailconf.py > site.py > PyObjC fetchmailconf.pyc > site.pyc > PyObjC.pth fetchmailconf.pyo > site.pyo > PyRSS2Gen-1.0.0-py2.5.egg-info libsvn > svn > PyRSS2Gen.py macholib > twisted > PyRSS2Gen.pyc > macholib-1.2.1.dev-py2.5.egg-info wx > Twisted-2.4.0-py2.5.egg-info modulegraph > wx-2.8-mac-unicode > Twisted_Words-0.4.0-py2.5.egg-info > modulegraph-0.7.2.dev-py2.5.egg-info wxPython > Twisted_Xish-0.4.0-py2.5.egg-info numpy > wxPython_common-2.8.4.0-py2.5.egg-info > altgraph numpy-1.0.1-py2.5.egg-info > wxaddons > altgraph-0.6.8.dev-py2.5.egg-info pkg_resources.py > wxaddons-2.8.4.0-py2.5.egg-info > bdist_mpkg pkg_resources.pyc > wxversion.py > bdist_mpkg-0.4.3.dev-py2.5.egg-info py2app > wxversion.pyc > bonjour > py2app-0.4.1.dev-py2.5.egg-info xattr > bonjour_py-0.2-py2.5.egg-info pyOpenSSL-0.6-py2.5.egg-info > xattr-0.5-py2.5.egg-info > dateutil > python_dateutil-1.2-py2.5.egg-info zope > easy_install.py setuptools > zope.interface-3.3.0-py2.5.egg-info > > At first I was very excited to see that Apple is including things like > setuptools, numpy, twisted...... > > But then I saw that some of the versions are quite old. Naturally, I > grabbed the latest numpy version from svn and did a python setup.py > install. Everything built just fine, but when I did an import numpy, > I got Apples version: > > >>> import numpy > >>> print numpy.__file__ > > /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/numpy/__init__.pyc > > After looking at sys.path, it became clear what the problem is: > > Running python setup.py install on Leopard causes packages to be > installed in the usual: > > /Library/Python/2.5/site-packages > > But, Apple put this directory _after_ > > /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python > > in sys.path. This, even if a user installs a newer version of one of > these packages, the builtin python will always use Apple's older > version. > > Obviously, I can set PYTHONPATH to get > /Library/Python/2.5/site-packages earlier in the sys.path list, but > this is going to bite many people. Is this a bug that I should report > to Apple, or is there a better work around? Also, this information > needs to be posted somewhere so we don't see this question a billion > times on various lists over the lifetime of Leopard. > > Cheers, > > Brian > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Nov 1 16:04:22 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 01 Nov 2007 15:04:22 -0500 Subject: [Numpy-discussion] Problem with numpy on Leopard In-Reply-To: References: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> Message-ID: <472A3146.8000303@gmail.com> Matthieu Brucher wrote: > Hi, > > The problem will arise for every package, not only numpy, so Apple > fixing this is the best solution IMHO. It's unlikely they are going to. If they put that stuff there, it's because they are using it for something, not as an (in)convenience to you. I don't recommend using the Python.framework in /System for anything except for distributing lightweight .apps. In that case, you can control your sys.path very well. For regular work, use the binary from www.python.org. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ellisonbg.net at gmail.com Thu Nov 1 16:44:32 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 1 Nov 2007 14:44:32 -0600 Subject: [Numpy-discussion] Problem with numpy on Leopard In-Reply-To: <472A3146.8000303@gmail.com> References: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> <472A3146.8000303@gmail.com> Message-ID: <6ce0ac130711011344p1ff676f7wd6f3b53dab170a5@mail.gmail.com> > It's unlikely they are going to. If they put that stuff there, it's because they > are using it for something, not as an (in)convenience to you. I don't recommend > using the Python.framework in /System for anything except for distributing > lightweight .apps. In that case, you can control your sys.path very well. For > regular work, use the binary from www.python.org. I agree that they probably won't fix this, but with Leopard, Apple has gone out of their way to integrate the builtin python with PyObjC, the new scripting bridge and Xcode. The python.org binary doesn't have those things, which could be a real barrier to some people. I am going to file a bug report. Brian > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From ellisonbg.net at gmail.com Thu Nov 1 16:52:36 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 1 Nov 2007 14:52:36 -0600 Subject: [Numpy-discussion] Problem with numpy on Leopard In-Reply-To: <472A3146.8000303@gmail.com> References: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> <472A3146.8000303@gmail.com> Message-ID: <6ce0ac130711011352o519c43b5v4f40b36a16a9516@mail.gmail.com> More evidence that just using the python.org python binary isn't a universal fix for everyone: >From a thread on one of the python-dev lists: > Which reminds me -- what version of Python is in Leopard? 2.5.1 + most of the patches that will be in 2.5.2 + some additional patches by Apple. AFAIK the latter are some patches to support their build machinery and patches that add support for DTrace. On 11/1/07, Robert Kern wrote: > Matthieu Brucher wrote: > > Hi, > > > > The problem will arise for every package, not only numpy, so Apple > > fixing this is the best solution IMHO. > > It's unlikely they are going to. If they put that stuff there, it's because they > are using it for something, not as an (in)convenience to you. I don't recommend > using the Python.framework in /System for anything except for distributing > lightweight .apps. In that case, you can control your sys.path very well. For > regular work, use the binary from www.python.org. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From tim.hochberg at ieee.org Thu Nov 1 18:06:50 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 1 Nov 2007 15:06:50 -0700 Subject: [Numpy-discussion] vectorizing loops In-Reply-To: <73A6DD54-4D2C-4D95-9C1F-AE50AF424A8A@physics.mcmaster.ca> References: <472123B6.1050609@jpl.nasa.gov> <200710311118.15239.faltet@carabos.com> <200711011356.38449.faltet@carabos.com> <73A6DD54-4D2C-4D95-9C1F-AE50AF424A8A@physics.mcmaster.ca> Message-ID: On Nov 1, 2007 7:14 AM, David M. Cooke wrote: > On Nov 1, 2007, at 08:56 , Francesc Altet wrote: > > > A Wednesday 31 October 2007, Timothy Hochberg escrigu?: > >> On Oct 31, 2007 3:18 AM, Francesc Altet wrote: > >> > >> [SNIP] > >> > >>> Incidentally, all the improvements of the PyTables flavor of > >>> numexpr have been reported to the original authors, but, for the > >>> sake of keeping numexpr simple, they decided to implement only some > >>> of them. However, people is encouraged to try out the Pytables > >>> flavor from: > >> > >> My original concern was that we'd start overflowing the code cache > >> and slow things down. Some experiments seemed to confirm this on some > >> processors, but it then turned out those were in error. At least > >> that's my recollection. Because of that, and because PyTables is, as > >> far as I know, the major user of numexpr, I suspect I'd be fine > >> putting those changes in now. I say "suspect" since it's been a long > >> time since I looked at the relevant patches, and I should probably > >> look at those again before I commit myself. I just haven't had the > >> free time and motivation to go back and look at the patches. I can't > >> speak for David though. > > > > If I remember correctly, another point where you (specially David) > > were > > not very keen to include was the support for strings, arguing that > > numexpr is meant mainly to deal with numerical data, not textual. > > However, our experience is that adding this support was not too > > complicated (mainly due to NumPy flexibility), and can be helpful in > > some instances. As for one, we use them for evaluating expressions > > like 'array_string == "some_string"', and this can be very convenient > > to use when you are in the middle of potentially complex boolean > > expressions that you want to evaluate *fast*. > > > > At any rate, we would be glad if you would like to integrate our > > patches > > in the main numexpr, as there is not much sense to have different > > implementations of numexpr (most specially when it seems that there > > are > > not much users out there). So, count on us for any question you may > > have in this regard. > > Well, I don't have much time to work on it, but if you make sure your > patches on the scipy Trac apply clean, I'll have a quick look at them > and apply them. Since you've had them working in production code, they > should be good ;-) > > Another issue is that numexpr is still in the scipy sandbox, so only > those who enable it will use it (or use it through PyTables). One > problem with moving it out is that Tim reports the compile times on > Windows are ridiculous (20 mins!). While this is true at the default optimization (O2), it compiles reasonably quickly at O1 and I've never been able to detect a speed difference between versions compiled with O1 versus O2. It would probably be sufficient to crank back the optimization on Windows. > Maybe numexpr should become a > scikit? It certainly doesn't need the rest of scipy. > > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Nov 1 19:12:02 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 01 Nov 2007 16:12:02 -0700 Subject: [Numpy-discussion] Problem with numpy on Leopard In-Reply-To: <472A3146.8000303@gmail.com> References: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> <472A3146.8000303@gmail.com> Message-ID: <472A5D42.5070500@noaa.gov> Robert Kern wrote: >> The problem will arise for every package, not only numpy, so Apple >> fixing this is the best solution IMHO. > > It's unlikely they are going to. If they put that stuff there, it's because they > are using it for something, not as an (in)convenience to you. I don't recommend > using the Python.framework in /System for anything except for distributing > lightweight .apps. AARRGG! This has come up before, when IBM was putting python in systems for some reason. Also, anyone remember RedHat, when upgrading python all their stuff would break. The problem there was that they refused, for some unknown reason, to specify a python version in their #! lines, nor did they specify a path (i.e., they used /usr/bin/env python) -- so you couldn't change the default python without all of RedHat's utilities breaking. The issue here is that if Apple had put site-packages on the path before their Extras dir, then when someone did something like install an upgrade to numpy (what a good idea) there is a chance that some Apple-supplied utility that relies on numpy would break. Hence Roberts solution: treat the Apple Python as a system only tool, only to be added to by Apple themselves. I guess that's OK, but it's really silly that it has to be that way. The solution: support versioning of packages in python! This came up some years ago, when wxPython developed a versioning system -- it would have made much more sense for their to be a standard way to do it, but the core folks on python-dev didn't see the point. oh well. The way I see it, python is a very good, robust, general purpose tool. It's a great thing for OS suppliers to provide python for system and users use -- I'd love to see it treated as other system libraries and utilities are, but to do that, there must be a versioning system for packages -- so Apple can use the version they installed, and a user can install and upgrade along side it, and not break anything. It's analygous to shared libraries -- it's insane to use shared libs without versions - we'd have to statically link everything. Having to install all of Python, and all those packages because you want to upgrade numpy just seems crazy. Sorry about that! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From wbaxter at gmail.com Thu Nov 1 19:49:37 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 2 Nov 2007 08:49:37 +0900 Subject: [Numpy-discussion] C or C++ package like NumPy? Message-ID: Does anyone know of a C or C++ library that's similar to NumPy? Seems like all the big C++ efforts are focused on linear algebra rather than general purpose multidimensional arrays. I've written a multidimensional array class in the D programming language with an API modeled loosely after NumPy's. It works ok but there are a lot of differences between a statically typed language like C++ or D and a dynamic one like Python. So I was looking around for API inspiration from other C/C++ libraries. But the projects I know about are all focused on linear algebra (like ublas and MTL) and don't support general N-dimensional arrays. Thanks for your comments. --bb From nad at acm.org Thu Nov 1 19:46:01 2007 From: nad at acm.org (Ned Deily) Date: Thu, 01 Nov 2007 16:46:01 -0700 Subject: [Numpy-discussion] Problem with numpy on Leopard References: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> <472A3146.8000303@gmail.com> <472A5D42.5070500@noaa.gov> Message-ID: In article <472A5D42.5070500 at noaa.gov>, Christopher Barker wrote: > [...] AARRGG! > [...] > Hence Roberts solution: treat the Apple Python as a system only tool, > only to be added to by Apple themselves. I guess that's OK, but it's > really silly that it has to be that way. > > The solution: support versioning of packages in python! This came up > some years ago, when wxPython developed a versioning system -- it would > have made much more sense for their to be a standard way to do it, but > the core folks on python-dev didn't see the point. oh well. Ah, but there is a de-facto standard multi-platform Python versioning system out there in ever increasing use: setuptools (a.k.a easy_install). It's just *not* *quite* there yet, as in not yet being in the standard library. Still, it is designed to be easily bootstrapped into vanilla systems, it has the best chance of making it there of anything else out there, and the fact that it is plug-compatible with distutils for most applications is a huge plus. > The way I see it, python is a very good, robust, general purpose tool. > It's a great thing for OS suppliers to provide python for system and > users use -- I'd love to see it treated as other system libraries and > utilities are, but to do that, there must be a versioning system for > packages -- so Apple can use the version they installed, and a user can > install and upgrade along side it, and not break anything. > > It's analygous to shared libraries -- it's insane to use shared libs > without versions - we'd have to statically link everything. Having to > install all of Python, and all those packages because you want to > upgrade numpy just seems crazy. There's another way to look at the issue in the OSX world, though. Apple strongly encourages the use of application packages and the tools provided for Python on OSX, i.e. py2app, make it easy for developers to provide robust, stand-alone applications without dependencies on shared libraries. Yes, that may lead to some wasted disk space, with multiple copies of otherwise potentially sharable libraries hidden under the covers of the application, but delivering a stand-alone application via a single drag-and-drop disk image rather than with an installer that has to manage library dependencies makes the extra disk space a small price to pay. Or, at least, that's what we're encouraged to believe. I do after seeing problems like the one you're wrestling with now. -- Ned Deily, nad at acm.org From focke at slac.stanford.edu Thu Nov 1 20:06:07 2007 From: focke at slac.stanford.edu (Warren Focke) Date: Thu, 1 Nov 2007 17:06:07 -0700 (PDT) Subject: [Numpy-discussion] C or C++ package like NumPy? In-Reply-To: References: Message-ID: http://www.oonumerics.org/blitz/ On Fri, 2 Nov 2007, Bill Baxter wrote: > Does anyone know of a C or C++ library that's similar to NumPy? > Seems like all the big C++ efforts are focused on linear algebra > rather than general purpose multidimensional arrays. > > I've written a multidimensional array class in the D programming > language with an API modeled loosely after NumPy's. It works ok but > there are a lot of differences between a statically typed language > like C++ or D and a dynamic one like Python. So I was looking around > for API inspiration from other C/C++ libraries. But the projects I > know about are all focused on linear algebra (like ublas and MTL) and > don't support general N-dimensional arrays. > > Thanks for your comments. > --bb > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From wbaxter at gmail.com Thu Nov 1 20:29:58 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 2 Nov 2007 09:29:58 +0900 Subject: [Numpy-discussion] C or C++ package like NumPy? In-Reply-To: References: Message-ID: Ah, ok. Thanks. That does look like a good example. I've heard of it, but never looked too closely for some reason. I guess I always thought of it as the library that pioneered expression templates but that no one actually uses. --bb On Nov 2, 2007 9:06 AM, Warren Focke wrote: > http://www.oonumerics.org/blitz/ > > > On Fri, 2 Nov 2007, Bill Baxter wrote: > > > Does anyone know of a C or C++ library that's similar to NumPy? > > Seems like all the big C++ efforts are focused on linear algebra > > rather than general purpose multidimensional arrays. > > > > I've written a multidimensional array class in the D programming > > language with an API modeled loosely after NumPy's. It works ok but > > there are a lot of differences between a statically typed language > > like C++ or D and a dynamic one like Python. So I was looking around > > for API inspiration from other C/C++ libraries. But the projects I > > know about are all focused on linear algebra (like ublas and MTL) and > > don't support general N-dimensional arrays. > > > > Thanks for your comments. > > --bb > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Thu Nov 1 21:50:29 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Nov 2007 19:50:29 -0600 Subject: [Numpy-discussion] numpy FFT memory accumulation In-Reply-To: <6.2.3.4.2.20071101105313.058fbf88@rjs.org> References: <6.2.3.4.2.20071101105313.058fbf88@rjs.org> Message-ID: On 11/1/07, Ray S wrote: > > At 09:00 AM 11/1/2007, Chuck wrote: > > In Python, collections.deque makes a pretty good circular buffer. > Numpy will > make an array out of it, which involves a copy, but it might be > better than what you are doing now. > > hmmm, I'll think more about that - and the copy is only at program > start, it seems > the fft can always be fft(d[:-N]) > >>> from collections import deque > >>> import numpy as N > >>> d = deque(N.zeros(10,)) > >>> d.extend(N.ones(4,)) > >>> d > deque([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, > 1.0, 1.0]) > >>> [d.pop() for i in range(4)] > [1.0, 1.0, 1.0, 1.0] Yeah, I noticed that inconvenience. >>> d > deque([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]) > > An additional complication is that I pass the numpy (or Numeric) > array address to the ctypes library call so that the data is placed > directly into the array from the call. I use the if/else end wrap > logic to determine whether I need to do a split and copy if the new > data wraps. OK. Hmm, I wonder if you would lose much by taking a straight forward radix-2 fft and teaching it to use modular indices? Probably not worth the trouble, but an fft tailored to a ring buffer might be useful for other things. Probably the easiest thing is to just copy the ring buffer out into a linear array. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 1 21:52:12 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Nov 2007 19:52:12 -0600 Subject: [Numpy-discussion] numpy FFT memory accumulation In-Reply-To: <6.2.3.4.2.20071101111616.06d30b28@rjs.org> References: <6.2.3.4.2.20071101111616.06d30b28@rjs.org> Message-ID: On 11/1/07, Ray S wrote: > > At 09:00 AM 11/1/2007, you wrote: > I saw that Numeric did also (I still use Numeric for smaller array > speed) but much more slowly. > I will try to repeat with a small demo and post. > > It turns out to be some aspect of mixing numpy and Numeric; > > the attached *Stable.py files allocate memory that stays exactly the > same > > mixedGrows.py grows by about 50kB/s while changing slightly, > randomly; note that the array is assigned as numpy and operated on > with Numeric functions. > > My main app also uses mmap/numpy to assign the arrays to shared > memory for multiple processes, and does other operations as well, > possibly accounting for the faster memory growth. > I'll re-factor to an all-numpy solution and test. > > I used Numeric functions for the ~40% speed increase, but I don't I know that numarray was slow in creating small arrays, but is Numpy really that bad compared to Numeric? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 1 22:14:48 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Nov 2007 20:14:48 -0600 Subject: [Numpy-discussion] C or C++ package like NumPy? In-Reply-To: References: Message-ID: On 11/1/07, Bill Baxter wrote: > > Ah, ok. Thanks. That does look like a good example. > I've heard of it, but never looked too closely for some reason. I > guess I always thought of it as the library that pioneered expression > templates but that no one actually uses. I believe it is no longer maintained. I might be wrong about that, though. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg.net at gmail.com Fri Nov 2 00:11:36 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 1 Nov 2007 22:11:36 -0600 Subject: [Numpy-discussion] [Pythonmac-SIG] Problem with numpy on Leopard In-Reply-To: <24d517dd0711011647u1a05d42ckbae5355a6f0ef74f@mail.gmail.com> References: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> <472A3146.8000303@gmail.com> <472A5D42.5070500@noaa.gov> <24d517dd0711011647u1a05d42ckbae5355a6f0ef74f@mail.gmail.com> Message-ID: <6ce0ac130711012111hde8356ew47a17c506698b449@mail.gmail.com> > It's not entirely silly. This has been the advice given to app > developers on this list and the PyObjC list for years now. It's nice > to have a better system Python for quick scripts, but it's still the > System Python. It's Apples, for their stuff that uses Python. And it > is specific to OS 10.5. If you use it to build your app, your app is > pretty much guaranteed not to work when someone tries to run it on > 10.6, etc. At some level I agree - I haven't used the system python on Tiger for a long time now. But, that is not how Apple "sells" the python on their system. > Maybe they have changed how py2app works in 10.5, but the old behavior > was to embed Python into your app, UNLESS it was the system python. > Unless there is evidence to the contrary (I don't have Leopard yet), I > assume that is still true. > > The old advice still holds: For building apps, install your own > framework build of Python, put it first in your path, and add any > libraries you require to it. Then you have control over what goes > into your app, not Apple. > > --Dethe > _______________________________________________ > Pythonmac-SIG maillist - Pythonmac-SIG at python.org > http://mail.python.org/mailman/listinfo/pythonmac-sig > From matthieu.brucher at gmail.com Fri Nov 2 02:50:05 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 2 Nov 2007 07:50:05 +0100 Subject: [Numpy-discussion] C or C++ package like NumPy? In-Reply-To: References: Message-ID: It is not although it is used in scipy (without everything to use it with Visual Studio) You can look at Vigra (but I don't know if there is linear algebra, but there are views, multidimensional containers, ...). Matthieu 2007/11/2, Charles R Harris : > > > > On 11/1/07, Bill Baxter wrote: > > > > Ah, ok. Thanks. That does look like a good example. > > I've heard of it, but never looked too closely for some reason. I > > guess I always thought of it as the library that pioneered expression > > templates but that no one actually uses. > > > I believe it is no longer maintained. I might be wrong about that, though. > > > Chuck > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Fri Nov 2 02:57:02 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 2 Nov 2007 15:57:02 +0900 Subject: [Numpy-discussion] C or C++ package like NumPy? In-Reply-To: References: Message-ID: On Nov 2, 2007 3:50 PM, Matthieu Brucher wrote: > You can look at Vigra (but I don't know if there is linear algebra, but > there are views, multidimensional containers, ...). Thanks for the link. Hadn't heard of that one. --bb From nad at acm.org Fri Nov 2 02:59:11 2007 From: nad at acm.org (Ned Deily) Date: Thu, 01 Nov 2007 23:59:11 -0700 Subject: [Numpy-discussion] Problem with numpy on Leopard References: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> <472A3146.8000303@gmail.com> <472A5D42.5070500@noaa.gov> <24d517dd0711011647u1a05d42ckbae5355a6f0ef74f@mail.gmail.com> <6ce0ac130711012111hde8356ew47a17c506698b449@mail.gmail.com> <07Nov1.205537pst."57996"@synergy1.parc.xerox.com> Message-ID: In article <07Nov1.205537pst."57996"@synergy1.parc.xerox.com>, Bill Janssen wrote: > This is really interesting. For my apps, I use the system Python on > Tiger, and expect to do it again with Leopard. If I have to have a > specific version of an extension that's needed, I make sure the > directory where I keep it is on the sys.path of my application before > the standard directories. That seems to be about all that's required. > I'm not sure what all the FUD about the system Python is... For your own apps on your own machines, sure - you can manage the whole environment and the system Python should be fine (that is, if and until you need a newer build of Python itself). I think the "FUD" you refer to concerns a different problem: developing and distributing multiple-component Python apps to multiple users on multiple machines. Controlling all those environments isn't so easy. Deploying apps and components as eggs using setuptools is one way to simplify that problem; building standalone OSX apps with embedded stripped-down Python runtimes and components using py2app is another. -- Ned Deily, nad at acm.org From matthieu.brucher at gmail.com Fri Nov 2 03:00:55 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 2 Nov 2007 08:00:55 +0100 Subject: [Numpy-discussion] C or C++ package like NumPy? In-Reply-To: References: Message-ID: Sorry : http://kogs-www.informatik.uni-hamburg.de/~koethe/vigra/ It has some publications written about the design it uses (iterators and such), really well done. Matthieu 2007/11/2, Bill Baxter : > > On Nov 2, 2007 3:50 PM, Matthieu Brucher > wrote: > > > You can look at Vigra (but I don't know if there is linear algebra, but > > there are views, multidimensional containers, ...). > > Thanks for the link. Hadn't heard of that one. > > --bb > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Fri Nov 2 03:08:08 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 2 Nov 2007 16:08:08 +0900 Subject: [Numpy-discussion] C or C++ package like NumPy? In-Reply-To: References: Message-ID: Oh, I didn't realize you didn't give a link. I just googled reflexively. Anyway, that makes me think of the other generic image library I've heard of -- Adobe's GIL. Never really looked at it in much detail but checking now, it looks like it does support N-dim images. http://opensource.adobe.com/gil/html/gildesignguide.html#ImageSectionDG --bb On Nov 2, 2007 4:00 PM, Matthieu Brucher wrote: > Sorry : http://kogs-www.informatik.uni-hamburg.de/~koethe/vigra/ > It has some publications written about the design it uses (iterators and > such), really well done. > > Matthieu > > 2007/11/2, Bill Baxter : > > > > > > > > On Nov 2, 2007 3:50 PM, Matthieu Brucher > wrote: > > > > > You can look at Vigra (but I don't know if there is linear algebra, but > > > there are views, multidimensional containers, ...). > > > > Thanks for the link. Hadn't heard of that one. > > > > --bb > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > French PhD student > Website : http://miles.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From faltet at carabos.com Fri Nov 2 09:40:27 2007 From: faltet at carabos.com (Francesc Altet) Date: Fri, 2 Nov 2007 14:40:27 +0100 Subject: [Numpy-discussion] vectorizing loops In-Reply-To: <73A6DD54-4D2C-4D95-9C1F-AE50AF424A8A@physics.mcmaster.ca> References: <472123B6.1050609@jpl.nasa.gov> <200711011356.38449.faltet@carabos.com> <73A6DD54-4D2C-4D95-9C1F-AE50AF424A8A@physics.mcmaster.ca> Message-ID: <200711021440.28333.faltet@carabos.com> A Thursday 01 November 2007, David M. Cooke escrigu?: > > At any rate, we would be glad if you would like to integrate our > > patches > > in the main numexpr, as there is not much sense to have different > > implementations of numexpr (most specially when it seems that there > > are > > not much users out there). So, count on us for any question you > > may have in this regard. > > Well, I don't have much time to work on it, but if you make sure your > patches on the scipy Trac apply clean, I'll have a quick look at them > and apply them. Since you've had them working in production code, > they should be good ;-) Well, being in production and around 10000 tests (where numexpr is involved) in the pytables package seems a good guarantee to me too ;-) I've attached a clean patch to ticket #529 of SciPy site: http://scipy.org/scipy/scipy/ticket/529 However, I've committed a couple of mistakes during ticket creation, so: - in #529, I've uploaded the patch twice, so they are exactly the same - please mark #530 as invalid (duplicated) Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Fri Nov 2 09:55:21 2007 From: faltet at carabos.com (Francesc Altet) Date: Fri, 2 Nov 2007 14:55:21 +0100 Subject: [Numpy-discussion] vectorizing loops In-Reply-To: References: <472123B6.1050609@jpl.nasa.gov> <73A6DD54-4D2C-4D95-9C1F-AE50AF424A8A@physics.mcmaster.ca> Message-ID: <200711021455.21871.faltet@carabos.com> A Thursday 01 November 2007, Timothy Hochberg escrigu?: > On Nov 1, 2007 7:14 AM, David M. Cooke > > Another issue is that numexpr is still in the scipy sandbox, so > > only those who enable it will use it (or use it through PyTables). > > One problem with moving it out is that Tim reports the compile > > times on Windows are ridiculous (20 mins!). > > While this is true at the default optimization (O2), it compiles > reasonably quickly at O1 and I've never been able to detect a speed > difference between versions compiled with O1 versus O2. It would > probably be sufficient to crank back the optimization on Windows. Yes. This has been my experience too on Windows/MSVC boxes. > > Maybe numexpr should become a > > scikit? It certainly doesn't need the rest of scipy. Call me intrepid, but I've always felt that numexpr belongs more to numpy itself than scipy. However, I agree that perhaps it should be a bit more polished (but not much; perhaps just adding some functions like, exp, log, log10... would be enough) before being integrated. At any rate, Numexpr would be a extremely useful complement to NumPy. My two cents, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From ndbecker2 at gmail.com Fri Nov 2 10:09:25 2007 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 02 Nov 2007 10:09:25 -0400 Subject: [Numpy-discussion] C or C++ package like NumPy? References: Message-ID: Charles R Harris wrote: > On 11/1/07, Bill Baxter wrote: >> >> Ah, ok. Thanks. That does look like a good example. >> I've heard of it, but never looked too closely for some reason. I >> guess I always thought of it as the library that pioneered expression >> templates but that no one actually uses. > > > I believe it is no longer maintained. I might be wrong about that, though. > > Chuck It is being maintained, at least to some extent. See: blitz-support at lists.sourceforge.net From subscriber100 at rjs.org Fri Nov 2 10:43:46 2007 From: subscriber100 at rjs.org (Ray Schumacher) Date: Fri, 02 Nov 2007 06:43:46 -0800 Subject: [Numpy-discussion] numpy FFT memory accumulation In-Reply-To: References: Message-ID: <6.2.3.4.2.20071102062412.03514eb0@rjs.org> At 10:57 PM 11/1/2007, Charles R Harris wrote: > > An additional complication is that I pass the numpy (or Numeric) > > array address to the ctypes library call so that the data is placed > > directly into the array from the call. I use the if/else end wrap > > logic to determine whether I need to do a split and copy if the new > > data wraps. > >OK. Hmm, I wonder if you would lose much by taking a straight forward >radix-2 fft and teaching it to use modular indices? Probably not worth the >trouble, but an fft tailored to a ring buffer might be useful for other >things. The problem is, I once compiled my own FFT dll to call from Python and it was 2x slower than FFTPACK - I'm not math-smart enough to make all of the caching and numerical shortcuts. That, and Intel's optimized FFT is 3x faster than FFTPACK... I may still try to do a "zoom/range FFT" which does not compute all bins, and maybe only with a sine transform, which (I think) should be sufficient to determine peak real frequency and maybe use fewer cycles. >Probably the easiest thing is to just copy the ring buffer out into >a linear array. I do that for the buffer-wrap condition, while simply assigning a slice (no copy) to the temp array otherwise. > > I used Numeric functions for the ~40% speed increase, but I don't > >I know that numarray was slow in creating small arrays, but is Numpy >really that bad compared to Numeric? I just saw the # of FFTs/sec go from 390 to 550 just by switching numpy to Numeric (Intel Core Duo). Add a timer to my previous code posts and see how your results look. For the mega-arrays a lot of the numpy developers work with it is much faster, and I now find Numeric is lacking many other niceties, like frombuffer(). Ray -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.15.18/1104 - Release Date: 11/1/2007 6:47 PM From ellisonbg.net at gmail.com Fri Nov 2 11:33:11 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 2 Nov 2007 09:33:11 -0600 Subject: [Numpy-discussion] [Pythonmac-SIG] Problem with numpy on Leopard In-Reply-To: <2964868492913213348@unknownmsgid> References: <6ce0ac130711011250o73d50c1du944fb62a3263c6ad@mail.gmail.com> <472A3146.8000303@gmail.com> <472A5D42.5070500@noaa.gov> <24d517dd0711011647u1a05d42ckbae5355a6f0ef74f@mail.gmail.com> <6ce0ac130711012111hde8356ew47a17c506698b449@mail.gmail.com> <2964868492913213348@unknownmsgid> Message-ID: <6ce0ac130711020833l34f4c881r1813941aeb38acf@mail.gmail.com> On 11/1/07, Bill Janssen wrote: > > > It's not entirely silly. This has been the advice given to app > > > developers on this list and the PyObjC list for years now. It's nice > > > to have a better system Python for quick scripts, but it's still the > > > System Python. It's Apples, for their stuff that uses Python. And it > > > is specific to OS 10.5. If you use it to build your app, your app is > > > pretty much guaranteed not to work when someone tries to run it on > > > 10.6, etc. > > > > At some level I agree - I haven't used the system python on Tiger for > > a long time now. But, that is not how Apple "sells" the python on > > their system. > > This is really interesting. For my apps, I use the system Python on > Tiger, and expect to do it again with Leopard. If I have to have a > specific version of an extension that's needed, I make sure the > directory where I keep it is on the sys.path of my application before > the standard directories. That seems to be about all that's required. > I'm not sure what all the FUD about the system Python is... There is little fear, uncertainty or doubt about this. The problem is that this represents a huge change of behavior for users making the transition to Leopard. On Tiger, a user could download any basically any python pacakge and do "python setup.py install" and things would work. Now under Leopard, this becomes "python setup.py install" + muck with PYTHONPATH or .pth files. I personally have no problem doing this, but many of the casual python users with whom I work will be tripped up by this....and they will come and ask me what to do....over and over again.... Brian From ellisonbg.net at gmail.com Fri Nov 2 12:16:36 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 2 Nov 2007 10:16:36 -0600 Subject: [Numpy-discussion] Round 2 with Leopard+Python Message-ID: <6ce0ac130711020916pf348e27xc5543dbd49f5fce0@mail.gmail.com> Hi, In the process of working through the issues with sys.path on Leopard, I have found another potential Leopard bug that is particularly nasty. In Tiger, sudo preserves environment variables: $ export FOO=/tmp $ python -c "import os; print os.environ['FOO']" /tmp $ sudo python -c "import os; print os.environ['FOO']" /tmp But, in Leopard, sudo does not perserve environment variables: $ export FOO=/tmp $ python -c "import os; print os.environ['FOO']" /tmp $ sudo python -c "import os; print os.environ['FOO']" Password: Traceback (most recent call last): File "", line 1, in File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/UserDict.py", line 22, in __getitem__ raise KeyError(key) KeyError: 'FOO' This is a big problem. First, if you have set PYTHONPATH to point sys.path at the site-packages in /Library, this setting will be lost when you do: sudo python setup.py install On another package. I encountered this in building pytables, which requires numpy >= 1.0.3. I had installed numpy 1.0.4, and set my PYTHONPATH to point to it. But, the pytables setup.py script failts because PYTHONPATH is lost and it only sees the older (1.0.1) builtin numpy. Second, some setup.py scripts use environment variables to determine how things are built, find other dependencies, etc. Currently, this will fail on Leopard if such packages are installed into locations that require sudo. I haven't tried it yet, but I expect that this will also hold true for other python installations. The behavior also shows up with ruby on Leopard. The solution currently is to install all packages to locations that don't require sudo to write to. I will file a bug report, but until the bug is fixed, we should explore putting a note on the numpy/scipy site - and even possibly on the python.org site to describe the problem and its workaround. Brian From bblais at bryant.edu Fri Nov 2 12:58:33 2007 From: bblais at bryant.edu (Brian Blais) Date: Fri, 2 Nov 2007 12:58:33 -0400 Subject: [Numpy-discussion] weird numpy/pickle problem Message-ID: <4641B5EC-552D-4B2A-872C-86A6D602C52D@bryant.edu> Hello, I encountered a peculiar numpy and pickle problem. My version: Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04) Mac OS X Tiger In [2]:numpy.__version__ Out[2]:'1.0.4.dev3869' I pickle a matrix, and reload it. Some operations work ok, but others give a hardware error and a crash! I boiled it down to the code below. Can anyone reproduce (or not) this error? thanks, bb -- Brian Blais bblais at bryant.edu http://web.bryant.edu/~bblais ======================================================================= from numpy import matrix,random import pickle w_test=matrix(random.rand(3,21)) fid=open('blah0_test.pickle','wb') pickle.dump(w_test,fid,0) fid.close() fid=open('blah0_test.pickle','rb') w_xh=pickle.load(fid) fid.close() # this works print w_xh*w_xh.T #this works x=matrix(random.rand(2,21)) print x*w_xh.T # this gives a hardware error! x=matrix([0]*21) print x*w_xh.T -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.martinsen-burrell at wartburg.edu Fri Nov 2 13:32:54 2007 From: n.martinsen-burrell at wartburg.edu (Neil Martinsen-Burrell) Date: Fri, 02 Nov 2007 12:32:54 -0500 Subject: [Numpy-discussion] weird numpy/pickle problem In-Reply-To: Message-ID: On 11/2/07 12:00, "numpy-discussion-request at scipy.org" wrote: > Date: Fri, 2 Nov 2007 12:58:33 -0400 > From: Brian Blais > Subject: [Numpy-discussion] weird numpy/pickle problem > To: numpy-discussion at scipy.org > > I boiled it down to the code below. Can anyone reproduce (or not) > this error? Works for me on OS X 10.4.9, Python 2.5.0, numpy 1.0.4dev3977. -Neil -- "The theory of probabilities is at bottom nothing but common sense reduced to calculus." -- Pierre Simon de Laplace From stefan at sun.ac.za Fri Nov 2 14:31:13 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 2 Nov 2007 20:31:13 +0200 Subject: [Numpy-discussion] weird numpy/pickle problem In-Reply-To: <4641B5EC-552D-4B2A-872C-86A6D602C52D@bryant.edu> References: <4641B5EC-552D-4B2A-872C-86A6D602C52D@bryant.edu> Message-ID: <20071102183113.GA911@mentat.za.net> On Fri, Nov 02, 2007 at 12:58:33PM -0400, Brian Blais wrote: > I encountered a peculiar numpy and pickle problem. My version: > > Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04) Mac OS X Tiger > In [2]:numpy.__version__ > Out[2]:'1.0.4.dev3869' > > I pickle a matrix, and reload it. Some operations work ok, but others give a > hardware error and a crash! > > I boiled it down to the code below. Can anyone reproduce (or not) > this error? This ticket looks similar: http://projects.scipy.org/scipy/numpy/ticket/551 Cheers St?fan From hep.sebastien.binet at gmail.com Fri Nov 2 18:53:30 2007 From: hep.sebastien.binet at gmail.com (Sebastien Binet) Date: Fri, 2 Nov 2007 15:53:30 -0700 Subject: [Numpy-discussion] C or C++ package like NumPy? In-Reply-To: References: Message-ID: <200711021553.37279.binet@cern.ch> Hi, > Does anyone know of a C or C++ library that's similar to NumPy? > Seems like all the big C++ efforts are focused on linear algebra > rather than general purpose multidimensional arrays. > > I've written a multidimensional array class in the D programming > language with an API modeled loosely after NumPy's. It works ok but > there are a lot of differences between a statically typed language > like C++ or D and a dynamic one like Python. So I was looking around > for API inspiration from other C/C++ libraries. But the projects I > know about are all focused on linear algebra (like ublas and MTL) and > don't support general N-dimensional arrays. Maybe Boost.MultiArray could also be considered. http://www.boost.org/libs/multi_array/doc/user.html Cheers, Sebastien. -- ################################### # Dr. Sebastien Binet # # Lawrence Berkeley National Lab. # # 1 Cyclotron Road # # Berkeley, CA 94720 # ################################### -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From millman at berkeley.edu Sat Nov 3 05:02:50 2007 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 3 Nov 2007 02:02:50 -0700 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release Message-ID: Hello, NumPy 1.0.4 is almost out! I will be tagging the release this Sunday (11/4), so please be extra careful about anything you check-in over the next 24 hours. Once the tag is made, the trunk will be open for changes. I plan to announce the 1.0.4 release by Wednesday (11/7). I am hoping that everyone will be able to take the time to test the NumPy trunk this weekend and alert me of any problems. Please check-out the trunk, build it, and run the full test suite: svn co http://svn.scipy.org/svn/numpy/trunk ./numpy-trunk cd numpy-trunk python setup.py build sudo python setup.py install python import numpy numpy.test(10,10) Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Sun Nov 4 14:15:41 2007 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 4 Nov 2007 11:15:41 -0800 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: <014401c81e4c$9dec8390$a885e892@sun.ac.za> References: <014401c81e4c$9dec8390$a885e892@sun.ac.za> Message-ID: On Nov 3, 2007 11:06 AM, Albert Strasheim wrote: > Things mostly look good as far as the buildslaves are concerned, except on > Windows x64, where one of the regression tests is failing: > > http://buildbot.scipy.org/Windows%20XP%20x86_64%20MSVC/builds/163/step-shell_2/0 Hello, Can someone take a look at the problem with NumPy on Windows XP? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From woiski at gmail.com Sun Nov 4 14:53:51 2007 From: woiski at gmail.com (Emanuel Woiski) Date: Sun, 4 Nov 2007 17:53:51 -0200 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: References: <014401c81e4c$9dec8390$a885e892@sun.ac.za> Message-ID: <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> On Nov 4, 2007 5:15 PM, Jarrod Millman wrote: > On Nov 3, 2007 11:06 AM, Albert Strasheim wrote: > > Things mostly look good as far as the buildslaves are concerned, except > on > > Windows x64, where one of the regression tests is failing: > > > > > http://buildbot.scipy.org/Windows%20XP%20x86_64%20MSVC/builds/163/step-shell_2/0 > > Hello, > > Can someone take a look at the problem with NumPy on Windows XP? > Dear Sirs I am havig a lot of trouble for quite some time wih numpy - scipy Windows binaries for old AMD (pre-XP) processors. Since I teach in a lab with lots of machines using that kind of processor, I got stuck with numpy 1.0rc2 and scipy 0.5.1, which is, believe me, the only combination that woks...Well not only tests fail but Python crashes! FYI there are many people over the world still using not-so-recent computers... Yes I have tried s0.6 and n1.0.3.1 without success. > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > Regards woiski -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Sun Nov 4 15:18:48 2007 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 4 Nov 2007 12:18:48 -0800 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> References: <014401c81e4c$9dec8390$a885e892@sun.ac.za> <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> Message-ID: On Nov 4, 2007 11:53 AM, Emanuel Woiski wrote: > I am havig a lot of trouble for quite some time wih numpy - scipy Windows > binaries for old AMD (pre-XP) processors. Since I teach in a lab with lots > of machines using that kind of processor, I got stuck with numpy 1.0rc2 and > scipy 0.5.1, which is, believe me, the only combination that woks...Well not > only tests fail but Python crashes! > FYI there are many people over the world still using not-so-recent > computers... > Yes I have tried s0.6 and n1.0.3.1 without success. Are you building from source? If so, have you tried the most recent development code? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From bmschwar at fas.harvard.edu Sun Nov 4 15:51:27 2007 From: bmschwar at fas.harvard.edu (Benjamin M. Schwartz) Date: Sun, 04 Nov 2007 15:51:27 -0500 Subject: [Numpy-discussion] Making a minimalist NumPy Message-ID: <472E30CF.5030102@fas.harvard.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 NumPy is included in the OLPC operating system, which is very constrained in space. Therefore, it would be nice to remove some subpackages to save a few megabytes. For example, the system does not include any Fortran code or compiler, so f2py (3.6 MB) seems superfluous. I also think the distutils subpackage (1.9M) is probably not necessary. Therefore, I have two questions. 1. Which packages do you think are necessary to have a functioning NumPy? 2. What is the easiest way to make (or get) a minimal NumPy installation? For example, would the scons/autoconf branch make this easier? - --Ben Schwartz OLPC volunteer -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHLjDPUJT6e6HFtqQRAnG/AKCYfJ/RZwg3+5i6ofHt6PpwhgusiQCfRkZa QYvFDDRXuLyy4lxt7VswnOs= =uzX/ -----END PGP SIGNATURE----- From efiring at hawaii.edu Sun Nov 4 16:13:48 2007 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 04 Nov 2007 11:13:48 -1000 Subject: [Numpy-discussion] segfault Message-ID: <472E360C.90609@hawaii.edu> A quick scan of the tickets did not show me anything like the following, but I might have missed it. The attached script generates a segfault on my ubuntu feisty system with svn numpy. Running inside of ipython, the segfault occurs upon exiting ipython, not upon running the script. Running the script from the command line gives the attached trace. Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: numpysegfault.py Type: text/x-python Size: 73 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: segfaultdump.txt URL: From matthieu.brucher at gmail.com Sun Nov 4 16:21:06 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sun, 4 Nov 2007 22:21:06 +0100 Subject: [Numpy-discussion] Numpy book and PyArray_CastToType Message-ID: Hi, I'm trying to use PyArray_CastToType(), but according to the book, there are two arguments, a PyArrayObject* and a PyArrayDescr*, but according to the header file, there are three arguments, an additional int. Does someone know its use ? Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Nov 4 17:55:39 2007 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 05 Nov 2007 00:55:39 +0200 Subject: [Numpy-discussion] segfault In-Reply-To: <472E360C.90609@hawaii.edu> References: <472E360C.90609@hawaii.edu> Message-ID: <1194216939.5136.2.camel@localhost.localdomain> su, 2007-11-04 kello 11:13 -1000, Eric Firing kirjoitti: > A quick scan of the tickets did not show me anything like the following, > but I might have missed it. The attached script generates a segfault on > my ubuntu feisty system with svn numpy. Running inside of ipython, the > segfault occurs upon exiting ipython, not upon running the script. > Running the script from the command line gives the attached trace. Also on numpy 1.0.3.1, self-built on Ubuntu gutsy. Valgrind shows this: $ valgrind --suppressions=/home/pauli/.valgrind/valgrind-python.supp python -c 'import numpy; u = numpy.array([]); g = numpy.array([True, True]); u[g] = 0' ==17983== Memcheck, a memory error detector. ==17983== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al. ==17983== Using LibVEX rev 1732, a library for dynamic binary translation. ==17983== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP. ==17983== Using valgrind-3.2.3-Debian, a dynamic binary instrumentation framework. ==17983== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al. ==17983== For more details, rerun with: -v ==17983== ==17983== Invalid write of size 4 ==17983== at 0x46275AF: DOUBLE_copyswap (arraytypes.inc.src:999) ==17983== by 0x4655C80: iter_ass_subscript (arrayobject.c:9348) ==17983== by 0x4658620: array_ass_sub (arrayobject.c:2971) ==17983== by 0x80C550D: PyEval_EvalFrameEx (ceval.c:1497) ==17983== by 0x80CA114: PyEval_EvalCodeEx (ceval.c:2831) ==17983== by 0x80CA186: PyEval_EvalCode (ceval.c:494) ==17983== by 0x80EA371: PyRun_StringFlags (pythonrun.c:1273) ==17983== by 0x80EA434: PyRun_SimpleStringFlags (pythonrun.c:900) ==17983== by 0x8058DBD: Py_Main (main.c:512) ==17983== by 0x80588C1: main (python.c:23) ==17983== Address 0x433C390 is 0 bytes after a block of size 8 alloc'd ==17983== at 0x4021765: malloc (vg_replace_malloc.c:149) ==17983== by 0x4634A77: PyArray_NewFromDescr (arrayobject.c:5430) ==17983== by 0x463BA49: PyArray_FromAny (arrayobject.c:7428) ==17983== by 0x464C025: PyArray_CheckFromAny (arrayobject.c:8607) ==17983== by 0x464C261: _array_fromobject (multiarraymodule.c:5640) ==17983== by 0x80C8EEB: PyEval_EvalFrameEx (ceval.c:3564) ==17983== by 0x80CA114: PyEval_EvalCodeEx (ceval.c:2831) ==17983== by 0x80CA186: PyEval_EvalCode (ceval.c:494) ==17983== by 0x80EA371: PyRun_StringFlags (pythonrun.c:1273) ==17983== by 0x80EA434: PyRun_SimpleStringFlags (pythonrun.c:900) ==17983== by 0x8058DBD: Py_Main (main.c:512) ==17983== by 0x80588C1: main (python.c:23) ==17983== ==17983== Invalid write of size 4 ==17983== at 0x46275B4: DOUBLE_copyswap (arraytypes.inc.src:999) ==17983== by 0x4655C80: iter_ass_subscript (arrayobject.c:9348) ==17983== by 0x4658620: array_ass_sub (arrayobject.c:2971) ==17983== by 0x80C550D: PyEval_EvalFrameEx (ceval.c:1497) ==17983== by 0x80CA114: PyEval_EvalCodeEx (ceval.c:2831) ==17983== by 0x80CA186: PyEval_EvalCode (ceval.c:494) ==17983== by 0x80EA371: PyRun_StringFlags (pythonrun.c:1273) ==17983== by 0x80EA434: PyRun_SimpleStringFlags (pythonrun.c:900) ==17983== by 0x8058DBD: Py_Main (main.c:512) ==17983== by 0x80588C1: main (python.c:23) ==17983== Address 0x433C394 is 4 bytes after a block of size 8 alloc'd ==17983== at 0x4021765: malloc (vg_replace_malloc.c:149) ==17983== by 0x4634A77: PyArray_NewFromDescr (arrayobject.c:5430) ==17983== by 0x463BA49: PyArray_FromAny (arrayobject.c:7428) ==17983== by 0x464C025: PyArray_CheckFromAny (arrayobject.c:8607) ==17983== by 0x464C261: _array_fromobject (multiarraymodule.c:5640) ==17983== by 0x80C8EEB: PyEval_EvalFrameEx (ceval.c:3564) ==17983== by 0x80CA114: PyEval_EvalCodeEx (ceval.c:2831) ==17983== by 0x80CA186: PyEval_EvalCode (ceval.c:494) ==17983== by 0x80EA371: PyRun_StringFlags (pythonrun.c:1273) ==17983== by 0x80EA434: PyRun_SimpleStringFlags (pythonrun.c:900) ==17983== by 0x8058DBD: Py_Main (main.c:512) ==17983== by 0x80588C1: main (python.c:23) ==17983== ==17983== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1395 from 8) ==17983== malloc/free: in use at exit: 2,042,554 bytes in 691 blocks. ==17983== malloc/free: 73,854 allocs, 73,163 frees, 35,375,892 bytes allocated. ==17983== For counts of detected errors, rerun with: -v ==17983== searching for pointers to 691 not-freed blocks. ==17983== checked 3,292,072 bytes. ==17983== ==17983== LEAK SUMMARY: ==17983== definitely lost: 0 bytes in 0 blocks. ==17983== possibly lost: 36,150 bytes in 57 blocks. ==17983== still reachable: 2,006,404 bytes in 634 blocks. ==17983== suppressed: 0 bytes in 0 blocks. ==17983== Rerun with --leak-check=full to see details of leaked memory. From cookedm at physics.mcmaster.ca Sun Nov 4 18:42:26 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sun, 4 Nov 2007 18:42:26 -0500 Subject: [Numpy-discussion] Making a minimalist NumPy In-Reply-To: <472E30CF.5030102@fas.harvard.edu> References: <472E30CF.5030102@fas.harvard.edu> Message-ID: <0CA2C5AC-A902-4405-9B58-BA4B479653B9@physics.mcmaster.ca> On Nov 4, 2007, at 15:51 , Benjamin M. Schwartz wrote: > NumPy is included in the OLPC operating system, which is very > constrained in > space. Therefore, it would be nice to remove some subpackages to > save a few > megabytes. For example, the system does not include any Fortran > code or > compiler, so f2py (3.6 MB) seems superfluous. I also think the > distutils > subpackage (1.9M) is probably not necessary. Therefore, I have two > questions. > > 1. Which packages do you think are necessary to have a functioning > NumPy? > > 2. What is the easiest way to make (or get) a minimal NumPy > installation? For > example, would the scons/autoconf branch make this easier? The *biggest* single optimisation for space that you could make is not to have both .pyc and .pyo files. AFAIK, the only difference between the two now is that .pyo files don't have asserts included. Testing on build 625 of the OLPC runnng in VMWare, that removes about 3MB from the numpy package right there (and even more when done globally -- about 25MB.). [btw, the Python test/ directory would be another 14MB.] After that, 1) remove f2py -- as you say, no Fortran, no need (2.2MB) 2) remove test directories "find . -name tests -type d -exec rm -rf {} ;' (1MB) 3) remove distutils (1MB) Now, it's down to about 3.5 MB; there's not much more that can be done after that. While I'm confident that removing the f2py and test directories can be done safely (by just removing the directories), I'm not so sure about numpy.distutils -- it really depends on what other software you're using. It could be trimmed a bit: the Fortran compiler descriptions in distutils/fcompiler/ could be removed, and system_info.py could be cut down. Although you can make up for the extra space it uses by removing Numeric (2MB). -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From david at ar.media.kyoto-u.ac.jp Sun Nov 4 21:58:41 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 05 Nov 2007 11:58:41 +0900 Subject: [Numpy-discussion] Round 2 with Leopard+Python In-Reply-To: <6ce0ac130711020916pf348e27xc5543dbd49f5fce0@mail.gmail.com> References: <6ce0ac130711020916pf348e27xc5543dbd49f5fce0@mail.gmail.com> Message-ID: <472E86E1.8000407@ar.media.kyoto-u.ac.jp> Brian Granger wrote: > Hi, > > In the process of working through the issues with sys.path on Leopard, > I have found another potential Leopard bug that is particularly nasty. > > In Tiger, sudo preserves environment variables: > > $ export FOO=/tmp > $ python -c "import os; print os.environ['FOO']" > /tmp > $ sudo python -c "import os; print os.environ['FOO']" > /tmp > > But, in Leopard, sudo does not perserve environment variables: > > $ export FOO=/tmp > $ python -c "import os; print os.environ['FOO']" > /tmp > $ sudo python -c "import os; print os.environ['FOO']" > Password: > Traceback (most recent call last): > File "", line 1, in > File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/UserDict.py", > line 22, in __getitem__ > raise KeyError(key) > KeyError: 'FOO' > > This is a big problem. First, if you have set PYTHONPATH to point > sys.path at the site-packages in /Library, this setting will be lost > when you do: > > sudo python setup.py install > > On another package. I encountered this in building pytables, which > requires numpy >= 1.0.3. I had installed numpy 1.0.4, and set my > PYTHONPATH to point to it. But, the pytables setup.py script failts > because PYTHONPATH is lost and it only sees the older (1.0.1) builtin > numpy. > > Second, some setup.py scripts use environment variables to determine > how things are built, find other dependencies, etc. Currently, this > will fail on Leopard if such packages are installed into locations > that require sudo. I haven't tried it yet, but I expect that this > will also hold true for other python installations. The behavior also > shows up with ruby on Leopard. > > The solution currently is to install all packages to locations that > don't require sudo to write to. I will file a bug report, but until > the bug is fixed, we should explore putting a note on the numpy/scipy > site - and even possibly on the python.org site to describe the > problem and its workaround. > Have you tried the env_reset option (man sudoers) ? IMHO, the Leopard behaviour looks saner than Tiger, from what you are saying. Having PYTHONPATH overridable by the user looks like a good tool for unwanted privileges escalation... Actually, looking a bit at sudo NEWS file, you can see that PYTHONPATH was added as an env variable to disable something like 2 years ago, which explains the behaviour (sudo has been updated, I suppose, and Tiger is a bit more than 2 years old if I remember correctly). Do you *really* need to install numpy in a location only writable through sudo ? cheers, David From devnew at gmail.com Mon Nov 5 02:37:27 2007 From: devnew at gmail.com (devnew at gmail.com) Date: Sun, 04 Nov 2007 23:37:27 -0800 Subject: [Numpy-discussion] how to convert btw rgb and pixel value Message-ID: <1194248247.421519.106550@y27g2000pre.googlegroups.com> hi i wish to convert an rgb image into an array of double values..is there a method for that in numpy? also i want to create an array of doubles into corresponding rgb tuples of an image can anyone guide me? dn From matthieu.brucher at gmail.com Mon Nov 5 03:26:21 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 5 Nov 2007 09:26:21 +0100 Subject: [Numpy-discussion] how to convert btw rgb and pixel value In-Reply-To: <1194248247.421519.106550@y27g2000pre.googlegroups.com> References: <1194248247.421519.106550@y27g2000pre.googlegroups.com> Message-ID: It depends on the transformation you want to use, there is no method available in Numpy, but you can create your own in an instant. Matthieu 2007/11/5, devnew at gmail.com : > > hi > i wish to convert an rgb image into an array of double values..is > there a method for that in numpy? > also i want to create an array of doubles into corresponding rgb > tuples of an image > can anyone guide me? > dn > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Mon Nov 5 05:58:45 2007 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 05 Nov 2007 12:58:45 +0200 Subject: [Numpy-discussion] how to convert btw rgb and pixel value In-Reply-To: <1194248247.421519.106550@y27g2000pre.googlegroups.com> References: <1194248247.421519.106550@y27g2000pre.googlegroups.com> Message-ID: <1194260325.30441.2.camel@nadav.envision.co.il> If the image is in the form of a standard image format (png, bmp, jpeg...) you can use the PIL library. png file can be read by pylab's imread function. Nadav. On Sun, 2007-11-04 at 23:37 -0800, devnew at gmail.com wrote: > hi > i wish to convert an rgb image into an array of double values..is > there a method for that in numpy? > also i want to create an array of doubles into corresponding rgb > tuples of an image > can anyone guide me? > dn > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From woiski at gmail.com Mon Nov 5 05:54:22 2007 From: woiski at gmail.com (Emanuel Woiski) Date: Mon, 5 Nov 2007 08:54:22 -0200 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: References: <014401c81e4c$9dec8390$a885e892@sun.ac.za> <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> Message-ID: <9c08f0d10711050254s35ee5431u64d3991eb7e14da@mail.gmail.com> On Nov 4, 2007 6:18 PM, Jarrod Millman wrote: > On Nov 4, 2007 11:53 AM, Emanuel Woiski wrote: > > I am havig a lot of trouble for quite some time wih numpy - scipy > Windows > > binaries for old AMD (pre-XP) processors. Since I teach in a lab with > lots > > of machines using that kind of processor, I got stuck with numpy 1.0rc2and > > scipy 0.5.1, which is, believe me, the only combination that woks...Well > not > > only tests fail but Python crashes! > > FYI there are many people over the world still using not-so-recent > > computers... > > Yes I have tried s0.6 and n1.0.3.1 without success. > > Are you building from source? If so, have you tried the most recent > development code? > Nope, just WinXP (and WIn2000) binaries.... Thanks woiski > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdh2358 at gmail.com Mon Nov 5 15:57:11 2007 From: jdh2358 at gmail.com (John Hunter) Date: Mon, 5 Nov 2007 14:57:11 -0600 Subject: [Numpy-discussion] boolean masks & lists Message-ID: <88e473830711051257y79594cb1uc93fe779537d2e41@mail.gmail.com> A colleague of mine just asked for help with a pesky bug that turned out to be caused by his use of a list of booleans rather than an array of booleans as his logical indexing mask. I assume this is a feature and not a bug, but it certainly surprised him: In [58]: mask = [True, False, False, False, True] In [59]: maska = n.array(mask, n.bool) In [60]: x = arange(5) In [61]: x[mask] Out[61]: array([1, 0, 0, 0, 1]) In [62]: x[maska] Out[62]: array([0, 4]) From david at ar.media.kyoto-u.ac.jp Tue Nov 6 02:25:28 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 06 Nov 2007 16:25:28 +0900 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: <9c08f0d10711050254s35ee5431u64d3991eb7e14da@mail.gmail.com> References: <014401c81e4c$9dec8390$a885e892@sun.ac.za> <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> <9c08f0d10711050254s35ee5431u64d3991eb7e14da@mail.gmail.com> Message-ID: <473016E8.9060801@ar.media.kyoto-u.ac.jp> Emanuel Woiski wrote: > > > On Nov 4, 2007 6:18 PM, Jarrod Millman > wrote: > > On Nov 4, 2007 11:53 AM, Emanuel Woiski > wrote: > > I am havig a lot of trouble for quite some time wih numpy - > scipy Windows > > binaries for old AMD (pre-XP) processors. Since I teach in a lab > with lots > > of machines using that kind of processor, I got stuck with numpy > 1.0rc2 and > > scipy 0.5.1, which is, believe me, the only combination that > woks...Well not > > only tests fail but Python crashes! > > FYI there are many people over the world still using not-so-recent > > computers... > > Yes I have tried s0.6 and n1.0.3.1 without success. > > Are you building from source? If so, have you tried the most recent > development code? > > > Nope, just WinXP (and WIn2000) binaries.... The problem is a distribution problem (that is how the binaries were built). Do you need optimized BLAS/LAPACK (for fast linear algebra) ? If not, would you mind trying those ? http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.0.3.1.win32-py2.5.exe http://www.ar.media.kyoto-u.ac.jp/members/david/archives/scipy-0.6.0.win32-py2.5.exe Note that those do not use any optimized libraries (no ATLAS, no fftw, etc...). This is the bare minimal to make it work on any x86 machine (no need for SSE/SSE2). cheers, David From woiski at gmail.com Tue Nov 6 07:05:28 2007 From: woiski at gmail.com (Emanuel Woiski) Date: Tue, 6 Nov 2007 10:05:28 -0200 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: <473016E8.9060801@ar.media.kyoto-u.ac.jp> References: <014401c81e4c$9dec8390$a885e892@sun.ac.za> <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> <9c08f0d10711050254s35ee5431u64d3991eb7e14da@mail.gmail.com> <473016E8.9060801@ar.media.kyoto-u.ac.jp> Message-ID: <9c08f0d10711060405r263de2fci78c03fa99bc3e682@mail.gmail.com> Thanks. I will try them later and let you know. Any chance to have those for 2.4 as well?:) If it works, I will be able to upgrade to matplotlib 0.90.1.... Regards woiski On Nov 6, 2007 5:25 AM, David Cournapeau wrote: > Emanuel Woiski wrote: > > > > > > On Nov 4, 2007 6:18 PM, Jarrod Millman > > wrote: > > > > On Nov 4, 2007 11:53 AM, Emanuel Woiski > > wrote: > > > I am havig a lot of trouble for quite some time wih numpy - > > scipy Windows > > > binaries for old AMD (pre-XP) processors. Since I teach in a lab > > with lots > > > of machines using that kind of processor, I got stuck with numpy > > 1.0rc2 and > > > scipy 0.5.1, which is, believe me, the only combination that > > woks...Well not > > > only tests fail but Python crashes! > > > FYI there are many people over the world still using not-so-recent > > > computers... > > > Yes I have tried s0.6 and n1.0.3.1 without success. > > > > Are you building from source? If so, have you tried the most recent > > development code? > > > > > > Nope, just WinXP (and WIn2000) binaries.... > The problem is a distribution problem (that is how the binaries were > built). Do you need optimized BLAS/LAPACK (for fast linear algebra) ? If > not, would you mind trying those ? > > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.0.3.1.win32-py2.5.exe > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/scipy-0.6.0.win32-py2.5.exe > > Note that those do not use any optimized libraries (no ATLAS, no fftw, > etc...). This is the bare minimal to make it work on any x86 machine (no > need for SSE/SSE2). > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Nov 6 06:59:51 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 06 Nov 2007 20:59:51 +0900 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: <9c08f0d10711060405r263de2fci78c03fa99bc3e682@mail.gmail.com> References: <014401c81e4c$9dec8390$a885e892@sun.ac.za> <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> <9c08f0d10711050254s35ee5431u64d3991eb7e14da@mail.gmail.com> <473016E8.9060801@ar.media.kyoto-u.ac.jp> <9c08f0d10711060405r263de2fci78c03fa99bc3e682@mail.gmail.com> Message-ID: <47305737.3030804@ar.media.kyoto-u.ac.jp> Emanuel Woiski wrote: > Thanks. I will try them later and let you know. Any chance to have > those for 2.4 as well?:) > If it works, I will be able to upgrade to matplotlib 0.90.1.... Well, you will have to find someone else. Using windows is already painful enough: I don't have the motivation to build for every supported python versions :) cheers, David From woiski at gmail.com Tue Nov 6 07:44:13 2007 From: woiski at gmail.com (Emanuel Woiski) Date: Tue, 6 Nov 2007 10:44:13 -0200 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: <47305737.3030804@ar.media.kyoto-u.ac.jp> References: <014401c81e4c$9dec8390$a885e892@sun.ac.za> <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> <9c08f0d10711050254s35ee5431u64d3991eb7e14da@mail.gmail.com> <473016E8.9060801@ar.media.kyoto-u.ac.jp> <9c08f0d10711060405r263de2fci78c03fa99bc3e682@mail.gmail.com> <47305737.3030804@ar.media.kyoto-u.ac.jp> Message-ID: <9c08f0d10711060444x3a355f09tbdb57c6b1d9aa1ad@mail.gmail.com> yep I have no choice - those are a bunch of lab machines....:) thanks anyway regards woiski On Nov 6, 2007 9:59 AM, David Cournapeau wrote: > Emanuel Woiski wrote: > > Thanks. I will try them later and let you know. Any chance to have > > those for 2.4 as well?:) > > If it works, I will be able to upgrade to matplotlib 0.90.1.... > Well, you will have to find someone else. Using windows is already > painful enough: I don't have the motivation to build for every supported > python versions :) > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Nov 6 07:57:02 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 06 Nov 2007 21:57:02 +0900 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: <9c08f0d10711060444x3a355f09tbdb57c6b1d9aa1ad@mail.gmail.com> References: <014401c81e4c$9dec8390$a885e892@sun.ac.za> <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> <9c08f0d10711050254s35ee5431u64d3991eb7e14da@mail.gmail.com> <473016E8.9060801@ar.media.kyoto-u.ac.jp> <9c08f0d10711060405r263de2fci78c03fa99bc3e682@mail.gmail.com> <47305737.3030804@ar.media.kyoto-u.ac.jp> <9c08f0d10711060444x3a355f09tbdb57c6b1d9aa1ad@mail.gmail.com> Message-ID: <4730649E.8070702@ar.media.kyoto-u.ac.jp> Emanuel Woiski wrote: > yep I have no choice - those are a bunch of lab machines....:) > thanks anyway > regards > woiski Can't you build it yourself ? You just have to install mingw, and compiling blas/lapack is not too difficult. cheers, David From dalcinl at gmail.com Tue Nov 6 09:22:44 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 6 Nov 2007 11:22:44 -0300 Subject: [Numpy-discussion] boolean masks & lists In-Reply-To: <88e473830711051257y79594cb1uc93fe779537d2e41@mail.gmail.com> References: <88e473830711051257y79594cb1uc93fe779537d2e41@mail.gmail.com> Message-ID: Mmm... It looks as it 'mask' is being inernally converted from [True, False, False, False, True] to [1, 0, 0, 0, 1] so your are finally getting x[1], x[0], x[0], x[0], x[1] On 11/5/07, John Hunter wrote: > A colleague of mine just asked for help with a pesky bug that turned > out to be caused by his use of a list of booleans rather than an array > of booleans as his logical indexing mask. I assume this is a feature > and not a bug, but it certainly surprised him: > > In [58]: mask = [True, False, False, False, True] > > In [59]: maska = n.array(mask, n.bool) > > In [60]: x = arange(5) > > In [61]: x[mask] > Out[61]: array([1, 0, 0, 0, 1]) > > In [62]: x[maska] > Out[62]: array([0, 4]) > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From jdh2358 at gmail.com Tue Nov 6 09:31:11 2007 From: jdh2358 at gmail.com (John Hunter) Date: Tue, 6 Nov 2007 08:31:11 -0600 Subject: [Numpy-discussion] boolean masks & lists In-Reply-To: References: <88e473830711051257y79594cb1uc93fe779537d2e41@mail.gmail.com> Message-ID: <88e473830711060631k46dc798ahbb39be0aa38c206f@mail.gmail.com> On Nov 6, 2007 8:22 AM, Lisandro Dalcin wrote: > Mmm... > It looks as it 'mask' is being inernally converted from > [True, False, False, False, True] > to > [1, 0, 0, 0, 1] Yep, clearly. The question is: is this the desired behavior because it leads to a "silent failure" for people who are expecting sequences of booleans to behave like arrays of booleans. JDH From tim.hochberg at ieee.org Tue Nov 6 09:33:19 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 6 Nov 2007 07:33:19 -0700 Subject: [Numpy-discussion] boolean masks & lists In-Reply-To: References: <88e473830711051257y79594cb1uc93fe779537d2e41@mail.gmail.com> Message-ID: On Nov 6, 2007 7:22 AM, Lisandro Dalcin wrote: > Mmm... > It looks as it 'mask' is being inernally converted from > [True, False, False, False, True] > to > [1, 0, 0, 0, 1] > > so your are finally getting > > x[1], x[0], x[0], x[0], x[1] That would be my guess as well. And, it looks wrong to me. Given that array(mask) gives you the boolean mask, there's every expectation to expect the the list and array cases should be the same. I would file a bug report. > > > > On 11/5/07, John Hunter wrote: > > A colleague of mine just asked for help with a pesky bug that turned > > out to be caused by his use of a list of booleans rather than an array > > of booleans as his logical indexing mask. I assume this is a feature > > and not a bug, but it certainly surprised him: > > > > In [58]: mask = [True, False, False, False, True] > > > > In [59]: maska = n.array(mask, n.bool) > > > > In [60]: x = arange(5) > > > > In [61]: x[mask] > > Out[61]: array([1, 0, 0, 0, 1]) > > > > In [62]: x[maska] > > Out[62]: array([0, 4]) > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Nov 6 11:02:13 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 06 Nov 2007 10:02:13 -0600 Subject: [Numpy-discussion] boolean masks & lists In-Reply-To: <88e473830711051257y79594cb1uc93fe779537d2e41@mail.gmail.com> References: <88e473830711051257y79594cb1uc93fe779537d2e41@mail.gmail.com> Message-ID: <47309005.80000@enthought.com> John Hunter wrote: > A colleague of mine just asked for help with a pesky bug that turned > out to be caused by his use of a list of booleans rather than an array > of booleans as his logical indexing mask. I assume this is a feature > and not a bug, but it certainly surprised him: > > In [58]: mask = [True, False, False, False, True] > > In [59]: maska = n.array(mask, n.bool) > > In [60]: x = arange(5) > > In [61]: x[mask] > Out[61]: array([1, 0, 0, 0, 1]) > > In [62]: x[maska] > Out[62]: array([0, 4]) > The issues is how to determine what behavior is desired and how to manage that with all the possibilities. Right now the rule is that only boolean arrays are treated as masks and integer arrays mean indexing. Lists are always interpreted as integer indexing (except in certain special cases where the list has slice objects in it). Changing this to something like: lists are interpreted either as boolean or integer arrays, may be reasonable, but I don't see how we can change it until 1.1 because the rule has already been specified and is consistent if not obvious in this simple case. The question is: is anybody using boolean lists specifically according to the rule in place now? -Travis From pgmdevlist at gmail.com Tue Nov 6 11:56:03 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 6 Nov 2007 11:56:03 -0500 Subject: [Numpy-discussion] Assessing the use of packages Message-ID: <200711061156.03652.pgmdevlist@gmail.com> All, It's evaluation time in my department (Bio. & Ag. Engng, UGA), and I'd need to document the impact of my Python contributions on the scientific community at large, or more realistically on the numpy/scipy user community... * Is there a way to estimate how many people installed one particular package from the SVN ? * if it's too tricky, would anybody using maskedarray and/or timeseries and/or pyloess (the SVN packages I helped implementing) mind dropping me a line off-list, with a very short description of the main field of research/use ? I'd basically need some kind of numbers to give to the People-In-Charge. Thanks a lot in advance for your time. P. PS: Sorry for the cross-posting.. From robert.kern at gmail.com Tue Nov 6 13:25:31 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 06 Nov 2007 12:25:31 -0600 Subject: [Numpy-discussion] Assessing the use of packages In-Reply-To: <200711061156.03652.pgmdevlist@gmail.com> References: <200711061156.03652.pgmdevlist@gmail.com> Message-ID: <4730B19B.5010007@gmail.com> Pierre GM wrote: > All, > > It's evaluation time in my department (Bio. & Ag. Engng, UGA), and I'd need to > document the impact of my Python contributions on the scientific community at > large, or more realistically on the numpy/scipy user community... > > * Is there a way to estimate how many people installed one particular package > from the SVN ? No. We could only record who has checked them out from SVN. However, everyone who has checked out the scipy trunk will have gotten the packages you are concerned with. Whether or not they've built them or used them is another matter which we cannot determine. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Tue Nov 6 18:18:24 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 06 Nov 2007 17:18:24 -0600 Subject: [Numpy-discussion] Making a minimalist NumPy In-Reply-To: <472E30CF.5030102@fas.harvard.edu> References: <472E30CF.5030102@fas.harvard.edu> Message-ID: <4730F640.5010100@enthought.com> Benjamin M. Schwartz wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > NumPy is included in the OLPC operating system, which is very constrained in > space. Therefore, it would be nice to remove some subpackages to save a few > megabytes. For example, the system does not include any Fortran code or > compiler, so f2py (3.6 MB) seems superfluous. I also think the distutils > subpackage (1.9M) is probably not necessary. Therefore, I have two questions. > > 1. Which packages do you think are necessary to have a functioning NumPy? > > 2. What is the easiest way to make (or get) a minimal NumPy installation? For > example, would the scons/autoconf branch make this easier? > * You can get rid of f2py, oldnumeric, numarray, and testing. * If you don't need to support building of c-extensions then distutils can also be tossed. To make it, you should be able to just edit the numpy/numpy/setup.py script to remove adding those sub-packages. Then, python setup.py install should work. Let me know if you need further help. -Travis O. From haase at msg.ucsf.edu Wed Nov 7 08:08:05 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 7 Nov 2007 14:08:05 +0100 Subject: [Numpy-discussion] SWIGed function does not like my boolean array Message-ID: Hi, I have a templated function written in C++. My SWIG typemap instantiates this for many argument array types (such as unit8, int16, uint16, int32, float32,float64,...) All works well, except when feeding in a boolean array as in seb.mmms(a>6) I get this error: NotImplementedError: No matching function for overloaded 'mmms' What should I do ? Preferably I would avoid having to add another type-instantiation into the library (it looks already quite bloated having 6+ versions of every function). Isn't bool just a synonym for int32 ? Thanks Sebastian Haase From matthieu.brucher at gmail.com Wed Nov 7 08:54:25 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 7 Nov 2007 14:54:25 +0100 Subject: [Numpy-discussion] SWIGed function does not like my boolean array In-Reply-To: References: Message-ID: Hi, No, a bool is not an int32. Try just sizeof(bool) to be sure (on my box, it's one). Besides, if you use a std::vector of bool, be aware of the fact that it is not like the other vectors. Matthieu 2007/11/7, Sebastian Haase : > > Hi, > I have a templated function written in C++. > My SWIG typemap instantiates this for many argument array types (such > as unit8, int16, uint16, int32, float32,float64,...) > All works well, except when feeding in a boolean array as in > seb.mmms(a>6) > I get this error: > NotImplementedError: No matching function for overloaded 'mmms' > > What should I do ? > Preferably I would avoid having to add another type-instantiation into > the library (it looks already quite bloated having 6+ versions of > every function). Isn't bool just a synonym for int32 ? > > Thanks > Sebastian Haase > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Nov 7 08:45:37 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 07 Nov 2007 22:45:37 +0900 Subject: [Numpy-discussion] numpy.scons: 2nd alpha Message-ID: <4731C181.7090604@ar.media.kyoto-u.ac.jp> Hi, I have just finished to implement everything I wanted to do for a 2nd alpha for numpy.scons. Most of the work since 1st alpha was infrastructure polishing (performance library, fortran): now, most common performance libraries are fully supported (MKL and ATLAS on linux, sunperf on solaris, vecLib/Accelerate on Mac OS X). I would appreciate people testing the branch http://projects.scipy.org/scipy/numpy/wiki/NumpyScons#Gettingthecode There are minor problem preventing the thing to build on windows for some reasons, but this will be fixed soon. Basically, the whole design is now done, and it should be able to support all the platforms (and more) currently supported by numpy.distutils. I would really like to hear about people intimated with numpy.distutils, and package developers for some things I may have missed. I think numpy.scons is now much better and easier as a build system, but I am obviously biased. Most significant changes since 1st alpha: Documentation ============= I started to put more documentation on the numpy wiki: http://projects.scipy.org/scipy/numpy/wiki/NumpyScons There is also some basic description on the design: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/distutils/scons/doc/DESIGN Configuration check =================== - performance libraries are globally supported (default and site.cfg configuration): MKL, ATLAS, Accelerate/vecLib and Sunperf are implemented. Adding a new performance library can be as easy as a 5 lines code. - 'meta' checkers are now reusing code from performance libraries: Both CBLAS and LAPACK are implemented. This means that numpy can now be entirely built with scons, using optimized libraries on the following platforms: - solaris with sunperf and sunstudio (this one has not been extensively tested: it looks like every solaris machine I have access to behaves really differently). - mac os X and Gnu toolchain - linux + ATLAS/MKL + gnu toolchain I am now relatively happy with the code for BLAS/LAPACK and performance libraries implementation. It is easily extensible and quite flexible. Generci library check ===================== I have updated and improved the basic checker, NumpyCheckLibAndHeader. This should cover most basic needs (check for a library with some functions and some headers), and should be able to replace system_info for many cases. For example, the following will check for the function sf_open in the library sndfile, with header sndfile.h: config.NumpyCheckLibAndHeader(*'sndfile'*, *'sf_open'*, *'sndfile.h'*, section = *'sndfile'*) section is optional, and tells the checker to use sndfile section in site.cfg if found. Fortran support =============== I have polished fortran support: in perticular, name mangling for F77 and F90 is implemented and has been tested with g77/gfortran/ifort/sunfort on various platforms. It also defines variables usable by f2py (http://projects.scipy.org/scipy/numpy/wiki/NumpySconsExtExamples#UsingFortran). cheers, David From haase at msg.ucsf.edu Wed Nov 7 10:08:39 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 7 Nov 2007 16:08:39 +0100 Subject: [Numpy-discussion] SWIGed function does not like my boolean array In-Reply-To: References: Message-ID: O.K., so sizeof(bool) is 1. But I already have a function instantiation for uint8. The problem is that without doing "some magic" the compiler (?, or numpy ?) would never allow to use anything but a "dedicated" bool-typed function for bool. Even though the CPU treats bool exactly like an integer of same byte-width. How can I have numpy (or is SWIG the problem ??) use my existing integer function for a bool-type array ? (Without making a differently-typed copy of course ...) Thanks, Sebastian PS: I'm not using the C++ std library. On Nov 7, 2007 2:54 PM, Matthieu Brucher wrote: > Hi, > > No, a bool is not an int32. Try just sizeof(bool) to be sure (on my box, > it's one). > Besides, if you use a std::vector of bool, be aware of the fact that it is > not like the other vectors. > > Matthieu > > 2007/11/7, Sebastian Haase : > > > > > > > > Hi, > > I have a templated function written in C++. > > My SWIG typemap instantiates this for many argument array types (such > > as unit8, int16, uint16, int32, float32,float64,...) > > All works well, except when feeding in a boolean array as in > > seb.mmms(a>6) > > I get this error: > > NotImplementedError: No matching function for overloaded 'mmms' > > > > What should I do ? > > Preferably I would avoid having to add another type-instantiation into > > the library (it looks already quite bloated having 6+ versions of > > every function). Isn't bool just a synonym for int32 ? > > > > Thanks > > Sebastian Haase > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > French PhD student > Website : http://miles.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From matthieu.brucher at gmail.com Wed Nov 7 10:45:14 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 7 Nov 2007 16:45:14 +0100 Subject: [Numpy-discussion] SWIGed function does not like my boolean array In-Reply-To: References: Message-ID: Unfortunately, operations on boolean are not the same as operations on integers, so you can't replace one with another. Matthieu 2007/11/7, Sebastian Haase : > > O.K., so sizeof(bool) is 1. > But I already have a function instantiation for uint8. > > The problem is that without doing "some magic" the compiler (?, or > numpy ?) would never allow to use anything but a "dedicated" > bool-typed function for bool. > Even though the CPU treats bool exactly like an integer of same > byte-width. > > How can I have numpy (or is SWIG the problem ??) use my existing > integer function for a bool-type array ? > (Without making a differently-typed copy of course ...) > > Thanks, > Sebastian > > PS: I'm not using the C++ std library. > > > On Nov 7, 2007 2:54 PM, Matthieu Brucher > wrote: > > Hi, > > > > No, a bool is not an int32. Try just sizeof(bool) to be sure (on my box, > > it's one). > > Besides, if you use a std::vector of bool, be aware of the fact that it > is > > not like the other vectors. > > > > Matthieu > > > > 2007/11/7, Sebastian Haase : > > > > > > > > > > > > Hi, > > > I have a templated function written in C++. > > > My SWIG typemap instantiates this for many argument array types (such > > > as unit8, int16, uint16, int32, float32,float64,...) > > > All works well, except when feeding in a boolean array as in > > > seb.mmms(a>6) > > > I get this error: > > > NotImplementedError: No matching function for overloaded 'mmms' > > > > > > What should I do ? > > > Preferably I would avoid having to add another type-instantiation into > > > the library (it looks already quite bloated having 6+ versions of > > > every function). Isn't bool just a synonym for int32 ? > > > > > > Thanks > > > Sebastian Haase > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > -- > > French PhD student > > Website : http://miles.developpez.com/ > > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > > LinkedIn : http://www.linkedin.com/in/matthieubrucher > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Wed Nov 7 11:20:20 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 7 Nov 2007 17:20:20 +0100 Subject: [Numpy-discussion] SWIGed function does not like my boolean array In-Reply-To: References: Message-ID: On Nov 7, 2007 4:45 PM, Matthieu Brucher wrote: > Unfortunately, operations on boolean are not the same as operations on > integers, so you can't replace one with another. I don't understand. I'm thinking of most math functions in the C-library. In C a boolean is just an integer of 0 or 1 (quasi, by definition). Could you explain what you mean ? Thanks, Sebastian > > 2007/11/7, Sebastian Haase < haase at msg.ucsf.edu>: > > O.K., so sizeof(bool) is 1. > > But I already have a function instantiation for uint8. > > > > The problem is that without doing "some magic" the compiler (?, or > > numpy ?) would never allow to use anything but a "dedicated" > > bool-typed function for bool. > > Even though the CPU treats bool exactly like an integer of same > byte-width. > > > > How can I have numpy (or is SWIG the problem ??) use my existing > > integer function for a bool-type array ? > > (Without making a differently-typed copy of course ...) > > > > Thanks, > > Sebastian > > > > PS: I'm not using the C++ std library. > > > > > > On Nov 7, 2007 2:54 PM, Matthieu Brucher > wrote: > > > Hi, > > > > > > No, a bool is not an int32. Try just sizeof(bool) to be sure (on my box, > > > it's one). > > > Besides, if you use a std::vector of bool, be aware of the fact that it > is > > > not like the other vectors. > > > > > > Matthieu > > > > > > 2007/11/7, Sebastian Haase < haase at msg.ucsf.edu>: > > > > > > > > > > > > > > > > Hi, > > > > I have a templated function written in C++. > > > > My SWIG typemap instantiates this for many argument array types (such > > > > as unit8, int16, uint16, int32, float32,float64,...) > > > > All works well, except when feeding in a boolean array as in > > > > seb.mmms(a>6) > > > > I get this error: > > > > NotImplementedError: No matching function for overloaded 'mmms' > > > > > > > > What should I do ? > > > > Preferably I would avoid having to add another type-instantiation into > > > > the library (it looks already quite bloated having 6+ versions of > > > > every function). Isn't bool just a synonym for int32 ? > > > > > > > > Thanks > > > > Sebastian Haase > > > > _______________________________________________ > > > > Numpy-discussion mailing list > > > > Numpy-discussion at scipy.org > > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > > > > -- > > > French PhD student > > > Website : http://miles.developpez.com/ > > > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > > > LinkedIn : http://www.linkedin.com/in/matthieubrucher > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > French PhD student > Website : http://miles.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From matthieu.brucher at gmail.com Wed Nov 7 11:23:04 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 7 Nov 2007 17:23:04 +0100 Subject: [Numpy-discussion] SWIGed function does not like my boolean array In-Reply-To: References: Message-ID: > > I don't understand. I'm thinking of most math functions in the > C-library. In C a boolean is just an integer of 0 or 1 (quasi, by > definition). > > Could you explain what you mean ? > In C++, bool is a new type that has two values, true and false. If you add true and true, it is still true, and not 2. In C, everything that is not 0 is true, not in C++. Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Wed Nov 7 12:35:41 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 7 Nov 2007 18:35:41 +0100 Subject: [Numpy-discussion] SWIGed function does not like my boolean array In-Reply-To: References: Message-ID: On Nov 7, 2007 5:23 PM, Matthieu Brucher wrote: > > > I don't understand. I'm thinking of most math functions in the > > C-library. In C a boolean is just an integer of 0 or 1 (quasi, by > > definition). > > > > Could you explain what you mean ? > > > > In C++, bool is a new type that has two values, true and false. If you add > true and true, it is still true, and not 2. In C, everything that is not 0 > is true, not in C++. Yes, I know this. But my situation is "the other way around". Lets say I want to count "foreground pixels" in an image: I would want to "sum" all the true values, i.e. a true *is* a 1 and a false *is* a 0. In other words, I'm really thinking of (older kind of) C, where there *was* no bool. I assume this thinking still applies to the internal arithmetic of CPUs today. Also the "bit-values" of a boolean array (in memory) are set this way already anyway ! How can I simply call my functions looking at these bit values ? (essentially interpreting a boolean true as 1 and false as 0) -Sebastian From tim.hochberg at ieee.org Wed Nov 7 12:46:21 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 7 Nov 2007 10:46:21 -0700 Subject: [Numpy-discussion] SWIGed function does not like my boolean array In-Reply-To: References: Message-ID: On Nov 7, 2007 10:35 AM, Sebastian Haase wrote: > On Nov 7, 2007 5:23 PM, Matthieu Brucher > wrote: > > > > > I don't understand. I'm thinking of most math functions in the > > > C-library. In C a boolean is just an integer of 0 or 1 (quasi, by > > > definition). > > > > > > Could you explain what you mean ? > > > > > > > In C++, bool is a new type that has two values, true and false. If you > add > > true and true, it is still true, and not 2. In C, everything that is not > 0 > > is true, not in C++. > > Yes, I know this. But my situation is "the other way around". Lets say > I want to count "foreground pixels" in an image: I would want to "sum" > all the true values, i.e. a true *is* a 1 and a false *is* a 0. > > In other words, I'm really thinking of (older kind of) C, where there > *was* no bool. > I assume this thinking still applies to the internal arithmetic of CPUs > today. > Also the "bit-values" of a boolean array (in memory) are set this way > already anyway ! > > How can I simply call my functions looking at these bit values ? > (essentially interpreting a boolean true as 1 and false as 0) I'm not sure how well this would work, but could you change the dtype before passing the array to your function? If you wanted a copy, you could just to the equivalent of a.astype(unit8). However, if you didn't want a copy, you could set the dtype to unit8, operate on the array and then reset it to bool: >>> a = np.array([True, True, False, True]) >>> a array([ True, True, False, True], dtype=bool) >>> a.dtype = np.uint8 >>> a array([1, 1, 0, 1], dtype=uint8) >>> # do something with 'a' here >>> a.dtype = bool >>> a array([ True, True, False, True], dtype=bool) This assumes everything is single threaded. If you have multiple threads accessing 'a', this could be a problem... And, you probably want to do this in C, so translate as appropriate. -tim > > > -Sebastian > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From pschmittml at gmail.com Wed Nov 7 13:55:54 2007 From: pschmittml at gmail.com (Peter Schmitt) Date: Wed, 7 Nov 2007 11:55:54 -0700 Subject: [Numpy-discussion] [f2py] mailing lists, bug, and .f2py_f2cmap, Oh My! Message-ID: Hi, I have one short question, a possible bug report, and one longer question: 1. (short question) Which mailing list is appropriate for f2py discussion: numpy-discussion or f2py-users? 2. (bug report?) I'm using numpy from svn (revision 4410). When using f2py to compile, I got an error: File "/home/pschmitt/usr/local/lib/python2.5/site-packages/numpy/f2py/rules.py", line 1228, in buildmodule for l in '\n\n'.join(funcwrappers2)+'\n'.split('\n'): TypeError: cannot concatenate 'str' and 'list' objects I made what I think should be a fix: $ svn diff numpy/numpy/f2py/rules.py Index: numpy/numpy/f2py/rules.py =================================================================== --- numpy/numpy/f2py/rules.py (revision 4410) +++ numpy/numpy/f2py/rules.py (working copy) @@ -1219,7 +1219,7 @@ f.write('! This file is autogenerated with f2py (version:%s)\n'%(f2py_version)) f.write('! It contains Fortran 90 wrappers to fortran functions.\n') lines = [] - for l in '\n\n'.join(funcwrappers2)+'\n'.split('\n'): + for l in ('\n\n'.join(funcwrappers2)+'\n').split('\n'): if len(l)>72 and l[0]==' ': lines.append(l[:72]+'&\n &') l = l[72:] but I'm not sure if this is what Pearu wanted with that block of code.... Should I submit a bug report??? 3. (Longer question) I have 3 source files: a) types_QG.f90, which specifies the "types_QG" module: "integer, parameter :: WP=4, intdim=selected_int_kind(8)" b) fftpack.f which is a f77 program to perform fourier transforms in 1D c) fft_lib.f90 which is a f90 program (a f90 module, actually) that uses fftpack.f to do 1,2 or 3D fourier transforms. I want access to access the functions in fft_lib.f90 via Python... So I use f2py! :) I just learned about ".f2py_f2cmap" to handle fortran90 (kind=KIND(..)) statements and I'm trying to implement it into my code [http://cens.ioc.ee/projects/f2py2e/FAQ.html#q-what-if-fortran-90-code-uses-type-spec-kind-kind]. f2py successfully applies the f2py_f2cmap: make fft gfortran -fPIC -c types_QG.f90 gfortran -fPIC -c fftpack.f f2py2.5 --fcompiler=gfortran -m fft_lib -c types_QG.o fftpack.o fft_lib.f90 Reading .f2py_f2cmap ... Mapping "real(kind=wp)" to "float" Mapping "real(kind=WP)" to "float" Mapping "integer(kind=intdim)" to "int" Succesfully applied user defined changes from .f2py_f2cmap ... However, f2py complains about the "kind=" syntax in every single function I have defined in the module within fft_lib.f90. The errors look something like this: > /tmp/tmpeP_IaX/src.linux-i686-2.5/fft_lib-f2pywrappers2.f90:9.16: > > real(kind=wp) a(2,2,2) > 1 > Error: Parameter 'wp' at (1) has not been declared or is a variable, which does not reduce to a constant expression If I remove every function from the fft_lib.f90 source code, f2py succesfully compiles the Python module, and I can call all the subroutines in fft_lib. Any ideas on what's going on with f2py, the "kind" statement, .f2py_f2cmap, and functions within a f90 module? Thanks, Pete From pschmittml at gmail.com Wed Nov 7 17:10:28 2007 From: pschmittml at gmail.com (Peter Schmitt) Date: Wed, 7 Nov 2007 15:10:28 -0700 Subject: [Numpy-discussion] [f2py] mailing lists, bug, and .f2py_f2cmap, Oh My! In-Reply-To: References: Message-ID: Here's a simple example to document my problem: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!! types.f90 !!! !!!!!!!!!!!!!!!!!!!!! module types integer, parameter :: WP=4, intdim=selected_int_kind(8) end module types ! end types.f90 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!! hello.f90 !!! !!!!!!!!!!!!!!!!!!!! module hello use types contains subroutine foo (a, b) real(WP), intent(inout) :: a integer(intdim) :: b print*, "Hello from subroutine 'foo'!" a = a * 5.0 print*, " a*5.0=", a print*, " b=", b end subroutine foo function bar(a,b) result(c) real(WP) :: a, b, c print*, "hello from function 'bar'!" c = a*b print*, " a*b=", c end function end module hello !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ################################## ### .f2py_f2cmap: ### {'integer':{'intdim':'int'},'real':{'WP':'float'}} ################################## ################################## ### build the library with gfortran & f2py ### gfortran -fPIC -c types.f90 f2py2.5 --fcompiler=gfortran -m hello -c types.o hello.f90 Now, the f2py2.5 command fails with errors like: > /tmp/tmpdugvMz/src.linux-i686-2.5/hello-f2pywrappers2.f90:7.16: > > real(kind=wp) a > 1 > Error: Parameter 'wp' at (1) has not been declared or is a variable, which does not > reduce to a constant expression I can successfully compile with the f2py2.5 command by commenting out everything within "function bar .... end function", and the f90 subroutine works fine in Python. Given this example, any ideas of why I'm getting all the "real(kind=wp)" errors? Thanks, Pete On Nov 7, 2007 11:55 AM, Peter Schmitt wrote: > I have 3 source files: > a) types_QG.f90, which specifies the "types_QG" module: "integer, > parameter :: WP=4, intdim=selected_int_kind(8)" > b) fftpack.f which is a f77 program to perform fourier transforms in 1D > c) fft_lib.f90 which is a f90 program (a f90 module, actually) > that uses fftpack.f to do 1,2 or 3D fourier transforms. I want access > to access the functions in fft_lib.f90 via Python... So I use f2py! > :) > > I just learned about ".f2py_f2cmap" to handle fortran90 > (kind=KIND(..)) statements and I'm trying to implement it into my code > [http://cens.ioc.ee/projects/f2py2e/FAQ.html#q-what-if-fortran-90-code-uses-type-spec-kind-kind]. > f2py successfully applies the f2py_f2cmap: > > make fft > gfortran -fPIC -c types_QG.f90 > gfortran -fPIC -c fftpack.f > f2py2.5 --fcompiler=gfortran -m fft_lib -c types_QG.o fftpack.o fft_lib.f90 > Reading .f2py_f2cmap ... > Mapping "real(kind=wp)" to "float" > Mapping "real(kind=WP)" to "float" > Mapping "integer(kind=intdim)" to "int" > Succesfully applied user defined changes from .f2py_f2cmap > ... > > However, f2py complains about the "kind=" syntax in every single > function I have defined in the module within fft_lib.f90. The errors > look something like this: > > > /tmp/tmpeP_IaX/src.linux-i686-2.5/fft_lib-f2pywrappers2.f90:9.16: > > > > real(kind=wp) a(2,2,2) > > 1 > > Error: Parameter 'wp' at (1) has not been declared or is a variable, which does not reduce to a constant expression > > If I remove every function from the fft_lib.f90 source code, f2py > succesfully compiles the Python module, and I can call all the > subroutines in fft_lib. > > Any ideas on what's going on with f2py, the "kind" statement, > .f2py_f2cmap, and functions within a f90 module? > > Thanks, > Pete > From millman at berkeley.edu Wed Nov 7 19:21:02 2007 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 7 Nov 2007 16:21:02 -0800 Subject: [Numpy-discussion] ANN: NumPy 1.0.4 Message-ID: I'm pleased to announce the release of NumPy 1.0.4. NumPy is the fundamental package needed for scientific computing with Python. It contains: * a powerful N-dimensional array object * sophisticated (broadcasting) functions * basic linear algebra functions * basic Fourier transforms * sophisticated random number capabilities * tools for integrating Fortran code. Besides it's obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide-variety of databases. This is largely a bug fix release with one notable improvements: - The NumPy and SciPy developers have decided to adopt the Python naming convention for classes. So as of this release, TestCase classes may now be prefixed with either 'test' or 'Test'. This will allow us to write TestCase classes using the CapCase words, while still accepting the old style names. For information, please see the release notes: http://sourceforge.net/project/shownotes.php?release_id=552568&group_id=1369 Thank you to everybody who contributed to the recent release. Enjoy, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david at ar.media.kyoto-u.ac.jp Thu Nov 8 02:45:31 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 08 Nov 2007 16:45:31 +0900 Subject: [Numpy-discussion] Why is g++ used for link with mingw ? Message-ID: <4732BE9B.2030306@ar.media.kyoto-u.ac.jp> Hi, I was wondering why the linking was done with g++ with mingw in numpy.distutils ? (I got some trouble because some function in math library are only defined for gcc and not g++, for example atanhf). This also means that g++ is required to build numpy, which is a bit strange since there is not a single C++ line; is there an issue I am missing, or is this legacy (the mingw tools in numpy.distutils seems to be 6 years old from the comments). cheers, David From haase at msg.ucsf.edu Thu Nov 8 05:28:51 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 8 Nov 2007 11:28:51 +0100 Subject: [Numpy-discussion] SWIGed function does not like my boolean array In-Reply-To: References: Message-ID: On Nov 7, 2007 6:46 PM, Timothy Hochberg wrote: > > > > > On Nov 7, 2007 10:35 AM, Sebastian Haase wrote: > > > > On Nov 7, 2007 5:23 PM, Matthieu Brucher > wrote: > > > > > > > I don't understand. I'm thinking of most math functions in the > > > > C-library. In C a boolean is just an integer of 0 or 1 (quasi, by > > > > definition). > > > > > > > > Could you explain what you mean ? > > > > > > > > > > In C++, bool is a new type that has two values, true and false. If you > add > > > true and true, it is still true, and not 2. In C, everything that is not > 0 > > > is true, not in C++. > > > > Yes, I know this. But my situation is "the other way around". Lets say > > I want to count "foreground pixels" in an image: I would want to "sum" > > all the true values, i.e. a true *is* a 1 and a false *is* a 0. > > > > In other words, I'm really thinking of (older kind of) C, where there > > *was* no bool. > > I assume this thinking still applies to the internal arithmetic of CPUs > today. > > Also the "bit-values" of a boolean array (in memory) are set this way > > already anyway ! > > > > How can I simply call my functions looking at these bit values ? > > (essentially interpreting a boolean true as 1 and false as 0) > > I'm not sure how well this would work, but could you change the dtype before > passing the array to your function? If you wanted a copy, you could just to > the equivalent of a.astype(unit8). However, if you didn't want a copy, you > could set the dtype to unit8, operate on the array and then reset it to > bool: > >>> a = np.array([True, True, False, True]) > >>> a > array([ True, True, False, True], dtype=bool) > >>> a.dtype = np.uint8 > >>> a > array([1, 1, 0, 1], dtype=uint8) > >>> # do something with 'a' here > >>> a.dtype = bool > >>> a > array([ True, True, False, True], dtype=bool) > This assumes everything is single threaded. If you have multiple threads > accessing 'a', this could be a problem... And, you probably want to do this > in C, so translate as appropriate. > > -tim > Thanks Tim, this sound like a good idea. How about creating an a = a.view() before changing dtype. This should make the proposed solution thread safe again. How "expensive" is the creation of a view (performance wise, e.g. compared to calling a trivial C-function) ? Thanks, Sebastian From meine at informatik.uni-hamburg.de Thu Nov 8 07:34:33 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 8 Nov 2007 13:34:33 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays Message-ID: <200711081334.33674.meine@informatik.uni-hamburg.de> Hi! I wonder why simple elementwise operations like "a * 2" or "a + 1" are not performed in order of increasing memory addresses in order to exploit CPU caches etc. - as it is now, their speed drops by a factor of around 3 simply by transpose()ing. Similarly (but even less logical), copy() and even the constructor are affected (yes, I understand that copy() creates contiguous arrays, but shouldn't it respect/retain the order nevertheless?): ### constructor ### In [89]: %timeit -r 10 -n 1000000 numpy.ndarray((3,3,3)) 1000000 loops, best of 10: 1.19 s per loop In [90]: %timeit -r 10 -n 1000000 numpy.ndarray((3,3,3), order="f") 1000000 loops, best of 10: 2.19 s per loop ### copy 3x3x3 array ### In [85]: a = numpy.ndarray((3,3,3)) In [86]: %timeit -r 10 a.copy() 1000000 loops, best of 10: 1.14 s per loop In [87]: a = numpy.ndarray((3,3,3), order="f") In [88]: %timeit -r 10 -n 1000000 a.copy() 1000000 loops, best of 10: 3.39 s per loop ### copy 256x256x256 array ### In [74]: a = numpy.ndarray((256,256,256)) In [75]: %timeit -r 10 a.copy() 10 loops, best of 10: 119 ms per loop In [76]: a = numpy.ndarray((256,256,256), order="f") In [77]: %timeit -r 10 a.copy() 10 loops, best of 10: 274 ms per loop ### fill ### In [79]: a = numpy.ndarray((256,256,256)) In [80]: %timeit -r 10 a.fill(0) 10 loops, best of 10: 60.2 ms per loop In [81]: a = numpy.ndarray((256,256,256), order="f") In [82]: %timeit -r 10 a.fill(0) 10 loops, best of 10: 60.2 ms per loop ### power ### In [151]: a = numpy.ndarray((256,256,256)) In [152]: %timeit -r 10 a ** 2 10 loops, best of 10: 124 ms per loop In [153]: a = numpy.asfortranarray(a) In [154]: %timeit -r 10 a ** 2 10 loops, best of 10: 458 ms per loop ### addition ### In [160]: a = numpy.ndarray((256,256,256)) In [161]: %timeit -r 10 a + 1 10 loops, best of 10: 139 ms per loop In [162]: a = numpy.asfortranarray(a) In [163]: %timeit -r 10 a + 1 10 loops, best of 10: 465 ms per loop ### fft ### In [146]: %timeit -r 10 numpy.fft.fft(vol, axis=0) 10 loops, best of 10: 1.16 s per loop In [148]: %timeit -r 10 numpy.fft.fft(vol0, axis=2) 10 loops, best of 10: 1.16 s per loop In [149]: vol.flags Out[149]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [150]: vol0.flags Out[150]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [9]: %timeit -r 10 numpy.fft.fft(vol0, axis=0) 10 loops, best of 10: 939 ms per loop ### mean ### In [173]: %timeit -r 10 vol.mean() 10 loops, best of 10: 272 ms per loop In [174]: %timeit -r 10 vol0.mean() 10 loops, best of 10: 683 ms per loop ### max ### In [175]: %timeit -r 10 vol.max() 10 loops, best of 10: 63.8 ms per loop In [176]: %timeit -r 10 vol0.max() 10 loops, best of 10: 475 ms per loop ### min ### In [177]: %timeit -r 10 vol.min() 10 loops, best of 10: 63.8 ms per loop In [178]: %timeit -r 10 vol0.min() 10 loops, best of 10: 476 ms per loop ### rot90 ### In [10]: %timeit -r 10 numpy.rot90(vol) 100000 loops, best of 10: 6.97 s per loop In [12]: %timeit -r 10 numpy.rot90(vol0) 100000 loops, best of 10: 6.92 s per loop -- Ciao, / / /--/ / / ANS From david at ar.media.kyoto-u.ac.jp Thu Nov 8 07:44:59 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 08 Nov 2007 21:44:59 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <200711081334.33674.meine@informatik.uni-hamburg.de> References: <200711081334.33674.meine@informatik.uni-hamburg.de> Message-ID: <473304CB.3000103@ar.media.kyoto-u.ac.jp> Hans Meine wrote: > Hi! > > I wonder why simple elementwise operations like "a * 2" or "a + 1" are not > performed in order of increasing memory addresses in order to exploit CPU > caches etc. - as it is now, their speed drops by a factor of around 3 simply > by transpose()ing. Because it is not trivial to do so in all cases, I guess. It is a problem which comes back time to time on the ML, but AFAIK, nobody had a fix for it. Fundamentally, for many element-wise operations, either you have to implement the thing for every possible case, or you abstract it through an iterator, which gives you a decrease of performances in some cases. There are also cases where the current implementation is far from optimal, for lack of man power I guess (taking a look at PyArray_Mean, for example, shows that it uses PyArray_GenericReduceFunction, which is really slow compare to a straight C implementation). > Similarly (but even less logical), copy() and even the > constructor are affected (yes, I understand that copy() creates contiguous > arrays, but shouldn't it respect/retain the order nevertheless?): > I don't see why it is illogical: when you do a copy, you don't preserve memory layout, and so a simple memcpy of the whole buffer is not possible. cheers, David > ### constructor ### > In [89]: %timeit -r 10 -n 1000000 numpy.ndarray((3,3,3)) > 1000000 loops, best of 10: 1.19 s per loop > > In [90]: %timeit -r 10 -n 1000000 numpy.ndarray((3,3,3), order="f") > 1000000 loops, best of 10: 2.19 s per loop > > ### copy 3x3x3 array ### > In [85]: a = numpy.ndarray((3,3,3)) > > In [86]: %timeit -r 10 a.copy() > 1000000 loops, best of 10: 1.14 s per loop > > In [87]: a = numpy.ndarray((3,3,3), order="f") > > In [88]: %timeit -r 10 -n 1000000 a.copy() > 1000000 loops, best of 10: 3.39 s per loop > > ### copy 256x256x256 array ### > In [74]: a = numpy.ndarray((256,256,256)) > > In [75]: %timeit -r 10 a.copy() > 10 loops, best of 10: 119 ms per loop > > In [76]: a = numpy.ndarray((256,256,256), order="f") > > In [77]: %timeit -r 10 a.copy() > 10 loops, best of 10: 274 ms per loop > > ### fill ### > In [79]: a = numpy.ndarray((256,256,256)) > > In [80]: %timeit -r 10 a.fill(0) > 10 loops, best of 10: 60.2 ms per loop > > In [81]: a = numpy.ndarray((256,256,256), order="f") > > In [82]: %timeit -r 10 a.fill(0) > 10 loops, best of 10: 60.2 ms per loop > > ### power ### > In [151]: a = numpy.ndarray((256,256,256)) > > In [152]: %timeit -r 10 a ** 2 > 10 loops, best of 10: 124 ms per loop > > In [153]: a = numpy.asfortranarray(a) > > In [154]: %timeit -r 10 a ** 2 > 10 loops, best of 10: 458 ms per loop > > ### addition ### > In [160]: a = numpy.ndarray((256,256,256)) > > In [161]: %timeit -r 10 a + 1 > 10 loops, best of 10: 139 ms per loop > > In [162]: a = numpy.asfortranarray(a) > > In [163]: %timeit -r 10 a + 1 > 10 loops, best of 10: 465 ms per loop > > ### fft ### > In [146]: %timeit -r 10 numpy.fft.fft(vol, axis=0) > 10 loops, best of 10: 1.16 s per loop > > In [148]: %timeit -r 10 numpy.fft.fft(vol0, axis=2) > 10 loops, best of 10: 1.16 s per loop > > In [149]: vol.flags > Out[149]: > C_CONTIGUOUS : True > F_CONTIGUOUS : False > OWNDATA : True > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > > In [150]: vol0.flags > Out[150]: > C_CONTIGUOUS : False > F_CONTIGUOUS : True > OWNDATA : False > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > > In [9]: %timeit -r 10 numpy.fft.fft(vol0, axis=0) > 10 loops, best of 10: 939 ms per loop > > ### mean ### > In [173]: %timeit -r 10 vol.mean() > 10 loops, best of 10: 272 ms per loop > > In [174]: %timeit -r 10 vol0.mean() > 10 loops, best of 10: 683 ms per loop > > ### max ### > In [175]: %timeit -r 10 vol.max() > 10 loops, best of 10: 63.8 ms per loop > > In [176]: %timeit -r 10 vol0.max() > 10 loops, best of 10: 475 ms per loop > > ### min ### > In [177]: %timeit -r 10 vol.min() > 10 loops, best of 10: 63.8 ms per loop > > In [178]: %timeit -r 10 vol0.min() > 10 loops, best of 10: 476 ms per loop > > ### rot90 ### > In [10]: %timeit -r 10 numpy.rot90(vol) > 100000 loops, best of 10: 6.97 s per loop > > In [12]: %timeit -r 10 numpy.rot90(vol0) > 100000 loops, best of 10: 6.92 s per loop > > From meine at informatik.uni-hamburg.de Thu Nov 8 08:13:19 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 8 Nov 2007 14:13:19 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <473304CB.3000103@ar.media.kyoto-u.ac.jp> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <473304CB.3000103@ar.media.kyoto-u.ac.jp> Message-ID: <200711081413.19779.meine@informatik.uni-hamburg.de> Am Donnerstag, 08. November 2007 13:44:59 schrieb David Cournapeau: > Hans Meine wrote: > > I wonder why simple elementwise operations like "a * 2" or "a + 1" are > > not performed in order of increasing memory addresses in order to exploit > > CPU caches etc. - as it is now, their speed drops by a factor of around 3 > > simply by transpose()ing. > > Because it is not trivial to do so in all cases, I guess. It is a > problem which comes back time to time on the ML, but AFAIK, nobody had a > fix for it. Fundamentally, for many element-wise operations, either you > have to implement the thing for every possible case, or you abstract it > through an iterator, which gives you a decrease of performances in some > cases. OK, so far I have not looked at the NumPy internals, but I would expect sth. like the following: [elementwise operation on array 'a'] 1) ii = [i for i, s in sorted(enumerate(ia.strides), key = lambda (i, s): -s)] 2) aprime = a.transpose(ii) # aprime will be "as C-contiguous as it gets" 3) bprime = perform_operation(aprime) # fast elementwise operation 4) jj = [ii.index(i) for i in range(bprime.ndim)] # inverse ii 5) b = bprime.transpose(jj) # let result have the same order as the input Apart from the fact that this is more or less pseudo-code and that the last step brings a behaviour change (which is intended, but probably not very welcome), am I overlooking a problem? > > Similarly (but even less logical), copy() and even the > > constructor are affected (yes, I understand that copy() creates > > contiguous arrays, but shouldn't it respect/retain the order > > nevertheless?): > > I don't see why it is illogical: when you do a copy, you don't preserve > memory layout, and so a simple memcpy of the whole buffer is not possible. As I wrote above, I don't think this is good. A fortran-order-contiguous array is still contiguous, and not inferior in any way to C-order arrays. So I actually expect copy() to return an array of unchanged order. The numpy book only says "copy() - Return a copy of the array (which is always single-segment, and ALIGNED).", which would be fulfilled also if the memory order was preserved as by my proposed method from above. -- Ciao, / / /--/ / / ANS From tim.hochberg at ieee.org Thu Nov 8 10:04:17 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 8 Nov 2007 08:04:17 -0700 Subject: [Numpy-discussion] SWIGed function does not like my boolean array In-Reply-To: References: Message-ID: On Nov 8, 2007 3:28 AM, Sebastian Haase wrote: > On Nov 7, 2007 6:46 PM, Timothy Hochberg wrote: > > > > > > > > > > On Nov 7, 2007 10:35 AM, Sebastian Haase wrote: > > > > > > On Nov 7, 2007 5:23 PM, Matthieu Brucher > > wrote: > > > > > > > > > I don't understand. I'm thinking of most math functions in the > > > > > C-library. In C a boolean is just an integer of 0 or 1 (quasi, by > > > > > definition). > > > > > > > > > > Could you explain what you mean ? > > > > > > > > > > > > > In C++, bool is a new type that has two values, true and false. If > you > > add > > > > true and true, it is still true, and not 2. In C, everything that is > not > > 0 > > > > is true, not in C++. > > > > > > Yes, I know this. But my situation is "the other way around". Lets say > > > I want to count "foreground pixels" in an image: I would want to "sum" > > > all the true values, i.e. a true *is* a 1 and a false *is* a 0. > > > > > > In other words, I'm really thinking of (older kind of) C, where there > > > *was* no bool. > > > I assume this thinking still applies to the internal arithmetic of > CPUs > > today. > > > Also the "bit-values" of a boolean array (in memory) are set this way > > > already anyway ! > > > > > > How can I simply call my functions looking at these bit values ? > > > (essentially interpreting a boolean true as 1 and false as 0) > > > > I'm not sure how well this would work, but could you change the dtype > before > > passing the array to your function? If you wanted a copy, you could just > to > > the equivalent of a.astype(unit8). However, if you didn't want a copy, > you > > could set the dtype to unit8, operate on the array and then reset it to > > bool: > > >>> a = np.array([True, True, False, True]) > > >>> a > > array([ True, True, False, True], dtype=bool) > > >>> a.dtype = np.uint8 > > >>> a > > array([1, 1, 0, 1], dtype=uint8) > > >>> # do something with 'a' here > > >>> a.dtype = bool > > >>> a > > array([ True, True, False, True], dtype=bool) > > This assumes everything is single threaded. If you have multiple threads > > accessing 'a', this could be a problem... And, you probably want to do > this > > in C, so translate as appropriate. > > > > -tim > > > Thanks Tim, this sound like a good idea. How about creating an > a = a.view() > before changing dtype. Yeah. That's a better idea. You should be able to just use "a.view(np.uint8 )". > This should make the proposed solution thread safe again. > How "expensive" is the creation of a view (performance wise, e.g. > compared to calling a trivial C-function) ? It should be pretty cheap in theory, but I have no idea really. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkb.teichmann at gmail.com Thu Nov 8 10:25:57 2007 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Thu, 8 Nov 2007 15:25:57 +0000 (UTC) Subject: [Numpy-discussion] saving and loading as PNG Message-ID: Hi all, i am working with numpy and mostly 16 bit matrices, and so I was looking for a standard way of saving them. I found that PNG actually supports that, but I could not find a way to use this feature from within python. I thought that it actually is a cool way of storing matrices and thus I wrote a little pyrex program that does the job. I figured this might be interesting to a broader audience, so I posted it here, maybe for inclusion in numpy? Some more thoughts: * Other implementations: There is other people who have done such a thing. the PIL knows how to read and write PNG, but only 8 bit. The same holds for matplotlib. * Problems: until now, numpy does not link agains libpng. This should not be a problem on linux-like systems, as those normally all have libpng installed. Windows however, I don't know. Same thing with pyrex. Numpy doesn't use pyrex until now. * other places to put: numpy could be considered to basic to have such a functionality, one could put it in scipy. Well, but png is not really a _scientific_ application (even though I personally use it to do science). Matplotlib would be another nice place to put this feature, big advantage is that matplotlib already links against libpng. So, now I shared my thoughts with you, comments? Greetings, Martin And here the code (188 lines are hopefully ok for a mailinglist?) cimport c_numpy import numpy cdef extern from "stdlib.h": cdef void *malloc(int) cdef void free(void *) cdef extern from "stdio.h": cdef struct file: pass ctypedef file FILE cdef FILE *fopen(char *, char *) cdef int fclose(FILE *) cdef extern from "errno.h": cdef int errno cdef extern from "png.h": cdef struct jmp_buf: pass cdef int setjmp(jmp_buf) cdef struct png_struct: jmp_buf jmpbuf ctypedef png_struct *png_structp cdef struct png_info: int width int height int bit_depth int color_type int channels ctypedef png_info *png_infop cdef struct png_text: int compression char *key char *text int text_length ctypedef png_text *png_textp cdef png_structp png_create_read_struct(char *, void *, void *, void*) cdef void png_init_io(png_structp, FILE *) cdef void png_set_sig_bytes(png_structp, int) cdef png_infop png_create_info_struct(png_structp) cdef void png_read_info(png_structp, png_infop) cdef int png_set_interlace_handling(png_structp) cdef void png_read_update_info(png_structp, png_infop) cdef void png_read_image(png_structp, unsigned char **) cdef void png_read_png(png_structp, png_infop, int, void *) cdef void png_read_end(png_structp, png_infop) cdef void png_destroy_read_struct(png_structp *, png_infop *, void *) cdef png_structp png_create_write_struct(char *, void *, void *, void*) cdef void png_set_IHDR(png_structp, png_infop, unsigned int, unsigned int, int, int, int, int, int) cdef void png_write_info(png_structp, png_infop) cdef void png_write_image(png_structp, unsigned char **) cdef void png_write_end(png_structp, png_infop) cdef void png_destroy_write_struct(png_structp *,png_infop *) cdef void png_set_swap(png_structp) cdef void png_set_rows(png_structp, png_infop, unsigned char **) cdef void png_write_png(png_structp, png_infop, int, void *) cdef void png_set_text(png_structp, png_infop, png_textp, int) cdef int PNG_COLOR_TYPE_GRAY cdef int PNG_COLOR_TYPE_PALETTE cdef int PNG_COLOR_TYPE_RGB cdef int PNG_COLOR_TYPE_RGB_ALPHA cdef int PNG_COLOR_TYPE_GRAY_ALPHA cdef int PNG_INTERLACE_NONE cdef int PNG_COMPRESSION_TYPE_BASE cdef int PNG_FILTER_TYPE_BASE cdef int PNG_TRANSFORM_IDENTITY cdef int PNG_TRANSFORM_SWAP_ENDIAN cdef char *PNG_LIBPNG_VER_STRING def load(fn): """ load a PNG file into a NumPy array. This function can read all kinds of PNG images in a senseful way: Depending on the number of bits per pixel, we chose numpy.bool, numpy.unit8 or numpy.uint16 as base type. For gray scale images or pallettes, this returns a 2-D array, for everything else a 3-D array, where the third dimension has a size of 2 for gray+alpha images, 3 for RGB and 4 for RGBA images. """ cdef FILE *f f = fopen(fn, "rb") if f == NULL: raise IOError(errno) cdef png_structp p p = png_create_read_struct(PNG_LIBPNG_VER_STRING, NULL, NULL, NULL) if setjmp(p.jmpbuf): fclose(f) raise RuntimeError("libpng reported error") cdef png_infop i i = png_create_info_struct(p) png_init_io(p, f) png_read_info(p, i) if i.color_type == PNG_COLOR_TYPE_GRAY: shape = (i.height, i.width) elif i.color_type == PNG_COLOR_TYPE_GRAY_ALPHA: shape = (i.height, i.width, 2) elif i.color_type == PNG_COLOR_TYPE_PALETTE: shape = (i.height, i.width) elif i.color_type == PNG_COLOR_TYPE_RGB: shape = (i.height, i.width, 3) elif i.color_type == PNG_COLOR_TYPE_RGB_ALPHA: shape = (i.height, i.width, 4) if i.bit_depth == 1: dtype = numpy.bool elif i.bit_depth == 8: dtype = numpy.uint8 elif i.bit_depth == 16: dtype = numpy.uint16 png_set_interlace_handling(p) png_read_update_info(p, i) r = numpy.zeros(shape, dtype) cdef c_numpy.ndarray s s = r cdef unsigned char **rp rp = malloc(sizeof(char *) * i.height) cdef int j for j from 0 <= j < i.height: rp[j] = &s.data[(j * i.channels * i.width * i.bit_depth) / 8] png_set_swap(p) png_read_image(p, rp) free(rp) png_destroy_read_struct(&p, &i, NULL) fclose(f) return r def save(fn, data): """ Save a NumPy array to PNG This function saves the NumPy array in data to the file fn. It saves the data the best way it can: 2-D arrays are saved as gray scale images, 3-D arrays are saved as gray+alpha, rgb or rgba depending on the size of the 3rd dimension. """ cdef c_numpy.ndarray d d = numpy.ascontiguousarray(data) cdef int j cdef int width if data.itemsize > 2: raise ValueError("PNG cannot handle BPP > 16") if data.ndim == 2: width = data.shape[1] * data.itemsize elif data.ndim == 3: width = data.shape[2] * data.shape[1] * data.itemsize else: raise ValueError("Can only save 2-D or 3-D arrays to PNG") cdef FILE *f f = fopen(fn, "wb") cdef png_structp p p = png_create_write_struct(PNG_LIBPNG_VER_STRING, NULL, NULL, NULL) cdef unsigned char **rp rp = NULL if setjmp(p.jmpbuf): free(rp) fclose(f) raise RuntimeError("libpng reported error") cdef png_infop i i = png_create_info_struct(p) png_init_io(p, f) if data.ndim == 2: color = PNG_COLOR_TYPE_GRAY elif data.shape[2] == 2: color = PNG_COLOR_TYPE_GRAY_ALPHA elif data.shape[2] == 3: color = PNG_COLOR_TYPE_RGB elif data.shape[2] == 4: color = PNG_COLOR_TYPE_RGB_ALPHA else: raise ValueError("Third dimension must be <= 4 for saving to PNG") png_set_IHDR(p, i, data.shape[1], data.shape[0], data.itemsize*8, color, PNG_INTERLACE_NONE, PNG_COMPRESSION_TYPE_BASE, PNG_FILTER_TYPE_BASE) rp = malloc(sizeof(char *) * data.shape[0]) for j from 0 <= j < data.shape[0]: rp[j] = &d.data[j * width] png_set_rows(p, i, rp) png_write_png(p, i, PNG_TRANSFORM_SWAP_ENDIAN, NULL) free(rp) png_destroy_write_struct(&p, &i) fclose(f) From cournape at gmail.com Thu Nov 8 10:37:06 2007 From: cournape at gmail.com (David Cournapeau) Date: Fri, 9 Nov 2007 00:37:06 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <200711081413.19779.meine@informatik.uni-hamburg.de> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <473304CB.3000103@ar.media.kyoto-u.ac.jp> <200711081413.19779.meine@informatik.uni-hamburg.de> Message-ID: <5b8d13220711080737y76cf3fe6ybd1396ebb2aa0397@mail.gmail.com> On Nov 8, 2007 10:13 PM, Hans Meine wrote: > Am Donnerstag, 08. November 2007 13:44:59 schrieb David Cournapeau: > > Hans Meine wrote: > > > I wonder why simple elementwise operations like "a * 2" or "a + 1" are > > > not performed in order of increasing memory addresses in order to exploit > > > CPU caches etc. - as it is now, their speed drops by a factor of around 3 > > > simply by transpose()ing. > > > > Because it is not trivial to do so in all cases, I guess. It is a > > problem which comes back time to time on the ML, but AFAIK, nobody had a > > fix for it. Fundamentally, for many element-wise operations, either you > > have to implement the thing for every possible case, or you abstract it > > through an iterator, which gives you a decrease of performances in some > > cases. > > OK, so far I have not looked at the NumPy internals, but I would expect sth. > like the following: > > [elementwise operation on array 'a'] > 1) ii = [i for i, s in sorted(enumerate(ia.strides), key = lambda (i, s): -s)] > 2) aprime = a.transpose(ii) # aprime will be "as C-contiguous as it gets" > 3) bprime = perform_operation(aprime) # fast elementwise operation > 4) jj = [ii.index(i) for i in range(bprime.ndim)] # inverse ii > 5) b = bprime.transpose(jj) # let result have the same order as the input The problem is not F vs C storage: for element-wise operation, it does not matter at all; you just apply the same function (perform_operation) over and over on every element of the array. The order does not matter at all. But what if you have segmented buffers, non aligned, etc... arrays ? All this has to be taken care of, and this means handling reference count and other things which are always delicate to handle well... Or you just use the current situation, which let python handle it (through PyObject_Call and a callable, at a C level). I agree that in some cases, it would be useful to have better performances: in particular, having fast code paths for common cases (aligned, and single segmented) would be nice. One example is the PyArray_Clip function (in multiarray.c), whose speed was too slow for my taste in my use cases: the general idea is to get to the point where you have a C single buffer at which point you can call a trivial C function (fast_clip). As you can see, this is not trivial (~150 lines of C code): since the point is to go faster, you really want to avoid copying things, which means you have to be pretty careful. Speed-wise, it definitely worths it, though: I don't remember the exact numbers, but in some cases (which are very common), it was a several times speed. > > Apart from the fact that this is more or less pseudo-code and that the last > step brings a behaviour change (which is intended, but probably not very > welcome), am I overlooking a problem? > > > > Similarly (but even less logical), copy() and even the > > > constructor are affected (yes, I understand that copy() creates > > > contiguous arrays, but shouldn't it respect/retain the order > > > nevertheless?): > > > > I don't see why it is illogical: when you do a copy, you don't preserve > > memory layout, and so a simple memcpy of the whole buffer is not possible. > > As I wrote above, I don't think this is good. A fortran-order-contiguous > array is still contiguous, and not inferior in any way to C-order arrays. > So I actually expect copy() to return an array of unchanged order. Maybe this behaviour was kept for compatibility with numarray ? If you look at the docstring, it is said that copy may not return the same order than its input. It kind of make sense to me: C order is the default of many numpy operations. David From meine at informatik.uni-hamburg.de Thu Nov 8 10:42:47 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 8 Nov 2007 16:42:47 +0100 Subject: [Numpy-discussion] saving and loading as PNG In-Reply-To: References: Message-ID: <200711081642.48278.meine@informatik.uni-hamburg.de> Am Donnerstag, 08. November 2007 16:25:57 schrieb Martin Teichmann: > Some more thoughts: > * Other implementations: There is other people who have done such a thing. > the PIL knows how to read and write PNG, but only 8 bit. The same holds > for matplotlib. Our VIGRA imaging library can read and save PNGs with 8/16 bit data, as well as several kinds of TIFF files (including float data). We are currently trying to re-model our python bindings (which have not yet been officially released) using NumPy containers for the images. Unfortunately, we seem to get a performance penalty by the common convention that the order of the indices is x,y, e.g. colorImage[x,y] == [r,g,b]. See the other current thread. -- Ciao, / / /--/ / / ANS From meine at informatik.uni-hamburg.de Thu Nov 8 10:50:42 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 8 Nov 2007 16:50:42 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <5b8d13220711080737y76cf3fe6ybd1396ebb2aa0397@mail.gmail.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081413.19779.meine@informatik.uni-hamburg.de> <5b8d13220711080737y76cf3fe6ybd1396ebb2aa0397@mail.gmail.com> Message-ID: <200711081650.42183.meine@informatik.uni-hamburg.de> Am Donnerstag, 08. November 2007 16:37:06 schrieb David Cournapeau: > The problem is not F vs C storage: for element-wise operation, it does > not matter at all; you just apply the same function > (perform_operation) over and over on every element of the array. The > order does not matter at all. Yet Fortran order leads to several times slower operations, see the figures in my original post. :-( > But what if you have segmented buffers, non aligned, etc... arrays ? The code I posted should deal with it - by sorting the indices by decreasing stride, I simply ensure that all (source and target) segments are traversed in order of increasing memory addresses. It does not affect segments or alignment. > All this has to be taken care of, Right - my "perform_operation(aprime)" step would need to apply the operation on every memory segment. > and this means handling reference > count and other things which are always delicate to handle well... I am not sure that I understand where refcounting comes into play here. > Or > you just use the current situation, which let python handle it > (through PyObject_Call and a callable, at a C level). I need to look at the code to see what you mean here. Probably, I have a wrong picture of where the slowness comes from (I thought that the order of the loops was wrong). > > As I wrote above, I don't think this is good. A fortran-order-contiguous > > array is still contiguous, and not inferior in any way to C-order arrays. > > So I actually expect copy() to return an array of unchanged order. > > Maybe this behaviour was kept for compatibility with numarray ? If you > look at the docstring, it is said that copy may not return the same > order than its input. It kind of make sense to me: C order is the > default of many numpy operations. That is very sad, because Fortran order is much more natural for handling images, where you're absolutely used to index with [x,y], and x being the faster-changing index in memory. -- Ciao, / / /--/ / / ANS From zyzhu2000 at gmail.com Thu Nov 8 11:19:35 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Thu, 8 Nov 2007 10:19:35 -0600 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang Message-ID: Good morning. I just installed the Windows binary of numpy 1.04. When I ran numpy.test() in IDLE (the Python shell that comes with Python), the program hang (or at least is running for half an hour). I am using Windows XP, duel core intel CPU. Does anyone know what is going on? Thanks, Geoffrey From cournape at gmail.com Thu Nov 8 11:31:40 2007 From: cournape at gmail.com (David Cournapeau) Date: Fri, 9 Nov 2007 01:31:40 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <200711081650.42183.meine@informatik.uni-hamburg.de> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081413.19779.meine@informatik.uni-hamburg.de> <5b8d13220711080737y76cf3fe6ybd1396ebb2aa0397@mail.gmail.com> <200711081650.42183.meine@informatik.uni-hamburg.de> Message-ID: <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> On Nov 9, 2007 12:50 AM, Hans Meine wrote: > Am Donnerstag, 08. November 2007 16:37:06 schrieb David Cournapeau: > > The problem is not F vs C storage: for element-wise operation, it does > > not matter at all; you just apply the same function > > (perform_operation) over and over on every element of the array. The > > order does not matter at all. > > Yet Fortran order leads to several times slower operations, see the figures in > my original post. :-( This is because the current implementation for at least some of the operations you are talking about are using PyArray_GenericReduce and other similar functions, which are really high level (they use python callable, etc..). This is easier, because you don't have to care about anything (type, etc...), but this means that python runtime is handling all the thing. Instead, you should use a pure C implementation (by pure C, I mean a C function totally independant of python, only dealing with standard C types). This would already lead a significant performance leap. Once you do that, you should not see difference between F and C storage, normally. > > > But what if you have segmented buffers, non aligned, etc... arrays ? > > The code I posted should deal with it - by sorting the indices by decreasing > stride, I simply ensure that all (source and target) segments are traversed > in order of increasing memory addresses. It does not affect segments or > alignment. If you have segmented addresses, I don't think the ordering matters much anymore, for memory access, no ? For example, > > > All this has to be taken care of, > > Right - my "perform_operation(aprime)" step would need to apply the operation > on every memory segment. > > > and this means handling reference > > count and other things which are always delicate to handle well... > > I am not sure that I understand where refcounting comes into play here. > > > Or > > you just use the current situation, which let python handle it > > (through PyObject_Call and a callable, at a C level). > > I need to look at the code to see what you mean here. Probably, I have a > wrong picture of where the slowness comes from (I thought that the order of > the loops was wrong). > > > > As I wrote above, I don't think this is good. A fortran-order-contiguous > > > array is still contiguous, and not inferior in any way to C-order arrays. > > > So I actually expect copy() to return an array of unchanged order. > > > > Maybe this behaviour was kept for compatibility with numarray ? If you > > look at the docstring, it is said that copy may not return the same > > order than its input. It kind of make sense to me: C order is the > > default of many numpy operations. > > That is very sad, because Fortran order is much more natural for handling > images, where you're absolutely used to index with [x,y], and x being the > faster-changing index in memory. > > > -- > Ciao, / / > /--/ > / / ANS > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From cournape at gmail.com Thu Nov 8 11:31:54 2007 From: cournape at gmail.com (David Cournapeau) Date: Fri, 9 Nov 2007 01:31:54 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081413.19779.meine@informatik.uni-hamburg.de> <5b8d13220711080737y76cf3fe6ybd1396ebb2aa0397@mail.gmail.com> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> Message-ID: <5b8d13220711080831q42dd78et70d5ab22b32417cd@mail.gmail.com> On Nov 9, 2007 1:31 AM, David Cournapeau wrote: > On Nov 9, 2007 12:50 AM, Hans Meine wrote: > > Am Donnerstag, 08. November 2007 16:37:06 schrieb David Cournapeau: > > > The problem is not F vs C storage: for element-wise operation, it does > > > not matter at all; you just apply the same function > > > (perform_operation) over and over on every element of the array. The > > > order does not matter at all. > > > > Yet Fortran order leads to several times slower operations, see the figures in > > my original post. :-( > This is because the current implementation for at least some of the > operations you are talking about are using PyArray_GenericReduce and > other similar functions, which are really high level (they use python > callable, etc..). This is easier, because you don't have to care about > anything (type, etc...), but this means that python runtime is > handling all the thing. Instead, you should use a pure C > implementation (by pure C, I mean a C function totally independant of > python, only dealing with standard C types). This would already lead a > significant performance leap. > > Once you do that, you should not see difference between F and C > storage, normally. > > > > > But what if you have segmented buffers, non aligned, etc... arrays ? > > > > The code I posted should deal with it - by sorting the indices by decreasing > > stride, I simply ensure that all (source and target) segments are traversed > > in order of increasing memory addresses. It does not affect segments or > > alignment. > If you have segmented addresses, I don't think the ordering matters > much anymore, for memory access, no ? For example, > > > > > > All this has to be taken care of, > > > > Right - my "perform_operation(aprime)" step would need to apply the operation > > on every memory segment. > > > > > and this means handling reference > > > count and other things which are always delicate to handle well... > > > > I am not sure that I understand where refcounting comes into play here. > > > > > Or > > > you just use the current situation, which let python handle it > > > (through PyObject_Call and a callable, at a C level). > > > > I need to look at the code to see what you mean here. Probably, I have a > > wrong picture of where the slowness comes from (I thought that the order of > > the loops was wrong). > > > > > > As I wrote above, I don't think this is good. A fortran-order-contiguous > > > > array is still contiguous, and not inferior in any way to C-order arrays. > > > > So I actually expect copy() to return an array of unchanged order. > > > > > > Maybe this behaviour was kept for compatibility with numarray ? If you > > > look at the docstring, it is said that copy may not return the same > > > order than its input. It kind of make sense to me: C order is the > > > default of many numpy operations. > > > > That is very sad, because Fortran order is much more natural for handling > > images, where you're absolutely used to index with [x,y], and x being the > > faster-changing index in memory. > > > > > > -- > > Ciao, / / > > /--/ > > / / ANS > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > From meine at informatik.uni-hamburg.de Thu Nov 8 11:55:13 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 8 Nov 2007 17:55:13 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> Message-ID: <200711081755.14084.meine@informatik.uni-hamburg.de> Am Donnerstag, 08. November 2007 17:31:40 schrieb David Cournapeau: > This is because the current implementation for at least some of the > operations you are talking about are using PyArray_GenericReduce and > other similar functions, which are really high level (they use python > callable, etc..). This is easier, because you don't have to care about > anything (type, etc...), but this means that python runtime is > handling all the thing. I suspected that after your last post, but that's really bad for pointwise operations on a contiguous, aligned array. A simple transpose() should really not make any difference here. > Instead, you should use a pure C > implementation (by pure C, I mean a C function totally independant of > python, only dealing with standard C types). This would already lead a > significant performance leap. AFAICS, it would be much more elegant and easier to implement this using C++ templates. We have a lot of experience with such a design from our VIGRA library ( http://kogs-www.informatik.uni-hamburg.de/~koethe/vigra/ ), which is an imaging library based on the STL concepts (and some necessary and convenient extensions for higher-dimensional arrays and a more flexible API). I am not very keen on writing hundreds of lines of C code for things that can easily be handled with C++ functors. But I don't think that I am the first to propose this, and I know that C has some advantages (faster compilation; are there more? ;-) ) - what is the opinion on this in the SciPy community? > If you have segmented addresses, I don't think the ordering matters > much anymore, for memory access, no ? Yes, I think it does. It probably depends on the sizes of the segments though. If you have a multi-segment box-sub-range of a large dataset (3D volume or even only 2D), processing each contiguous "row" (column/...) at once within the inner loop definitely makes a difference. I.e. as long as one dimension is not strided (and the data's extent in this dimension is not too small), it should be handled in the inner loop. The other loops probably don't make a big difference. -- Ciao, / / /--/ / / ANS From cournape at gmail.com Thu Nov 8 12:16:26 2007 From: cournape at gmail.com (David Cournapeau) Date: Fri, 9 Nov 2007 02:16:26 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <200711081755.14084.meine@informatik.uni-hamburg.de> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> Message-ID: <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> On Nov 9, 2007 1:55 AM, Hans Meine wrote: > Am Donnerstag, 08. November 2007 17:31:40 schrieb David Cournapeau: > > This is because the current implementation for at least some of the > > operations you are talking about are using PyArray_GenericReduce and > > other similar functions, which are really high level (they use python > > callable, etc..). This is easier, because you don't have to care about > > anything (type, etc...), but this means that python runtime is > > handling all the thing. > > I suspected that after your last post, but that's really bad for pointwise > operations on a contiguous, aligned array. A simple transpose() should > really not make any difference here. It should not, but in practise, it is not so easy to do. AFAIK, even matlab has the same problem, with less difference, though. And they have much more ressources than numpy. Not to say that we cannot do better, but this takes time. > > > Instead, you should use a pure C > > implementation (by pure C, I mean a C function totally independant of > > python, only dealing with standard C types). This would already lead a > > significant performance leap. > > AFAICS, it would be much more elegant and easier to implement this using C++ > templates. We have a lot of experience with such a design from our VIGRA > library ( http://kogs-www.informatik.uni-hamburg.de/~koethe/vigra/ ), which > is an imaging library based on the STL concepts (and some necessary and > convenient extensions for higher-dimensional arrays and a more flexible API). > > I am not very keen on writing hundreds of lines of C code for things that can > easily be handled with C++ functors. But I don't think that I am the first > to propose this, and I know that C has some advantages (faster compilation; > are there more? ;-) ) The advantages of C over C++ are much more than compilation speed: it is actually portable, is simple, and easily callable from other languages, 3 significant points that C++ does not meet at all. Generally, I don't see much benefit for using low level C++ in numerical code ; I consider most numerical-based container a total failure (the fact that after 20 years, there is still nothing close to a standard for even a matrix concept in C++ is for me quite striking). I think C++ 's aspect which would be more useful in numpy is RAII (I must confess that I personnally don't like C++ very much, in particular when template are involved). This is only my opinion, I don't know the opinion on other people on this. > > Yes, I think it does. It probably depends on the sizes of the segments > though. If you have a multi-segment box-sub-range of a large dataset (3D > volume or even only 2D), processing each contiguous "row" (column/...) at > once within the inner loop definitely makes a difference. I.e. as long as > one dimension is not strided (and the data's extent in this dimension is not > too small), it should be handled in the inner loop. The other loops > probably don't make a big difference. If you have contiguous segments in subranges, then this is already handled through the ufunc mechanism, but I don't know anything about their actual implementation. Other people much more knowledgable than me can give you more info here. David From robert.kern at gmail.com Thu Nov 8 13:12:42 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 08 Nov 2007 12:12:42 -0600 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: References: Message-ID: <4733519A.1050609@gmail.com> Geoffrey Zhu wrote: > Good morning. > > I just installed the Windows binary of numpy 1.04. When I ran > numpy.test() in IDLE (the Python shell that comes with Python), the > program hang (or at least is running for half an hour). I am using > Windows XP, duel core intel CPU. > > Does anyone know what is going on? No. Run numpy.test(verbosity=2) so it will print out each test name before running the test. Then we might have some idea about where the hang is coming from. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Thu Nov 8 13:28:20 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 08 Nov 2007 10:28:20 -0800 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> Message-ID: <47335544.6060004@noaa.gov> This discussion makes me wonder if the basic element-wise operations could (should?) be special cased for contiguous arrays, reducing them to simple pointer incrementing from the start to the finish of the data block. The same code would work for C and Fortran order arrays, and be pretty simple. This would address Hans' issue, no? It's a special case but a common one. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mforbes at physics.ubc.ca Thu Nov 8 13:32:42 2007 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Thu, 8 Nov 2007 10:32:42 -0800 Subject: [Numpy-discussion] Warning or error on conversion from complex to float. Message-ID: <72D07F2D-305B-4698-8E2E-B35D91A29CE3@physics.ubc.ca> Hi, Is it possible or easy to add a warning and/or error when array assignments are made that lose information? I just got caught with the following type of code: def f(): return numpy.array([1j,2.0]) x = numpy.empty((2,),dtype=float) x[:] = f() I am pre-allocating arrays for speed, but made a change to f() resulting in complex results. My code now silently ignores the imaginary part. It would have been very helpful if the assignment check to make sure that the conversion was not going to lose information (presumably with an assert so that performance is not affected outside of debugging). Any simple suggestions on how to avoid this problem in the future? How hard would it be to add such a check in numpy? Thanks, Michael. From cournape at gmail.com Thu Nov 8 13:39:37 2007 From: cournape at gmail.com (David Cournapeau) Date: Fri, 9 Nov 2007 03:39:37 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <47335544.6060004@noaa.gov> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> <47335544.6060004@noaa.gov> Message-ID: <5b8d13220711081039m485fa6fax8ccc3c060fdc572f@mail.gmail.com> On Nov 9, 2007 3:28 AM, Christopher Barker wrote: > This discussion makes me wonder if the basic element-wise operations > could (should?) be special cased for contiguous arrays, reducing them to > simple pointer incrementing from the start to the finish of the data > block. I don't see why it could not. I think that most of the implementation could be common to most functions, because they share a lot (handling out argument, axis argument, and sometimes dtype argument). The only different part would be the actual compuation by a C function, which is trivial as you pointed. cheers, David From nwagner at iam.uni-stuttgart.de Thu Nov 8 13:48:24 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 08 Nov 2007 19:48:24 +0100 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: <4733519A.1050609@gmail.com> References: <4733519A.1050609@gmail.com> Message-ID: On Thu, 08 Nov 2007 12:12:42 -0600 Robert Kern wrote: > Geoffrey Zhu wrote: >> Good morning. >> >> I just installed the Windows binary of numpy 1.04. When >>I ran >> numpy.test() in IDLE (the Python shell that comes with >>Python), the >> program hang (or at least is running for half an hour). >>I am using >> Windows XP, duel core intel CPU. >> >> Does anyone know what is going on? > > No. Run numpy.test(verbosity=2) so it will print out >each test name before > running the test. Then we might have some idea about >where the hang is coming from. > > -- > Robert Kern > > "I have come to believe that the whole world is an >enigma, a harmless enigma > that is made terrible by our own mad attempt to >interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion I can confirm the problem on an AMD x86_64 with python2.5.1. The test hangs at check_cdouble (numpy.tests.test_linalg.test_det) Nils From zyzhu2000 at gmail.com Thu Nov 8 13:48:46 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Thu, 8 Nov 2007 12:48:46 -0600 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: <4733519A.1050609@gmail.com> References: <4733519A.1050609@gmail.com> Message-ID: On Nov 8, 2007 12:12 PM, Robert Kern wrote: > > Geoffrey Zhu wrote: > > Good morning. > > > > I just installed the Windows binary of numpy 1.04. When I ran > > numpy.test() in IDLE (the Python shell that comes with Python), the > > program hang (or at least is running for half an hour). I am using > > Windows XP, duel core intel CPU. > > > > Does anyone know what is going on? > > No. Run numpy.test(verbosity=2) so it will print out each test name before > running the test. Then we might have some idea about where the hang is coming from. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Thanks for the hint, Robert. It hangs on check_cdouble (numpy.tests.test_linalg.test_det). Also testip_not_allclose() had three warnings. I guess that's probably okay. testip_not_allclose (numpy.core.tests.test_numeric.test_allclose_inf) ... Warning: invalid value encountered in absolute Warning: invalid value encountered in absolute Warning: invalid value encountered in less_equal ok Thanks, Geoffrey From meine at informatik.uni-hamburg.de Thu Nov 8 14:30:20 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 8 Nov 2007 20:30:20 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <47335544.6060004@noaa.gov> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> <47335544.6060004@noaa.gov> Message-ID: <200711082030.24978.meine@informatik.uni-hamburg.de> On Donnerstag 08 November 2007, Christopher Barker wrote: > This discussion makes me wonder if the basic element-wise operations > could (should?) be special cased for contiguous arrays, reducing them to > simple pointer incrementing from the start to the finish of the data > block. The same code would work for C and Fortran order arrays, and be > pretty simple. This is exactly what I am thinking and talking about. I did not want to imply that NumPy is bad because it does not yet do so - it's an impressive piece of well-designed software AFAICS - but I do think that this is a really common and important use case that should not stay as inefficiently handled as it is now. Ciao, / / .o. /--/ ..o / / ANS ooo -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From berthe.loic at gmail.com Thu Nov 8 15:15:03 2007 From: berthe.loic at gmail.com (LB) Date: Thu, 08 Nov 2007 20:15:03 -0000 Subject: [Numpy-discussion] Improving numpy.interp Message-ID: <1194552903.150362.210130@e34g2000pro.googlegroups.com> I often need to make a linear interpolation for a single scalar value but this is not very simple with numpy.interp : >>> import numpy as n >>> n.__version__ '1.0.5.dev4420' >>> xp = n.arange(10) >>> yp = 2.5 + xp**2 -x >>> x = 3.2 >>> n.interp(x, xp, yp) Traceback (most recent call last): File "", line 1, in ValueError: object of too small depth for desired array So I use the following trick (which is really ugly) : >>> n.interp([x], xp, yp)[0] 7.7000000000000011 Did I miss an obvious way to do it ? If not, I'd be tempted to patch interp to let it accept scalar values as first argument as follow : >>> def interp(x, *args, **kwds): ... if type(x) in (float, int): ... return n.interp([x], *args, **kwds).item() ... else : ... return n.interp(x, *args, **kwds) ... >>> >>> interp(x, xp, yp) 7.7000000000000011 >>> interp([x,2*x], xp, yp) array([ 7.7, 38.5]) -- LB From oliphant at enthought.com Thu Nov 8 18:16:12 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 08 Nov 2007 17:16:12 -0600 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <200711081334.33674.meine@informatik.uni-hamburg.de> References: <200711081334.33674.meine@informatik.uni-hamburg.de> Message-ID: <473398BC.5070501@enthought.com> Hans Meine wrote: > Hi! > > I wonder why simple elementwise operations like "a * 2" or "a + 1" are not > performed in order of increasing memory addresses in order to exploit CPU > caches etc. C-order is "special" in NumPy due to the history. I agree that it doesn't need to be and we have taken significant steps in that direction. Right now, the fundamental problem is probably due to the fact that the output arrays are being created as C-order arrays when the input is a Fortran order array. Once that is changed then we can cause Fortran-order inputs to use the simplest path through the code. Fortran order arrays can be preserved but it takes a little extra work because backward compatible expectations had to be met. See for example the order argument to the copy method of arrays. > - as it is now, their speed drops by a factor of around 3 simply > by transpose()ing. Similarly (but even less logical), copy() and even the > constructor are affected (yes, I understand that copy() creates contiguous > arrays, but shouldn't it respect/retain the order nevertheless?): > As mentioned, it can preserve order with the 'order' argument a.copy('A') > ### constructor ### > In [89]: %timeit -r 10 -n 1000000 numpy.ndarray((3,3,3)) > 1000000 loops, best of 10: 1.19 s per loop > > In [90]: %timeit -r 10 -n 1000000 numpy.ndarray((3,3,3), order="f") > 1000000 loops, best of 10: 2.19 s per loop > > I bet what you are seeing here is simply the overhead of processing the order argument. Try the first one with order='c' to see what I mean. > ### copy 3x3x3 array ### > In [85]: a = numpy.ndarray((3,3,3)) > > In [86]: %timeit -r 10 a.copy() > 1000000 loops, best of 10: 1.14 s per loop > > In [87]: a = numpy.ndarray((3,3,3), order="f") > > In [88]: %timeit -r 10 -n 1000000 a.copy() > 1000000 loops, best of 10: 3.39 s per loop > Use the 'a' argument to allow copying in "fortran" order. > ### copy 256x256x256 array ### > In [74]: a = numpy.ndarray((256,256,256)) > > In [75]: %timeit -r 10 a.copy() > 10 loops, best of 10: 119 ms per loop > > In [76]: a = numpy.ndarray((256,256,256), order="f") > > In [77]: %timeit -r 10 a.copy() > 10 loops, best of 10: 274 ms per loop > Same thing here. Nobody is opposed to having faster code as long as we don't in the process break code bases. There is also the matter of implementation.... -Travis O. From oliphant at enthought.com Thu Nov 8 18:18:48 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 08 Nov 2007 17:18:48 -0600 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <47335544.6060004@noaa.gov> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> <47335544.6060004@noaa.gov> Message-ID: <47339958.4070700@enthought.com> Christopher Barker wrote: > This discussion makes me wonder if the basic element-wise operations > could (should?) be special cased for contiguous arrays, reducing them to > simple pointer incrementing from the start to the finish of the data > block. The same code would work for C and Fortran order arrays, and be > pretty simple. > > This would address Hans' issue, no? > > It's a special case but a common one. > > There is a special case for this already. It's just that the specific operations he is addressing requires creation of output arrays that by default are in C-order. This would need to change in order to take advantage of the special case. -Travis O. From charlesr.harris at gmail.com Thu Nov 8 19:16:23 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 8 Nov 2007 17:16:23 -0700 Subject: [Numpy-discussion] Warning or error on conversion from complex to float. In-Reply-To: <72D07F2D-305B-4698-8E2E-B35D91A29CE3@physics.ubc.ca> References: <72D07F2D-305B-4698-8E2E-B35D91A29CE3@physics.ubc.ca> Message-ID: On Nov 8, 2007 11:32 AM, Michael McNeil Forbes wrote: > Hi, > > Is it possible or easy to add a warning and/or error when array > assignments are made that lose information? I just got caught with > the following type of code: > > def f(): > return numpy.array([1j,2.0]) > > x = numpy.empty((2,),dtype=float) > x[:] = f() > > I am pre-allocating arrays for speed, but made a change to f() > resulting in complex results. My code now silently ignores the > imaginary part. It would have been very helpful if the assignment > check to make sure that the conversion was not going to lose > information (presumably with an assert so that performance is not > affected outside of debugging). > > Any simple suggestions on how to avoid this problem in the future? > How hard would it be to add such a check in numpy? > This seems to be a generic problem. In [1]: a = empty((10,), dtype=float32) In [2]: a[:] = 10.0 In [3]: a[:] = float32(1) In [4]: b = empty((10,), dtype=float64) In [5]: a[:] = b[:] In the case of the scalar assignments downcasting is a convenience because having to explicitly use the correct data type in an assignment gets to be a hassle when the type isn't native to Python. It might have been more correct to require matching data types, but convenience won out in that case. That said, it might be useful to warn on such an unexpected conversion as complex to float, and maybe for float to integer conversions also. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Nov 8 23:05:16 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 09 Nov 2007 13:05:16 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <47339958.4070700@enthought.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> <47335544.6060004@noaa.gov> <47339958.4070700@enthought.com> Message-ID: <4733DC7C.1010307@ar.media.kyoto-u.ac.jp> Travis E. Oliphant wrote: > Christopher Barker wrote: >> This discussion makes me wonder if the basic element-wise operations >> could (should?) be special cased for contiguous arrays, reducing them to >> simple pointer incrementing from the start to the finish of the data >> block. The same code would work for C and Fortran order arrays, and be >> pretty simple. >> >> This would address Hans' issue, no? >> >> It's a special case but a common one. >> >> > There is a special case for this already. It's just that the specific > operations he is addressing requires creation of output arrays that by > default are in C-order. This would need to change in order to take > advantage of the special case. For copy and array creation, I understand this, but for element-wise operations (mean, min, and max), this is not enough to explain the difference, no ? For example, I can understand a 50 % or 100 % time increase for simple operations (by simple, I mean one element operation taking only a few CPU cycles), because of copies, but a 5 fold time increase seems too big, no (mayb a cache problem, though) ? Also, the fact that mean is slower than min/max for both cases (F vs C) seems a bit counterintuitive (maybe cache effects are involved somehow ?). Again, I see huge differences between my Xeon PIV @ 3.2 Ghz and my pentium M @ 1.2 Ghz for those operations: pentium M gives more "intuitive results (and is almost as fast, and sometimes even faster than my Xeon for arrays which can stay in cache). cheers, David From david at ar.media.kyoto-u.ac.jp Thu Nov 8 23:06:49 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 09 Nov 2007 13:06:49 +0900 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: References: <4733519A.1050609@gmail.com> Message-ID: <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> Geoffrey Zhu wrote: > On Nov 8, 2007 12:12 PM, Robert Kern wrote: > >> Geoffrey Zhu wrote: >> >>> Good morning. >>> >>> I just installed the Windows binary of numpy 1.04. When I ran >>> numpy.test() in IDLE (the Python shell that comes with Python), the >>> program hang (or at least is running for half an hour). I am using >>> Windows XP, duel core intel CPU. >>> >>> Does anyone know what is going on? >>> >> No. Run numpy.test(verbosity=2) so it will print out each test name before >> running the test. Then we might have some idea about where the hang is coming from. >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless enigma >> that is made terrible by our own mad attempt to interpret it as though it had >> an underlying truth." >> -- Umberto Eco >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> >> > > Thanks for the hint, Robert. > > It hangs on check_cdouble (numpy.tests.test_linalg.test_det). > > > Also testip_not_allclose() had three warnings. I guess that's probably okay. > > testip_not_allclose (numpy.core.tests.test_numeric.test_allclose_inf) ... > Warning: invalid value encountered in absolute > Warning: invalid value encountered in absolute > Warning: invalid value encountered in less_equal > ok > Are you on x86-64, too ? Which BLAS are you using ? This smells like a C/Fortran problem (because it happens with complex values only). cheers, David From oliphant at enthought.com Thu Nov 8 23:44:11 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 08 Nov 2007 22:44:11 -0600 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <4733DC7C.1010307@ar.media.kyoto-u.ac.jp> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> <47335544.6060004@noaa.gov> <47339958.4070700@enthought.com> <4733DC7C.1010307@ar.media.kyoto-u.ac.jp> Message-ID: <4733E59B.5040001@enthought.com> David Cournapeau wrote: > Travis E. Oliphant wrote: > >> Christopher Barker wrote: >> >>> This discussion makes me wonder if the basic element-wise operations >>> could (should?) be special cased for contiguous arrays, reducing them to >>> simple pointer incrementing from the start to the finish of the data >>> block. The same code would work for C and Fortran order arrays, and be >>> pretty simple. >>> >>> This would address Hans' issue, no? >>> >>> It's a special case but a common one. >>> >>> >>> >> There is a special case for this already. It's just that the specific >> operations he is addressing requires creation of output arrays that by >> default are in C-order. This would need to change in order to take >> advantage of the special case. >> > For copy and array creation, I understand this, but for element-wise > operations (mean, min, and max), this is not enough to explain the > difference, no ? For example, I can understand a 50 % or 100 % time > increase for simple operations (by simple, I mean one element operation > taking only a few CPU cycles), because of copies, but a 5 fold time > increase seems too big, no (mayb a cache problem, though) ? Also, the > fact that mean is slower than min/max for both cases (F vs C) seems a > bit counterintuitive (maybe cache effects are involved somehow ?). > > Again, I see huge differences between my Xeon PIV @ 3.2 Ghz and my > pentium M @ 1.2 Ghz for those operations: pentium M gives more > "intuitive results (and is almost as fast, and sometimes even faster > than my Xeon for arrays which can stay in cache). > > I wasn't talking about the min, mean, and max methods specifically. These are all implemented with the reduce method of a ufunc. But, there are special cases for the reduce method as well and so relatively smooth pathways for optimization. -Travis O. From david at ar.media.kyoto-u.ac.jp Fri Nov 9 00:10:08 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 09 Nov 2007 14:10:08 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <4733E59B.5040001@enthought.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> <47335544.6060004@noaa.gov> <47339958.4070700@enthought.com> <4733DC7C.1010307@ar.media.kyoto-u.ac.jp> <4733E59B.5040001@enthought.com> Message-ID: <4733EBB0.9060107@ar.media.kyoto-u.ac.jp> Travis E. Oliphant wrote: > > I wasn't talking about the min, mean, and max methods specifically. > These are all implemented with the reduce method of a ufunc. > Ah, my mistake, I wrongly understood only some of them were implemented through ufunc. But the ufunc machinery has nothing to do with output array output, right ? So is the 5 time speed increase mainly happening inside ufunc ? David From oliphant at enthought.com Fri Nov 9 00:38:39 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 08 Nov 2007 23:38:39 -0600 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <4733EBB0.9060107@ar.media.kyoto-u.ac.jp> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> <47335544.6060004@noaa.gov> <47339958.4070700@enthought.com> <4733DC7C.1010307@ar.media.kyoto-u.ac.jp> <4733E59B.5040001@enthought.com> <4733EBB0.9060107@ar.media.kyoto-u.ac.jp> Message-ID: <4733F25F.6040004@enthought.com> David Cournapeau wrote: > Travis E. Oliphant wrote: > >> I wasn't talking about the min, mean, and max methods specifically. >> These are all implemented with the reduce method of a ufunc. >> >> > Ah, my mistake, I wrongly understood only some of them were implemented > through ufunc. But the ufunc machinery has nothing to do with output > array output, right ? So is the 5 time speed increase mainly happening > inside ufunc ? > I'm not sure. Ufuncs don't do everything in NumPy as you have noted. There are less well publicized (and less structured) array functions for each data-type that also implement things. Not all of these places have "optimized" paths, but there was some thought given to it in several places of the code. One area that could be improved that may be exposed in some of the timing tests, is that iterator-based algorithms always use a "C-contiguous" iterator. There is no "Fortran-contiguous" iterator (although there could be). -Travis O. From peridot.faceted at gmail.com Fri Nov 9 01:00:13 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 9 Nov 2007 01:00:13 -0500 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <4733DC7C.1010307@ar.media.kyoto-u.ac.jp> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> <47335544.6060004@noaa.gov> <47339958.4070700@enthought.com> <4733DC7C.1010307@ar.media.kyoto-u.ac.jp> Message-ID: On 08/11/2007, David Cournapeau wrote: > For copy and array creation, I understand this, but for element-wise > operations (mean, min, and max), this is not enough to explain the > difference, no ? For example, I can understand a 50 % or 100 % time > increase for simple operations (by simple, I mean one element operation > taking only a few CPU cycles), because of copies, but a 5 fold time > increase seems too big, no (mayb a cache problem, though) ? Also, the > fact that mean is slower than min/max for both cases (F vs C) seems a > bit counterintuitive (maybe cache effects are involved somehow ?). I have no doubt at all that cache effects are involved: for an int array, each data element is four bytes, but typical CPUs need to load 64 bytes at a time into cache. If you read (or write) the rest of those bytes in the next iterations through the loop, the (large) cost of a memory read is amortized. If you jump to the next row of the array, some large number of bytes away, those 64 bytes basically need to be purged to make room for another 64 bytes, of which you'll use 4. If you're reading from a FORTRAN-order array and writing to a C-order one, there's no way around doing this on one end or another: you're effectively doing a transpose, which is pretty much always expensive. Is there any reason not to let ufuncs pick whichever order for newly-allocated arrays they want? The natural choice would be the same as the bigger input array, if it's a contiguous block of memory (whether or not the contiguous flags are set). Failing that, the same as the other input array (if any); failing that, C order's as good a default as any. How difficult would this be to implement? Anne From david at ar.media.kyoto-u.ac.jp Fri Nov 9 01:30:45 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 09 Nov 2007 15:30:45 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711081650.42183.meine@informatik.uni-hamburg.de> <5b8d13220711080831r46b6b348k16f867ff990bd435@mail.gmail.com> <200711081755.14084.meine@informatik.uni-hamburg.de> <5b8d13220711080916l7acb63cdne5c896277b08bb55@mail.gmail.com> <47335544.6060004@noaa.gov> <47339958.4070700@enthought.com> <4733DC7C.1010307@ar.media.kyoto-u.ac.jp> Message-ID: <4733FE95.7080902@ar.media.kyoto-u.ac.jp> Anne Archibald wrote: > On 08/11/2007, David Cournapeau wrote: > >> For copy and array creation, I understand this, but for element-wise >> operations (mean, min, and max), this is not enough to explain the >> difference, no ? For example, I can understand a 50 % or 100 % time >> increase for simple operations (by simple, I mean one element operation >> taking only a few CPU cycles), because of copies, but a 5 fold time >> increase seems too big, no (mayb a cache problem, though) ? Also, the >> fact that mean is slower than min/max for both cases (F vs C) seems a >> bit counterintuitive (maybe cache effects are involved somehow ?). > > I have no doubt at all that cache effects are involved: for an int > array, each data element is four bytes, but typical CPUs need to load > 64 bytes at a time into cache. If you read (or write) the rest of > those bytes in the next iterations through the loop, the (large) cost > of a memory read is amortized. If you jump to the next row of the > array, some large number of bytes away, those 64 bytes basically need > to be purged to make room for another 64 bytes, of which you'll use 4. > If you're reading from a FORTRAN-order array and writing to a C-order > one, there's no way around doing this on one end or another: you're > effectively doing a transpose, which is pretty much always expensive. This is obviously part of the picture, and there is indeed no way around the simple fact that on sequential memory access is extremely slow on modern hardware. The fact that the array iterator does not support (yet) the Fortran order would largely explain this (I wrongly assumed the iterator already did that). > > Is there any reason not to let ufuncs pick whichever order for > newly-allocated arrays they want? The natural choice would be the same > as the bigger input array, if it's a contiguous block of memory > (whether or not the contiguous flags are set). Failing that, the same > as the other input array (if any); failing that, C order's as good a > default as any. How difficult would this be to implement? I think part of the problem is that it is difficult to know which is faster a priori (is copying multi-segmented parts in a single buffer faster for processing faster than direct processing ?). As mentioned previously, on the same platform (same OS/compiler combination), there are already huge differences between two CPU (pentium4 vs pentium M, the former being noticeably pig-slow when SSE FPU is not used). Does ufunc use buffer for segmented memory, or do they only use them for misbehaved arrays ? Also, from my very limited experience, I found array iterators to be significantly slower than simple array indexing on contiguous, single segment arrays. Do ufunc always use array iterator, or do they use simple array indexing when possible ? When I implemented the fast clip function (which does not use ufunc), I noticed that the following matter: - avoiding unnecessary copies. - quickly determine which cases can be optimized wrt to the operations you are using (memmove vs memcpy, etc...) - fast memory copying (numpy do not have those yet: by using optimized memory copying, you can often gain a 50 % speed increase on recent archs; also, having information about alignment could help, too, and we go back to aligned allocators discussion :) ). The problem is to find a good API to put those optimizations at one place. I unfortunately do not know much about ufunc, maybe they already implement most of those. cheers, David From david at ar.media.kyoto-u.ac.jp Fri Nov 9 02:59:55 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 09 Nov 2007 16:59:55 +0900 Subject: [Numpy-discussion] Suggestions on the best way to customize compilation flags Message-ID: <4734137B.9040008@ar.media.kyoto-u.ac.jp> Hi, I would like know the opinion from other people on the best way to customize compilation flags for building numpy. I am talking about the end-user point of view (e.g. I want to use the --optimize--so-aggressively-that-my-cpu-burns-into-flames flags, but without breaking build of python extensions because I forgot -fPIC). One of the goal of using scons within distutils was to separate the logic between compilation options which are necessary to make the binary work at all (basically, what are the options to build a python extension on supported platforms, how to build a dll for ctypes, etc...) and the ones which are 'secondary' (optimization, debug, warnings). I have started working in this direction, using compiler configuration objects, which are just dictionaries: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/distutils/scons/core/default.py The idea is that: - those configurations can be easily changed during the configure step of a package (for example, use sse unit instead of x87, etc...) - those configuration are not used directly when building projects: they are first copied to the actually used flags. The idea is that if necessary, you could request build environments without some warning flags (think pyrex/swig compilation), and only add them on user request. - those configurations are totally independent from the options necessary to build python extensions (those are not user accessible; I have not yet cleaned everything, but the goal is to have those flags set outside the extension builder logic, to avoid the distutils mess, which put and modifying flags everywhere deep down in the code). thanks, David From meine at informatik.uni-hamburg.de Fri Nov 9 06:37:12 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Fri, 9 Nov 2007 12:37:12 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <473398BC.5070501@enthought.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <473398BC.5070501@enthought.com> Message-ID: <200711091237.12969.meine@informatik.uni-hamburg.de> Am Freitag, 09. November 2007 00:16:12 schrieb Travis E. Oliphant: > C-order is "special" in NumPy due to the history. I agree that it > doesn't need to be and we have taken significant steps in that > direction. Thanks for this clarifying statement. > Right now, the fundamental problem is probably due to the > fact that the output arrays are being created as C-order arrays when the > input is a Fortran order array. Once that is changed then we can cause > Fortran-order inputs to use the simplest path through the code. That looks like a plan. I have read about __array_wrap__ in your book, and it sounds exactly like what we need for our imaging library; this way we can ensure that the result of operations on images is again an image. Here, it is crucial that the resulting image has Fortran order, too (for C/C++ algorithms to agree with the Python idea of the images' orientation). > Fortran order arrays can be preserved but it takes a little extra work > because backward compatible expectations had to be met. See for example > the order argument to the copy method of arrays. What do you mean exactly (if you have something specific in mind at all)? Should the order be configurable for all operations, and default to "C" for backwards compatibility? (BTW: copy() has no arguments yet in my numpybook, page 57.) At last, I had a look at the relevant code for adding arrays; this is what I found: In core/src/umathmodule.c.src, the code templates for the relevant ufuncs (e.g. add) are defined, which will get expanded to e.g. DOUBLE_add. This will get wrapped into an ufunc in 'InitOperators' using f = PyUFunc_FromFuncAndData(add_functions, add_data, add_signatures, 18, 2, 1, PyUFunc_Zero, "add", "adds the arguments elementwise.", 0); and inserted as "add" into a dictionary which is later passed to PyArray_SetNumericOps. This will install the ufunc in the static NumericOps struct 'n_ops', which is used in the 'array_add' function installed in the PyTypeObject. Thus, it is the ufunc code (not suprisingly), which seems to call DOUBLE_add for the inner loop: | static void | DOUBLE_add(char **args, intp *dimensions, intp *steps, void *func) | { | register intp i; | intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; | char *i1=args[0], *i2=args[1], *op=args[2]; | for(i=0; i References: <200711081334.33674.meine@informatik.uni-hamburg.de> <473398BC.5070501@enthought.com> <200711091237.12969.meine@informatik.uni-hamburg.de> Message-ID: Hi, this comment might be counterproductive to this discussion: However. I'm also using numpy as a basis for putting together an image analysis "platform"/library. This includes both 2D and higher dimensional images. Since all my code, if not n Python, is written in C or C++ and not Fortran, I decided early on that I had to get used to "invese indexing", as in image[y,x] or image[z,y,x] "Inverse" only refers to the common (alphabetecally oriented) intuition to say "at point x,y,z" rather than "z,y,x". I also argueed that mathematical matrices (mostly / often) use an index order as "row , column" which essentially is "y, x" (if the matrix is seen as an image) In any case, getting used to a "z,y,x" order saved me lots of technical troubles, and after all, most code is or will be C/C++ and not Fortran. Hope I was not entirey misguided, Sebastian Haase On Nov 9, 2007 12:37 PM, Hans Meine wrote: > Am Freitag, 09. November 2007 00:16:12 schrieb Travis E. Oliphant: > > C-order is "special" in NumPy due to the history. I agree that it > > doesn't need to be and we have taken significant steps in that > > direction. > > Thanks for this clarifying statement. > > > Right now, the fundamental problem is probably due to the > > fact that the output arrays are being created as C-order arrays when the > > input is a Fortran order array. Once that is changed then we can cause > > Fortran-order inputs to use the simplest path through the code. > > That looks like a plan. I have read about __array_wrap__ in your book, and it > sounds exactly like what we need for our imaging library; this way we can > ensure that the result of operations on images is again an image. Here, it > is crucial that the resulting image has Fortran order, too (for C/C++ > algorithms to agree with the Python idea of the images' orientation). > > > Fortran order arrays can be preserved but it takes a little extra work > > because backward compatible expectations had to be met. See for example > > the order argument to the copy method of arrays. > > What do you mean exactly (if you have something specific in mind at all)? > Should the order be configurable for all operations, and default to "C" for > backwards compatibility? > (BTW: copy() has no arguments yet in my numpybook, page 57.) > > At last, I had a look at the relevant code for adding arrays; this is what I > found: In core/src/umathmodule.c.src, the code templates for the relevant > ufuncs (e.g. add) are defined, which will get expanded to e.g. DOUBLE_add. > This will get wrapped into an ufunc in 'InitOperators' using > > f = PyUFunc_FromFuncAndData(add_functions, add_data, add_signatures, 18, > 2, 1, PyUFunc_Zero, "add", > "adds the arguments elementwise.", 0); > > and inserted as "add" into a dictionary which is later passed to > PyArray_SetNumericOps. This will install the ufunc in the static > NumericOps struct 'n_ops', which is used in the 'array_add' function > installed in the PyTypeObject. > > Thus, it is the ufunc code (not suprisingly), which seems to call > DOUBLE_add for the inner loop: > > | static void > | DOUBLE_add(char **args, intp *dimensions, intp *steps, void *func) > | { > | register intp i; > | intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; > | char *i1=args[0], *i2=args[1], *op=args[2]; > | for(i=0; i | *((double *)op)=*((double *)i1) + *((double *)i2); > | } > | } > > If I understood David correctly, he suggested an unstrided variant of > this inner loop, which simply uses pointer increments. While this is > a good idea (also probably quite some work), the real thing bugging me is > that the above DOUBLE_add could (and should!) be called by the ufunc > framework in such a way that it is equally efficient for C and Fortran > arrays. (I.e. with steps set to sizeof(double), without prior > copying/conversion.) It could even be used with negative step sizes when the > input and output arrays have different orders. Disclaimer: So far, I did not > look at the ufunc code yet, so I do not know which code paths exist. > > BTW: What are the "void *func" in core/src/umathmodule.c.src arguments > for? Shouldn't that be "void * /*func*/", since the parameters are > never used? > > -- > Ciao, / / > /--/ > / / ANS > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From meine at informatik.uni-hamburg.de Fri Nov 9 07:43:05 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Fri, 9 Nov 2007 13:43:05 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711091237.12969.meine@informatik.uni-hamburg.de> Message-ID: <200711091343.06414.meine@informatik.uni-hamburg.de> Am Freitag, 09. November 2007 13:04:24 schrieb Sebastian Haase: > Since all my code, if not n Python, is written in C or C++ and not > Fortran, I decided early on that I had to get used to "invese > indexing", as in > image[y,x] or image[z,y,x] We cannot do that here, since a) we use the opposite order in our C++ VIGRA, using operator(): img(x, y) is similar to img[y][x] in other libraries, and b) we have had Python bindings for quite some time (only not published), where we used img[x, y] already. Of course, we want to have the same order in both languages in order to facilitate porting algorithms after rapid prototyping. > "Inverse" only refers to the common (alphabetecally oriented) > intuition to say "at point x,y,z" rather than "z,y,x". Yes, this is the typical notation for coordinates and vectors. Nobody would write (z, y, x) vectors. ;-) > I also argueed that mathematical matrices (mostly / often) use an > index order as "row , column" which essentially is "y, x" (if the > matrix is seen as an image) That's right, it's different between matrices and images. In fact, we use "C" order for matrices in VIGRA: http://kogs-www.informatik.uni-hamburg.de/~koethe/vigra/doc/vigra/classvigra_1_1linalg_1_1Matrix.html -- Ciao, / / /--/ / / ANS From haase at msg.ucsf.edu Fri Nov 9 08:06:34 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 9 Nov 2007 14:06:34 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <200711091343.06414.meine@informatik.uni-hamburg.de> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711091237.12969.meine@informatik.uni-hamburg.de> <200711091343.06414.meine@informatik.uni-hamburg.de> Message-ID: On Nov 9, 2007 1:43 PM, Hans Meine wrote: > Am Freitag, 09. November 2007 13:04:24 schrieb Sebastian Haase: > Of course, we want to have the same order in both languages in order to > facilitate porting algorithms after rapid prototyping. > Yes, I understand your situation. > > "Inverse" only refers to the common (alphabetecally oriented) > > intuition to say "at point x,y,z" rather than "z,y,x". > > Yes, this is the typical notation for coordinates and vectors. Nobody would > write (z, y, x) vectors. ;-) This is of course my fate now - you have to choose your poisson. ;-) I find it worse having to think in Fortran-order ... That's were I was saying "I hope I wasn't misguided" Cheers, Sebastian From oliphant at enthought.com Fri Nov 9 11:12:06 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 09 Nov 2007 10:12:06 -0600 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <200711091237.12969.meine@informatik.uni-hamburg.de> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <473398BC.5070501@enthought.com> <200711091237.12969.meine@informatik.uni-hamburg.de> Message-ID: <473486D6.4080904@enthought.com> Hans Meine wrote: > > > >> Fortran order arrays can be preserved but it takes a little extra work >> because backward compatible expectations had to be met. See for example >> the order argument to the copy method of arrays. >> > > What do you mean exactly (if you have something specific in mind at all)? > I meant a.copy('A') will preserver the order of 'a' on copy. > Should the order be configurable for all operations, and default to "C" for > backwards compatibility? > Yes, basically that should be the case when it matters. > (BTW: copy() has no arguments yet in my numpybook, page 57.) > > At last, I had a look at the relevant code for adding arrays; this is what I > found: In core/src/umathmodule.c.src, the code templates for the relevant > ufuncs (e.g. add) are defined, which will get expanded to e.g. DOUBLE_add. > This will get wrapped into an ufunc in 'InitOperators' using > > f = PyUFunc_FromFuncAndData(add_functions, add_data, add_signatures, 18, > 2, 1, PyUFunc_Zero, "add", > "adds the arguments elementwise.", 0); > > and inserted as "add" into a dictionary which is later passed to > PyArray_SetNumericOps. This will install the ufunc in the static > NumericOps struct 'n_ops', which is used in the 'array_add' function > installed in the PyTypeObject. > Yep, you've got the code path correct. > Thus, it is the ufunc code (not suprisingly), which seems to call > DOUBLE_add for the inner loop: > Indeed. The ufuncs store inner loops for all the data-types that are supported by the ufunc. These inner loops allow for "strided" memory access and are called by the general ufunc code. The loop structure holds the information needed to loop over the ufuncs. > | static void > | DOUBLE_add(char **args, intp *dimensions, intp *steps, void *func) > | { > | register intp i; > | intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; > | char *i1=args[0], *i2=args[1], *op=args[2]; > | for(i=0; i | *((double *)op)=*((double *)i1) + *((double *)i2); > | } > | } > > If I understood David correctly, he suggested an unstrided variant of > this inner loop, which simply uses pointer increments. I'm not sure if he was saying that or not. But, more generally, I'm all for optimizations at this level. But, they will have to be done differently for different cases with this loop as the fall-back. > While this is > a good idea (also probably quite some work), the real thing bugging me is > that the above DOUBLE_add could (and should!) be called by the ufunc > framework in such a way that it is equally efficient for C and Fortran > arrays. Yes, that's what I was talking about. There is actually a path through the ufunc code where this loop is called only once. The requirement right now is that all the arrays are C-contiguous, but this should be changed to all arrays have the same contiguousness (and the output-array creation code changed to create Fortran-order arrays when the inputs are all Fortran-order). > BTW: What are the "void *func" in core/src/umathmodule.c.src arguments > for? Shouldn't that be "void * /*func*/", since the parameters are > never used? > Sure, it could be void * /* func*/ You'll have to forgive my ignorance, but does that matter? Are we using up a register or something by naming an unused argument? -Travis O. From zyzhu2000 at gmail.com Fri Nov 9 11:14:32 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Fri, 9 Nov 2007 10:14:32 -0600 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> References: <4733519A.1050609@gmail.com> <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> Message-ID: On Nov 8, 2007 10:06 PM, David Cournapeau wrote: > > Geoffrey Zhu wrote: > > On Nov 8, 2007 12:12 PM, Robert Kern wrote: > > > >> Geoffrey Zhu wrote: > >> > >>> Good morning. > >>> > >>> I just installed the Windows binary of numpy 1.04. When I ran > >>> numpy.test() in IDLE (the Python shell that comes with Python), the > >>> program hang (or at least is running for half an hour). I am using > >>> Windows XP, duel core intel CPU. > >>> > >>> Does anyone know what is going on? > >>> > >> No. Run numpy.test(verbosity=2) so it will print out each test name before > >> running the test. Then we might have some idea about where the hang is coming from. > >> > >> -- > >> Robert Kern > >> > >> "I have come to believe that the whole world is an enigma, a harmless enigma > >> that is made terrible by our own mad attempt to interpret it as though it had > >> an underlying truth." > >> -- Umberto Eco > >> _______________________________________________ > >> Numpy-discussion mailing list > >> Numpy-discussion at scipy.org > >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > >> > >> > > > > Thanks for the hint, Robert. > > > > It hangs on check_cdouble (numpy.tests.test_linalg.test_det). > > > > > > Also testip_not_allclose() had three warnings. I guess that's probably okay. > > > > testip_not_allclose (numpy.core.tests.test_numeric.test_allclose_inf) ... > > Warning: invalid value encountered in absolute > > Warning: invalid value encountered in absolute > > Warning: invalid value encountered in less_equal > > ok > > > Are you on x86-64, too ? Which BLAS are you using ? This smells like a > C/Fortran problem (because it happens with complex values only). > > cheers, > > David > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Hi David, Although the processor is Intel Duo Core E6700, which supports x86-64, I am only using 32-bit Windows XP. I am not using my own BLAS. I am simply using the pre-compiled binary. I installed 1.04 over the old 1.031. I don't know if it is caused by some files that are left behind. Thanks, Geoffrey From zyzhu2000 at gmail.com Fri Nov 9 11:52:55 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Fri, 9 Nov 2007 10:52:55 -0600 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: References: <4733519A.1050609@gmail.com> <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> Message-ID: On Nov 9, 2007 10:14 AM, Geoffrey Zhu wrote: > > On Nov 8, 2007 10:06 PM, David Cournapeau wrote: > > > > Geoffrey Zhu wrote: > > > On Nov 8, 2007 12:12 PM, Robert Kern wrote: > > > > > >> Geoffrey Zhu wrote: > > >> > > >>> Good morning. > > >>> > > >>> I just installed the Windows binary of numpy 1.04. When I ran > > >>> numpy.test() in IDLE (the Python shell that comes with Python), the > > >>> program hang (or at least is running for half an hour). I am using > > >>> Windows XP, duel core intel CPU. > > >>> > > >>> Does anyone know what is going on? > > >>> > > >> No. Run numpy.test(verbosity=2) so it will print out each test name before > > >> running the test. Then we might have some idea about where the hang is coming from. > > >> > > >> -- > > >> Robert Kern > > >> > > >> "I have come to believe that the whole world is an enigma, a harmless enigma > > >> that is made terrible by our own mad attempt to interpret it as though it had > > >> an underlying truth." > > >> -- Umberto Eco > > >> _______________________________________________ > > >> Numpy-discussion mailing list > > >> Numpy-discussion at scipy.org > > >> http://projects.scipy.org/mailman/listinfo/numpy-discussion > > >> > > >> > > > > > > Thanks for the hint, Robert. > > > > > > It hangs on check_cdouble (numpy.tests.test_linalg.test_det). > > > > > > > > > Also testip_not_allclose() had three warnings. I guess that's probably okay. > > > > > > testip_not_allclose (numpy.core.tests.test_numeric.test_allclose_inf) ... > > > Warning: invalid value encountered in absolute > > > Warning: invalid value encountered in absolute > > > Warning: invalid value encountered in less_equal > > > ok > > > > > Are you on x86-64, too ? Which BLAS are you using ? This smells like a > > C/Fortran problem (because it happens with complex values only). > > > > cheers, > > > > David > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > Hi David, > > Although the processor is Intel Duo Core E6700, which supports x86-64, > I am only using 32-bit Windows XP. I am not using my own BLAS. I am > simply using the pre-compiled binary. I installed 1.04 over the old > 1.031. I don't know if it is caused by some files that are left > behind. > > Thanks, > Geoffrey > Okay. I verified (by looking at which DLL python.exe actually loads) that it is using _dotblas.pyd under C:\Python25\Lib\site-packages\numpy\core that came with the pre-built 1.04 binary. It is not using any left-over from 1.031. From zyzhu2000 at gmail.com Fri Nov 9 12:31:54 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Fri, 9 Nov 2007 11:31:54 -0600 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: References: <4733519A.1050609@gmail.com> <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> Message-ID: Very interesting! If I use the MSI file, numpy.test() hangs. If, however, I use the EGG file, it is actually fine. From nwagner at iam.uni-stuttgart.de Fri Nov 9 12:42:31 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 09 Nov 2007 18:42:31 +0100 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: References: <4733519A.1050609@gmail.com> <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, 9 Nov 2007 11:31:54 -0600 "Geoffrey Zhu" wrote: > Very interesting! If I use the MSI file, numpy.test() >hangs. If, > however, I use the EGG file, it is actually fine. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion How do I install an EGG ? I am testing Windows XP SP2. Nils From oliphant at enthought.com Fri Nov 9 12:45:27 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 09 Nov 2007 11:45:27 -0600 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: References: <4733519A.1050609@gmail.com> <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> Message-ID: <47349CB7.5050209@enthought.com> Geoffrey Zhu wrote: > Very interesting! If I use the MSI file, numpy.test() hangs. If, > however, I use the EGG file, it is actually fine. > Can you find the md5sum of these files? There is a md5sum.exe at http://www.etree.org/md5com.html It would be good to verify that you have the correct files. Thanks, -Travis From zyzhu2000 at gmail.com Fri Nov 9 13:01:24 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Fri, 9 Nov 2007 12:01:24 -0600 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: <47349CB7.5050209@enthought.com> References: <4733519A.1050609@gmail.com> <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> <47349CB7.5050209@enthought.com> Message-ID: On Nov 9, 2007 11:45 AM, Travis E. Oliphant wrote: > Geoffrey Zhu wrote: > > Very interesting! If I use the MSI file, numpy.test() hangs. If, > > however, I use the EGG file, it is actually fine. > > > > Can you find the md5sum of these files? > > There is a md5sum.exe at > > http://www.etree.org/md5com.html > > It would be good to verify that you have the correct files. > > > Thanks, > > -Travis > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Sure. Here they are: a3be796f61ee63c06675524f3dda8836 *numpy-1.0.4.win32-py2.5.msi 07bb650726caa946abfa464d8968437c *numpy-1.0.4-py2.5-win32.egg I downloaded them from sourceforge. By the way, something is gong on with the sourceforge download page. When I tried to download the MSI file, sourceforge always comes up with a connection error. I figured it was in the ajax and managed to click the direct download link really quickly. Another sign is that all the download counts for 1.04 are zero. From zyzhu2000 at gmail.com Fri Nov 9 13:05:00 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Fri, 9 Nov 2007 12:05:00 -0600 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: References: <4733519A.1050609@gmail.com> <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> Message-ID: On Nov 9, 2007 11:42 AM, Nils Wagner wrote: > On Fri, 9 Nov 2007 11:31:54 -0600 > "Geoffrey Zhu" wrote: > > > Very interesting! If I use the MSI file, numpy.test() > >hangs. If, > > however, I use the EGG file, it is actually fine. > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > How do I install an EGG ? I am testing Windows XP SP2. > > Nils > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > I downloaded and ran ez_setup.py from here: http://peak.telecommunity.com/DevCenter/EasyInstall . And then I downloaded the egg file and ran easy_install c:\mypath\numpy-1.0.4-py2.5-win32.egg. Do not put the egg file in any of your python package directories, or easy_install won't unpack it. I haven't figured out why. From rex at nosyntax.com Fri Nov 9 14:16:17 2007 From: rex at nosyntax.com (rex) Date: Fri, 9 Nov 2007 11:16:17 -0800 Subject: [Numpy-discussion] NumPy 1.04, MKL 10.0, & Intel 10.1 icc & ifort Message-ID: <20071109191617.GA17405@nosyntax.net> Build was successful after a change in distutils. Core 2 Duo, Debian Etch-32, Python 2.5, icc 10.1, ifort 10.1, & mkl 10.0. MKL & the compilers were installed to their default locations: /opt/intel/mkl/, /opt/intel/cc/, /opt/intel/fc/ Installation will not interfere with earlier versions. NumPy site.cfg for MKL 10.0 ========================================== [DEFAULT] library_dirs=/opt/intel/mkl/10.0.011/lib/32 include_dirs=/opt/intel/mkl/10.0.011/include [mkl] libraries=mkl,guide [lapack_info] libraries=mkl_lapack,mkl,guide ========================================== Changes in distutils: ========================================================= In MKL 9.1 & 10.0, mkl_lapack32 & mkl_lapack64 no longer exist. They were replaced by /32/mkl_lapack and /64/mkl_lapack. As a result, numpy-1.04/numpy/distutils/system_info.py needs to be changed: #lapack_libs = self.get_libs('lapack_libs',['mkl_lapack32','mkl_lapack64']) lapack_libs = self.get_libs('lapack_libs',['mkl_lapack']) If the above change is not made numpy builds, but without using MKL. In numpy-1.04/numpy/distutils/ccompiler.py change: #cc_exe = 'icc' #works, but is suboptimal cc_exe = 'icc -msse3 -fast' #adjust to suit your cpu ========================================================= Running the command below (all one line) from the numpy-1.04 directory python setup.py config --compiler=intel build_clib --compiler=intel build_ext --compiler=intel install > inst.log Gave this inst.log: ===================================================== F2PY Version 2_4412 blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl', 'pthread', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] FOUND: libraries = ['mkl', 'pthread', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] lapack_opt_info: lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl', 'pthread', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] FOUND: libraries = ['mkl_lapack', 'mkl', 'pthread', 'mkl', 'guide', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] FOUND: libraries = ['mkl_lapack', 'mkl', 'pthread', 'mkl', 'guide', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] running config running build_clib running build_ext running build_src building py_modules sources creating build [...] =========================================================== For reference, here are the respective lib/32 files for MKL 9.1 and 10.0: c2d0:/opt/intel/mkl/9.1/lib/32# ls libguide.a libmkl_gfortran.a libmkl_ias.so libmkl_p3.so libmkl_p4.so libmkl_vml_def.so libmkl_vml_p4p.so libguide.so libmkl_gfortran.so libmkl_lapack.a libmkl_p4m.so libmkl.so libmkl_vml_p3.so libmkl_vml_p4.so libmkl_def.so libmkl_ia32.a libmkl_lapack.so libmkl_p4p.so libmkl_solver.a libmkl_vml_p4m.so libvml.so c2d0:/opt/intel/mkl/10.0.011/lib/32# ls libguide.a libmkl_cdft_core.a libmkl_intel.a libmkl_p4.so libmkl_vml_ia.so libguide.so libmkl_core.a libmkl_intel.so libmkl_scalapack.a libmkl_vml_p3.so libiomp5.a libmkl_core.so libmkl_intel_thread.a libmkl_scalapack_core.a libmkl_vml_p4m2.so libiomp5.so libmkl_def.so libmkl_intel_thread.so libmkl_sequential.a libmkl_vml_p4m.s o libmkl_blacs.a libmkl_gf.a libmkl_lapack.a libmkl_sequential.so libmkl_vml_p4p.so libmkl_blacs_intelmpi20.a libmkl_gf.so libmkl_lapack.so libmkl.so libmkl_vml_p4.so libmkl_blacs_intelmpi.a libmkl_gnu_thread.a libmkl_p3.so libmkl_solver.a libmkl_blacs_openmpi.a libmkl_gnu_thread.so libmkl_p4m.so libmkl_solver_sequential.a libmkl_cdft.a libmkl_ia32.a libmkl_p4p.so libmkl_vml_def.so Thanks to all who have helped me with earlier versions. -rex From meine at informatik.uni-hamburg.de Fri Nov 9 14:39:59 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Fri, 9 Nov 2007 20:39:59 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <473486D6.4080904@enthought.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711091237.12969.meine@informatik.uni-hamburg.de> <473486D6.4080904@enthought.com> Message-ID: <200711092039.59828.meine@informatik.uni-hamburg.de> On Freitag 09 November 2007, Travis E. Oliphant wrote: > > While this is > > a good idea (also probably quite some work), the real thing bugging me is > > that the above DOUBLE_add could (and should!) be called by the ufunc > > framework in such a way that it is equally efficient for C and Fortran > > arrays. > > Yes, that's what I was talking about. There is actually a path through > the ufunc code where this loop is called only once. The requirement > right now is that all the arrays are C-contiguous, but this should be > changed to all arrays have the same contiguousness (and the output-array > creation code changed to create Fortran-order arrays when the inputs are > all Fortran-order). After some measurements, I must say that even the slower Fortran variant is competitive (read: faster ;-) ), compared with our very flexible dynamic functors used in the current interactive VIGRA. IOW: Good job. :-) > > BTW: What are the "void *func" in core/src/umathmodule.c.src arguments > > for? Shouldn't that be "void * /*func*/", since the parameters are > > never used? > > Sure, it could be void * /* func*/ > > You'll have to forgive my ignorance, but does that matter? Are we > using up a register or something by naming an unused argument? No, but it triggers a compiler warning ("unused parameter foo", if warnings are enabled), which I even think to be useful. At least in g++, but that obviously showed my ignorance of C, since gcc -W does not complain. ;-) BTW: I am getting an interesting warning for numpy/core/src/scalartypes.inc.src:288 (empty 'if' body), which indeed looks suspicious. The rest of the warnings can be neglected I AFAICS. Ciao, / / .o. /--/ ..o / / ANS ooo -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From cournape at gmail.com Fri Nov 9 21:23:02 2007 From: cournape at gmail.com (David Cournapeau) Date: Sat, 10 Nov 2007 11:23:02 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <473486D6.4080904@enthought.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <473398BC.5070501@enthought.com> <200711091237.12969.meine@informatik.uni-hamburg.de> <473486D6.4080904@enthought.com> Message-ID: <5b8d13220711091823r1e22c14cj750d988ba2f9a7b3@mail.gmail.com> On Nov 10, 2007 1:12 AM, Travis E. Oliphant wrote: > Hans Meine wrote: > > | static void > > | DOUBLE_add(char **args, intp *dimensions, intp *steps, void *func) > > | { > > | register intp i; > > | intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; > > | char *i1=args[0], *i2=args[1], *op=args[2]; > > | for(i=0; i > | *((double *)op)=*((double *)i1) + *((double *)i2); > > | } > > | } > > > > If I understood David correctly, he suggested an unstrided variant of > > this inner loop, which simply uses pointer increments. > I'm not sure if he was saying that or not. But, more generally, I'm all > for optimizations at this level. But, they will have to be done > differently for different cases with this loop as the fall-back. I was not really clear, but yes, his was part of my argument: I don't think compilers can optimize the above well when there is no stride (more exactly when the stride could be done by simply using array indexing). This would need some benchmarks, but I have always read that using pointer arithmetics should be avoided when speed matters (e.g. *a + n * sizeof(*a) compared to a[n]), because it becomes much more difficult for the compiler to optimize, Generally, if you can get to a function which does the thing the "obvious way", this is better. Of course, you have to separate the case where this is possible and where it is not. But such work would also be really helpful if/when we optimize some basic things with MMX/SSE and co, and I think the above is impossible to auto vectorize (gcc 4.3, not released yet, gives some really helpful analysis for that, and can tell you when it fails to auto-vectorize, and why). > > While this is > > a good idea (also probably quite some work), the real thing bugging me is > > that the above DOUBLE_add could (and should!) be called by the ufunc > > framework in such a way that it is equally efficient for C and Fortran > > arrays. > Yes, that's what I was talking about. There is actually a path through > the ufunc code where this loop is called only once. The requirement > right now is that all the arrays are C-contiguous, but this should be > changed to all arrays have the same contiguousness (and the output-array > creation code changed to create Fortran-order arrays when the inputs are > all Fortran-order). This was my other point. For some of my own C code, that's what I do: I have some function to detect whether the array can be treated sequentially, for cases where C vs F does not matter at all. I don't see difficulty to provide such a facility in numpy, at least for the common operations we are talking about, but maybe I am missing something. I should take a look a the ufunc machinery David From cournape at gmail.com Fri Nov 9 21:33:10 2007 From: cournape at gmail.com (David Cournapeau) Date: Sat, 10 Nov 2007 11:33:10 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <200711092039.59828.meine@informatik.uni-hamburg.de> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711091237.12969.meine@informatik.uni-hamburg.de> <473486D6.4080904@enthought.com> <200711092039.59828.meine@informatik.uni-hamburg.de> Message-ID: <5b8d13220711091833u1f575e59me46f663e24fdbbd5@mail.gmail.com> On Nov 10, 2007 4:39 AM, Hans Meine wrote: > On Freitag 09 November 2007, Travis E. Oliphant wrote: > > > While this is > > > a good idea (also probably quite some work), the real thing bugging me is > > > that the above DOUBLE_add could (and should!) be called by the ufunc > > > framework in such a way that it is equally efficient for C and Fortran > > > arrays. > > > > Yes, that's what I was talking about. There is actually a path through > > the ufunc code where this loop is called only once. The requirement > > right now is that all the arrays are C-contiguous, but this should be > > changed to all arrays have the same contiguousness (and the output-array > > creation code changed to create Fortran-order arrays when the inputs are > > all Fortran-order). > > After some measurements, I must say that even the slower Fortran variant is > competitive (read: faster ;-) ), compared with our very flexible dynamic > functors used in the current interactive VIGRA. IOW: Good job. :-) I am not suprised at all: although I've seen many times people saying that C++ can provide good abstractions without cost, this is not my experience, at least with g++. Whatever the method/library used (blitz, the ones in boost, etc...). I am really convinced that C++ provides the bad abstraction for numerical works (and the fundamental design decision of C++ to avoid extending the language to implement things in libraries instead is flawed, at least for numerical works). cheers, David From cournape at gmail.com Fri Nov 9 22:09:07 2007 From: cournape at gmail.com (David Cournapeau) Date: Sat, 10 Nov 2007 12:09:07 +0900 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <5b8d13220711091823r1e22c14cj750d988ba2f9a7b3@mail.gmail.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <473398BC.5070501@enthought.com> <200711091237.12969.meine@informatik.uni-hamburg.de> <473486D6.4080904@enthought.com> <5b8d13220711091823r1e22c14cj750d988ba2f9a7b3@mail.gmail.com> Message-ID: <5b8d13220711091909rce0c29m9534c4439e7ba1fc@mail.gmail.com> On Nov 10, 2007 11:23 AM, David Cournapeau wrote: > > This would need some benchmarks, but I have always read that using > pointer arithmetics should be avoided when speed matters (e.g. *a + n > * sizeof(*a) compared to a[n]), because it becomes much more difficult > for the compiler to optimize, Generally, if you can get to a function > which does the thing the "obvious way", this is better. Of course, you > have to separate the case where this is possible and where it is not. > But such work would also be really helpful if/when we optimize some > basic things with MMX/SSE and co, and I think the above is impossible > to auto vectorize (gcc 4.3, not released yet, gives some really > helpful analysis for that, and can tell you when it fails to > auto-vectorize, and why). Actually, gcc 4.2 already supports the option: -ftree-vectorizer-verbose=n. cheers, David > > > > While this is > > > a good idea (also probably quite some work), the real thing bugging me is > > > that the above DOUBLE_add could (and should!) be called by the ufunc > > > framework in such a way that it is equally efficient for C and Fortran > > > arrays. > > Yes, that's what I was talking about. There is actually a path through > > the ufunc code where this loop is called only once. The requirement > > right now is that all the arrays are C-contiguous, but this should be > > changed to all arrays have the same contiguousness (and the output-array > > creation code changed to create Fortran-order arrays when the inputs are > > all Fortran-order). > > This was my other point. For some of my own C code, that's what I do: > I have some function to detect whether the array can be treated > sequentially, for cases where C vs F does not matter at all. I don't > see difficulty to provide such a facility in numpy, at least for the > common operations we are talking about, but maybe I am missing > something. I should take a look a the ufunc machinery > > David > From charlesr.harris at gmail.com Fri Nov 9 23:36:34 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 9 Nov 2007 21:36:34 -0700 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <5b8d13220711091823r1e22c14cj750d988ba2f9a7b3@mail.gmail.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <473398BC.5070501@enthought.com> <200711091237.12969.meine@informatik.uni-hamburg.de> <473486D6.4080904@enthought.com> <5b8d13220711091823r1e22c14cj750d988ba2f9a7b3@mail.gmail.com> Message-ID: On Nov 9, 2007 7:23 PM, David Cournapeau wrote: > On Nov 10, 2007 1:12 AM, Travis E. Oliphant > wrote: > > Hans Meine wrote: > > > | static void > > > | DOUBLE_add(char **args, intp *dimensions, intp *steps, void *func) > > > | { > > > | register intp i; > > > | intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; > > > | char *i1=args[0], *i2=args[1], *op=args[2]; > > > | for(i=0; i > > | *((double *)op)=*((double *)i1) + *((double *)i2); > > > | } > > > | } > > > > > > If I understood David correctly, he suggested an unstrided variant of > > > this inner loop, which simply uses pointer increments. > > I'm not sure if he was saying that or not. But, more generally, I'm all > > for optimizations at this level. But, they will have to be done > > differently for different cases with this loop as the fall-back. > > I was not really clear, but yes, his was part of my argument: I don't > think compilers can optimize the above well when there is no stride > (more exactly when the stride could be done by simply using array > indexing). > > This would need some benchmarks, but I have always read that using > pointer arithmetics should be avoided when speed matters (e.g. *a + n > * sizeof(*a) compared to a[n]), because it becomes much more difficult > for the compiler to optimize, Generally, if you can get to a function > which does the thing the "obvious way", this is better. Of course, you > have to separate the case where this is possible and where it is not. > But such work would also be really helpful if/when we optimize some > basic things with MMX/SSE and co, and I think the above is impossible > to auto vectorize (gcc 4.3, not released yet, gives some really > helpful analysis for that, and can tell you when it fails to > auto-vectorize, and why). > The relative speed of pointers vs indexing depends on a lot of things: 1) the architecture (registers, instruction sets, implementation) 2) the compiler 3) the code (number of base addresses, etc.) My subjective feel is that on newer intel type hardware and using a recent compiler, indexing is generally faster. But it does depend. I wrote an indexed version of the code above: typedef int intp; void DOUBLE_add_ptr(char **args, intp *dimensions, intp *steps, void *func) { register intp i; intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; char *i1=args[0], *i2=args[1], *op=args[2]; for(i=0; i From meine at informatik.uni-hamburg.de Sat Nov 10 10:26:43 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Sat, 10 Nov 2007 16:26:43 +0100 Subject: [Numpy-discussion] Unnecessarily bad performance of elementwise operators with Fortran-arrays In-Reply-To: <5b8d13220711091833u1f575e59me46f663e24fdbbd5@mail.gmail.com> References: <200711081334.33674.meine@informatik.uni-hamburg.de> <200711092039.59828.meine@informatik.uni-hamburg.de> <5b8d13220711091833u1f575e59me46f663e24fdbbd5@mail.gmail.com> Message-ID: <200711101626.48348.meine@informatik.uni-hamburg.de> On Samstag 10 November 2007, David Cournapeau wrote: > > After some measurements, I must say that even the slower Fortran variant > > is competitive (read: faster ;-) ), compared with our very flexible > > dynamic functors used in the current interactive VIGRA. IOW: Good job. > > :-) > > I am not suprised at all: although I've seen many times people saying > that C++ can provide good abstractions without cost, this is not my > experience, at least with g++. Oh, I tried to prevent this misunderstanding; this is about the interactive VIGRA. Here, we have a functor grammar parser which composes functors (objects with virtual methods etc.) according to the syntax tree, in order to compute complex operations in one go. In case you're interested in our API, here's the relevant part of a mini tutorial I wrote lately: http://kogs-www.informatik.uni-hamburg.de/~meine/vigra_interactive/#processing-each-pixel This is not to be confused with STL-like C++ functors; i.e. in the normal C++ VIGRA we use functors a lot, and our experience is very positive. E.g. our copyImage which gets two 2D source iterators (upper left, lower right), a source accessor (which may perform additional conversions or pull pixel components from different sources together), and a destination iterator + accessor has exactly the speed of a memcpy when standard (pass-through) accessors are used. (In fact, it was slightly faster, but most probably within measurement uncertainty.) > Whatever the method/library used (blitz, the ones in boost, etc...). I like boost a lot, but I think in some places it may stress compilers too much. OTOH, my experience is that compilation takes ages, but the resulting code is quite fast. But that may depend on the (part of) library you're talking about. > I am really convinced that C++ > provides the bad abstraction for numerical works (and the fundamental > design decision of C++ to avoid extending the language to implement > things in libraries instead is flawed, at least for numerical works). I cannot follow you here, but I think there's not much point having yet another "which language is best" discussion, especially not when it becomes off-topic. I just wanted to tell you that my above statement is not an example that "C++ can [not] provide abstractions without cost". The same statement in C++ would be much faster than NumPy, but it's comparing apples and oranges (C++ vs. Python). I just compared our Python extension module with NumPy, and both use a very different approach (cf. the above link). Ciao, / / .o. /--/ ..o / / ANS ooo -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From matthieu.brucher at gmail.com Sat Nov 10 12:33:26 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 10 Nov 2007 18:33:26 +0100 Subject: [Numpy-discussion] Compilation with Visual Studio 2003 Message-ID: Hi, I want to add some support to ACML, but currently, I can't even compile numpy (Visual Studio and g77), the current SVN fails with : running build_ext No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler customize MSVCCompiler using build_ext building 'numpy.core.multiarray' extension compiling C sources D:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ibuild\src.win32-2.5\numpy\core\src-Inumpy\core\include -Ibuild\src.win32-2.5\numpy\core -Inumpy\core\src -Inumpy\core\include -ID:\Pyth on25\include -ID:\Python25\PC /Tcnumpy\core\src\multiarraymodule.c /Fobuild\temp .win32-2.5\Release\numpy\core\src\multiarraymodule.obj multiarraymodule.c d:\Travail\numpy\numpy\core\src\arrayobject.c(662) : warning C4244: '=' : conver sion de 'npy_longlong' en 'long', perte possible de donn?es numpy\core\src\multiarraymodule.c(7592) : error C2065: 'NPY_ALLOW_THREADS' : ide ntificateur non d?clar? error: Command "D:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.ex e /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ibuild\src.win32-2.5\numpy\core\src-Inum py\core\include -Ibuild\src.win32-2.5\numpy\core -Inumpy\core\src -Inumpy\core\i nclude -ID:\Python25\include -ID:\Python25\PC /Tcnumpy\core\src\multiarraymodule .c /Fobuild\temp.win32-2.5\Release\numpy\core\src\multiarraymodule.obj" failed w ith exit status 2 Is there something I forgot to add in the site.cfg ? Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From lxander.m at gmail.com Sat Nov 10 13:25:39 2007 From: lxander.m at gmail.com (Alexander Michael) Date: Sat, 10 Nov 2007 13:25:39 -0500 Subject: [Numpy-discussion] numpy 1.04 numpy.test() hang In-Reply-To: References: <4733519A.1050609@gmail.com> <4733DCD9.3000307@ar.media.kyoto-u.ac.jp> Message-ID: <525f23e80711101025o38d14b26u92ecf27b6b871fa1@mail.gmail.com> On 11/9/07, Geoffrey Zhu wrote: > And then I downloaded the egg file and ran easy_install > c:\mypath\numpy-1.0.4-py2.5-win32.egg. Do not put the egg file in any > of your python package directories, or easy_install won't unpack it. I > haven't figured out why. Most likely because "installing an egg" for the most part simply means putting it on your python path. I say mostly, because of the issue of unzipping that your experiencing as well as possibly adding it to your easy_install.pth file and maybe even an issue with the part of the python path the egg being on needing to be one that allows .pth files. This is the long answer (and should likely be considered a bug), the short answer is "don't do that". :) You can also install setuptools with the windows installer at: . From mforbes at physics.ubc.ca Sat Nov 10 17:33:57 2007 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Sat, 10 Nov 2007 14:33:57 -0800 Subject: [Numpy-discussion] Warnings as exceptions? Message-ID: Why are numpy warnings printed rather than issued using the standard warnings library? I know that the behaviour can be controlled by seterr(), but it seem rather unpythonic not to use the warnings library. Is there an explicit reason for this choice? (It seems like a pretty trivial modification of util.py will catch most of the cases.) If performance is an issue, one could add a 'print' mode which behaves as the current 'warn'. Michael. (P.S. I am sure this has been discussed somewhere before, but I can't seem to find a reference.) From cournape at gmail.com Sat Nov 10 21:30:59 2007 From: cournape at gmail.com (David Cournapeau) Date: Sun, 11 Nov 2007 11:30:59 +0900 Subject: [Numpy-discussion] Compilation with Visual Studio 2003 In-Reply-To: References: Message-ID: <5b8d13220711101830g66d3958et51f72a802ab65e71@mail.gmail.com> On Nov 11, 2007 2:33 AM, Matthieu Brucher wrote: > Hi, > > I want to add some support to ACML, but currently, I can't even compile > numpy (Visual Studio and g77), the current SVN fails with : > running build_ext > No module named msvccompiler in numpy.distutils; trying from distutils > customize MSVCCompiler > customize MSVCCompiler using build_ext > building 'numpy.core.multiarray' extension > compiling C sources > D:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo > /Ox > /MD /W3 /GX /DNDEBUG -Ibuild\src.win32-2.5\numpy\core\src > -Inumpy\core\include > -Ibuild\src.win32-2.5\numpy\core -Inumpy\core\src -Inumpy\core\include > -ID:\Pyth > on25\include -ID:\Python25\PC /Tcnumpy\core\src\multiarraymodule.c > /Fobuild\temp > .win32-2.5\Release\numpy\core\src\multiarraymodule.obj > multiarraymodule.c > d:\Travail\numpy\numpy\core\src\arrayobject.c(662) : warning C4244: '=' : > conver > sion de 'npy_longlong' en 'long', perte possible de donn?es > numpy\core\src\multiarraymodule.c(7592) : error C2065: 'NPY_ALLOW_THREADS' : > ide > ntificateur non d?clar? > error: Command "D:\Program Files\Microsoft Visual Studio .NET > 2003\Vc7\bin\cl.ex > e /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ibuild\src.win32- 2.5\numpy\core\src > -Inum > py\core\include -Ibuild\src.win32-2.5\numpy\core -Inumpy\core\src > -Inumpy\core\i > nclude -ID:\Python25\include -ID:\Python25\PC > /Tcnumpy\core\src\multiarraymodule > .c /Fobuild\temp.win32-2.5\Release\numpy\core\src\multiarraymodule.obj " > failed w > ith exit status 2 > > Is there something I forgot to add in the site.cfg ? > No, it looks like NPY_ALLOW_THREADS is not defined: this pre processor symbol is defined in config.h, which is auto generated by the setup.py from numpy.core. Can you paste its content (should be somewhere in your build directory). David From matthieu.brucher at gmail.com Sun Nov 11 03:16:51 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sun, 11 Nov 2007 09:16:51 +0100 Subject: [Numpy-discussion] Compilation with Visual Studio 2003 In-Reply-To: <5b8d13220711101830g66d3958et51f72a802ab65e71@mail.gmail.com> References: <5b8d13220711101830g66d3958et51f72a802ab65e71@mail.gmail.com> Message-ID: > No, it looks like NPY_ALLOW_THREADS is not defined: this pre processor > symbol is defined in config.h, which is auto generated by the setup.py > from numpy.core. Can you paste its content (should be somewhere in > your build directory). > > David Here it is : /* #define SIZEOF_SHORT 2 */ /* #define SIZEOF_INT 4 */ /* #define SIZEOF_LONG 4 */ /* #define SIZEOF_FLOAT 4 */ /* #define SIZEOF_DOUBLE 8 */ #define SIZEOF_LONG_DOUBLE 8 #define SIZEOF_PY_INTPTR_T 4 /* #define SIZEOF_LONG_LONG 8 */ #define SIZEOF_PY_LONG_LONG 8 /* #define CHAR_BIT 8 */ Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Sun Nov 11 16:24:45 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sun, 11 Nov 2007 22:24:45 +0100 Subject: [Numpy-discussion] Compilation with Visual Studio 2003 In-Reply-To: References: <5b8d13220711101830g66d3958et51f72a802ab65e71@mail.gmail.com> Message-ID: Strange, I tested again and it worked this time... Matthieu 2007/11/11, Matthieu Brucher : > > > No, it looks like NPY_ALLOW_THREADS is not defined: this pre processor > > symbol is defined in config.h, which is auto generated by the setup.py > > from numpy.core. Can you paste its content (should be somewhere in > > your build directory). > > > > David > > > Here it is : > > /* #define SIZEOF_SHORT 2 */ > /* #define SIZEOF_INT 4 */ > /* #define SIZEOF_LONG 4 */ > /* #define SIZEOF_FLOAT 4 */ > /* #define SIZEOF_DOUBLE 8 */ > #define SIZEOF_LONG_DOUBLE 8 > #define SIZEOF_PY_INTPTR_T 4 > /* #define SIZEOF_LONG_LONG 8 */ > #define SIZEOF_PY_LONG_LONG 8 > /* #define CHAR_BIT 8 */ > > Matthieu > -- > French PhD student > Website : http://miles.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Sun Nov 11 21:27:33 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 12 Nov 2007 11:27:33 +0900 Subject: [Numpy-discussion] Compilation with Visual Studio 2003 In-Reply-To: References: <5b8d13220711101830g66d3958et51f72a802ab65e71@mail.gmail.com> Message-ID: <4737BA15.5070502@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > Strange, I tested again and it worked this time... > I sometimes noticed this kind of behaviour too, when I cancelled the build at some point. Maybe the build does not update the config.h if you cancelled just at the middle of its generation, or something like that. David From D.Hendriks at tue.nl Mon Nov 12 03:51:55 2007 From: D.Hendriks at tue.nl (D.Hendriks (Dennis)) Date: Mon, 12 Nov 2007 09:51:55 +0100 Subject: [Numpy-discussion] weibull distribution has only one parameter? Message-ID: <4738142B.9080508@tue.nl> According to (for instance) http://en.wikipedia.org/wiki/Weibull_distribution the Weibull distribution has two parameters: lambda > 0 is the scale parameter (real) and k > 0 is the shape parameter (real). However, the numpy.random.weibull function has only a single 'a' parameter (except for the size parameter which indicates the size of the array to fill with values - this is NOT a parameter of the distribution itself). My question is how this 'a' parameter translates to the Weibull distribution as it 'normally' is and how to sample the distribution when I have the lambda and k parameters? From robert.kern at gmail.com Mon Nov 12 04:01:17 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Nov 2007 03:01:17 -0600 Subject: [Numpy-discussion] weibull distribution has only one parameter? In-Reply-To: <4738142B.9080508@tue.nl> References: <4738142B.9080508@tue.nl> Message-ID: <4738165D.5030701@gmail.com> D.Hendriks (Dennis) wrote: > According to (for instance) > http://en.wikipedia.org/wiki/Weibull_distribution the Weibull > distribution has two parameters: lambda > 0 is the scale parameter > (real) and k > 0 is the shape parameter (real). However, the > numpy.random.weibull function has only a single 'a' parameter (except > for the size parameter which indicates the size of the array to fill > with values - this is NOT a parameter of the distribution itself). My > question is how this 'a' parameter translates to the Weibull > distribution as it 'normally' is and how to sample the distribution when > I have the lambda and k parameters? lambda * numpy.random.weibull(k) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From D.Hendriks at tue.nl Mon Nov 12 04:15:07 2007 From: D.Hendriks at tue.nl (D.Hendriks (Dennis)) Date: Mon, 12 Nov 2007 10:15:07 +0100 Subject: [Numpy-discussion] weibull distribution has only one parameter? In-Reply-To: <4738165D.5030701@gmail.com> References: <4738142B.9080508@tue.nl> <4738165D.5030701@gmail.com> Message-ID: <4738199B.3000300@tue.nl> Robert Kern wrote: >D.Hendriks (Dennis) wrote: > > >>According to (for instance) >>http://en.wikipedia.org/wiki/Weibull_distribution the Weibull >>distribution has two parameters: lambda > 0 is the scale parameter >>(real) and k > 0 is the shape parameter (real). However, the >>numpy.random.weibull function has only a single 'a' parameter (except >>for the size parameter which indicates the size of the array to fill >>with values - this is NOT a parameter of the distribution itself). My >>question is how this 'a' parameter translates to the Weibull >>distribution as it 'normally' is and how to sample the distribution when >>I have the lambda and k parameters? >> >> > >lambda * numpy.random.weibull(k) > > > Thanks for the quick replay. However, when I look at the image of the probability density function at http://en.wikipedia.org/wiki/Weibull_distribution I see a red line and a green line, both with k=2. The red line is for lambda=0.5 and the green for lambda=1.0. The green line is not only half the height of the red one (while double the lambda factor!), but also has its mean a bit more to the right. Looking at the formulas on the same page, this makes sense. All of this makes me doubt the correctness of the formula you proposed... -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Nov 12 06:39:20 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 12 Nov 2007 20:39:20 +0900 Subject: [Numpy-discussion] Merging new (optional) build system in the trunk ? Message-ID: <47383B68.1000606@ar.media.kyoto-u.ac.jp> Hi, I would appreciate to get some comment on whether there is any chance to get my numpy.scons branch merge into the trunk at some near future. I feel to have reached the point where the only big thing missing is more testing. I tried to test it on many platforms, but there is a limit to what I can test just by myself. The branch has been conceived such as by default, the current numpy.distutils is used to build, and the scons-based build is used only by explicit request (another setup.py), so this does not force any use now: except the work related to numpyconfig (which can be tested separately, since it was done in a different branch, and is only a few lines of code), everything else is exactly the same than before. Thanks, David P.S: Just as a reminder, here are some of the already implemented goals currently: - all the facilities of scons are available to build extensions (parallel builds, automatic dependency, easy extension through of new builders and configuration checks, etc...) - ability to build shared libraries and Ctypes-based extension in a cross platform way. - optimization flags logic cleanly separated from the other flags: you can now easily replace, add, remove optimization flags without having to handle platform specific flags such as -fPIC, etc... - facility replacing system_info which is easier to use (one function call to support most common cases) - performance library checks are stronger, for more reliable build results. MKL, Sunperf, ATLAS, vecLib and Accelerate are already supported; new libraries can generally be added in a few lines, as they all use the same infrastructure. - the current (successfully) tested platforms are : Mac OS X Intel (gfortran + gcc), Linux (g77 + gcc, gfortran + gcc, intel compilers), MS Windows (mingw, VS 2003 + g77, VS 2005 + g77), Solaris (with sun compilers). - Generally, I tried to follow the autoconf philosophy (do not check for version, check for facilities). The fortran runtime, for example, is found automatically, as is the fortran name mangling. - easier maintenance: no more distutils monkey patching because upstream distutils does not have a clean API, the current codebase is a bit below 3000 lines of code, not counting scons (as a comparison, system_info.py is almost 2000 lines by itself), most of the code is totally cross platform, and platform specific are encapsulated; there are a few hacks, but they are implementation-only hacks. So concretely, I believe that most of the things stated in http://projects.scipy.org/scipy/numpy/wiki/DistutilsRevamp are available *now*. More concrete info here: http://projects.scipy.org/scipy/numpy/wiki/NumpyScons From aisaac at american.edu Mon Nov 12 10:09:53 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 12 Nov 2007 10:09:53 -0500 Subject: [Numpy-discussion] weibull distribution has only one parameter? In-Reply-To: <4738199B.3000300@tue.nl> References: <4738142B.9080508@tue.nl> <4738165D.5030701@gmail.com><4738199B.3000300@tue.nl> Message-ID: On Mon, 12 Nov 2007, "D.Hendriks (Dennis)" apparently wrote: > All of this makes me doubt the correctness of the formula > you proposed. It is always a good idea to hesitate before doubting Robert. hth, Alan Isaac From D.Hendriks at tue.nl Mon Nov 12 11:44:36 2007 From: D.Hendriks at tue.nl (D.Hendriks (Dennis)) Date: Mon, 12 Nov 2007 17:44:36 +0100 Subject: [Numpy-discussion] weibull distribution has only one parameter? In-Reply-To: References: <4738142B.9080508@tue.nl> <4738165D.5030701@gmail.com><4738199B.3000300@tue.nl> Message-ID: <473882F4.5060305@tue.nl> Alan G Isaac wrote: >On Mon, 12 Nov 2007, "D.Hendriks (Dennis)" apparently wrote: > > >>All of this makes me doubt the correctness of the formula >>you proposed. >> >> >It is always a good idea to hesitate before doubting Robert. > > >hth, >Alan Isaac > > So, you are saying that it was indeed correct? That still leaves the question why I can't seem to confirm that in the figure I mentioned (red and green lines). Also, if you refer to X = lambda*(-ln(U))^(1/k) as 'proof' for the validity of the formula, I have to ask if Weibull(a,Size) does actually correspond to (-ln(U))^(1/a)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Mon Nov 12 11:51:36 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 12 Nov 2007 17:51:36 +0100 Subject: [Numpy-discussion] Numpy book and PyArray_CastToType In-Reply-To: References: Message-ID: Nobody can answer this ? Matthieu 2007/11/4, Matthieu Brucher : > > Hi, > > I'm trying to use PyArray_CastToType(), but according to the book, there > are two arguments, a PyArrayObject* and a PyArrayDescr*, but according to > the header file, there are three arguments, an additional int. Does someone > know its use ? > > Matthieu > -- > French PhD student > Website : http://miles.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Mon Nov 12 11:54:59 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 12 Nov 2007 17:54:59 +0100 Subject: [Numpy-discussion] Getting an array interface and using it Message-ID: Hi, I have an object that exposes an array interface. I want to modify the data it contains, but using numpy.array(myObject) seems to copy the data and thus my object is not modified. Am I mistaken or did I make a mistake in my array interface ? Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Mon Nov 12 11:55:21 2007 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 12 Nov 2007 18:55:21 +0200 Subject: [Numpy-discussion] Numpy book and PyArray_CastToType In-Reply-To: References: Message-ID: <20071112165521.GA3962@dogbert.sdsl.sun.ac.za> Hello, Have you considered looking at the source for PyArray_CastToType in core/src/arrayobject.c? Cheers, Albert On Mon, 12 Nov 2007, Matthieu Brucher wrote: > Nobody can answer this ? > > Matthieu > > 2007/11/4, Matthieu Brucher : > > > > Hi, > > > > I'm trying to use PyArray_CastToType(), but according to the book, there > > are two arguments, a PyArrayObject* and a PyArrayDescr*, but according to > > the header file, there are three arguments, an additional int. Does someone > > know its use ? > > > > Matthieu From rmay at ou.edu Mon Nov 12 12:21:31 2007 From: rmay at ou.edu (Ryan May) Date: Mon, 12 Nov 2007 11:21:31 -0600 Subject: [Numpy-discussion] weibull distribution has only one parameter? In-Reply-To: <473882F4.5060305@tue.nl> References: <4738142B.9080508@tue.nl> <4738165D.5030701@gmail.com> <4738199B.3000300@tue.nl> <473882F4.5060305@tue.nl> Message-ID: <47388B9B.6090703@ou.edu> D.Hendriks (Dennis) wrote: > Alan G Isaac wrote: >> On Mon, 12 Nov 2007, "D.Hendriks (Dennis)" apparently wrote: >> >>> All of this makes me doubt the correctness of the formula >>> you proposed. >>> >> It is always a good idea to hesitate before doubting Robert. >> >> >> hth, >> Alan Isaac >> > So, you are saying that it was indeed correct? That still leaves the > question why I can't seem to confirm that in the figure I mentioned (red > and green lines). Also, if you refer to X = lambda*(-ln(U))^(1/k) as > 'proof' for the validity of the formula, I have to ask if > Weibull(a,Size) does actually correspond to (-ln(U))^(1/a)? > Have you actually looked at a histogram of the random variates generated this way to see if they are wrong? Multiplying the the individual random values by a number changes the distribution differently than multiplying the distribution/density function by a number. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From robert.kern at gmail.com Mon Nov 12 12:58:57 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Nov 2007 11:58:57 -0600 Subject: [Numpy-discussion] weibull distribution has only one parameter? In-Reply-To: <473882F4.5060305@tue.nl> References: <4738142B.9080508@tue.nl> <4738165D.5030701@gmail.com><4738199B.3000300@tue.nl> <473882F4.5060305@tue.nl> Message-ID: <47389461.7080100@gmail.com> D.Hendriks (Dennis) wrote: > Alan G Isaac wrote: >> On Mon, 12 Nov 2007, "D.Hendriks (Dennis)" apparently wrote: >> >>> All of this makes me doubt the correctness of the formula >>> you proposed. >>> >> It is always a good idea to hesitate before doubting Robert. >> >> >> hth, >> Alan Isaac >> > So, you are saying that it was indeed correct? That still leaves the > question why I can't seem to confirm that in the figure I mentioned (red > and green lines). Also, if you refer to X = lambda*(-ln(U))^(1/k) as > 'proof' for the validity of the formula, I have to ask if > Weibull(a,Size) does actually correspond to (-ln(U))^(1/a)? double rk_standard_exponential(rk_state *state) { /* We use -log(1-U) since U is [0, 1) */ return -log(1.0 - rk_double(state)); } double rk_weibull(rk_state *state, double a) { return pow(rk_standard_exponential(state), 1./a); } Like Ryan says, multiplying a random deviate by a number is different from multiplying the PDF by a number. Multiplying the random deviate by lambda is equivalent to transforming pdf(x) to pdf(x/lambda) not lambda*pdf(x). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From p.e.creasey.00 at googlemail.com Mon Nov 12 13:10:31 2007 From: p.e.creasey.00 at googlemail.com (Peter Creasey) Date: Mon, 12 Nov 2007 18:10:31 +0000 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? Message-ID: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> Hi all, The following code calling numpy v1.0.4 fails to terminate on my machine, which was not the case with v1.0.3.1 from numpy import arange, float64 from numpy.linalg import eig a = arange(13*13, dtype = float64) a.shape = (13,13) a = a%17 eig(a) Regards, Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Mon Nov 12 13:37:36 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 12 Nov 2007 10:37:36 -0800 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> Message-ID: On Nov 12, 2007 10:10 AM, Peter Creasey wrote: > The following code calling numpy v1.0.4 fails to terminate on my machine, > which was not the case with v1.0.3.1 > > from numpy import arange, float64 > from numpy.linalg import eig > a = arange(13*13, dtype = float64) > a.shape = (13,13) > a = a%17 > eig(a) It sounds like the same problem that was reported in this thread: http://thread.gmane.org/gmane.comp.python.numeric.general/17456/focus=17465 A friend of mine says the windows binary of 1.0.4 also hangs on eigh and lstsq (so linalg in general). I don't have that problem on 1.0.5 compiled on GNU/Linux. From lou_boog2000 at yahoo.com Mon Nov 12 13:37:50 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Mon, 12 Nov 2007 10:37:50 -0800 (PST) Subject: [Numpy-discussion] Problem with numpy.linalg.eig? Message-ID: <34500.26617.qm@web34406.mail.mud.yahoo.com> Works fine on my computer (Mac OS X 10.4), Python 2.4. Runs in a second or so. -- Lou Pecora ---Peter wrote: Hi all, The following code calling numpy v1.0.4 fails to terminate on my machine, which was not the case with v1.0.3.1 from numpy import arange, float64 from numpy.linalg import eig a = arange(13*13, dtype = float64) a.shape = (13,13) a = a%17 eig(a) Regards, Peter __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From matthieu.brucher at gmail.com Mon Nov 12 13:42:30 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 12 Nov 2007 19:42:30 +0100 Subject: [Numpy-discussion] Numpy book and PyArray_CastToType In-Reply-To: <20071112165521.GA3962@dogbert.sdsl.sun.ac.za> References: <20071112165521.GA3962@dogbert.sdsl.sun.ac.za> Message-ID: 2007/11/12, Albert Strasheim : > > Hello, > > Have you considered looking at the source for PyArray_CastToType in > core/src/arrayobject.c? > > Cheers, > > Albert Well no :( I didn't know where to look (and this should be fixed in the book as well...) Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Nov 12 13:51:09 2007 From: cournape at gmail.com (David Cournapeau) Date: Tue, 13 Nov 2007 03:51:09 +0900 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> Message-ID: <5b8d13220711121051y76e56705m73163281f2d2897f@mail.gmail.com> On Nov 13, 2007 3:37 AM, Keith Goodman wrote: > > On Nov 12, 2007 10:10 AM, Peter Creasey wrote: > > The following code calling numpy v1.0.4 fails to terminate on my machine, > > which was not the case with v1.0.3.1 > > > > from numpy import arange, float64 > > from numpy.linalg import eig > > a = arange(13*13, dtype = float64) > > a.shape = (13,13) > > a = a%17 > > eig(a) > > It sounds like the same problem that was reported in this thread: > > http://thread.gmane.org/gmane.comp.python.numeric.general/17456/focus=17465 > > A friend of mine says the windows binary of 1.0.4 also hangs on eigh > and lstsq (so linalg in general). I don't have that problem on 1.0.5 > compiled on GNU/Linux. Could you friend try the binaries there: http://projects.scipy.org/pipermail/numpy-discussion/2007-November/029811.html This may be a problem related to the blas/lapack used for the binaries. The binaries posted above use non optimized BLAS/LAPACK (I can update to 1.0.4 if this is a problem). cheers, David From Chris.Barker at noaa.gov Mon Nov 12 14:04:32 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 12 Nov 2007 11:04:32 -0800 Subject: [Numpy-discussion] Getting an array interface and using it In-Reply-To: References: Message-ID: <4738A3C0.1070802@noaa.gov> Matthieu Brucher wrote: > I have an object that exposes an array interface. I want to modify the > data it contains, but using numpy.array(myObject) seems to copy the data > and thus my object is not modified. Am I mistaken or did I make a > mistake in my array interface ? I think numpy.array(object) always makes a copy. You want numpy.asarray(object) which will make a view if object exposes the array interface and matches the type and sizes requested. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From matthieu.brucher at gmail.com Mon Nov 12 14:10:15 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 12 Nov 2007 20:10:15 +0100 Subject: [Numpy-discussion] Getting an array interface and using it In-Reply-To: <4738A3C0.1070802@noaa.gov> References: <4738A3C0.1070802@noaa.gov> Message-ID: 2007/11/12, Christopher Barker : > > Matthieu Brucher wrote: > > I have an object that exposes an array interface. I want to modify the > > data it contains, but using numpy.array(myObject) seems to copy the data > > and thus my object is not modified. Am I mistaken or did I make a > > mistake in my array interface ? > > I think numpy.array(object) always makes a copy. > > You want numpy.asarray(object) which will make a view if object exposes > the array interface and matches the type and sizes requested. Thank you for the tip, I'll try it this evening. I checked that the data is actually copied, so this should be what I need ;) Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Mon Nov 12 14:28:18 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 12 Nov 2007 14:28:18 -0500 Subject: [Numpy-discussion] Getting an array interface and using it In-Reply-To: References: <4738A3C0.1070802@noaa.gov> Message-ID: <200711121428.18140.pgmdevlist@gmail.com> > > I think numpy.array(object) always makes a copy. > > > > You want numpy.asarray(object) which will make a view if object exposes > > the array interface and matches the type and sizes requested. FYI, numpy.asarray is a shortcut for numpy.array(copy=False), numpy.asanyarray for numpy.array(copy=False, subok=True)... In other terms, you can stick to array, as long as you provide the proper keywords. From kwgoodman at gmail.com Mon Nov 12 14:48:57 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 12 Nov 2007 11:48:57 -0800 Subject: [Numpy-discussion] numpy-1.0.5.dev4445.egg-info Message-ID: I noticed that python setup.py install --prefix=/usr/local installs a new file called numpy-1.0.5.dev4445.egg-info. It used to be that files were only installed in /usr/local/lib/python2.4/site-packages/numpy. What is the file used for? $ cat /usr/local/lib/python2.4/site-packages/numpy-1.0.5.dev4445.egg-info Metadata-Version: 1.0 Name: numpy Version: 1.0.5.dev4445 Summary: NumPy: array processing for numbers, strings, records, and objects. Home-page: http://numeric.scipy.org Author: NumPy Developers Author-email: numpy-discussion at lists.sourceforge.net License: BSD Download-URL: http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Description: NumPy is a general-purpose array-processing package designed to efficiently manipulate large multi-dimensional arrays of arbitrary records without sacrificing too much speed for small multi-dimensional arrays. NumPy is built on the Numeric code base and adds features introduced by numarray as well as an extended C-API and the ability to create arrays of arbitrary type which also makes NumPy suitable for interfacing with general-purpose data-base applications. There are also basic facilities for discrete fourier transform, basic linear algebra and random number generation. Platform: Windows Platform: Linux Platform: Solaris Platform: Mac OS-X Platform: Unix Classifier: Development Status :: 4 - Beta Classifier: Intended Audience :: Science/Research Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved Classifier: Programming Language :: C Classifier: Programming Language :: Python Classifier: Topic :: Software Development Classifier: Topic :: Scientific/Engineering Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: POSIX Classifier: Operating System :: Unix Classifier: Operating System :: MacOS From matthieu.brucher at gmail.com Mon Nov 12 14:55:30 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 12 Nov 2007 20:55:30 +0100 Subject: [Numpy-discussion] numpy-1.0.5.dev4445.egg-info In-Reply-To: References: Message-ID: It allows the check on egg dependencies IIRC Matthieu 2007/11/12, Keith Goodman : > > I noticed that > > python setup.py install --prefix=/usr/local > > installs a new file called numpy-1.0.5.dev4445.egg-info. It used to be > that files were only installed in > /usr/local/lib/python2.4/site-packages/numpy. > > What is the file used for? > > $ cat /usr/local/lib/python2.4/site-packages/numpy-1.0.5.dev4445.egg-info > Metadata-Version: 1.0 > Name: numpy > Version: 1.0.5.dev4445 > Summary: NumPy: array processing for numbers, strings, records, and > objects. > Home-page: http://numeric.scipy.org > Author: NumPy Developers > Author-email: numpy-discussion at lists.sourceforge.net > License: BSD > Download-URL: > http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > Description: NumPy is a general-purpose array-processing package designed > to > efficiently manipulate large multi-dimensional arrays of arbitrary > records without sacrificing too much speed for small > multi-dimensional > arrays. NumPy is built on the Numeric code base and adds features > introduced by numarray as well as an extended C-API and the > ability to > create arrays of arbitrary type which also makes NumPy suitable > for > interfacing with general-purpose data-base applications. > > There are also basic facilities for discrete fourier transform, > basic linear algebra and random number generation. > > Platform: Windows > Platform: Linux > Platform: Solaris > Platform: Mac OS-X > Platform: Unix > Classifier: Development Status :: 4 - Beta > Classifier: Intended Audience :: Science/Research > Classifier: Intended Audience :: Developers > Classifier: License :: OSI Approved > Classifier: Programming Language :: C > Classifier: Programming Language :: Python > Classifier: Topic :: Software Development > Classifier: Topic :: Scientific/Engineering > Classifier: Operating System :: Microsoft :: Windows > Classifier: Operating System :: POSIX > Classifier: Operating System :: Unix > Classifier: Operating System :: MacOS > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From zyzhu2000 at gmail.com Mon Nov 12 16:02:38 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Mon, 12 Nov 2007 15:02:38 -0600 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> Message-ID: On Nov 12, 2007 12:37 PM, Keith Goodman wrote: > > On Nov 12, 2007 10:10 AM, Peter Creasey wrote: > > The following code calling numpy v1.0.4 fails to terminate on my machine, > > which was not the case with v1.0.3.1 > > > > from numpy import arange, float64 > > from numpy.linalg import eig > > a = arange(13*13, dtype = float64) > > a.shape = (13,13) > > a = a%17 > > eig(a) > > It sounds like the same problem that was reported in this thread: > > http://thread.gmane.org/gmane.comp.python.numeric.general/17456/focus=17465 > > A friend of mine says the windows binary of 1.0.4 also hangs on eigh > and lstsq (so linalg in general). I don't have that problem on 1.0.5 > compiled on GNU/Linux. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > The code hangs on my machine too. In the thread you mentioned above, I wrote that using the EGG instead of MSI appears to fix the numpy.test() problem, but maybe it just somehow hides it. From stefan at sun.ac.za Mon Nov 12 16:08:40 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 12 Nov 2007 23:08:40 +0200 Subject: [Numpy-discussion] how to convert btw rgb and pixel value In-Reply-To: <1194260325.30441.2.camel@nadav.envision.co.il> References: <1194248247.421519.106550@y27g2000pre.googlegroups.com> <1194260325.30441.2.camel@nadav.envision.co.il> Message-ID: <20071112210840.GK7580@mentat.za.net> Since PIL Images now have array interfaces, it has become a lot simpler. The following should do the job: from numpy import array from PIL import Image def imread(fname,flatten=False): """Return a copy of a PIL image as a numpy array. *Parameters*: im : PIL image Input image. flatten : bool If true, convert the output to grey-scale. *Returns*: img_array : ndarray The different colour bands/channels are stored in the third dimension, such that a grey-image is MxN, an RGB-image MxNx3 and an RGBA-image MxNx4. """ im = Image.open(fname) if flatten: im = im.convert('F') return array(im) Cheers St?fan On Mon, Nov 05, 2007 at 12:58:45PM +0200, Nadav Horesh wrote: > If the image is in the form of a standard image format (png, bmp, jpeg...) you > can use the PIL library. png file can be read by pylab's imread function. > > Nadav. > > On Sun, 2007-11-04 at 23:37 -0800, devnew at gmail.com wrote: > > hi > i wish to convert an rgb image into an array of double values..is > there a method for that in numpy? > also i want to create an array of doubles into corresponding rgb > tuples of an image > can anyone guide me? > dn From david at ar.media.kyoto-u.ac.jp Mon Nov 12 23:26:56 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 13 Nov 2007 13:26:56 +0900 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> Message-ID: <47392790.3040703@ar.media.kyoto-u.ac.jp> Geoffrey Zhu wrote: > On Nov 12, 2007 12:37 PM, Keith Goodman wrote: >> On Nov 12, 2007 10:10 AM, Peter Creasey wrote: >>> The following code calling numpy v1.0.4 fails to terminate on my machine, >>> which was not the case with v1.0.3.1 >>> >>> from numpy import arange, float64 >>> from numpy.linalg import eig >>> a = arange(13*13, dtype = float64) >>> a.shape = (13,13) >>> a = a%17 >>> eig(a) >> It sounds like the same problem that was reported in this thread: >> >> http://thread.gmane.org/gmane.comp.python.numeric.general/17456/focus=17465 > > The code hangs on my machine too. In the thread you mentioned above, I > wrote that using the EGG instead of MSI appears to fix the > numpy.test() problem, but maybe it just somehow hides it. When you use the MSI, can you always reproduce the problem ? As I said previously, it is hard to know for sure without being able to reproduce the bug on our own workstation, but if this is a problem between fortran and C argument passing, then the result can be pretty random since the problem reduced to a pointer pointing at a wrong address (crash, wrong value, etc...). cheers, David From listservs at mac.com Tue Nov 13 00:09:13 2007 From: listservs at mac.com (Chris) Date: Tue, 13 Nov 2007 05:09:13 +0000 (UTC) Subject: [Numpy-discussion] f2py cannot find gfortran compiler on OSX 10.5 Message-ID: I've just upgraded my OSX system to Leopard, and have successfully build numpy from scratch. I am trying to build some code, which includes f2py extensions, that build successfully on 10.4. I have the gfortran compiler installed, and am building my code using: python setup.py config_fc --fcompiler gnu95 build However, when I run this, I get the following error when the Fortran code compiles: AttributeError: 'NoneType' object has no attribute 'compile' Is there any reason that gnu95 no longer would find gfortran? I have confirmed that the compiler is in my path. Thanks. From oliphant at enthought.com Tue Nov 13 00:30:52 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 12 Nov 2007 23:30:52 -0600 Subject: [Numpy-discussion] Merging new (optional) build system in the trunk ? In-Reply-To: <47383B68.1000606@ar.media.kyoto-u.ac.jp> References: <47383B68.1000606@ar.media.kyoto-u.ac.jp> Message-ID: <4739368C.3070908@enthought.com> David Cournapeau wrote: > Hi, > > I would appreciate to get some comment on whether there is any > chance to get my numpy.scons branch merge into the trunk at some near > future. I feel to have reached the point where the only big thing > missing is more testing. I tried to test it on many platforms, but there > is a limit to what I can test just by myself. The branch has been > conceived such as by default, the current numpy.distutils is used to > build, and the scons-based build is used only by explicit request > (another setup.py), so this does not force any use now: except the work > related to numpyconfig (which can be tested separately, since it was > done in a different branch, and is only a few lines of code), everything > else is exactly the same than before. > I think there is a chance. I'm generally favorable to the idea. I was mainly waiting for the 1.0.4 release. I think we should be able to merge it over now if there are no serious objections. I'm also interested in moving scipy.weave into numpy (without the blitz converters which will stay in scipy). For 1.0.5, I would also like to see aligned memory and some steps towards optimization using intrinsics. -Travis O. From david at ar.media.kyoto-u.ac.jp Tue Nov 13 00:34:35 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 13 Nov 2007 14:34:35 +0900 Subject: [Numpy-discussion] Merging new (optional) build system in the trunk ? In-Reply-To: <4739368C.3070908@enthought.com> References: <47383B68.1000606@ar.media.kyoto-u.ac.jp> <4739368C.3070908@enthought.com> Message-ID: <4739376B.9070008@ar.media.kyoto-u.ac.jp> Travis E. Oliphant wrote: > David Cournapeau wrote: >> Hi, >> >> I would appreciate to get some comment on whether there is any >> chance to get my numpy.scons branch merge into the trunk at some near >> future. I feel to have reached the point where the only big thing >> missing is more testing. I tried to test it on many platforms, but there >> is a limit to what I can test just by myself. The branch has been >> conceived such as by default, the current numpy.distutils is used to >> build, and the scons-based build is used only by explicit request >> (another setup.py), so this does not force any use now: except the work >> related to numpyconfig (which can be tested separately, since it was >> done in a different branch, and is only a few lines of code), everything >> else is exactly the same than before. >> > > I think there is a chance. I'm generally favorable to the idea. I was > mainly waiting for the 1.0.4 release. I think we should be able to > merge it over now if there are no serious objections. Ok, great. Just tell me when you want to merge it, because I did a few things to make testing on the buildbot easier: - I inverted the old and new setup.py (to be able to test my branch on the buildbot). - There is also some debugging on by default, which should be disabled in the trunk. This ia a 2 second change, but just to avoid any surprises for people regularly using the trunk Also, what is the timeline for 1.0.5 ? Ideally, once the scons work is merged into the trunk, I would like to work on a scipy branch which uses scons, so that both numpy and scipy next (1.0.5 and 0.7 ?) releases can both be built using scons. > > I'm also interested in moving scipy.weave into numpy (without the blitz > converters which will stay in scipy). > > For 1.0.5, I would also like to see aligned memory and some steps > towards optimization using intrinsics. Are you speaking about SIMD intrisics ? cheers, David From millman at berkeley.edu Tue Nov 13 01:09:20 2007 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 12 Nov 2007 22:09:20 -0800 Subject: [Numpy-discussion] I removed some duplicate code from weave Message-ID: Hey, I just noticed that dumb_shelve.py and dumbdbm_patched.py were in both scipy.io and scipy.weave. Since I assume that this was just an oversight, I went ahead and cleaned it up. First, I made sure that the newest code was in scipy.io: http://projects.scipy.org/scipy/scipy/changeset/3521 http://projects.scipy.org/scipy/scipy/changeset/3522 And then I removed the version in scipy.weave: http://projects.scipy.org/scipy/scipy/changeset/3524 Please let me know if this change causes any major problems. Cheers, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From zyzhu2000 at gmail.com Tue Nov 13 02:49:00 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Tue, 13 Nov 2007 01:49:00 -0600 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: <47392790.3040703@ar.media.kyoto-u.ac.jp> References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> Message-ID: On Nov 12, 2007 10:26 PM, David Cournapeau wrote: > Geoffrey Zhu wrote: > > On Nov 12, 2007 12:37 PM, Keith Goodman wrote: > >> On Nov 12, 2007 10:10 AM, Peter Creasey wrote: > >>> The following code calling numpy v1.0.4 fails to terminate on my machine, > >>> which was not the case with v1.0.3.1 > >>> > >>> from numpy import arange, float64 > >>> from numpy.linalg import eig > >>> a = arange(13*13, dtype = float64) > >>> a.shape = (13,13) > >>> a = a%17 > >>> eig(a) > >> It sounds like the same problem that was reported in this thread: > >> > >> http://thread.gmane.org/gmane.comp.python.numeric.general/17456/focus=17465 > > > > The code hangs on my machine too. In the thread you mentioned above, I > > wrote that using the EGG instead of MSI appears to fix the > > numpy.test() problem, but maybe it just somehow hides it. > When you use the MSI, can you always reproduce the problem ? As I said > previously, it is hard to know for sure without being able to reproduce > the bug on our own workstation, but if this is a problem between fortran > and C argument passing, then the result can be pretty random since the > problem reduced to a pointer pointing at a wrong address (crash, wrong > value, etc...). > > cheers, > > David > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Yes, with the MSI I can always reproduce the problem with numpy.test(). It always hangs.With the egg it does not hang. Pointer problems are usually random, but not random if we are using the same binaries in EGG and MSI and variables are always initialized to certain value. From david at ar.media.kyoto-u.ac.jp Tue Nov 13 03:37:11 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 13 Nov 2007 17:37:11 +0900 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> Message-ID: <47396237.60209@ar.media.kyoto-u.ac.jp> Geoffrey Zhu wrote: > > Yes, with the MSI I can always reproduce the problem with > numpy.test(). It always hangs.With the egg it does not hang. Pointer > problems are usually random, but not random if we are using the same > binaries in EGG and MSI and variables are always initialized to > certain value. Ok, could you try this: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.0.4.win32-py2.5.msi I built it with mingwin, and with NETLIB BLAS/LAPACK. This is just for testing, do not use it for anything else. David From haase at msg.ucsf.edu Tue Nov 13 07:11:33 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 13 Nov 2007 13:11:33 +0100 Subject: [Numpy-discussion] Using arr.dtype.type to check byteorder-independed dtype fails for bool Message-ID: Hi, I need to check the array dtype in a way that it is ignoring differences coming only from big-endian vs. little-endian. So far I was using a test like this successfully: if arr.dtype.type == N.uint8: .... This would test positively for both byte-orders independent of the system's byteorder. I just noticed however, that if arr.dtype.type == N.bool: ... always fails (evaluates to False) Testing arr.dtype == N.bool, on the other hand works (Sitting at my Windows-PC cannot test the byte-order problem though !) While writing this email, I just found that if arr.dtype.type == N.bool_: ... works. (Note the trailing underscore !). What is this ? Thanks for help,hints and comments, Sebastian Haase From p.e.creasey.00 at googlemail.com Tue Nov 13 07:22:23 2007 From: p.e.creasey.00 at googlemail.com (Peter Creasey) Date: Tue, 13 Nov 2007 12:22:23 +0000 (UTC) Subject: [Numpy-discussion] Problem with numpy.linalg.eig? References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> Message-ID: > > Yes, with the MSI I can always reproduce the problem with > numpy.test(). It always hangs.With the egg it does not hang. Pointer > problems are usually random, but not random if we are using the same > binaries in EGG and MSI and variables are always initialized to > certain value. > I can consistently reproduce this problem with the EGG. It disappears, however, when I replace the lapack_lite.pyd with the version from the 1.0.3.1 EGG (python 2.4). cheers, Peter From haase at msg.ucsf.edu Tue Nov 13 07:39:29 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 13 Nov 2007 13:39:29 +0100 Subject: [Numpy-discussion] Merging new (optional) build system in the trunk ? In-Reply-To: <4739368C.3070908@enthought.com> References: <47383B68.1000606@ar.media.kyoto-u.ac.jp> <4739368C.3070908@enthought.com> Message-ID: On Nov 13, 2007 6:30 AM, Travis E. Oliphant wrote: > David Cournapeau wrote: > I'm also interested in moving scipy.weave into numpy (without the blitz > converters which will stay in scipy). > Hi, I'm excited to hear that weave would get "elevated" to a more exposed place ! However, I was wondering, if this fits with the idea to keep numpy minimalistic !? Does this mean that weave is "simpler" that I always thought ? [I always thought it was short of magic ;-) ] Regards, Sebastian Haase From stefan at sun.ac.za Tue Nov 13 08:18:26 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 13 Nov 2007 15:18:26 +0200 Subject: [Numpy-discussion] Using arr.dtype.type to check byteorder-independed dtype fails for bool In-Reply-To: References: Message-ID: <20071113131826.GC7379@mentat.za.net> Hi Sebastian On Tue, Nov 13, 2007 at 01:11:33PM +0100, Sebastian Haase wrote: > Hi, > I need to check the array dtype in a way that it is ignoring > differences coming only from big-endian vs. little-endian. Does N.issubdtype(first_dtype, second_dtype) work? Cheers St?fan From vel.accel at gmail.com Tue Nov 13 08:29:40 2007 From: vel.accel at gmail.com (dieter h.) Date: Tue, 13 Nov 2007 08:29:40 -0500 Subject: [Numpy-discussion] Proposing Ubiquity Message-ID: <1e52e0880711130529k267c0b70mbb6147a598a2d8fc@mail.gmail.com> Hi all, Would it make sense for all functionality in Numpy/Scipy to have ubiquitous returns? In that I'm proposing that every func/method (where appropriate) have a flag in its arg list establishing a return for either [INDEX, BOOLEAN, VALUE=default]. this could be by a singular Enum flag, I believe. Obvioulsy, this proposal gives the option for a return of Index/Bool to be used for Indexing arrays, or the actual values. Since I mosty index existing arrays, this would be a huge plus for me. Thanks -Dieter From haase at msg.ucsf.edu Tue Nov 13 08:53:32 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 13 Nov 2007 14:53:32 +0100 Subject: [Numpy-discussion] Bug in arange dtype ">f" was: Using arr.dtype.type to check byteorder-independed dtype fails for bool Message-ID: On Nov 13, 2007 2:18 PM, Stefan van der Walt wrote: > Hi Sebastian > > On Tue, Nov 13, 2007 at 01:11:33PM +0100, Sebastian Haase wrote: > > Hi, > > I need to check the array dtype in a way that it is ignoring > > differences coming only from big-endian vs. little-endian. > > Does > > N.issubdtype(first_dtype, second_dtype) > > work? Hi St?fan, trying to anwer your question with a quick "arange" test, I ran into more confusion: >> a = N.arange(.5, dtype=">f") >>> `a.dtype` 'dtype('float32')' >>> a = N.arange(.5, dtype=">> `a.dtype` 'dtype('float32')' Both equal positively with N.float32 now ! >>> N.__version__ '1.0.1' This is (was?) apparently a bug in arange It works as expected with empty(): >>> N.empty(5, dtype=">f").dtype==N.float32 False >>> N.empty(5, dtype=" References: <20071113131826.GC7379@mentat.za.net> Message-ID: On Nov 13, 2007 2:18 PM, Stefan van der Walt wrote: > Hi Sebastian > > On Tue, Nov 13, 2007 at 01:11:33PM +0100, Sebastian Haase wrote: > > Hi, > > I need to check the array dtype in a way that it is ignoring > > differences coming only from big-endian vs. little-endian. > > Does > > N.issubdtype(first_dtype, second_dtype) > > work? Hi St?fan ! It appears to work : >>> N.empty(5, dtype=">f").dtype==N.float32 False >>> N.empty(5, dtype=">> a = N.empty(5, dtype=">f") >>> a.dtype >f4 >>> a.dtype.type >>> N.issubdtype(a.dtype, N.float32) True >>> a = N.empty(5, dtype=">?") >>> a.dtype bool >>> a = N.empty(5, dtype=">> a.dtype bool >>> a = N.empty(5, dtype=">?") >>> N.issubdtype(a.dtype, N.bool) True >>> N.issubdtype(a.dtype, N.bool_) True Furthermore however, >>> N.empty(5, dtype=">?").dtype == N.empty(5, dtype=" References: Message-ID: <6ce0ac130711130700t58f2fbe5l6d496b609f0195ef@mail.gmail.com> Please see the other active thread on this topic on the scipy-users list. This is a known issue. Brian On Nov 12, 2007 10:09 PM, Chris wrote: > I've just upgraded my OSX system to Leopard, and have successfully build > numpy from scratch. I am trying to build some code, which includes f2py > extensions, that build successfully on 10.4. I have the gfortran compiler > installed, and am building my code using: > > python setup.py config_fc --fcompiler gnu95 build > > However, when I run this, I get the following error when the Fortran code > compiles: > > AttributeError: 'NoneType' object has no attribute 'compile' > > Is there any reason that gnu95 no longer would find gfortran? I have > confirmed that the compiler is in my path. > > Thanks. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From tim.hochberg at ieee.org Tue Nov 13 11:17:13 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 13 Nov 2007 09:17:13 -0700 Subject: [Numpy-discussion] Using arr.dtype.type to check byteorder-independed dtype fails for bool In-Reply-To: References: <20071113131826.GC7379@mentat.za.net> Message-ID: On Nov 13, 2007 6:57 AM, Sebastian Haase wrote: > On Nov 13, 2007 2:18 PM, Stefan van der Walt wrote: > > Hi Sebastian > > > > On Tue, Nov 13, 2007 at 01:11:33PM +0100, Sebastian Haase wrote: > > > Hi, > > > I need to check the array dtype in a way that it is ignoring > > > differences coming only from big-endian vs. little-endian. > > > > Does > > > > N.issubdtype(first_dtype, second_dtype) > > > > work? > > Hi St?fan ! > It appears to work : > > >>> N.empty(5, dtype=">f").dtype==N.float32 > False > >>> N.empty(5, dtype=" True > >>> a = N.empty(5, dtype=">f") > >>> a.dtype > >f4 > >>> a.dtype.type > > >>> N.issubdtype(a.dtype, N.float32) > True > >>> a = N.empty(5, dtype=">?") > >>> a.dtype > bool > >>> a = N.empty(5, dtype=" >>> a.dtype > bool > >>> a = N.empty(5, dtype=">?") > >>> N.issubdtype(a.dtype, N.bool) > True > >>> N.issubdtype(a.dtype, N.bool_) > True > > Furthermore however, > >>> N.empty(5, dtype=">?").dtype == N.empty(5, dtype=" True > > So, for "symmetry reasons" with my existing non-bool code, I would > likely use "arr.dtype.type == N.bool_". > Still wondering what the "_" means here. (reading the book did not > enlighten....) Out of curiosity, does it work with arr.dtype.type == N.float? I would expect not, and I imagine that is the same reason that it doesn't work for N.bool. Which is that N.bool is __builtin__.bool or, in other words, N.boolis just Python's boolean type. I believe this is to protect those foolish enough to use "from x import *". Anyway, N.bool_ is the real, numpy boolean dtype that is analogous to unit8 and friends. > > > Thanks, > Sebastian > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Tue Nov 13 11:23:39 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 13 Nov 2007 09:23:39 -0700 Subject: [Numpy-discussion] Warnings as exceptions? In-Reply-To: References: Message-ID: On Nov 10, 2007 3:33 PM, Michael McNeil Forbes wrote: > Why are numpy warnings printed rather than issued using the standard > warnings library? I know that the behaviour can be controlled by > seterr(), but it seem rather unpythonic not to use the warnings library. > > Is there an explicit reason for this choice? (It seems like a pretty > trivial modification of util.py will catch most of the cases.) If > performance is an issue, one could add a 'print' mode which behaves > as the current 'warn'. I believe that performance is at least part of the reason. I suspect that routing things through the Python warnings module is somewhat expensive and there are some things that one usually want to ignore (underflow for example). It's not the cases where warnings are printed that are a performance issue; it's the cases where something dodgy happens but you are intentionally ignoring it that could cause performance issues. In principle, I think that you could get away with just using seterr to turn warnings on and off and use the warnings framework to do everything else. However, the warnings framework has always been harder for me to work with than seterr, so I'm not keen on that myself. What might well make sense is to route things through the warnings module only when seterr is set to "warn" for that error type. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Nov 13 11:46:04 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 13 Nov 2007 10:46:04 -0600 Subject: [Numpy-discussion] Warnings as exceptions? In-Reply-To: References: Message-ID: <4739D4CC.1060402@enthought.com> Michael McNeil Forbes wrote: > Why are numpy warnings printed rather than issued using the standard > warnings library? I know that the behaviour can be controlled by > seterr(), but it seem rather unpythonic not to use the warnings library. > > The "warn" option explicitly allows you to use the warnings library. There is already the "print" mode which is in fact the default. It is trivial to change your default to "warn" if you prefer. The standard default is "print" because it is less intrusive and does not require users to setup the warnings module which is not always done by many users. -Travis O. From kwgoodman at gmail.com Tue Nov 13 12:17:34 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 13 Nov 2007 09:17:34 -0800 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: <5b8d13220711121051y76e56705m73163281f2d2897f@mail.gmail.com> References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <5b8d13220711121051y76e56705m73163281f2d2897f@mail.gmail.com> Message-ID: On Nov 12, 2007 10:51 AM, David Cournapeau wrote: > > On Nov 13, 2007 3:37 AM, Keith Goodman wrote: > > > > On Nov 12, 2007 10:10 AM, Peter Creasey wrote: > > > The following code calling numpy v1.0.4 fails to terminate on my machine, > > > which was not the case with v1.0.3.1 > > > > > > from numpy import arange, float64 > > > from numpy.linalg import eig > > > a = arange(13*13, dtype = float64) > > > a.shape = (13,13) > > > a = a%17 > > > eig(a) > > > > It sounds like the same problem that was reported in this thread: > > > > http://thread.gmane.org/gmane.comp.python.numeric.general/17456/focus=17465 > > > > A friend of mine says the windows binary of 1.0.4 also hangs on eigh > > and lstsq (so linalg in general). I don't have that problem on 1.0.5 > > compiled on GNU/Linux. > Could you friend try the binaries there: > > http://projects.scipy.org/pipermail/numpy-discussion/2007-November/029811.html > > This may be a problem related to the blas/lapack used for the > binaries. The binaries posted above use non optimized BLAS/LAPACK (I > can update to 1.0.4 if this is a problem). He tried. But he is using python 2.4. (He said your binary was for 2.5). From robert.kern at gmail.com Tue Nov 13 12:52:02 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 13 Nov 2007 11:52:02 -0600 Subject: [Numpy-discussion] Proposing Ubiquity In-Reply-To: <1e52e0880711130529k267c0b70mbb6147a598a2d8fc@mail.gmail.com> References: <1e52e0880711130529k267c0b70mbb6147a598a2d8fc@mail.gmail.com> Message-ID: <4739E442.3000703@gmail.com> dieter h. wrote: > Hi all, > > Would it make sense for all functionality in Numpy/Scipy to have > ubiquitous returns? In that I'm proposing that every func/method > (where appropriate) have a flag in its arg list establishing a return > for either [INDEX, BOOLEAN, VALUE=default]. this could be by a > singular Enum flag, I believe. Obvioulsy, this proposal gives the > option for a return of Index/Bool to be used for Indexing arrays, or > the actual values. Since I mosty index existing arrays, this would be > a huge plus for me. I'm afraid that I don't understand this proposal. Can you show some examples of what you would like to see? Preferably, can you write some working code that demonstrates this feature? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From zyzhu2000 at gmail.com Tue Nov 13 12:43:27 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Tue, 13 Nov 2007 11:43:27 -0600 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: <47396237.60209@ar.media.kyoto-u.ac.jp> References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> <47396237.60209@ar.media.kyoto-u.ac.jp> Message-ID: On Nov 13, 2007 2:37 AM, David Cournapeau wrote: > Geoffrey Zhu wrote: > > > > Yes, with the MSI I can always reproduce the problem with > > numpy.test(). It always hangs.With the egg it does not hang. Pointer > > problems are usually random, but not random if we are using the same > > binaries in EGG and MSI and variables are always initialized to > > certain value. > Ok, could you try this: > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.0.4.win32-py2.5.msi > > I built it with mingwin, and with NETLIB BLAS/LAPACK. This is just for > testing, do not use it for anything else. > > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > I just tried. Your new MSI does not hang on numpy.test() and the OP's test case, and it seems faster. The EGG version does not hang on numpy.test() but does on OP's test. The original MSI version hangs on numpy.test() if I open IDLE and type import numpy numpy.test() If I try the OP's test first, once it hang on "from numpy.linalg import eig" and the other time it ran successfully. After it ran successfully, it ran numpy.test() successfully, too. As you said, it is random. From mforbes at physics.ubc.ca Tue Nov 13 13:48:59 2007 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Tue, 13 Nov 2007 10:48:59 -0800 Subject: [Numpy-discussion] Warnings as exceptions? In-Reply-To: <4739D4CC.1060402@enthought.com> References: <4739D4CC.1060402@enthought.com> Message-ID: <3E4F9E8A-3AA0-4608-9CBE-CD7AD35FBFA7@physics.ubc.ca> On 13 Nov 2007, at 8:46 AM, Travis E. Oliphant wrote: > Michael McNeil Forbes wrote: >> Why are numpy warnings printed rather than issued using the standard >> warnings library? ... in util.py ... > The "warn" option explicitly allows you to use the warnings library. > There is already the "print" mode which is in fact the default. I see now that it works exactly like I expected. I got confused by the default and then looked in util.py in the old *numarray* code by mistake... Thanks. Michael. From tim.hochberg at ieee.org Tue Nov 13 14:02:42 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 13 Nov 2007 12:02:42 -0700 Subject: [Numpy-discussion] Warnings as exceptions? In-Reply-To: <3E4F9E8A-3AA0-4608-9CBE-CD7AD35FBFA7@physics.ubc.ca> References: <4739D4CC.1060402@enthought.com> <3E4F9E8A-3AA0-4608-9CBE-CD7AD35FBFA7@physics.ubc.ca> Message-ID: On Nov 13, 2007 11:48 AM, Michael McNeil Forbes wrote: > On 13 Nov 2007, at 8:46 AM, Travis E. Oliphant wrote: > > > Michael McNeil Forbes wrote: > >> Why are numpy warnings printed rather than issued using the standard > >> warnings library? ... in util.py ... > > The "warn" option explicitly allows you to use the warnings library. > > There is already the "print" mode which is in fact the default. > I see now that it works exactly like I expected. I got confused by > the default and then looked in util.py in the old *numarray* code by > mistake... > Ooops. I should have known that. The docstring for seterr should be adjusted since it does not list "print" as an option. I'll try to do it if no one gets to it first. > > Thanks. Michael. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Nov 13 16:41:59 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 13 Nov 2007 23:41:59 +0200 Subject: [Numpy-discussion] Bug in arange dtype ">f" was: Using arr.dtype.type to check byteorder-independed dtype fails for bool In-Reply-To: References: Message-ID: <20071113214158.GA15609@mentat.za.net> On Tue, Nov 13, 2007 at 02:53:32PM +0100, Sebastian Haase wrote: > trying to anwer your question with a quick "arange" test, I ran into > more confusion: > >> a = N.arange(.5, dtype=">f") > >>> `a.dtype` > 'dtype('float32')' > >>> a = N.arange(.5, dtype=" >>> `a.dtype` > 'dtype('float32')' Hrm, that doesn't look right. I'll take a look tomorrow. Cheers St?fan From david at ar.media.kyoto-u.ac.jp Tue Nov 13 23:42:49 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 14 Nov 2007 13:42:49 +0900 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <5b8d13220711121051y76e56705m73163281f2d2897f@mail.gmail.com> Message-ID: <473A7CC9.4040801@ar.media.kyoto-u.ac.jp> Keith Goodman wrote: > On Nov 12, 2007 10:51 AM, David Cournapeau wrote: > >> On Nov 13, 2007 3:37 AM, Keith Goodman wrote: >> >>> On Nov 12, 2007 10:10 AM, Peter Creasey wrote: >>> >>>> The following code calling numpy v1.0.4 fails to terminate on my machine, >>>> which was not the case with v1.0.3.1 >>>> >>>> from numpy import arange, float64 >>>> from numpy.linalg import eig >>>> a = arange(13*13, dtype = float64) >>>> a.shape = (13,13) >>>> a = a%17 >>>> eig(a) >>>> >>> It sounds like the same problem that was reported in this thread: >>> >>> http://thread.gmane.org/gmane.comp.python.numeric.general/17456/focus=17465 >>> >>> A friend of mine says the windows binary of 1.0.4 also hangs on eigh >>> and lstsq (so linalg in general). I don't have that problem on 1.0.5 >>> compiled on GNU/Linux. >>> >> Could you friend try the binaries there: >> >> http://projects.scipy.org/pipermail/numpy-discussion/2007-November/029811.html >> >> This may be a problem related to the blas/lapack used for the >> binaries. The binaries posted above use non optimized BLAS/LAPACK (I >> can update to 1.0.4 if this is a problem). >> > > He tried. But he is using python 2.4. (He said your binary was for 2.5). > Here we are: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.0.4.win32-py2.4.exe David From vel.accel at gmail.com Wed Nov 14 08:11:25 2007 From: vel.accel at gmail.com (dieter h.) Date: Wed, 14 Nov 2007 08:11:25 -0500 Subject: [Numpy-discussion] Proposing Ubiquity In-Reply-To: <4739E442.3000703@gmail.com> References: <1e52e0880711130529k267c0b70mbb6147a598a2d8fc@mail.gmail.com> <4739E442.3000703@gmail.com> Message-ID: <1e52e0880711140511m2a5ecf37le0b55e436279a85b@mail.gmail.com> On Nov 13, 2007 12:52 PM, Robert Kern wrote: > > dieter h. wrote: > > Hi all, > > > > Would it make sense for all functionality in Numpy/Scipy to have > > ubiquitous returns? In that I'm proposing that every func/method > > (where appropriate) have a flag in its arg list establishing a return > > for either [INDEX, BOOLEAN, VALUE=default]. this could be by a > > singular Enum flag, I believe. Obvioulsy, this proposal gives the > > option for a return of Index/Bool to be used for Indexing arrays, or > > the actual values. Since I mosty index existing arrays, this would be > > a huge plus for me. > > I'm afraid that I don't understand this proposal. Can you show some examples of > what you would like to see? Preferably, can you write some working code that > demonstrates this feature? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Hi Robert. Thank you for responding, but I made a mistake... so please disregard. -Dieter From kwgoodman at gmail.com Wed Nov 14 09:51:48 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 14 Nov 2007 06:51:48 -0800 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: <473A7CC9.4040801@ar.media.kyoto-u.ac.jp> References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <5b8d13220711121051y76e56705m73163281f2d2897f@mail.gmail.com> <473A7CC9.4040801@ar.media.kyoto-u.ac.jp> Message-ID: On Nov 13, 2007 8:42 PM, David Cournapeau wrote: > > Keith Goodman wrote: > > On Nov 12, 2007 10:51 AM, David Cournapeau wrote: > > > >> On Nov 13, 2007 3:37 AM, Keith Goodman wrote: > >> > >>> On Nov 12, 2007 10:10 AM, Peter Creasey wrote: > >>> > >>>> The following code calling numpy v1.0.4 fails to terminate on my machine, > >>>> which was not the case with v1.0.3.1 > >>>> > >>>> from numpy import arange, float64 > >>>> from numpy.linalg import eig > >>>> a = arange(13*13, dtype = float64) > >>>> a.shape = (13,13) > >>>> a = a%17 > >>>> eig(a) > >>>> > >>> It sounds like the same problem that was reported in this thread: > >>> > >>> http://thread.gmane.org/gmane.comp.python.numeric.general/17456/focus=17465 > >>> > >>> A friend of mine says the windows binary of 1.0.4 also hangs on eigh > >>> and lstsq (so linalg in general). I don't have that problem on 1.0.5 > >>> compiled on GNU/Linux. > >>> > >> Could you friend try the binaries there: > >> > >> http://projects.scipy.org/pipermail/numpy-discussion/2007-November/029811.html > >> > >> This may be a problem related to the blas/lapack used for the > >> binaries. The binaries posted above use non optimized BLAS/LAPACK (I > >> can update to 1.0.4 if this is a problem). > >> > > > > He tried. But he is using python 2.4. (He said your binary was for 2.5). > > > Here we are: > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.0.4.win32-py2.4.exe Thank you. He said it worked. He didn't even notice a slow down without ATLAS. On some calculations the results were different (between 1.0.2 and 1.0.4) in the last three decimal places. But that's to be expected, right? ATLAS must give different results than the non optimized alternative. From david at ar.media.kyoto-u.ac.jp Wed Nov 14 09:51:39 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 14 Nov 2007 23:51:39 +0900 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <5b8d13220711121051y76e56705m73163281f2d2897f@mail.gmail.com> <473A7CC9.4040801@ar.media.kyoto-u.ac.jp> Message-ID: <473B0B7B.3070102@ar.media.kyoto-u.ac.jp> Keith Goodman wrote: > On Nov 13, 2007 8:42 PM, David Cournapeau wrote: >> >> Here we are: >> >> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-1.0.4.win32-py2.4.exe > > Thank you. He said it worked. He didn't even notice a slow down > without ATLAS. On some calculations the results were different > (between 1.0.2 and 1.0.4) in the last three decimal places. But that's > to be expected, right? I don't think there is any chance to have exactly the same results (compiler/OS/CPU/BLAS all enter the equation). ATLAS will not change much the general performances of numpy: this only enter the equation for some functions (numpy.dot) and linear algebra of course, for relatively big numbers. For example, in my use case (linear algebra with maximum a few tens dimensions), ATLAS does not give much outside numpy.dot. And anyway, if you want good performance from atlas, you should compile it by yourself (ATLAS performance seems to really depend on the size of L1 cache, for example). So all in all, I think it worths considering just using netlib BLAS/LAPACK instead of ATLAS for binaries, at least on windows (I don't know who is responsible for the windows binaries); note that we still do not know why the official binaries hang, which is bothering. cheers, David From haase at msg.ucsf.edu Wed Nov 14 11:08:59 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 14 Nov 2007 17:08:59 +0100 Subject: [Numpy-discussion] how to assign to a "sub set" of an array Message-ID: Hi, First here is some test I ran, where I think the last command shows a bug: >>> a = N.arange(4); a.shape=2,2; a [[0 1] [2 3]] >>> aa = N.array((a,a,a)); aa [[[0 1] [2 3]] [[0 1] [2 3]] [[0 1] [2 3]]] >>> N.nonzero(a==0) ([0], [0]) >>> aa[N.nonzero(a==0)] = 5,5; aa [[[ 5 544434034] [ 2 3]] [[ 0 1] [ 2 3]] [[ 0 1] [ 2 3]]] What happend here ? The value 544434034 is different every time..... Background: I have a gray-scale 2D image (in this examples it's a 2x2 pixel image named 'a') I would like to show this image using grey-values EXCEPT I would like to show 0-values as blue. For use with pyOpenGL textures, I convert the image into an RGB image, in the example called 'aa', with shape 3,2,2. This is where I'm getting lost, and I tried assigning "something" (5,5) to aa "only at certain pixels".... Maybe someone can teach what the proper indexing-constuct, that I would need to do what I want. Did I find a bug ? Thanks, Sebastian Haase From tim.hochberg at ieee.org Wed Nov 14 11:59:00 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 14 Nov 2007 09:59:00 -0700 Subject: [Numpy-discussion] how to assign to a "sub set" of an array In-Reply-To: References: Message-ID: On Nov 14, 2007 9:08 AM, Sebastian Haase wrote: > Hi, > First here is some test I ran, where I think the last command shows a bug: > > >>> a = N.arange(4); a.shape=2,2; a > [[0 1] > [2 3]] > >>> aa = N.array((a,a,a)); aa > [[[0 1] > [2 3]] > > [[0 1] > [2 3]] > > [[0 1] > [2 3]]] > >>> N.nonzero(a==0) > ([0], [0]) > >>> aa[N.nonzero(a==0)] = 5,5; aa > [[[ 5 544434034] > [ 2 3]] > > [[ 0 1] > [ 2 3]] > > [[ 0 1] > [ 2 3]]] > > > What happend here ? The value 544434034 is different every time..... I'm not seeing that here on WinXP with numpy 1.0.4.dev3462 ; can you tell me what platform and version you are using? > > Background: > I have a gray-scale 2D image (in this examples it's a 2x2 pixel image > named 'a') > I would like to show this image using grey-values EXCEPT I would like > to show 0-values as blue. > For use with pyOpenGL textures, I convert the image into an RGB image, > in the example called 'aa', with shape 3,2,2. > > This is where I'm getting lost, and I tried assigning "something" > (5,5) to aa "only at certain pixels".... > Maybe someone can teach what the proper indexing-constuct, that I > would need to do what I want. > > Did I find a bug ? It looks like a bug, although it's possible it's been fixed. Or maybe it's a new one and I need to update my version of numpy to see it... > > > Thanks, > Sebastian Haase > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivilata at carabos.com Wed Nov 14 12:08:00 2007 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Wed, 14 Nov 2007 18:08:00 +0100 Subject: [Numpy-discussion] [ANN] Release of the first PyTables video Message-ID: <20071114170800.GD30891@tardis.terramar.selidor.net> ===================================== Release of the first PyTables video ===================================== `Carabos `_ is very proud to announce the first of a series of videos dedicated to introducing the main features of PyTables to the public in a visual and easy to grasp manner. http://www.carabos.com/videos/pytables-1-intro `PyTables `_ is a Free/Open Source package designed to handle massive amounts of data in a simple, but highly efficient way, using the HDF5 file format and NumPy data containers. This first video is an introductory overview of PyTables, covering the following topics: * HDF5 file creation * the object tree * homogeneous array storage * natural naming * working with attributes With a running length of little more than 10 minutes, you may sit back and watch it during any short break. More videos about PyTables will be published in the near future. Stay tuned on www.pytables.org for the announcement of the new videos. We would like to hear your opinion on the video so we can do it better the next time. We are also open to suggestions for the topics of future videos. Best regards, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 307 bytes Desc: Digital signature URL: From haase at msg.ucsf.edu Wed Nov 14 12:13:57 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 14 Nov 2007 18:13:57 +0100 Subject: [Numpy-discussion] how to assign to a "sub set" of an array In-Reply-To: References: Message-ID: On Nov 14, 2007 5:59 PM, Timothy Hochberg wrote: > > > > On Nov 14, 2007 9:08 AM, Sebastian Haase wrote: > > Hi, > > First here is some test I ran, where I think the last command shows a bug: > > > > >>> a = N.arange(4); a.shape=2,2; a > > [[0 1] > > [2 3]] > > >>> aa = N.array((a,a,a)); aa > > [[[0 1] > > [2 3]] > > > > [[0 1] > > [2 3]] > > > > [[0 1] > > [2 3]]] > > >>> N.nonzero(a==0) > > ([0], [0]) > > >>> aa[N.nonzero(a==0)] = 5,5; aa > > [[[ 5 544434034] > > [ 2 3]] > > > > [[ 0 1] > > [ 2 3]] > > > > [[ 0 1] > > [ 2 3]]] > > > > > > What happend here ? The value 544434034 is different every time..... > > I'm not seeing that here on WinXP with numpy 1.0.4.dev3462 ; can you tell me > what platform and version you are using? > > > > > > > > Background: > > I have a gray-scale 2D image (in this examples it's a 2x2 pixel image > named 'a') > > I would like to show this image using grey-values EXCEPT I would like > > to show 0-values as blue. > > For use with pyOpenGL textures, I convert the image into an RGB image, > > in the example called 'aa', with shape 3,2,2. > > > > This is where I'm getting lost, and I tried assigning "something" > > (5,5) to aa "only at certain pixels".... > > Maybe someone can teach what the proper indexing-constuct, that I > > would need to do what I want. > > > > Did I find a bug ? > > It looks like a bug, although it's possible it's been fixed. Or maybe it's a > new one and I need to update my version of numpy to see it... > I'm still on version 1.0,1 -- winXP. looks like it's fixed and I need to upgrade ;-) Thanks for checking, -Sebastian From Achim.Gaedke at physik.tu-darmstadt.de Wed Nov 14 13:31:38 2007 From: Achim.Gaedke at physik.tu-darmstadt.de (Achim Gaedke) Date: Wed, 14 Nov 2007 19:31:38 +0100 Subject: [Numpy-discussion] SegFault/double free with simple array mask operation Message-ID: <473B3F0A.50601@physik.tu-darmstadt.de> Hello everybody! Please have a look at the program below: # start import numpy t_array=numpy.ones(2048, dtype=numpy.float32) sinc_array=numpy.array((len(t_array),),dtype=numpy.float32) sinc_array[(t_array > 0.)]=1.0 # end If you execute this program, it crashes with Segmentation Fault or *** glibc detected *** python: double free or corruption (out): 0x081fe470 *** It depends on the circumstances, which error occurs, e.g. you must quit your interpreter if you are in interactvie mode. Obviously numpy.array() should be numpy.zeros() or numpy.empty() .... But this program should not crash with a core dump. Used Linux Versions are: Debian Testing with numpy 1.0.3, Debian Stable with numpy 1.0.1, Ubuntu Linux 6.10 with numpy 1.0 Also numpy-1.0.4 crashes. Yours, Achim From nwagner at iam.uni-stuttgart.de Wed Nov 14 13:44:35 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Wed, 14 Nov 2007 19:44:35 +0100 Subject: [Numpy-discussion] SegFault/double free with simple array mask operation In-Reply-To: <473B3F0A.50601@physik.tu-darmstadt.de> References: <473B3F0A.50601@physik.tu-darmstadt.de> Message-ID: On Wed, 14 Nov 2007 19:31:38 +0100 Achim Gaedke wrote: > Hello everybody! > > Please have a look at the program below: > > # start > import numpy > > t_array=numpy.ones(2048, dtype=numpy.float32) > sinc_array=numpy.array((len(t_array),),dtype=numpy.float32) > sinc_array[(t_array > 0.)]=1.0 > # end > > If you execute this program, it crashes with >Segmentation Fault or > *** glibc detected *** python: double free or corruption >(out): > 0x081fe470 *** > > It depends on the circumstances, which error occurs, >e.g. you must quit > your interpreter if you are in interactvie mode. > > Obviously numpy.array() should be numpy.zeros() or >numpy.empty() .... > But this program should not crash with a core dump. > > Used Linux Versions are: > Debian Testing with numpy 1.0.3, > Debian Stable with numpy 1.0.1, > Ubuntu Linux 6.10 with numpy 1.0 > Also numpy-1.0.4 crashes. > > Yours, Achim > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion I can confirm the problem on opensuse10.2 x86_64 using python2.5 >>> numpy.__version__ '1.0.5.dev4453' Here is a backtrace Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 46940023738768 (LWP 4279)] 0x00002ab1128cde47 in _PyObject_GC_Track () from /usr/lib64/libpython2.5.so.1.0 (gdb) bt #0 0x00002ab1128cde47 in _PyObject_GC_Track () from /usr/lib64/libpython2.5.so.1.0 #1 0x00002ab1128ce514 in PyGC_Collect () from /usr/lib64/libpython2.5.so.1.0 #2 0x00002ab1128c3ce7 in Py_Finalize () from /usr/lib64/libpython2.5.so.1.0 #3 0x00002ab1128ccf34 in Py_Main () from /usr/lib64/libpython2.5.so.1.0 #4 0x00002ab1133e7bc4 in __libc_start_main () from /lib64/libc.so.6 #5 0x00000000004006a9 in _start () Nils From gnchen at gmail.com Wed Nov 14 18:38:34 2007 From: gnchen at gmail.com (Gen-Nan Chen) Date: Wed, 14 Nov 2007 15:38:34 -0800 Subject: [Numpy-discussion] list as array item and casting problem problem Message-ID: <588944e60711141538i3805effbn1a5c7f148f8bf4da@mail.gmail.com> Hi! All, I have below strange behavior in numpy. Can anyone shed some light on this: I have an array with each item as a list object. You will get this when you use scipy.sparse In [369]: k Out[369]: array([[0], [1], [2], [3], [4], [5], [6], [7], [8], [9]], dtype=object) Assigning value by using slicing is working In [370]: k[0:3] = [[i+3] for i in range(3)] In [371]: k Out[371]: array([[3], [4], [5], [3], [4], [5], [6], [7], [8], [9]], dtype=object) But assigning value by using a list for slicing is NOT In [374]: k[[0, 1,2]] = [[i+3] for i in range(3)] In [375]: k Out[375]: array([3, 4, 5, [3], [4], [5], [6], [7], [8], [9]], dtype=object) You can see those list objects casted to integer. Is this a bug or a feature? if I want to slice it with random index without casting it, what should I do? -- Gen-Nan Chen, Ph. D. email: gnchen at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Nov 14 22:48:46 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 15 Nov 2007 12:48:46 +0900 Subject: [Numpy-discussion] SegFault/double free with simple array mask operation In-Reply-To: <473B3F0A.50601@physik.tu-darmstadt.de> References: <473B3F0A.50601@physik.tu-darmstadt.de> Message-ID: <473BC19E.30505@ar.media.kyoto-u.ac.jp> Achim Gaedke wrote: > Hello everybody! > > Please have a look at the program below: > > # start > import numpy > > t_array=numpy.ones(2048, dtype=numpy.float32) > sinc_array=numpy.array((len(t_array),),dtype=numpy.float32) > sinc_array[(t_array > 0.)]=1.0 > # end > > If you execute this program, it crashes with Segmentation Fault or > *** glibc detected *** python: double free or corruption (out): > 0x081fe470 *** > > It depends on the circumstances, which error occurs, e.g. you must quit > your interpreter if you are in interactvie mode. > > Obviously numpy.array() should be numpy.zeros() or numpy.empty() .... > But this program should not crash with a core dump. > > Used Linux Versions are: > Debian Testing with numpy 1.0.3, > Debian Stable with numpy 1.0.1, > Ubuntu Linux 6.10 with numpy 1.0 > Also numpy-1.0.4 crashes. > > Yours, Achim > Could you open a ticket on the numpy trac system ? (I can confirm the bug) cheers, David From Achim.Gaedke at physik.tu-darmstadt.de Thu Nov 15 02:24:30 2007 From: Achim.Gaedke at physik.tu-darmstadt.de (Achim Gaedke) Date: Thu, 15 Nov 2007 08:24:30 +0100 Subject: [Numpy-discussion] SegFault/double free with simple array mask operation In-Reply-To: <473BC19E.30505@ar.media.kyoto-u.ac.jp> References: <473B3F0A.50601@physik.tu-darmstadt.de> <473BC19E.30505@ar.media.kyoto-u.ac.jp> Message-ID: <473BF42E.6020407@physik.tu-darmstadt.de> David Cournapeau wrote: > Achim Gaedke wrote: > >> Hello everybody! >> >> Please have a look at the program below: >> >> # start >> import numpy >> >> t_array=numpy.ones(2048, dtype=numpy.float32) >> sinc_array=numpy.array((len(t_array),),dtype=numpy.float32) >> sinc_array[(t_array > 0.)]=1.0 >> # end >> .... > Could you open a ticket on the numpy trac system ? (I can confirm the bug) > > cheers, > > David > It is Ticket #614 . The version information in trac are outdated, I could not select version 1.0.3 or 1.0.4 . Yours, Achim From mforbes at physics.ubc.ca Thu Nov 15 05:47:32 2007 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Thu, 15 Nov 2007 02:47:32 -0800 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> <47396237.60209@ar.media.kyoto-u.ac.jp> Message-ID: On 13 Nov 2007, at 9:43 AM, Geoffrey Zhu wrote: > On Nov 13, 2007 2:37 AM, David Cournapeau u.ac.jp> wrote: >> Geoffrey Zhu wrote: >>> Pointer problems are usually random... ... > The original MSI version hangs on numpy.test() if I open IDLE and type > > import numpy > numpy.test() > > If I try the OP's test first, once it hang on "from numpy.linalg > import eig" and the other time it ran successfully. After it ran > successfully, it ran numpy.test() successfully, too. > > As you said, it is random. I have also been having random problems with the latest numpy from svn built on an Intel core 2 Duo Linux box running in 64 bit mode under Red Hat 3.4.6-8 with the gcc 3.4.6 20060404 and ATLAS 3.8.0. I am having a problem with numpy.linalg.eigh and complex Hermitian matrices. Randomly, I get seemingly correct answers, and then eigenvectors full of Nan's (though not completely. The first row the the eigenvectors seem to be numbers, but incorrect.) Sometimes, I can stop just after the error with pdb and "play". Calling eigh from the debugger sometimes gives a correct answer, and then other times gives eigenvalues and eigenvectors full of Nan's (not completely full mind you). For example: (Pdb) p numpy.linalg.eigh(HH) (array([-50.50589438, -45.86305013, -40.56713543, -35.57216233, 38.1497506 , 40.17291371, 43.35773763, 46.59527636, 49.42413434, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]), array([[-0.00072424 +0.j, -0.00136655 +0.j, 0.00200233 +0.j, ..., 0.00000000 +0.j, 0.00000000 +0.j, 0.00000000 +0.j], [ NaN NaNj, NaN NaNj, NaN NaNj, ..., NaN NaNj, NaN NaNj, NaN NaNj], [ NaN NaNj, NaN NaNj, NaN NaNj, ..., NaN NaNj, NaN NaNj, NaN NaNj], ..., [ NaN NaNj, NaN NaNj, NaN NaNj, ..., NaN NaNj, NaN NaNj, NaN NaNj], [ NaN NaNj, NaN NaNj, NaN NaNj, ..., NaN NaNj, NaN NaNj, NaN NaNj], [ NaN NaNj, NaN NaNj, NaN NaNj, ..., NaN NaNj, NaN NaNj, NaN NaNj]])) (Pdb) p numpy.linalg.eigh(HH) (array([-51.06208813, -48.50332834, -48.49643331, -46.25814405, -46.25813858, -44.33668063, -44.33668063, -42.73548661, -42.73548661, -41.45454929, -41.45454929, -40.49386126, -40.49386126, -39.85344006, -39.85344006, -39.53308677, -39.53308677, 37.91885011, 37.91885011, 38.2392034 , 38.2392034 , 38.8796246 , 38.8796246 , 39.84031263, 39.84031263, 41.12124995, 41.12124995, 42.72244397, 42.72244398, 44.64390192, 44.6439074 , 46.88219666, 46.88909168, 49.44785148]), array([[ -5.28060016e-04 +0.00000000e+00j, -3.92271866e-05 +0.00000000e+00j, 7.72453920e-04 +0.00000000e +00j, ..., -3.36896226e-01 +0.00000000e+00j, 6.28651296e-02 +0.00000000e+00j, -2.42202473e-01 +0.00000000e+00j], [ 1.48818848e-03 +2.78190640e-04j, 1.06069959e-03 +1.98279117e-04j, -1.88322135e-03 -3.52035081e-04j, ..., 2.86677919e-01 +5.35893907e-02j, -1.77188491e-01 -3.31222694e-02j, 2.38244862e-01 +4.45356831e-02j], [ -2.14234988e-03 -8.29950766e-04j, -2.44246082e-03 -9.46214364e-04j, 1.92200953e-03 +7.44590459e-04j, ..., -1.92999931e-01 -7.47685718e-02j, 2.55119386e-01 +9.88337767e-02j, -2.26238355e-01 -8.76452055e-02j], ..., [ 2.06281453e-01 -1.27724068e-01j, -2.32614835e-01 +1.44029008e-01j, -1.75975052e-01 +1.08959139e-01j, ..., 1.75246553e-03 -1.08508072e-03j, 2.22700685e-03 -1.37890426e-03j, 1.95336925e-03 -1.20947504e-03j], [ -2.26004880e-01 +8.75547569e-02j, 1.68085319e-01 -6.51165996e-02j, 2.71949658e-01 -1.05353859e-01j, ..., -1.78646965e-03 +6.92082029e-04j, -1.00620547e-03 +3.89806076e-04j, -1.41173185e-03 +5.46907831e-04j], [ 2.38078516e-01 -4.45045876e-02j, -6.17947313e-02 +1.15514373e-02j, -3.31159928e-01 +6.19045191e-02j, ..., 7.59301424e-04 -1.41938035e-04j, 3.85592692e-05 -7.20797663e-06j, 5.19068791e-04 -9.70307734e-05j]])) Here is the version info (Everything build from scratch, numpy from svn): >>> sys.version '2.5.1 (r251:54863, Nov 10 2007, 00:44:16) \n[GCC 3.4.6 20060404 (Red Hat 3.4.6-8)]' >>> numpy.version.version '1.0.5.dev4427' >>> scipy.version.version '0.7.0.dev3511' Using ATLAS-3.8.0. This is extremely annoying, and difficult to reproduce. I will try recompiling with some different versions and see if I can reproduce the problem. Running numpy.test() does *not* fail... Michael. From david at ar.media.kyoto-u.ac.jp Thu Nov 15 05:45:17 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 15 Nov 2007 19:45:17 +0900 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> <47396237.60209@ar.media.kyoto-u.ac.jp> Message-ID: <473C233D.3090309@ar.media.kyoto-u.ac.jp> Michael McNeil Forbes wrote: > On 13 Nov 2007, at 9:43 AM, Geoffrey Zhu wrote: > >> On Nov 13, 2007 2:37 AM, David Cournapeau > u.ac.jp> wrote: >> >>> Geoffrey Zhu wrote: >>> >>>> Pointer problems are usually random... >>>> > ... > >> The original MSI version hangs on numpy.test() if I open IDLE and type >> >> import numpy >> numpy.test() >> >> If I try the OP's test first, once it hang on "from numpy.linalg >> import eig" and the other time it ran successfully. After it ran >> successfully, it ran numpy.test() successfully, too. >> >> As you said, it is random. >> > > > I have also been having random problems with the latest numpy from > svn built on an Intel core 2 Duo Linux box running in 64 bit mode > under Red Hat 3.4.6-8 with the gcc 3.4.6 20060404 and ATLAS 3.8.0. > > I am having a problem with numpy.linalg.eigh and complex Hermitian > matrices. Randomly, I get seemingly correct answers, and then > eigenvectors full of Nan's (though not completely. The first row the > the eigenvectors seem to be numbers, but incorrect.) > Which fortran compiler are you using ? David From mforbes at physics.ubc.ca Thu Nov 15 06:28:44 2007 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Thu, 15 Nov 2007 03:28:44 -0800 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: <473C233D.3090309@ar.media.kyoto-u.ac.jp> References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> <47396237.60209@ar.media.kyoto-u.ac.jp> <473C233D.3090309@ar.media.kyoto-u.ac.jp> Message-ID: <5E87576B-FA51-4EFC-AAEB-D3BD0E4825A9@physics.ubc.ca> On 15 Nov 2007, at 2:45 AM, David Cournapeau wrote: > Which fortran compiler are you using ? GNU Fortran (GCC) 3.4.6 20060404 From gnurser at googlemail.com Thu Nov 15 10:17:06 2007 From: gnurser at googlemail.com (George Nurser) Date: Thu, 15 Nov 2007 15:17:06 +0000 Subject: [Numpy-discussion] array allocation using tuples gives views of same array Message-ID: <1d1e6ea70711150717x1d2da34cw971e24d73e7ec073@mail.gmail.com> I tried the (as I thought) nice compact form In [60]: a,b = (zeros((2,)),)*2 But... In [61]: b[0] = 2 In [62]: a Out[62]: array([ 2., 0.]) a and b are the _same_ array.... But In [68]: a,b = (zeros((2,)),zeros((2,))) In [69]: b[0] = 2 In [70]: a Out[70]: array([ 0., 0.]) is OK. a & b are independent in this case. I'm puzzled by this behaviour, I suspect because of my ignorance of how tuples work. It looks to me like a,b = (zeros((2,)),)*2 is equivalent to x= zeros((2,)) a,b=(x,)*2 If this is indeed a feature rather than a bug, is there an alternative compact way to allocate many arrays? Regards, George. From matthieu.brucher at gmail.com Thu Nov 15 10:23:27 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 15 Nov 2007 16:23:27 +0100 Subject: [Numpy-discussion] array allocation using tuples gives views of same array In-Reply-To: <1d1e6ea70711150717x1d2da34cw971e24d73e7ec073@mail.gmail.com> References: <1d1e6ea70711150717x1d2da34cw971e24d73e7ec073@mail.gmail.com> Message-ID: It's not a question of tuple, you made a tuple, but in each element, you put the same array, so this behaviour is to be expected. Matthieu 2007/11/15, George Nurser : > > I tried the (as I thought) nice compact form > In [60]: a,b = (zeros((2,)),)*2 > > But... > > In [61]: b[0] = 2 > > In [62]: a > Out[62]: array([ 2., 0.]) > > a and b are the _same_ array.... > > But > In [68]: a,b = (zeros((2,)),zeros((2,))) > > In [69]: b[0] = 2 > > In [70]: a > Out[70]: array([ 0., 0.]) > > is OK. a & b are independent in this case. > > I'm puzzled by this behaviour, I suspect because of my ignorance of > how tuples work. > > It looks to me like > a,b = (zeros((2,)),)*2 > is equivalent to > x= zeros((2,)) > a,b=(x,)*2 > > If this is indeed a feature rather than a bug, is there an alternative > compact way to allocate many arrays? > > > Regards, George. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From focke at slac.stanford.edu Thu Nov 15 10:29:12 2007 From: focke at slac.stanford.edu (Warren Focke) Date: Thu, 15 Nov 2007 07:29:12 -0800 (PST) Subject: [Numpy-discussion] array allocation using tuples gives views of same array In-Reply-To: <1d1e6ea70711150717x1d2da34cw971e24d73e7ec073@mail.gmail.com> References: <1d1e6ea70711150717x1d2da34cw971e24d73e7ec073@mail.gmail.com> Message-ID: On Thu, 15 Nov 2007, George Nurser wrote: > It looks to me like > a,b = (zeros((2,)),)*2 > is equivalent to > x= zeros((2,)) > a,b=(x,)*2 Correct. > If this is indeed a feature rather than a bug, is there an alternative > compact way to allocate many arrays? a, b = [zeros((2,)) for x in range(2)] From meine at informatik.uni-hamburg.de Thu Nov 15 11:11:35 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 15 Nov 2007 17:11:35 +0100 Subject: [Numpy-discussion] array allocation using tuples gives views of same array In-Reply-To: References: <1d1e6ea70711150717x1d2da34cw971e24d73e7ec073@mail.gmail.com> Message-ID: <200711151711.36344.meine@informatik.uni-hamburg.de> Am Donnerstag, 15. November 2007 16:29:12 schrieb Warren Focke: > On Thu, 15 Nov 2007, George Nurser wrote: > > It looks to me like > > a,b = (zeros((2,)),)*2 > > is equivalent to > > x= zeros((2,)) > > a,b=(x,)*2 > > Correct. > > > If this is indeed a feature rather than a bug, is there an alternative > > compact way to allocate many arrays? > > a, b = [zeros((2,)) for x in range(2)] Let me add that this is a standard Python caveat, which also happens with lists -- many of us have once tried to initialize an array of empty lists with ([], ) * N, which results in N references to the same empty list. Warren pointed out the standard solution above. -- Ciao, / / /--/ / / ANS From tim.hochberg at ieee.org Thu Nov 15 11:32:24 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 15 Nov 2007 09:32:24 -0700 Subject: [Numpy-discussion] array allocation using tuples gives views of same array In-Reply-To: <200711151711.36344.meine@informatik.uni-hamburg.de> References: <1d1e6ea70711150717x1d2da34cw971e24d73e7ec073@mail.gmail.com> <200711151711.36344.meine@informatik.uni-hamburg.de> Message-ID: On Nov 15, 2007 9:11 AM, Hans Meine wrote: > Am Donnerstag, 15. November 2007 16:29:12 schrieb Warren Focke: > > On Thu, 15 Nov 2007, George Nurser wrote: > > > It looks to me like > > > a,b = (zeros((2,)),)*2 > > > is equivalent to > > > x= zeros((2,)) > > > a,b=(x,)*2 > > > > Correct. > > > > > If this is indeed a feature rather than a bug, is there an alternative > > > compact way to allocate many arrays? > > > > a, b = [zeros((2,)) for x in range(2)] > > Let me add that this is a standard Python caveat, which also happens with > lists -- many of us have once tried to initialize an array of empty lists > with ([], ) * N, which results in N references to the same empty list. > Warren pointed out the standard solution above. And I'll just add that another option in this case is to avoid tuples and lists altogether and just unpack a larger array: a, b = zeros([2,2]) That's mostly inferior to the list comprehension solution in terms of clarity, but may appeal to the fans of the pithy. In some cases it also expresses intent better, but not often. Using lists for shapes is clearer both typographically: compare zeros((2,)) and zeros([2]), and conceptually: shapes are closer to lists (variable length, homogeneous) than tuples (fixed length, inhomogeneous). Tuples are used for the shape attribute since they need to be immutable, but there's no reason to type them that way; it just makes things hard to read. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From gnurser at googlemail.com Thu Nov 15 12:31:43 2007 From: gnurser at googlemail.com (George Nurser) Date: Thu, 15 Nov 2007 17:31:43 +0000 Subject: [Numpy-discussion] array allocation using tuples gives views of same array In-Reply-To: References: <1d1e6ea70711150717x1d2da34cw971e24d73e7ec073@mail.gmail.com> <200711151711.36344.meine@informatik.uni-hamburg.de> Message-ID: <1d1e6ea70711150931m758b0cf3l24d350660391aba0@mail.gmail.com> On 15/11/2007, Timothy Hochberg wrote: > > > On Nov 15, 2007 9:11 AM, Hans Meine wrote: > > Am Donnerstag, 15. November 2007 16:29:12 schrieb Warren Focke: > > > > > On Thu, 15 Nov 2007, George Nurser wrote: > > > > It looks to me like > > > > a,b = (zeros((2,)),)*2 > > > > is equivalent to > > > > x= zeros((2,)) > > > > a,b=(x,)*2 > > > > > > Correct. > > > > > > > If this is indeed a feature rather than a bug, is there an alternative > > > > compact way to allocate many arrays? > > > > > > a, b = [zeros((2,)) for x in range(2)] > > > > Let me add that this is a standard Python caveat, which also happens with > > lists -- many of us have once tried to initialize an array of empty lists > > with ([], ) * N, which results in N references to the same empty list. > > Warren pointed out the standard solution above. > > And I'll just add that another option in this case is to avoid tuples and > lists altogether and just unpack a larger array: > > a, b = zeros([2,2]) Pithy indeed. I didn't realize that one could unpack numpy arrays. > > > Using lists for shapes is clearer both typographically: compare zeros((2,)) > and zeros([2]), and conceptually: shapes are closer to lists (variable > length, homogeneous) than tuples (fixed length, inhomogeneous). Tuples are > used for the shape attribute since they need to be immutable, but there's no > reason to type them that way; it just makes things hard to read. > > Makes sense. Thanks to everybody for their help. George. From david at ar.media.kyoto-u.ac.jp Thu Nov 15 23:23:13 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 16 Nov 2007 13:23:13 +0900 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> <47396237.60209@ar.media.kyoto-u.ac.jp> Message-ID: <473D1B31.2050207@ar.media.kyoto-u.ac.jp> Michael McNeil Forbes wrote: > On 13 Nov 2007, at 9:43 AM, Geoffrey Zhu wrote: > >> On Nov 13, 2007 2:37 AM, David Cournapeau > u.ac.jp> wrote: >> >>> Geoffrey Zhu wrote: >>> >>>> Pointer problems are usually random... >>>> > ... > >> The original MSI version hangs on numpy.test() if I open IDLE and type >> >> import numpy >> numpy.test() >> >> If I try the OP's test first, once it hang on "from numpy.linalg >> import eig" and the other time it ran successfully. After it ran >> successfully, it ran numpy.test() successfully, too. >> >> As you said, it is random. >> > > > I have also been having random problems with the latest numpy from > svn built on an Intel core 2 Duo Linux box running in 64 bit mode > under Red Hat 3.4.6-8 with the gcc 3.4.6 20060404 and ATLAS 3.8.0. > Could you try without atlas ? Also, how did you configure atlas when building it ? It seems that atlas is definitely part of the problem (everybody having the problem does use atlas), and that it involves Core 2 duo. David From mforbes at physics.ubc.ca Fri Nov 16 00:22:30 2007 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Thu, 15 Nov 2007 21:22:30 -0800 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: <473D1B31.2050207@ar.media.kyoto-u.ac.jp> References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> <47396237.60209@ar.media.kyoto-u.ac.jp> <473D1B31.2050207@ar.media.kyoto-u.ac.jp> Message-ID: <89E31BE8-0370-4A58-B95B-B2D8B62717FA@physics.ubc.ca> >> I have also been having random problems with the latest numpy from >> svn built on an Intel core 2 Duo Linux box running in 64 bit mode >> under Red Hat 3.4.6-8 with the gcc 3.4.6 20060404 and ATLAS 3.8.0. >> > Could you try without atlas ? Also, how did you configure atlas when > building it ? It seems that atlas is definitely part of the problem > (everybody having the problem does use atlas), and that it involves > Core > 2 duo. I will try that. I made a mistake: it is a Core 2, not a core 2 duo... model name : Intel(R) Core(TM)2 CPU 6700 @ 2.66GHz Michael. From mforbes at physics.ubc.ca Fri Nov 16 04:46:31 2007 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Fri, 16 Nov 2007 01:46:31 -0800 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: <473D1B31.2050207@ar.media.kyoto-u.ac.jp> References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> <47396237.60209@ar.media.kyoto-u.ac.jp> <473D1B31.2050207@ar.media.kyoto-u.ac.jp> Message-ID: On 15 Nov 2007, at 8:23 PM, David Cournapeau wrote: > Could you try without atlas ? Also, how did you configure atlas when > building it ? It seems that atlas is definitely part of the problem > (everybody having the problem does use atlas), and that it involves > Core > 2 duo. > > David It seems to work fine without ATLAS, but then again, it is a somewhat "random" error. I will let some code run tonight and see if I detect anything. Michael. From woiski at gmail.com Fri Nov 16 11:44:20 2007 From: woiski at gmail.com (Emanuel Woiski) Date: Fri, 16 Nov 2007 14:44:20 -0200 Subject: [Numpy-discussion] NumPy trunk is frozen for upcoming 1.0.4 release In-Reply-To: <4730649E.8070702@ar.media.kyoto-u.ac.jp> References: <9c08f0d10711041153u170a6907p50b38a13e50089d7@mail.gmail.com> <9c08f0d10711050254s35ee5431u64d3991eb7e14da@mail.gmail.com> <473016E8.9060801@ar.media.kyoto-u.ac.jp> <9c08f0d10711060405r263de2fci78c03fa99bc3e682@mail.gmail.com> <47305737.3030804@ar.media.kyoto-u.ac.jp> <9c08f0d10711060444x3a355f09tbdb57c6b1d9aa1ad@mail.gmail.com> <4730649E.8070702@ar.media.kyoto-u.ac.jp> Message-ID: <9c08f0d10711160844x2591651bl1033544f04b38b36@mail.gmail.com> On Nov 6, 2007 10:57 AM, David Cournapeau wrote: > Emanuel Woiski wrote: > > yep I have no choice - those are a bunch of lab machines....:) > > thanks anyway > > regards > > woiski > Can't you build it yourself ? You just have to install mingw, and > compiling blas/lapack is not too difficult. > > cheers, > > Well, actually I did it and all the tests came out OK for 'my' numpy 1.0.4whereas crash (something related to msvcrt71.dll) for 'my' scipy 0.6.0. However, all my numpy/scipy codes seem to be working perfectly now :).... You can find them here (look down the page for numpy and scipy BUILT files): http://200.145.244.188/Disciplinas/teamspaces/material-de-python-comum-as-disciplinas(in Portuguese...) BTW I have used ATLAS binaries (PII) from the Scipy site and Enthought mingw installation... regards, woiski -------------- next part -------------- An HTML attachment was scrubbed... URL: From robince at gmail.com Fri Nov 16 13:25:12 2007 From: robince at gmail.com (Robin) Date: Fri, 16 Nov 2007 18:25:12 +0000 Subject: [Numpy-discussion] test failure: Ticket 396 Message-ID: Hello, Just thought I'd report this test failure with current svn (rev 4464) on Mac OS X 10.5 (could be a leopard issue?) since I hadn't seen it mentioned before. Oddly it only seems to fail in ipython, importing and running from the normal python interpreter all the tests pass. Actually, it only seems to fail when I start ipython with ipython -p scipy I haven't been able to recreate the problem with any other combination of startup and imports. Here is my ipythonrc-scipy: # load our basic configuration with generic options include ipythonrc # import ... # Load Numpy an SciPy by themselves so that 'help' works on them import_mod numpy scipy # from ... import ... import_some # Now we load all of SciPy # from ... import * import_all numpy import_all scipy ------------------------------------------- Here is the error: ====================================================================== ERROR: Ticket #396 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/tests/test_regression.py", line 600, in check_poly1d_nan_roots self.failUnlessRaises(N.linalg.LinAlgError,getattr,p,"r") File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/unittest.py", line 320, in failUnlessRaises callableObj(*args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/polynomial.py", line 623, in __getattr__ return roots(self.coeffs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/polynomial.py", line 124, in roots roots = _eigvals(A) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/polynomial.py", line 40, in _eigvals return eigvals(arg) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/linalg/decomp.py", line 378, in eigvals return eig(a,b=b,left=0,right=0,overwrite_a=overwrite_a) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/linalg/decomp.py", line 128, in eig a1 = asarray_chkfinite(a) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/function_base.py", line 398, in asarray_chkfinite raise ValueError, "array must not contain infs or NaNs" ValueError: array must not contain infs or NaNs ---------------------------------------------------------------------- Ran 692 tests in 0.456s FAILED (errors=1) Out[2]: Thanks, Robin From listservs at mac.com Fri Nov 16 14:54:38 2007 From: listservs at mac.com (Chris) Date: Fri, 16 Nov 2007 19:54:38 +0000 (UTC) Subject: [Numpy-discussion] Problems building numpy using setupegg Message-ID: I am trying to build some installers of numpy using setuptools and bdist_mpkg on Leopard using the following command: python setupegg.py config_fc --fcompiler gnu95 config -L../staticlibs build bdist_mpkg --open However, I get the following error towards the end of the build: Copying numpy.egg-info to build/bdist.macosx-10.3-i386/mpkg/ platlib/Library/Python/2.5/site-packages/numpy-1.0.5.dev4464-py2.5.egg-info running install_scripts error: cannot copy tree 'build/scripts.macosx-10.3-i386-2.5': not a directory This is a confusing error, since there is a "build/scripts.macosx-10.3-i386-2.5" and it is certainly a directory. From woiski at gmail.com Fri Nov 16 14:55:57 2007 From: woiski at gmail.com (Emanuel Woiski) Date: Fri, 16 Nov 2007 17:55:57 -0200 Subject: [Numpy-discussion] beginner question: rank-1 arrays In-Reply-To: <470B216B.5040808@gmx.net> References: <470B216B.5040808@gmx.net> Message-ID: <9c08f0d10711161155n30f67766r6842382f136a41d7@mail.gmail.com> Sorry for coming very late to the thread, but you mean something like: for i in range(len(cols): On Oct 9, 2007 4:36 AM, Sven Schreiber wrote: > Alan G Isaac schrieb: > > On Mon, 8 Oct 2007, Robin apparently wrote: > >> However in my code (I am converting from MATLAB) it is > >> important to maintain 2d arrays, and keep the difference > >> between row and column vectors. > > > > How about using matrices? > > help(numpy.mat) > > > > hth, > > Alan Isaac > > > > Robin, Alan is right, you want numpy matrices which are always 2d. Check > out numpy.matlib; if you replace > from numpy import [whatever] > by > from numpy.matlib import [whatever] > you get everything there is in numpy, and things like ones() zeros() > empty() etc. will always be 2d matrices. > > -sven > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From grh at mur.at Fri Nov 16 15:25:38 2007 From: grh at mur.at (Georg Holzmann) Date: Fri, 16 Nov 2007 21:25:38 +0100 Subject: [Numpy-discussion] swig numpy2carray converters Message-ID: <473DFCC2.1050704@mur.at> Hallo! Because I had some troubles in wrapping my C++ library in python/numpy, I did (another) numpy2carray.i file for SWIG. With that interface file it is possible to input/output arrays with or without copying data (I also included an example for an interface to fortran style arrays). I am still a numpy newbie, so this might not be perferct, but however, I finally found some time to put it online - maybe it is helpfull for other beginners: http://grh.mur.at/software/numpy2carray.html LG Georg From woiski at gmail.com Fri Nov 16 15:29:07 2007 From: woiski at gmail.com (Emanuel Woiski) Date: Fri, 16 Nov 2007 18:29:07 -0200 Subject: [Numpy-discussion] beginner question: rank-1 arrays In-Reply-To: <9c08f0d10711161155n30f67766r6842382f136a41d7@mail.gmail.com> References: <470B216B.5040808@gmx.net> <9c08f0d10711161155n30f67766r6842382f136a41d7@mail.gmail.com> Message-ID: <9c08f0d10711161229i37a37438vd34f14b259b8c6b4@mail.gmail.com> On Nov 16, 2007 5:55 PM, Emanuel Woiski wrote: > Sorry for coming very late to the thread, but you mean something like: > for i in range(len(cols): > > [sending too soon...] > > On Oct 9, 2007 4:36 AM, Sven Schreiber wrote: > > > Alan G Isaac schrieb: > > > On Mon, 8 Oct 2007, Robin apparently wrote: > > >> However in my code (I am converting from MATLAB) it is > > >> important to maintain 2d arrays, and keep the difference > > >> between row and column vectors. > > > > > > Well, I have noticed that numpy doesn't care very much about rows and cols. Mind you, if you slice along a col, you end up with a row - just try it and see. But how can you evaluate an expression such as a[i] - b[j], for all (i,j), with i for rows and j for cols? The trick here is 'newaxis' . With 'newaxis' you have a temporary dimension for a or b, without actually changing a or b shapes. Following the usual meaning, the expression becomes: c = a[:,newaxis] - b See: >>> a = arange(3.) >>> a array([ 0., 1., 2.]) >>> b = a # just an example... Now I want: >>> c1 = zeros((3,3)) >>> for i in range(3): for j in range (3): c1[i,j] = a[i] - b[j] >>> c1 array([[ 0., -1., -2.], [ 1., 0., -1.], [ 2., 1., 0.]]) That's exactly the same as the one-liner: >>> c2 = a[:,newaxis] - b >>> c2 array([[ 0., -1., -2.], [ 1., 0., -1.], [ 2., 1., 0.]]) Nice isn't it? Hope that help you somehow....:) cheers woiski -------------- next part -------------- An HTML attachment was scrubbed... URL: From rahulgarg44 at gmail.com Fri Nov 16 21:50:07 2007 From: rahulgarg44 at gmail.com (Rahul Garg) Date: Fri, 16 Nov 2007 19:50:07 -0700 Subject: [Numpy-discussion] numpy : your experiences? Message-ID: Hi. I have been a numpy user myself for some time now (though its my first message on this list). I am trying to do a informal survey for a univ related project .. I am a grad student btw working in comp sci. It would be awesome if you guys could respond to some of the following questions : a) Can you guys tell me briefly about the kind of problems you are tackling with numpy and scipy? b) Have you ever felt that numpy/scipy was slow and had to switch to C/C++/Fortran? c) Do you use any form of parallel processing? Multicores? SMPs? Clusters? If yes how did u utilize them? If you feel its not relevant to the list .. feel free to email me personally. I would be very interested in talking about these issues. thanks, rahul From peridot.faceted at gmail.com Sat Nov 17 02:07:34 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 17 Nov 2007 02:07:34 -0500 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: On 16/11/2007, Rahul Garg wrote: > It would be awesome if you guys could respond to some of the following > questions : > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? > b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? > c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? > > If you feel its not relevant to the list .. feel free to email me personally. > I would be very interested in talking about these issues. I think it would be interesting and on-topic to hear a few words from people to see what they do with numpy. a) I use python/numpy/scipy to work with astronomical observations of pulsars. This includes a range of tasks including: simple scripting to manage jobs on our computation cluster; minor calculations (like a better scientific calculator, though frink is sometimes better because it keeps track of units); gathering and plotting results; prototyping search algorithms and evaluating their statistical properties; providing tools for manipulating photon data; various other tasks. We also use the software package PRESTO for much of the heavy lifting; much of it is written in python. b) I have projects for which python is too slow, yes. Pulsar surveys are extremely compute-intensive (I estimated one that I'm involved with at two or three mega-core-hours), so any software that is going in the pipeline should have its programming-time/runtime tradeoff carefully examined. I wrote a search code in C, with bits in embedded assembler. All the driver code to shuffle files around and supply correct parameters is in python, though. PRESTO has a similar pattern, with more C code because it does more of the hard work. In most cases the communication between the heavy-lifting code and the python code is through the UNIX environment (the heavy-lifting code gets run as a separate executable) but PRESTO makes many functions available in python modules. On the other hand, I often write quick Monte Carlo simulations that run for a day or more, but since writing them takes about as long as running them, it's not worth writing them in a language that would run faster. c) Our problems tend to be embarrassingly parallel, so we tend not to use clever parallelism toolkits. For the survey I am working on, I wrote one (python) script to process a single beam on a single node (which takes about thirty hours), and another to keep the batch queue filled with an appropriate number of such jobs. I have thought about writing a more generic tool for managing this kind of job queue, but haven't invested the time yet. Anne From jtorrecilla at gmail.com Sat Nov 17 06:01:45 2007 From: jtorrecilla at gmail.com (Jesus Torrecilla Pinero) Date: Sat, 17 Nov 2007 12:01:45 +0100 Subject: [Numpy-discussion] segfault with dot Message-ID: <329bf94c0711170301t495baa08k77aae0eb28d97d37@mail.gmail.com> I am using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) under Windows XP and converting a program from Numeric to Numpy. If I have two arrays, say K and T and do dot(K,T) or K*T everything goes well, but if I have a vector b and try dot(K,b) or K*b I get a segfault. I have tried to run the test in tes_numeric.py and get the same result. I have tried too to convert b in a matrix with two columns, the second one being all zeroes, and numpy makes the multiplication correctly, so I think the problem is just with vectors Does this happens with other versions or do you have any idea to fix this problem? Thanks in advance -- Jes?s Torrecilla Pinero Universidad de Extremadura jtorrecilla at utcsl.com jtorreci at unex.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Sat Nov 17 06:05:46 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 17 Nov 2007 12:05:46 +0100 Subject: [Numpy-discussion] segfault with dot In-Reply-To: <329bf94c0711170301t495baa08k77aae0eb28d97d37@mail.gmail.com> References: <329bf94c0711170301t495baa08k77aae0eb28d97d37@mail.gmail.com> Message-ID: What CPU do you have and which version of numpy did you get and where did you get it from ? Matthieu 2007/11/17, Jesus Torrecilla Pinero : > > I am using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) under Windows > XP and converting a program from Numeric to Numpy. If I have two arrays, say > K and T and do dot(K,T) or K*T everything goes well, but if I have a vector > b and try dot(K,b) or K*b I get a segfault. > > I have tried to run the test in tes_numeric.py and get the same result. > > I have tried too to convert b in a matrix with two columns, the second one > being all zeroes, and numpy makes the multiplication correctly, so I think > the problem is just with vectors > > Does this happens with other versions or do you have any idea to fix this > problem? > > Thanks in advance > > -- > Jes?s Torrecilla Pinero > Universidad de Extremadura > jtorrecilla at utcsl.com > jtorreci at unex.es > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From Achim.Gaedke at physik.tu-darmstadt.de Sat Nov 17 06:55:57 2007 From: Achim.Gaedke at physik.tu-darmstadt.de (Achim Gaedke) Date: Sat, 17 Nov 2007 12:55:57 +0100 Subject: [Numpy-discussion] SegFault/double free with simple array mask operation In-Reply-To: <473BF42E.6020407@physik.tu-darmstadt.de> References: <473B3F0A.50601@physik.tu-darmstadt.de> <473BC19E.30505@ar.media.kyoto-u.ac.jp> <473BF42E.6020407@physik.tu-darmstadt.de> Message-ID: <473ED6CD.4020905@physik.tu-darmstadt.de> Achim Gaedke wrote: > David Cournapeau wrote: > >> Could you open a ticket on the numpy trac system ? (I can confirm the bug) >> >> cheers, >> >> David >> >> > It is Ticket #614 . The version information in trac are outdated, I > could not select version 1.0.3 or 1.0.4 . > Here is the solution for Segmentation Fault reported. It is basicly copied from the function iter_subscript_Bool, which alredy does the necessary range checks. Achim Index: arrayobject.c =================================================================== --- arrayobject.c (revision 4464) +++ arrayobject.c (working copy) @@ -9337,6 +9337,11 @@ return -1; } index = ind->dimensions[0]; + if (index > self->size) { + PyErr_SetString(PyExc_ValueError, + "too many boolean indices"); + return -1; + } strides = ind->strides[0]; dptr = ind->data; PyArray_ITER_RESET(self); From jtorrecilla at gmail.com Sat Nov 17 06:58:25 2007 From: jtorrecilla at gmail.com (Jesus Torrecilla Pinero) Date: Sat, 17 Nov 2007 12:58:25 +0100 Subject: [Numpy-discussion] segfault with dot In-Reply-To: References: <329bf94c0711170301t495baa08k77aae0eb28d97d37@mail.gmail.com> Message-ID: <329bf94c0711170358x7613f722kb0ff79555b89ffca@mail.gmail.com> numpy-1.0.4.win32-py2.5 from sourceforge (precompiled binary) CPU: AMD Athlon XP 2600+ 2.09 GHz, 1.00 Gb RAM The error report says: AppName: pythonw.exe AppVer: 0.0.0.0 ModName: _dotblas.pyd ModVer: 0.0.0.0 Offset: 0007ecf3 Jes?s Torrecilla Pinero -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Sat Nov 17 07:03:15 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 17 Nov 2007 13:03:15 +0100 Subject: [Numpy-discussion] segfault with dot In-Reply-To: <329bf94c0711170358x7613f722kb0ff79555b89ffca@mail.gmail.com> References: <329bf94c0711170301t495baa08k77aae0eb28d97d37@mail.gmail.com> <329bf94c0711170358x7613f722kb0ff79555b89ffca@mail.gmail.com> Message-ID: 2007/11/17, Jesus Torrecilla Pinero : > > numpy-1.0.4.win32-py2.5 from sourceforge (precompiled binary) > CPU: AMD Athlon XP 2600+ > 2.09 GHz, 1.00 Gb RAM > The error report says: > AppName: pythonw.exe AppVer: 0.0.0.0 ModName: _dotblas.pyd > ModVer: 0.0.0.0 Offset: 0007ecf3 OK, I think the problem is that you don't have the SSE2 instruction set and IIRC, this package needs it. David Cournapeau created a package that could solve your problem, but I don't remember the link, you can brows the archive for it ;) Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sat Nov 17 09:33:33 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 17 Nov 2007 16:33:33 +0200 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <20071117143333.GE16995@mentat.za.net> On Sat, Nov 17, 2007 at 02:07:34AM -0500, Anne Archibald wrote: > On 16/11/2007, Rahul Garg wrote: > > > It would be awesome if you guys could respond to some of the following > > questions : > > a) Can you guys tell me briefly about the kind of problems you are > > tackling with numpy and scipy? > > b) Have you ever felt that numpy/scipy was slow and had to switch to > > C/C++/Fortran? > > c) Do you use any form of parallel processing? Multicores? SMPs? > > Clusters? If yes how did u utilize them? > > > > If you feel its not relevant to the list .. feel free to email me personally. > > I would be very interested in talking about these issues. > > I think it would be interesting and on-topic to hear a few words from > people to see what they do with numpy. > > a) I use python/numpy/scipy to work with astronomical observations of > pulsars. This includes a range of tasks including: simple scripting to > manage jobs on our computation cluster; minor calculations (like a > better scientific calculator, though frink is sometimes better because > it keeps track of units); So does 'ipython -p physics': In [1]: x = 3 m/s^2 In [2]: y = 15 s In [3]: x*y Out[3]: 45 m/s Regards St?fan From lists.steve at arachnedesign.net Sat Nov 17 10:10:15 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Sat, 17 Nov 2007 10:10:15 -0500 Subject: [Numpy-discussion] [ANN] Release of the first PyTables video In-Reply-To: <20071114170800.GD30891@tardis.terramar.selidor.net> References: <20071114170800.GD30891@tardis.terramar.selidor.net> Message-ID: Hi, > ===================================== > Release of the first PyTables video > ===================================== > > `Carabos `_ is very proud to announce the > first of a series of videos dedicated to introducing the main features > of PyTables to the public in a visual and easy to grasp manner. I just got a chance to watch the video and wanted to thank you for putting that together. I've always been meaning to check out PyTables but haven't had the time to figure out how to work it on to potentially replace my hacked- together data storage schemes, so these videos are a great help. Looking forward to your next video! -steve From lists.steve at arachnedesign.net Sat Nov 17 10:28:05 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Sat, 17 Nov 2007 10:28:05 -0500 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: Hi Rahul, > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? I'm a grad student "doing" computational biology. I primarily use the NumPy/SciPy/matplotlib triumvirate as a post processing tool to analyze what the heck happened after we run some learning algorithms we develop (or canned ones, like libsvm (for example)) to look for some sense in the results. I've been working w/ analyzing interaction networks/graphs, so I also use NetworkX[1] quite a bit as well (it's also a nice package w/ responsive authors). Many of the folks (in my lab, and collaborators) like to use MATLAB, so I've found scipy's io.loadmat invaluable for making this a bit more seamless. So, in general, for me (so far) numpy/scipy are generally used to integrate various datasets together and see if things "look kosher" (before runs and after runs). > b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? Yes, for things like boosting, svm, graph mining, etc ... but that's no real surprise since their iterative and need to run on large datasets. You should also note that there are python interfaces to these things out there as well, but I (thus far) haven't taken much of advantage of those and usually pipe out data into the expected text input formats and pull them back in when the algo is done. > c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? I'd really like to (not just for Python), but I haven't. -steve [1] NetworkX: https://networkx.lanl.gov/wiki From srossross at gmail.com Sat Nov 17 03:37:25 2007 From: srossross at gmail.com (Sean Ross-Ross) Date: Sat, 17 Nov 2007 00:37:25 -0800 Subject: [Numpy-discussion] bug in numpy.apply_along_axis: numpy.__version__ '1.0.3.1' Message-ID: <2559AEE6-91F1-4C81-A315-08221CE04B35@gmail.com> Hi, I think I have found a bug in the function "apply_along_axis". recall that the function definition if apply_along_axis(func1d, axis, arr, *args) This bug occurs if func1d returns a python object without a __len__ attribute and is not a scalar, determined by numpy.core.numeric.isscalar. eg. a self contained example of the bug is: >>> from numpy import array, apply_along_axis >>> >>> # works >>> arr = array( [1,1] ) >>> apply_along_axis( lambda arr:arr[0], 0, arr ) >>> >>> # however this raises a type error >>> class Foo(object): pass >>> >>> arr = array( [ Foo(), Foo() ] ) >>> apply_along_axis( lambda arr:arr[0], 0, arr ) ------------------------------------------------------------------------ --- exceptions.TypeError Traceback (most recent call last) /sw/lib/python2.4/site-packages/numpy/lib/shape_base.py in apply_along_axis(func1d, axis, arr, *args) 52 holdshape = outshape 53 outshape = list(arr.shape) ---> 54 outshape[axis] = len(res) 55 outarr = zeros(outshape,asarray(res).dtype) 56 outarr[tuple(i.tolist())] = res TypeError: len() of unsized object >>> Thanks ~Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sat Nov 17 19:04:58 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 18 Nov 2007 02:04:58 +0200 Subject: [Numpy-discussion] Bug in arange dtype ">f" was: Using arr.dtype.type to check byteorder-independed dtype fails for bool In-Reply-To: References: Message-ID: <20071118000458.GD26080@mentat.za.net> On Tue, Nov 13, 2007 at 02:53:32PM +0100, Sebastian Haase wrote: > On Nov 13, 2007 2:18 PM, Stefan van der Walt wrote: > > Hi Sebastian > > > > On Tue, Nov 13, 2007 at 01:11:33PM +0100, Sebastian Haase wrote: > > > Hi, > > > I need to check the array dtype in a way that it is ignoring > > > differences coming only from big-endian vs. little-endian. > > > > Does > > > > N.issubdtype(first_dtype, second_dtype) > > > > work? > Hi St?fan, > > trying to anwer your question with a quick "arange" test, I ran into > more confusion: > >> a = N.arange(.5, dtype=">f") > >>> `a.dtype` > 'dtype('float32')' > >>> a = N.arange(.5, dtype=" >>> `a.dtype` > 'dtype('float32')' > > Both equal positively with N.float32 now ! > >>> N.__version__ > '1.0.1' This should now be fixed in SVN r4465. Regards St?fan From mforbes at physics.ubc.ca Sat Nov 17 21:19:04 2007 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Sat, 17 Nov 2007 18:19:04 -0800 Subject: [Numpy-discussion] Should array iterate over a set? Message-ID: <32CDED7E-3202-4F27-BFD0-A46879B19E41@physics.ubc.ca> My expectation was that array would iterate over a set. This is incorrect: >>> array(set([1,2,3])) array(set([1, 2, 3]), dtype=object) Is this the intended behaviour? A trivial work-around that does what I need is >>> array(list(set([1,2,3]))) array([1, 2, 3]) but I was wondering if this was by design or just a forgotten corner. (Maybe a vestige of the tuple special case for record arrays?) Michael. P.S. I just found that this was brought up by Ed Schofield on 2006-05-03, but there were no replies in that thread. From robert.kern at gmail.com Sat Nov 17 21:42:23 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 17 Nov 2007 20:42:23 -0600 Subject: [Numpy-discussion] Should array iterate over a set? In-Reply-To: <32CDED7E-3202-4F27-BFD0-A46879B19E41@physics.ubc.ca> References: <32CDED7E-3202-4F27-BFD0-A46879B19E41@physics.ubc.ca> Message-ID: <473FA68F.8060905@gmail.com> Michael McNeil Forbes wrote: > My expectation was that array would iterate over a set. This is > incorrect: > > >>> array(set([1,2,3])) > array(set([1, 2, 3]), dtype=object) > > Is this the intended behaviour? A trivial work-around that does what > I need is > > >>> array(list(set([1,2,3]))) > array([1, 2, 3]) > > but I was wondering if this was by design or just a forgotten > corner. (Maybe a vestige of the tuple special case for record arrays?) We can recognize most sequences (i.e. for all i in range(len(x)), x[i] responds correctly), but we cannot easily deal with arbitrary iterables which are not sequences in the array() function. There are a lot of special cases and heuristics going on in array() as it is. Instead, we have a fromiter() function which will take an iterable and construct an array from it. It is limited to 1D arrays, but this is by far the most common use for constructing an array from an iterable. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ivilata at carabos.com Sun Nov 18 11:51:56 2007 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Sun, 18 Nov 2007 17:51:56 +0100 Subject: [Numpy-discussion] [ANN] Release of the first PyTables video In-Reply-To: References: <20071114170800.GD30891@tardis.terramar.selidor.net> Message-ID: <20071118165156.GA17623@tardis.terramar.selidor.net> Steve Lianoglou (el 2007-11-17 a les 10:10:15 -0500) va dir:: > > `Carabos `_ is very proud to announce the > > first of a series of videos dedicated to introducing the main features > > of PyTables to the public in a visual and easy to grasp manner. > > I just got a chance to watch the video and wanted to thank you for > putting that together. Oh, it's a pleasure to hear that people enjoyed our video, specially since it's our vey first published attempt at screencasts. :) > I've always been meaning to check out PyTables but haven't had the > time to figure out how to work it on to potentially replace my hacked- > together data storage schemes, so these videos are a great help. Though we put a lot of effort in the written tutorials, we also thought that watching an interactive session would be an incomparable way of showing how it really feels to work with PyTables. It's also quite fun, so I recommend both developers and users to try and record and publish your own videos of your favorite tools! :) > Looking forward to your next video! We're currently working on it and it may get released in a couple of weeks or so. We'll probably only be announcing it in the PyTables lists and in our websites, to avoid flooding other lists with announces. Best regards, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 307 bytes Desc: Digital signature URL: From stefan at sun.ac.za Sun Nov 18 15:59:33 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 18 Nov 2007 22:59:33 +0200 Subject: [Numpy-discussion] SegFault/double free with simple array mask operation In-Reply-To: <473ED6CD.4020905@physik.tu-darmstadt.de> References: <473B3F0A.50601@physik.tu-darmstadt.de> <473BC19E.30505@ar.media.kyoto-u.ac.jp> <473BF42E.6020407@physik.tu-darmstadt.de> <473ED6CD.4020905@physik.tu-darmstadt.de> Message-ID: <20071118205932.GE26080@mentat.za.net> On Sat, Nov 17, 2007 at 12:55:57PM +0100, Achim Gaedke wrote: > Achim Gaedke wrote: > > David Cournapeau wrote: > > > >> Could you open a ticket on the numpy trac system ? (I can confirm the bug) > >> > >> cheers, > >> > >> David > >> > >> > > It is Ticket #614 . The version information in trac are outdated, I > > could not select version 1.0.3 or 1.0.4 . > > > Here is the solution for Segmentation Fault reported. > It is basicly copied from the function iter_subscript_Bool, which alredy > does the necessary range checks. Thanks, Achim. This is now fixed in SVN. Regards St?fan From efiring at hawaii.edu Sun Nov 18 16:18:10 2007 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 18 Nov 2007 11:18:10 -1000 Subject: [Numpy-discussion] SegFault/double free with simple array mask operation In-Reply-To: <20071118205932.GE26080@mentat.za.net> References: <473B3F0A.50601@physik.tu-darmstadt.de> <473BC19E.30505@ar.media.kyoto-u.ac.jp> <473BF42E.6020407@physik.tu-darmstadt.de> <473ED6CD.4020905@physik.tu-darmstadt.de> <20071118205932.GE26080@mentat.za.net> Message-ID: <4740AC12.7080705@hawaii.edu> Stefan, Ticket #607 should be closed now also. It looks like I can't do that, even though I created the ticket. I'm not sure whether it was the fix for #614 that did it, or whether it is the code it referred to, but now a proper exception is raised instead of a segfault. Eric Stefan van der Walt wrote: > On Sat, Nov 17, 2007 at 12:55:57PM +0100, Achim Gaedke wrote: >> Achim Gaedke wrote: >>> David Cournapeau wrote: >>> >>>> Could you open a ticket on the numpy trac system ? (I can confirm the bug) >>>> >>>> cheers, >>>> >>>> David >>>> >>>> >>> It is Ticket #614 . The version information in trac are outdated, I >>> could not select version 1.0.3 or 1.0.4 . >>> >> Here is the solution for Segmentation Fault reported. >> It is basicly copied from the function iter_subscript_Bool, which alredy >> does the necessary range checks. > > Thanks, Achim. This is now fixed in SVN. > > Regards > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From stefan at sun.ac.za Sun Nov 18 16:32:54 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 18 Nov 2007 23:32:54 +0200 Subject: [Numpy-discussion] SegFault/double free with simple array mask operation In-Reply-To: <4740AC12.7080705@hawaii.edu> References: <473B3F0A.50601@physik.tu-darmstadt.de> <473BC19E.30505@ar.media.kyoto-u.ac.jp> <473BF42E.6020407@physik.tu-darmstadt.de> <473ED6CD.4020905@physik.tu-darmstadt.de> <20071118205932.GE26080@mentat.za.net> <4740AC12.7080705@hawaii.edu> Message-ID: <20071118213253.GG26080@mentat.za.net> On Sun, Nov 18, 2007 at 11:18:10AM -1000, Eric Firing wrote: > Ticket #607 should be closed now also. It looks like I can't do that, > even though I created the ticket. > > I'm not sure whether it was the fix for #614 that did it, or whether it > is the code it referred to, but now a proper exception is raised instead > of a segfault. Thanks, Eric. I closed #607. Those reports referred to the same bug (indexing with too many booleans). Regards St?fan From millman at berkeley.edu Sun Nov 18 23:30:44 2007 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 18 Nov 2007 20:30:44 -0800 Subject: [Numpy-discussion] numpy distutils patch In-Reply-To: References: Message-ID: Hello, I never got any reply about the 'fix' for distutils.util.split_quoted in numpy/distutils/ccompiler.py. Can anyone confirm whether this fix is correct or necessary? If so, I would like to submit a patch upstream for this. Thanks, On Oct 29, 2007 2:17 AM, Jarrod Millman wrote: > Hey, > > I was looking at numpy/distutils/ccompiler.py and noticed that it has > a fix for distutils.util.split_quoted. > > Here is the relevant code from split_quoted in numpy.distutils.ccompiler: > ----------------------------------- > def split_quoted(s): > > > > if _has_white_re.search(s[beg+1:end-1]): > s = s[:beg] + s[beg+1:end-1] + s[end:] > pos = m.end() - 2 > else: > # Keeping quotes when a quoted word does not contain > # white-space. XXX: send a patch to distutils > pos = m.end() > > > ----------------------------------- > > Here is the relevant code from split_quoted in distutils.util: > ----------------------------------- > def split_quoted(s): > > > > s = s[:beg] + s[beg+1:end-1] + s[end:] > pos = m.end() - 2 > > > ----------------------------------- > > Does anyone know if a patch was ever submitted upstream? If not, is > there any reason that a patch shouldn't be submitted? > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From Chris.Barker at noaa.gov Mon Nov 19 17:55:37 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 19 Nov 2007 14:55:37 -0800 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <473DFCC2.1050704@mur.at> References: <473DFCC2.1050704@mur.at> Message-ID: <47421469.3050408@noaa.gov> Georg Holzmann wrote: > Because I had some troubles in wrapping my C++ library in python/numpy, > I did (another) numpy2carray.i file for SWIG. How is this better/different than numpy.i in: numpy/doc/swig/numpy.i > With that interface file it is possible to input/output arrays with or > without copying data numpy.i support that too. > (I also included an example for an interface to > fortran style arrays). That, it doesn't have. Just trying to keep too much effort from being duplicated.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rahulgarg44 at gmail.com Mon Nov 19 18:05:03 2007 From: rahulgarg44 at gmail.com (Rahul Garg) Date: Mon, 19 Nov 2007 16:05:03 -0700 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: hi. thanks for ur responses .. so it looks like python/numpy is used more for gluing things together or doing things like postprocessing. is anyone using it for core calculations .. as in long running python calculations? i used numpy myself for some nonlinear dynamics and chaos related calculations but they were usually very short running only for a few seconds at a time. thanks, rahul On Nov 17, 2007 8:28 AM, Steve Lianoglou wrote: > Hi Rahul, > > > a) Can you guys tell me briefly about the kind of problems you are > > tackling with numpy and scipy? > > I'm a grad student "doing" computational biology. I primarily use the > NumPy/SciPy/matplotlib triumvirate as a post processing tool to > analyze what the heck happened after we run some learning algorithms > we develop (or canned ones, like libsvm (for example)) to look for > some sense in the results. > > I've been working w/ analyzing interaction networks/graphs, so I also > use NetworkX[1] quite a bit as well (it's also a nice package w/ > responsive authors). > > Many of the folks (in my lab, and collaborators) like to use MATLAB, > so I've found scipy's io.loadmat invaluable for making this a bit more > seamless. > > So, in general, for me (so far) numpy/scipy are generally used to > integrate various datasets together and see if things "look > kosher" (before runs and after runs). > > > b) Have you ever felt that numpy/scipy was slow and had to switch to > > C/C++/Fortran? > > Yes, for things like boosting, svm, graph mining, etc ... but that's > no real surprise since their iterative and need to run on large > datasets. > > You should also note that there are python interfaces to these things > out there as well, but I (thus far) haven't taken much of advantage of > those and usually pipe out data into the expected text input formats > and pull them back in when the algo is done. > > > c) Do you use any form of parallel processing? Multicores? SMPs? > > Clusters? If yes how did u utilize them? > > I'd really like to (not just for Python), but I haven't. > > -steve > > [1] NetworkX: https://networkx.lanl.gov/wiki > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From fperez.net at gmail.com Mon Nov 19 19:19:02 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 19 Nov 2007 17:19:02 -0700 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: On Nov 19, 2007 4:05 PM, Rahul Garg wrote: > hi. > > thanks for ur responses .. so it looks like python/numpy is used more > for gluing things together or doing things like postprocessing. is > anyone using it for core calculations .. as in long running python > calculations? > i used numpy myself for some nonlinear dynamics and chaos related > calculations but they were usually very short running only for a few > seconds at a time. We use it for the core of long-running computations: http://dx.doi.org/10.1016/j.acha.2007.08.001 The innermost loops use numpy arrays, with some hand-coded C to do optimized dot-like operations without any error or type checking. The entire code is an amalgam of numpy, scipy, matplotlib, mayavi and pyx, with pyrex, some in-house Fortran wrapped with f2py, hand-coded C, and a bit of auto-generated C++ loaded at runtime with weave.inline(). While the times listed above are fairly short (small electrostatics examples), we are using this inside long-running quantum mechanics codes (papers in review now). Cheers, f From cookedm at physics.mcmaster.ca Mon Nov 19 21:09:06 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 19 Nov 2007 21:09:06 -0500 Subject: [Numpy-discussion] numpy distutils patch In-Reply-To: References: Message-ID: <059F8559-4A37-4BED-A000-2FF5C9DEFD13@physics.mcmaster.ca> On Nov 18, 2007, at 23:30 , Jarrod Millman wrote: > Hello, > > I never got any reply about the 'fix' for distutils.util.split_quoted > in numpy/distutils/ccompiler.py. Can anyone confirm whether this fix > is correct or necessary? If so, I would like to submit a patch > upstream for this. My opinion is that it's not necessary, or correct. The fix leaves quotes in if there is no whitespace, so '"Hi"' is converted to ['"Hi"'], while '"Hi there"' becomes ['Hi there']. I can't see when you'd want that behaviour. Also, it's only used by ccompiler (numpy.distutils.ccompiler replaces the version in distutils.ccompiler). numpy.distutils.fcompiler *doesn't* use this version, it uses distutils.utils.split_quoted. Since we run into more variety in terms of command lines with the Fortran compilers than the C compilers I think, and haven't been bitten by supposedly-bad quoting problems, I'll say we don't need our version. > On Oct 29, 2007 2:17 AM, Jarrod Millman wrote: >> Hey, >> >> I was looking at numpy/distutils/ccompiler.py and noticed that it has >> a fix for distutils.util.split_quoted. >> >> Here is the relevant code from split_quoted in >> numpy.distutils.ccompiler: >> ----------------------------------- >> def split_quoted(s): >> >> >> >> if _has_white_re.search(s[beg+1:end-1]): >> s = s[:beg] + s[beg+1:end-1] + s[end:] >> pos = m.end() - 2 >> else: >> # Keeping quotes when a quoted word does not contain >> # white-space. XXX: send a patch to distutils >> pos = m.end() >> >> >> ----------------------------------- >> >> Here is the relevant code from split_quoted in distutils.util: >> ----------------------------------- >> def split_quoted(s): >> >> >> >> s = s[:beg] + s[beg+1:end-1] + s[end:] >> pos = m.end() - 2 >> >> >> ----------------------------------- >> >> Does anyone know if a patch was ever submitted upstream? If not, is >> there any reason that a patch shouldn't be submitted? >> >> Thanks, >> >> -- >> Jarrod Millman >> Computational Infrastructure for Research Labs >> 10 Giannini Hall, UC Berkeley >> phone: 510.643.4014 >> http://cirl.berkeley.edu/ >> > > > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Oliver.Bock at barclaysglobal.com Mon Nov 19 21:42:30 2007 From: Oliver.Bock at barclaysglobal.com (Bock, Oliver BGI SYD) Date: Tue, 20 Nov 2007 13:42:30 +1100 Subject: [Numpy-discussion] Exception handling in ufuncs Message-ID: <1D2261C0DEDA024A8061E653AD9779880B681D@sydnte2k032.insidelive.net> Is it possible to trap non-floating point exceptions? For example, I store strings in ndarrays as objects, but I would like the result of invalid operations to be None. from numpy import * a = array(["a", "b"], dtype=object) b = array(["A", None], dtype=object) print a + b Traceback (most recent call last): File "C:\workspace\AlphaExplorer\ae1\test\_fiddle.py", line 225, in ? print a + b TypeError: cannot concatenate 'str' and 'NoneType' objects I have tried seterr() and seterrcall(), but these are not triggered (which did not surprise me because the documentation says they are used for floating point exceptions). Of course I can always write my own ufunc, but if a better way exists... def f(a, b): if a is None or b is None: return None return a + b print vectorize(f, otypes=[object])(a, b) -- This message and any attachments are confidential, proprietary, and may be privileged. If this message was misdirected, Barclays Global Investors (BGI) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of BGI, unless the author is authorized by BGI to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by BGI. Although BGI operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed. From listservs at mac.com Mon Nov 19 22:55:08 2007 From: listservs at mac.com (Chris) Date: Tue, 20 Nov 2007 03:55:08 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Building_installers_using_bdist=5Fmp?= =?utf-8?q?kg_on_Leopard?= Message-ID: There appears to be a bug in bdist_mpkg on Leopard: it requires an "admin" group that does not exist by default. The mpkg build dies with a ValueError saying that the group does not exist. Is it best just to add this group, or should the code be changed? From grh at mur.at Tue Nov 20 03:12:08 2007 From: grh at mur.at (Georg Holzmann) Date: Tue, 20 Nov 2007 09:12:08 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <47421469.3050408@noaa.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> Message-ID: <474296D8.9000405@mur.at> Hallo! > How is this better/different than numpy.i in: > > numpy/doc/swig/numpy.i The problem I had with numpy.i: - it copies the arrays on output (Argout Arrays) which was not possible for my algorithms (I have some very big matrices) - it is not possible to 2D or 3D Argout Arrays (why?), in the docs: ---8<--- Note that we support DATA_TYPE* argout typemaps in 1D, but not 2D or 3D. This because of a quirk with the SWIG typemap syntax and cannot be avoided. Note that for these types of 1D typemaps, the python function will take a single argument representing DIM1. ---8<--- - I also needed an output of fortran style arrays >> (I also included an example for an interface to >> fortran style arrays). > > That, it doesn't have. It has ;) ... look in numpy2carray.i, FARRAY2_OUT (line 175). But yes, sorry, I did no example in example.cpp ... > Just trying to keep too much effort from being duplicated.... Yes, of course. Actually I did not want to write that code ;). I was (and still am) a numpy beginner and needed to use my library in it - so all the people said it's so easy to do ... Then I tried boost.python and swig and I spent a lot of time trying to get that work, writing on maiiling lists ... - I did not think before that it would be that hard - and I don't understand why nobody else needed that before. Maybe this could also be integrated in numpy/doc/swig/numpy.i - but however, this is so much code and I didn't really understand how I should change that to my needs and also asked on the mailinglists ... OTOH the umfpack.i bindings in scipy where quite nice, much easier to read, but of course tuned to their task ... Therefore I made these wrappers, which should be easier to understand for beginners, how to write there own wrappers and tune them to their needs. LG Georg From schut at sarvision.nl Tue Nov 20 03:30:01 2007 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 20 Nov 2007 09:30:01 +0100 Subject: [Numpy-discussion] ndim argsort question Message-ID: <47429B09.40107@sarvision.nl> Hi, I'm getting a bit lost in numpy's fancy indexing, using argsort on a slice of a 4d arrays, and then using the resulting indices to sort other slices of that same array, preferably in one go. I have a data[d, b, y, x] array (for the curious: date, band, y, x: satellite image) Then I create a sorting index from band 0: idx = data[:, 0, :, :].argsort(axis=0) Now I want to sort all 7 bands accoring to this index, without looping, or at least avoid looping as far as possible. Something like sortedData = data[idx], but then for each 'b'. However, I cant get multidim indexing using the argsort result to work. Any hints? Hard to explain. I hope someone understands it, otherwise please let me know and I'll try to refrase :) Cheers, Vincent Schut. From charlesr.harris at gmail.com Tue Nov 20 03:35:31 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 20 Nov 2007 01:35:31 -0700 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: On Nov 19, 2007 4:05 PM, Rahul Garg wrote: > hi. > > thanks for ur responses .. so it looks like python/numpy is used more > for gluing things together or doing things like postprocessing. is > anyone using it for core calculations .. as in long running python > calculations? > i used numpy myself for some nonlinear dynamics and chaos related > calculations but they were usually very short running only for a few > seconds at a time. > It depends on what you mean by long running. A couple of hours seems long to me, others might think in terms of days and weeks. The longest running things I have done are Monte Carlo simulations and a genetic algorithm to find the best integer coefficients for a complex equal ripple filter starting with the floating point coefficients produced by a complex Remez algorithm. I have also used python and mixed numpy/C++ to program an image server to produce photometrically correct, blurred, widefield streaked star images using an indexed Tycho catalog. I mostly use python for quick and dirty programs, prototyping, and as an interface to C++ when speed is essential. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From David.L.Goldsmith at noaa.gov Tue Nov 20 03:42:20 2007 From: David.L.Goldsmith at noaa.gov (David.Goldsmith) Date: Tue, 20 Nov 2007 00:42:20 -0800 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <474296D8.9000405@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> Message-ID: <47429DEC.2040805@noaa.gov> Georg Holzmann wrote: >>> (I also included an example for an interface to >>> fortran style arrays). >>> >> That, it doesn't have. >> It has ;) ... look in numpy2carray.i, FARRAY2_OUT (line 175). >> But yes, sorry, I did no example in example.cpp ... >> I'm pretty sure Chris meant that numpy.i doesn't have it (and that consequently, your inclusion of it in your numpy2carray.i is quite welcome). :-) DG From matthieu.brucher at gmail.com Tue Nov 20 03:41:24 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 20 Nov 2007 09:41:24 +0100 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: > > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? Manifold learning, and thus unconstrianed optimizations b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? Not if it is done correctly (I just kept some piece of my old C++ code beacsue I didn't want to rewrite it) c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? Not for the moment, but I'm thinking about how to use it easily (free lunch) Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Tue Nov 20 03:47:30 2007 From: lbolla at gmail.com (lorenzo bolla) Date: Tue, 20 Nov 2007 09:47:30 +0100 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <80c99e790711200047s599fd3e9v26d4e0e7ccc34fa7@mail.gmail.com> > a) Can you guys tell me briefly about the kind of problems you are > > tackling with numpy and scipy? > > > Electromagnetic problems: eigenvalues finding, linear systems, > optimizations... > > b) Have you ever felt that numpy/scipy was slow and had to switch to > > C/C++/Fortran? > > > > I come from that world, and python gives me the programming speed I needed > with a more-than-reasonable executing speed. I mainly use python/numpy/scipy > as a better Matlab, which is platform independent and free, i.e. an > interface to numerical libraries like blas, lapack, fftw, umfpack, arpack > and so on (which are mainly written in Fortran/C). > > > c) Do you use any form of parallel processing? Multicores? SMPs? > > Clusters? If yes how did u utilize them? > > > I use MPI (mpi4py) on a shared memory multiprocessor SGI Altix. > Lorenzo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Nov 20 03:48:57 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 20 Nov 2007 01:48:57 -0700 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: On Nov 19, 2007 5:19 PM, Fernando Perez wrote: > On Nov 19, 2007 4:05 PM, Rahul Garg wrote: > > hi. > > > > thanks for ur responses .. so it looks like python/numpy is used more > > for gluing things together or doing things like postprocessing. is > > anyone using it for core calculations .. as in long running python > > calculations? > > i used numpy myself for some nonlinear dynamics and chaos related > > calculations but they were usually very short running only for a few > > seconds at a time. > > We use it for the core of long-running computations: > > http://dx.doi.org/10.1016/j.acha.2007.08.001 > Heh, *On subgroups of the Monster containing A5's* turns up as a related paper. You folks doing some fancy math there? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From schut at sarvision.nl Tue Nov 20 04:54:13 2007 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 20 Nov 2007 10:54:13 +0100 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <4742AEC5.4090906@sarvision.nl> Rahul Garg wrote: > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? mainly timeseries of Remote Sensing data ('satellite images') processing. No really fancy math, but huge (sometimes multiple gigabytes) multidimensional (date, bands, y, x: order of magnitude: [hundreds, some, tenthousands or more, idem]) datasets. Some tasks are long-running, not because of the complex math involved, but because of the amount of data. > b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? Yes, but usually the overhead of programming the whole thing in C is not worth the speedup of processing, especially with modern fast computers and easy ways to parallelize stuff (ipython and/or parallelpython). > c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? See above. Just started to use parallelpython here, currently not yet for production work, but in testing/prototyping phase. Using both smp, multicore and multiple machines ('cluster'). PP doesn't make any difference between them. How? Just cut the data in suitable pieces and throw them as a pp job to the cluster :-) It's really that simple nowadays. And most of our processing is very parallel in nature. Cheers, Vincent Schut. From schut at sarvision.nl Tue Nov 20 04:54:13 2007 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 20 Nov 2007 10:54:13 +0100 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <4742AEC5.4090906@sarvision.nl> Rahul Garg wrote: > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? mainly timeseries of Remote Sensing data ('satellite images') processing. No really fancy math, but huge (sometimes multiple gigabytes) multidimensional (date, bands, y, x: order of magnitude: [hundreds, some, tenthousands or more, idem]) datasets. Some tasks are long-running, not because of the complex math involved, but because of the amount of data. > b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? Yes, but usually the overhead of programming the whole thing in C is not worth the speedup of processing, especially with modern fast computers and easy ways to parallelize stuff (ipython and/or parallelpython). > c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? See above. Just started to use parallelpython here, currently not yet for production work, but in testing/prototyping phase. Using both smp, multicore and multiple machines ('cluster'). PP doesn't make any difference between them. How? Just cut the data in suitable pieces and throw them as a pp job to the cluster :-) It's really that simple nowadays. And most of our processing is very parallel in nature. Cheers, Vincent Schut. From wfspotz at sandia.gov Tue Nov 20 08:47:29 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Tue, 20 Nov 2007 06:47:29 -0700 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <474296D8.9000405@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> Message-ID: On Nov 20, 2007, at 1:12 AM, Georg Holzmann wrote: > The problem I had with numpy.i: > > - it copies the arrays on output (Argout Arrays) which was not > possible > for my algorithms (I have some very big matrices) Really? I worked pretty hard to avoid copies when they were not necessary. For the ARGOUT typemaps, I allocate an array of the requested size and then pass its data buffer to your function. If that is not what you want, then you should probably be using the INPLACE typemap. Do you have some different use case? If so, it would be far better for us (me) to add this to numpy.i, which covers more data types (especially dimension data types) and has a lot more error checking in it, rather than develop yet another numpy/swig interface file. > - it is not possible to 2D or 3D Argout Arrays (why?), in the docs: > ---8<--- > Note that we support DATA_TYPE* argout typemaps in 1D, but not 2D > or 3D. > This because of a quirk > with the SWIG typemap syntax and cannot be avoided. Note that for > these > types of 1D typemaps, the > python function will take a single argument representing DIM1. > ---8<--- I don't see where your numpy2carray.i file handles arrays greater than 1D either. Doing a search on "argout", the corresponding typemaps all apply to ttype*. > - I also needed an output of fortran style arrays This would be a good thing to add to numpy.i. > Yes, of course. Actually I did not want to write that code ;). > I was (and still am) a numpy beginner and needed to use my library > in it > - so all the people said it's so easy to do ... > Then I tried boost.python and swig and I spent a lot of time trying to > get that work, writing on maiiling lists ... - I did not think before > that it would be that hard - and I don't understand why nobody else > needed that before. I follow both the swig and numpy mail lists. I don't see any posts from you prior to 10/25/07. I probably just skimmed past your first posts because the subject referred to fortran, so I assumed you were dealing with fortran code. Sorry about that. > Maybe this could also be integrated in numpy/doc/swig/numpy.i - but > however, this is so much code and I didn't really understand how I > should change that to my needs and also asked on the mailinglists ... > OTOH the umfpack.i bindings in scipy where quite nice, much easier to > read, but of course tuned to their task ... The numpy.i file does get to be pretty hard to follow, largely because it uses nested macros to handle the large combination of cases. It is a question of how to handle all of the possible use cases. Even so, it would appear that you may have came up with one that wasn't covered, so you see my problem. > Therefore I made these wrappers, which should be easier to understand > for beginners, how to write there own wrappers and tune them to > their needs. In the end, though, I think it would be far better to add new capabilities to numpy.i rather than duplicate effort. This will never be as big as the Numeric/numarray split, but still.... Let me know what you are trying to do. If numpy.i can do it, I'll show you how. If not, I'll work to upgrade numpy.i. I suspect it can do everything except the fortran ordering. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From grh at mur.at Tue Nov 20 09:24:22 2007 From: grh at mur.at (Georg Holzmann) Date: Tue, 20 Nov 2007 15:24:22 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> Message-ID: <4742EE16.8030307@mur.at> Hallo! > Really? I worked pretty hard to avoid copies when they were not > necessary. For the ARGOUT typemaps, I allocate an array of the > requested size and then pass its data buffer to your function. If Yes but this means that you again allocate an array of the same size. E.g. in my algorithm I can have a very big internal matrix in C++ (say 700 MB - in fortran style). Now I want to have this matrix in numpy to plot some parts of it, get some data out of it ... whatever - if I again allocate an array of the same size, I am out of memory. Therefore I simply used the PyArray_FromDimsAndData() function to allocate the array. BTW: for me this also seems to be more the "numpy way", because usually numpy doesn't copy data. And if I want to copy this data I can simply call .copy() in python - however, there is of course the risk that this data is freed in C and you can have a crash ... so maybe this is no good idea for a general interface ... > that is not what you want, then you should probably be using the > INPLACE typemap. Yeah, but this results in the same as above ... > Do you have some different use case? If so, it would be far better > for us (me) to add this to numpy.i, which covers more data types > (especially dimension data types) and has a lot more error checking > in it, rather than develop yet another numpy/swig interface file. Yes I would very much appreciate that. My use case is (as described above) an ARGOUT 1D and/or 2D array in C and/or Fortran style storage without copying the data (and also not allocating it again). > I don't see where your numpy2carray.i file handles arrays greater > than 1D either. Doing a search on "argout", the corresponding > typemaps all apply to ttype*. Hm ... maybe this is a misunderstanding ? - I mean with 2D output a two dimensional numpy array (simply a matrix). In numpy2carray.i the macros are called ARRAY2_OUT and FARRAY2_OUT. > I follow both the swig and numpy mail lists. I don't see any posts > from you prior to 10/25/07. I probably just skimmed past your first > posts because the subject referred to fortran, so I assumed you were > dealing with fortran code. Sorry about that. I did not write to the swig list - I think it was the [C++-sig], but however, thats no longer important ... ;) > In the end, though, I think it would be far better to add new > capabilities to numpy.i rather than duplicate effort. This will of course ... > Let me know what you are trying to do. If numpy.i can do it, I'll > show you how. If not, I'll work to upgrade numpy.i. I suspect it > can do everything except the fortran ordering. Again, see above for my use case. But the fortran ordering should not be that hard (only setting the flags and strides right, as in FARRAY2_OUT in numpy2carray.i) - but of course someone has to do it ... ;) LG Georg From matthieu.brucher at gmail.com Tue Nov 20 09:31:43 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 20 Nov 2007 15:31:43 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <4742EE16.8030307@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> Message-ID: 2007/11/20, Georg Holzmann : > > Hallo! > > > Really? I worked pretty hard to avoid copies when they were not > > necessary. For the ARGOUT typemaps, I allocate an array of the > > requested size and then pass its data buffer to your function. If > > Yes but this means that you again allocate an array of the same size. Well, this is logical as you want a new argument in output. E.g. in my algorithm I can have a very big internal matrix in C++ (say > 700 MB - in fortran style). Now I want to have this matrix in numpy to > plot some parts of it, get some data out of it ... whatever - if I again > allocate an array of the same size, I am out of memory. > Therefore I simply used the PyArray_FromDimsAndData() function to > allocate the array. This is why you use INPLACE typemaps that will NOT copy your data. > that is not what you want, then you should probably be using the > > INPLACE typemap. > > Yeah, but this results in the same as above ... Are you sure ? Because the original object is not modified, so it is still the same data. If what you want is to provide a view from your C++ matrix, this is different. You must either : - propose the array interface - use a Python object inside your C++ matrix (this is to be done, I've a basic example in my blog) Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From lou_boog2000 at yahoo.com Tue Nov 20 09:33:11 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Tue, 20 Nov 2007 06:33:11 -0800 (PST) Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: Message-ID: <114769.63299.qm@web34415.mail.mud.yahoo.com> --- Rahul Garg wrote: > hi. > > thanks for ur responses .. so it looks like > python/numpy is used more > for gluing things together or doing things like > postprocessing. is > anyone using it for core calculations .. as in long > running python > calculations? > i used numpy myself for some nonlinear dynamics and > chaos related > calculations but they were usually very short > running only for a few > seconds at a time. > thanks, > rahul I've used Python a little to solve ODEs for chaotic systems. More for time series analysis (attractor reconstruction and associated data analysis problems). These ran rather fast on the order of seconds or minutes. Lately, I've been coding up a package to solved Schrodinger's Equation for 2D arbitrarily shaped, infinite wall potentials. I've settled on a Boundary Element Approach to get the eigenfunctions in these systems. The goal is to study phenomena associated with quantum chaos in the semiclassical regime. These calculations tend to run on the order of 10s of minutes to an hour. I eventually will be writing C extensions for the slower running functions (bottle necks). I've done that before and it's a big help for speed. -- Lou Pecora, my views are my own. ____________________________________________________________________________________ Be a better sports nut! Let your teams follow you with Yahoo Mobile. Try it now. http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ From grh at mur.at Tue Nov 20 09:56:44 2007 From: grh at mur.at (Georg Holzmann) Date: Tue, 20 Nov 2007 15:56:44 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> Message-ID: <4742F5AC.9090700@mur.at> Hallo! > E.g. in my algorithm I can have a very big internal matrix in C++ (say > 700 MB - in fortran style). Now I want to have this matrix in numpy to > plot some parts of it, get some data out of it ... whatever - if I again > allocate an array of the same size, I am out of memory. > Therefore I simply used the PyArray_FromDimsAndData() function to > allocate the array. > > > > This is why you use INPLACE typemaps that will NOT copy your data. [...] > Are you sure ? Because the original object is not modified, so it is > still the same data. Hm ... lets consider the same example as before (one 700MB matrix in C++). If I want to get this data with an INPLACE typemap I again have to allocate an 700 MB array in python, then passing it to my C++ library which puts in the data of it - so in the end I have to use two times 700 MB matrices ? (or maybe I don't understand something ;) ?) > If what you want is to provide a view from your C++ matrix, this is > different. You must either : > - propose the array interface > - use a Python object inside your C++ matrix (this is to be done, I've a > basic example in my blog) Yes, maybe thats what I need. Do you have a link to that blog ? LG GEorg From ndbecker2 at gmail.com Tue Nov 20 09:54:53 2007 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 20 Nov 2007 09:54:53 -0500 Subject: [Numpy-discussion] swig numpy2carray converters References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> Message-ID: Christopher Barker wrote: > > > Georg Holzmann wrote: >> Because I had some troubles in wrapping my C++ library in python/numpy, >> I did (another) numpy2carray.i file for SWIG. > > How is this better/different than numpy.i in: > > numpy/doc/swig/numpy.i > >> With that interface file it is possible to input/output arrays with or >> without copying data > > numpy.i support that too. > >> (I also included an example for an interface to >> fortran style arrays). > > That, it doesn't have. > > Just trying to keep too much effort from being duplicated.... > > -Chris > Is there any doc on numpy.i usage? From matthieu.brucher at gmail.com Tue Nov 20 10:05:13 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 20 Nov 2007 16:05:13 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <4742F5AC.9090700@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <4742F5AC.9090700@mur.at> Message-ID: > > > If what you want is to provide a view from your C++ matrix, this is > > different. You must either : > > - propose the array interface > > - use a Python object inside your C++ matrix (this is to be done, I've a > > basic example in my blog) > Of course : http://matt.eifelle.com/item/5 It's a basic version of the wrapper I use in my lab (pay attention to the constructor for instance), I hope you will be able to do something alike for your library. If it's not the case, you will have to fall back to the numpy way : allocating a new array, giving a pointer to the array, using it and stop using it after the function call is finished (with the wrapper I propose, you have the array for the life-time of your matrix instead). Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From grh at mur.at Tue Nov 20 11:21:16 2007 From: grh at mur.at (Georg Holzmann) Date: Tue, 20 Nov 2007 17:21:16 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <4742F5AC.9090700@mur.at> Message-ID: <4743097C.9060608@mur.at> Hallo! > Of course : http://matt.eifelle.com/item/5 > It's a basic version of the wrapper I use in my lab (pay attention to > the constructor for instance), I hope you will be able to do something Thanks ! But this assumes that the data in my C++ library is stored in a PyArrayObject ? This is of course not possible in my case - I need this library also in other situations, where I don't want to depend on python/numpy ... LG Georg From grh at mur.at Tue Nov 20 11:22:38 2007 From: grh at mur.at (Georg Holzmann) Date: Tue, 20 Nov 2007 17:22:38 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> Message-ID: <474309CE.7060600@mur.at> Hallo! > Is there any doc on numpy.i usage? yes there is a pdf in /numpy/doc/swig ! LG Georg From matthieu.brucher at gmail.com Tue Nov 20 11:38:10 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 20 Nov 2007 17:38:10 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <4743097C.9060608@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <4742F5AC.9090700@mur.at> <4743097C.9060608@mur.at> Message-ID: > > > Of course : http://matt.eifelle.com/item/5 > > It's a basic version of the wrapper I use in my lab (pay attention to > > the constructor for instance), I hope you will be able to do something > > Thanks ! > But this assumes that the data in my C++ library is stored in a > PyArrayObject ? Yes, but if your data storage is decoupled from the data use, you can replace the data storage by a Python one This is of course not possible in my case - I need this library also in > other situations, where I don't want to depend on python/numpy ... In this case, you use the default storage policy. If you only call functions and do not use classes (construct a class in Python and then call a method), this is an overkill. Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Tue Nov 20 12:10:36 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 20 Nov 2007 10:10:36 -0700 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: On Nov 20, 2007 1:48 AM, Charles R Harris wrote: > > > > On Nov 19, 2007 5:19 PM, Fernando Perez wrote: > > We use it for the core of long-running computations: > > > > http://dx.doi.org/10.1016/j.acha.2007.08.001 > > > > Heh, On subgroups of the Monster containing A5's turns up as a related > paper. You folks doing some fancy math there? Mmh, I'm not really sure why the Monster group paper turns up as 'related'. That kind of work sits squarely in the abstract algebra/group theory world, while what we do is far more applied. Think of it as a wavelet-inspired way of decomposing long-range potentials into near and far-field components to achieve better efficiency, along with tricks inspired from trapezoid rule integration to reduce the cost in higher dimensions. The first paper in the 'related' list, "Multiresolution separated representations of singular and weakly singular operators" is actually one that is really related, and has much of the theoretical background behind our work. It's actually nothing fancy at all from a theoretical standpoint, though rather complicated to implement (especially because speed comes from very aggressive truncations, so error analysis is difficult, critical and very easy to get wrong: you have to walk a fine line between speed and wrong results :). Cheers, f From zyzhu2000 at gmail.com Tue Nov 20 12:50:22 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Tue, 20 Nov 2007 11:50:22 -0600 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3D Surface Message-ID: Hi Everyone, This is off topic for this mailing list but I don't know where else to ask. I have N tabulated data points { (x_i, y_i, z_i) } that describes a 3D surface. The surface is pretty "smooth." However, the number of data points is too large to be stored and manipulated efficiently. To make it easier to deal with, I am looking for an easy method to compress and approximate the data. Maybe the approximation can be described by far fewer number of coefficients. If you can give me some hints about possible numpy or non-numpy solutions or let me know where is better to ask this kind of question, I would really appreciate it. Many thanks, Geoffrey From faltet at carabos.com Tue Nov 20 13:43:44 2007 From: faltet at carabos.com (Francesc Altet) Date: Tue, 20 Nov 2007 19:43:44 +0100 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3D Surface In-Reply-To: References: Message-ID: <200711201943.45543.faltet@carabos.com> A Tuesday 20 November 2007, Geoffrey Zhu escrigu?: > Hi Everyone, > > This is off topic for this mailing list but I don't know where else > to ask. > > I have N tabulated data points { (x_i, y_i, z_i) } that describes a > 3D surface. The surface is pretty "smooth." However, the number of > data points is too large to be stored and manipulated efficiently. To > make it easier to deal with, I am looking for an easy method to > compress and approximate the data. Maybe the approximation can be > described by far fewer number of coefficients. > > If you can give me some hints about possible numpy or non-numpy > solutions or let me know where is better to ask this kind of > question, I would really appreciate it. First, a good and easy try would be to use PyTables. It does support on-the-flight compression, that is, allows you to access compressed dataset slices without decompressing the complete dataset. This, in combination with a handy 'shuffle' filter (also included), allows for pretty good compression ratios on numerical data. See [1] [2] for a discussion on how to use and what you can expect from a compressor/shuffle process on PyTables. Also, if you can afford lossy compression, you may want to try truncation (quantization) before compressing as it does benefit the compression rate quite a lot. Feel free to experiment with the next function (Jeffrey Whittaker was the original author): def _quantize(data,least_significant_digit): """quantize data to improve compression. data is quantized using around(scale*data)/scale, where scale is 2**bits, and bits is determined from the least_significant_digit. For example, if least_significant_digit=1, bits will be 4.""" precision = 10.**-least_significant_digit exp = math.log(precision,10) if exp < 0: exp = int(math.floor(exp)) else: exp = int(math.ceil(exp)) bits = math.ceil(math.log(10.**-exp,2)) scale = 2.**bits return numpy.around(scale*data)/scale [1] http://www.pytables.org/docs/manual/ch05.html#compressionIssues [2] http://www.pytables.org/docs/manual/ch05.html#ShufflingOptim Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From wfspotz at sandia.gov Tue Nov 20 14:22:57 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Tue, 20 Nov 2007 12:22:57 -0700 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <4742EE16.8030307@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> Message-ID: <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> On Nov 20, 2007, at 7:24 AM, Georg Holzmann wrote: > Yes but this means that you again allocate an array of the same size. > E.g. in my algorithm I can have a very big internal matrix in C++ OK, so the key here is the *internal* matrix. I think you need to provide a way to extract that matrix from the C++ application as a numpy array. Then you can provide it to your function/method as an INPLACE array. No new memory will be allocated. >> that is not what you want, then you should probably be using the >> INPLACE typemap. > > Yeah, but this results in the same as above ... The INPLACE typemaps force you to provide the allocated memory. If you do it by accessing whatever your C++ app has already allocated, you should be fine. > Yes I would very much appreciate that. > My use case is (as described above) an ARGOUT 1D and/or 2D array in C > and/or Fortran style storage without copying the data (and also not > allocating it again). So assuming the INPLACE solution is sufficient for you, then the only new feature is fortran ordering. > Hm ... maybe this is a misunderstanding ? - I mean with 2D output a > two > dimensional numpy array (simply a matrix). > In numpy2carray.i the macros are called ARRAY2_OUT and FARRAY2_OUT. I guess my assumption was that in the general case you would want to retain the dimensions as input arguments, since it is logical for ARGOUT typemaps to allocate new memory. Since swig typemaps only allow numinputs=0 or numinputs=1, this was problematic. I guess the user could provide a sequence (tuple or list) as the single argument . . . don't know why I didn't think of that before. But I still think you want an INPLACE typemap. > Again, see above for my use case. > But the fortran ordering should not be that hard (only setting the > flags > and strides right, as in FARRAY2_OUT in numpy2carray.i) - but of > course > someone has to do it ... ;) Yes, it shouldn't be too hard. And I like your FARRAY notation for the interface. So your experience is that the flags and strides are all that have to be set differently? ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From David.L.Goldsmith at noaa.gov Tue Nov 20 15:28:08 2007 From: David.L.Goldsmith at noaa.gov (David.Goldsmith) Date: Tue, 20 Nov 2007 12:28:08 -0800 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> Message-ID: <47434358.4060901@noaa.gov> Bill Spotz wrote: >> Again, see above for my use case. >> But the fortran ordering should not be that hard (only setting the >> flags >> and strides right, as in FARRAY2_OUT in numpy2carray.i) - but of >> course >> someone has to do it ... ;) >> > > Yes, it shouldn't be too hard. And I like your FARRAY notation for > the interface. So your experience is that the flags and strides are > all that have to be set differently? > In my experience, so does the "zero-offset," (or is this controlled by flags in this context?) DG > ** Bill Spotz ** > ** Sandia National Laboratories Voice: (505)845-0170 ** > ** P.O. Box 5800 Fax: (505)284-0154 ** > ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From grh at mur.at Tue Nov 20 15:38:55 2007 From: grh at mur.at (Georg Holzmann) Date: Tue, 20 Nov 2007 21:38:55 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> Message-ID: <474345DF.8040202@mur.at> Hallo! > OK, so the key here is the *internal* matrix. I think you need to > provide a way to extract that matrix from the C++ application as a numpy > array. Then you can provide it to your function/method as an INPLACE > array. No new memory will be allocated. [...] > The INPLACE typemaps force you to provide the allocated memory. If you > do it by accessing whatever your C++ app has already allocated, you > should be fine. Hm ... I still don't understand that ... With INPLACE I have to allocate the numpy array before, then pass it to my getMyMatrix(my_inplace_numarray) method, then copy the data in that method to the my_inplace_numarray - right ? But I am still not convinced by the INPLACE method - the more natural way for me is to simply return the matrix ... Maybe the most usefull way would be to add also OUT_WITHOUTCOPY macros to numpy.i, where the data is allocated with PyArray_FromDimsAndData() ? (as a "long term" goal ...) >> Hm ... maybe this is a misunderstanding ? - I mean with 2D output a two >> dimensional numpy array (simply a matrix). >> In numpy2carray.i the macros are called ARRAY2_OUT and FARRAY2_OUT. > > I guess my assumption was that in the general case you would want to > retain the dimensions as input arguments, since it is logical for ARGOUT > typemaps to allocate new memory. Since swig typemaps only allow > numinputs=0 or numinputs=1, this was problematic. I guess the user > could provide a sequence (tuple or list) as the single argument . . . > don't know why I didn't think of that before. Again I do not see the problem - see e.g. ARRAY2_OUT_COPY in numpy2carray.i, shouldn't this be the same ? > Yes, it shouldn't be too hard. And I like your FARRAY notation for the > interface. So your experience is that the flags and strides are all > that have to be set differently? Yes, so far I had no crash ... ;) The only thing to do is the following: -------8<----------- PyArrayObject *tmp = (PyArrayObject*)obj; tmp->flags = NPY_FARRAY; int s = tmp->strides[1]; tmp->strides[0] = s; tmp->strides[1] = s * dim0[0]; -------8<----------- LG Georg From peridot.faceted at gmail.com Tue Nov 20 17:13:31 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 20 Nov 2007 17:13:31 -0500 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3D Surface In-Reply-To: References: Message-ID: On 20/11/2007, Geoffrey Zhu wrote: > I have N tabulated data points { (x_i, y_i, z_i) } that describes a 3D > surface. The surface is pretty "smooth." However, the number of data > points is too large to be stored and manipulated efficiently. To make > it easier to deal with, I am looking for an easy method to compress > and approximate the data. Maybe the approximation can be described by > far fewer number of coefficients. > > If you can give me some hints about possible numpy or non-numpy > solutions or let me know where is better to ask this kind of question, > I would really appreciate it. This is an important task in computer graphics, in particular, in the field of multiresolution modelling. If you look up "surface simplification" you'll find many references to articles. I don't know of a library offhand that does it, let alone one that is accessible from python, but you could try looking at a toolkit that does isosurface visualization - these are surfaces that can often be simplified enormously. In particular it looks like VTK might be able to do what you want. Anne From wfspotz at sandia.gov Tue Nov 20 17:20:11 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Tue, 20 Nov 2007 15:20:11 -0700 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <474345DF.8040202@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> Message-ID: <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> Here is what I am proposing you do: in your interface file, add something like PyObject * getMatrix() { npy_intp dims[2] = { /* Obtain the dimensions to your internal matrix */ }; double * data = /* Obtain the pointer to you internal matrix */; return PyArray_SimpleNewFromData(2, dims, NPY_DOUBLE, (void*) data); } For your function, use the INPLACE_ARRAY2 typemap: %apply (double* INPLACE_ARRAY2, int DIM1, int DIM2) {(double* matrix, int rows, int cols)}; void myFunction(double* matrix, int rows, int cols, double parameter, ...); And then in python you can do this: m = getMatrix() myFunction(m, 3.14, ...) However myFunction() alters your internal matrix, it will be reflected in the array m. What I don't understand is if double* matrix is *internal*, why is it in the argument list? Does myFunction() actually allocate the matrix? If so, then getMatrix() won't be able to get access to the matrix, but myFunction() is really a factory function or constructor. If this is true, then writing a simple function like getMatrix() as a SUBSTITUTE wrapper for myFunction() (i.e., that calls myFunction()) is MUCH, MUCH simpler than trying to maintain a general typemap that does the same thing. If my assumptions are wrong then you will need to explain to me exactly what myFunction() expects of its arguments, and why double* matrix is in the argument list at all. On Nov 20, 2007, at 1:38 PM, Georg Holzmann wrote: > Hallo! > >> OK, so the key here is the *internal* matrix. I think you need to >> provide a way to extract that matrix from the C++ application as a >> numpy array. Then you can provide it to your function/method as >> an INPLACE array. No new memory will be allocated. > [...] >> The INPLACE typemaps force you to provide the allocated memory. >> If you do it by accessing whatever your C++ app has already >> allocated, you should be fine. > > Hm ... I still don't understand that ... > With INPLACE I have to allocate the numpy array before, then pass > it to my getMyMatrix(my_inplace_numarray) method, then copy the > data in that method to the my_inplace_numarray - right ? No. INPLACE means "in place". The typemap expects that the array has already been allocated -- either you allocated something new or you obtained the array from existing data -- it doesn't care. It passes the pointer to the buffer to the function, which can alter the data IN PLACE, and you will see these changes to your array after the wrapper returns control to python. > But I am still not convinced by the INPLACE method - the more > natural way for me is to simply return the matrix ... This is only natural or makes sense if the function allocates the data for the matrix. Does it? If so, then multiple calls to it ensures duplicate allocations, which you indicate is not possible. If not, then the function expects the data to be allocated elsewhere. > Maybe the most usefull way would be to add also OUT_WITHOUTCOPY > macros to numpy.i, where the data is allocated with > PyArray_FromDimsAndData() ? > (as a "long term" goal ...) > Again I do not see the problem - see e.g. ARRAY2_OUT_COPY in > numpy2carray.i, shouldn't this be the same ? I do not understand the use case for which that typemap works. You create "ttype t1", then pass its address to the function as the pointer to the data. This has to be useless to the function. After the function returns, you access this pointer as if it points to meaningful data. How is this possible? The address of t1 isn't going to change, and only a single element is allocated. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From Chris.Barker at noaa.gov Tue Nov 20 19:41:03 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 20 Nov 2007 16:41:03 -0800 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3D Surface In-Reply-To: References: Message-ID: <47437E9F.2000102@noaa.gov> Anne Archibald wrote: >In particular it looks like VTK might be able > to do what you want. yes, it can. The way I've seen is to triangulate the surface, then do "decimation" -- remove the triangles that don't add "much" to the detail of the surface. I don't know of a way to do something similar keeping the Cartesian grid -- but you may not need to. Google "python triangle decimation" and VTK comes out on top, but there are some other things there that might be of interest too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Tue Nov 20 19:44:16 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 20 Nov 2007 16:44:16 -0800 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> Message-ID: <47437F60.1030008@noaa.gov> I'm a bit confused too. What would be great is a simple trimmed down example -- a small-as-you-can-make-it class with a method that shows what you need, perhaps with a little C++ sample that uses it. Then we can see how best to wrap it for python. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Tue Nov 20 19:45:40 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 20 Nov 2007 16:45:40 -0800 Subject: [Numpy-discussion] Building installers using bdist_mpkg on Leopard In-Reply-To: References: Message-ID: <47437FB4.2070403@noaa.gov> this looks like a question for the pythonmac list -- have you tried there? -Chris Chris wrote: > There appears to be a bug in bdist_mpkg on Leopard: it requires > an "admin" group that does not exist by default. The mpkg build > dies with a ValueError saying that the group does not exist. Is it > best just to add this group, or should the code be changed? > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Tue Nov 20 20:00:56 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 20 Nov 2007 17:00:56 -0800 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <47437F60.1030008@noaa.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <47437F60.1030008@noaa.gov> Message-ID: <47438348.5000509@noaa.gov> Christopher Barker wrote: > What would be great is a simple trimmed down example -- .. and then we'd have an example to put in the numpy.i docs and examples, too. By the way Bill, I haven't forgotten the examples I said I'd add to the docs. I've been distracted away from my SWIG work lately, but really hope to get back to it. I also had trouble being able to commit to SVN, but I'll address that when I have something worth committing. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From David.L.Goldsmith at noaa.gov Tue Nov 20 21:52:45 2007 From: David.L.Goldsmith at noaa.gov (David.Goldsmith) Date: Tue, 20 Nov 2007 18:52:45 -0800 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <47437F60.1030008@noaa.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <47437F60.1030008@noaa.gov> Message-ID: <47439D7D.1060905@noaa.gov> Chris, just to be clear, this is addressed to the OP, correct? DG Christopher Barker wrote: > I'm a bit confused too. > > What would be great is a simple trimmed down example -- a > small-as-you-can-make-it class with a method that shows what you need, > perhaps with a little C++ sample that uses it. Then we can see how best > to wrap it for python. > > -Chris > > > > From millman at berkeley.edu Wed Nov 21 01:31:54 2007 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 20 Nov 2007 22:31:54 -0800 Subject: [Numpy-discussion] numpy distutils patch In-Reply-To: <059F8559-4A37-4BED-A000-2FF5C9DEFD13@physics.mcmaster.ca> References: <059F8559-4A37-4BED-A000-2FF5C9DEFD13@physics.mcmaster.ca> Message-ID: On Nov 19, 2007 6:09 PM, David M. Cooke wrote: > My opinion is that it's not necessary, or correct. The fix leaves > quotes in if there is no whitespace, so '"Hi"' is converted to > ['"Hi"'], while '"Hi there"' becomes ['Hi there']. I can't see when > you'd want that behaviour. > > Also, it's only used by ccompiler (numpy.distutils.ccompiler replaces > the version in distutils.ccompiler). numpy.distutils.fcompiler > *doesn't* use this version, it uses distutils.utils.split_quoted. > Since we run into more variety in terms of command lines with the > Fortran compilers than the C compilers I think, and haven't been > bitten by supposedly-bad quoting problems, I'll say we don't need our > version. I think you are right, so I went ahead and removed the code: http://scipy.org/scipy/numpy/changeset/4481 Both NumPy and SciPy build just fine on my system. If anyone has any problems, please let me know ASAP. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From cimrman3 at ntc.zcu.cz Wed Nov 21 04:10:48 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 21 Nov 2007 10:10:48 +0100 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <4743F618.2040103@ntc.zcu.cz> Rahul Garg wrote: > It would be awesome if you guys could respond to some of the following > questions : > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? I am using both numpy and scipy to solve PDEs in the context of finite element method (elasticity, porous media, ...). > b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? I use bits of C code wrapped by SWIG either to address real bottle-necks (FE assembling, element matrix computations) or due to the fact that I have lots of "legacy" C code that I reuse from my previous projects. > c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? If I ever get myself to start, I will use petsc4py. > If you feel its not relevant to the list .. feel free to email me personally. > I would be very interested in talking about these issues. It is really interesting to see what others do. r. From karol.langner at kn.pl Wed Nov 21 04:27:33 2007 From: karol.langner at kn.pl (Karol Langner) Date: Wed, 21 Nov 2007 10:27:33 +0100 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <200711211027.33103.karol.langner@kn.pl> On Saturday 17 November 2007 03:50, Rahul Garg wrote: > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? Organizing jobs in computational chemistry, and parsing/analyzing the output. To some extent, also some actual calculations and visualization. > b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? Almost always when ab initio calculations are concerned. > c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? Not with Python. Cheers, Karol -- written by Karol Langner Wed Nov 21 10:25:22 CET 2007 From nadavh at visionsense.com Wed Nov 21 04:43:24 2007 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 21 Nov 2007 11:43:24 +0200 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3DSurface Message-ID: <710F2847B0018641891D9A21602763600B6E39@ex3.envision.co.il> Wouldn't a random or regular subsampling of the set will do the job? For data interpolation: 2D-Delaunay triangulation based method (I think you can find one in the scipy cookbook). Nadav. -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of Geoffrey Zhu Sent: Tue 20-Nov-07 19:50 To: Discussion of Numerical Python Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3DSurface Hi Everyone, This is off topic for this mailing list but I don't know where else to ask. I have N tabulated data points { (x_i, y_i, z_i) } that describes a 3D surface. The surface is pretty "smooth." However, the number of data points is too large to be stored and manipulated efficiently. To make it easier to deal with, I am looking for an easy method to compress and approximate the data. Maybe the approximation can be described by far fewer number of coefficients. If you can give me some hints about possible numpy or non-numpy solutions or let me know where is better to ask this kind of question, I would really appreciate it. Many thanks, Geoffrey _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From bernhard.voigt at gmail.com Wed Nov 21 05:27:35 2007 From: bernhard.voigt at gmail.com (bernhard.voigt at gmail.com) Date: Wed, 21 Nov 2007 02:27:35 -0800 (PST) Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? I'm using python with numpy,scipy, pytables and matplotlib for data analysis in the field of high energy particle physics. Most of the work is histograming millions of events, fitting functions to the distributions or applying cuts to yield optimized signal/background ratios. I often use the random number and optimization facilities for these purposes. Most of my colleagues use ROOT (root.cern.ch) which has also a python binding, however, I love the simplicity of numpy's ufuncs and indexing capabilities, which makes the code much denser and readable. Another part is developing toy simulation and reconstruction algorithms in python which later will be translated to C++ since the main software framework in our collaboration is written in C++. This includes log-likelihood track reconstruction algorithms based on the arrival times of photons measured with photomultipliers. Simulation involves particle generators and detector response simulations to test the reconstruction with known inputs. > b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? In particular for the simulation yes, depending on the level of detail of course. But only parts, eg. random number generation for certain distributions had to be coded in C/C++. Since the main software for my work is coded in C++, I often end up writing wrappers around parts of this software to extract the data I need for doing the analysis work in python. > c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? We have a cluster at our lab which I use for my computations. This is not very difficult since the data can be split into several files and each can be treated in the same way. One just needs to pass a script over and over again to the cluster, this is done in a shell script or with the tools provided by the cluster scheduling system. Cheers! Bernhard From Chris.Barker at noaa.gov Wed Nov 21 11:32:11 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 21 Nov 2007 08:32:11 -0800 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <47439D7D.1060905@noaa.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <47437F60.1030008@noaa.gov> <47439D7D.1060905@noaa.gov> Message-ID: <47445D8B.30907@noaa.gov> David.Goldsmith wrote: > Chris, just to be clear, this is addressed to the OP, correct? yes, but if anyone else want to come up with one, that would work too. -Chris >> What would be great is a simple trimmed down example -- a >> small-as-you-can-make-it class with a method that shows what you need, >> perhaps with a little C++ sample that uses it. Then we can see how best >> to wrap it for python. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Wed Nov 21 11:40:38 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 21 Nov 2007 08:40:38 -0800 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3DSurface In-Reply-To: <710F2847B0018641891D9A21602763600B6E39@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763600B6E39@ex3.envision.co.il> Message-ID: <47445F86.2090608@noaa.gov> Nadav Horesh wrote: > Wouldn't a random or regular subsampling of the set will do the job? > I have N tabulated data points { (x_i, y_i, z_i) } that describes a 3D > surface. The surface is pretty "smooth." If it's equally "smooth" everywhere, then yes, a subsampling would work fine, but I'm guessing the OP wants something smarter than that. > For data interpolation: 2D-Delaunay triangulation based method (I think you can find one in the scipy cookbook). yup -- but then you need the decimation to remove the "unneeded" points. I don't think Scipy has that. the GNU Triangulated Surface Library: http://gts.sourceforge.net/ should do what you want, but I don't know of any Python bindings -- you may be able to write some to the routines you need without too much pain. CGAL may have something too, and it does have Python bindings. http://cgal-python.gforge.inria.fr/ -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ellisonbg.net at gmail.com Wed Nov 21 11:55:34 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 21 Nov 2007 09:55:34 -0700 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: <114769.63299.qm@web34415.mail.mud.yahoo.com> References: <114769.63299.qm@web34415.mail.mud.yahoo.com> Message-ID: <6ce0ac130711210855j4dfe417cm895d88b2ff8684bb@mail.gmail.com> On Nov 20, 2007 7:33 AM, Lou Pecora wrote: > Lately, I've been coding up a package to solved > Schrodinger's Equation for 2D arbitrarily shaped, > infinite wall potentials. I've settled on a Boundary > Element Approach to get the eigenfunctions in these > systems. The goal is to study phenomena associated > with quantum chaos in the semiclassical regime. These > calculations tend to run on the order of 10s of > minutes to an hour. I eventually will be writing C > extensions for the slower running functions (bottle > necks). I've done that before and it's a big help for > speed. Very nice. Any plans to release that code? Brian > > > > -- Lou Pecora, my views are my own. > > > ____________________________________________________________________________________ > Be a better sports nut! Let your teams follow you > with Yahoo Mobile. Try it now. http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From grh at mur.at Wed Nov 21 12:07:58 2007 From: grh at mur.at (Georg Holzmann) Date: Wed, 21 Nov 2007 18:07:58 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> Message-ID: <474465EE.20305@mur.at> Hallo! As chris said, I need to make an example: http://grh.mur.at/software/numpy2carray.tar.gz I added the following class-example: class_example.h: the C++ code class_example.i: the SWIG interface file class_example_usage.py: example usage in python And some comments: Bill Spotz schrieb: > Here is what I am proposing you do: in your interface file, add > something like > > PyObject * getMatrix() > { > npy_intp dims[2] = { /* Obtain the dimensions to your internal > matrix */ }; > double * data = /* Obtain the pointer to you internal matrix */; > return PyArray_SimpleNewFromData(2, dims, NPY_DOUBLE, (void*)data); > } > > For your function, use the INPLACE_ARRAY2 typemap: > > %apply (double* INPLACE_ARRAY2, int DIM1, int DIM2) {(double* > matrix, int rows, int cols)}; > void myFunction(double* matrix, int rows, int cols, double > parameter, ...); > > And then in python you can do this: > > m = getMatrix() > myFunction(m, 3.14, ...) I see ... but what I mean with output of a matrix without copy is actually only your getMatrix() funtion I guess - this is basically what getBigData() does in class_example.h together with class_example.i. BTW: what is the difference between PyArray_SimpleNewFromData() and PyArray_FromDimsAndData() ? (I don't have this book ...) >> Again I do not see the problem - see e.g. ARRAY2_OUT_COPY in >> numpy2carray.i, shouldn't this be the same ? > > I do not understand the use case for which that typemap works. You > create "ttype t1", then pass its address to the function as the pointer > to the data. This has to be useless to the function. After the > function returns, you access this pointer as if it points to meaningful > data. How is this possible? The address of t1 isn't going to change, > and only a single element is allocated. See also the class_example_usage.py - there I used the ARRAY2_OUT_COPY for the getBigDataCopy() method. LG Georg From Chris.Barker at noaa.gov Wed Nov 21 12:32:19 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 21 Nov 2007 09:32:19 -0800 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <474465EE.20305@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <474465EE.20305@mur.at> Message-ID: <47446BA3.9070103@noaa.gov> Georg Holzmann wrote: > As chris said, I need to make an example: > http://grh.mur.at/software/numpy2carray.tar.gz Ah, I see now: /// @return internal big data without copying void getBigData(double **mtx, int *rows, int *cols) { *rows = drows; *cols = dcols; *mtx = very_big_data; } This is the one we've been talking about, correct? So you need to pass a pointer to a pointer in, and it gets set to the existing pointer -- thus no new data allocation. It's going to take Bill to figure out how to extend numpy.i to cover this case, but imagine it will be pretty straightforward. Thanks for the examples. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From strawman at astraw.com Wed Nov 21 14:40:24 2007 From: strawman at astraw.com (Andrew Straw) Date: Wed, 21 Nov 2007 11:40:24 -0800 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3DSurface In-Reply-To: <47445F86.2090608@noaa.gov> References: <710F2847B0018641891D9A21602763600B6E39@ex3.envision.co.il> <47445F86.2090608@noaa.gov> Message-ID: <474489A8.4080309@astraw.com> Christopher Barker wrote: >> For data interpolation: 2D-Delaunay triangulation based method (I think you can find one in the scipy cookbook). > > yup -- but then you need the decimation to remove the "unneeded" > points. I don't think Scipy has that. The sandbox does, thanks to Robert Kern. (And I should really submit a patch to move this into the main scipy.) http://www.scipy.org/Cookbook/Matplotlib/Gridding_irregularly_spaced_data From bobl at tricity.wsu.edu Wed Nov 21 14:41:35 2007 From: bobl at tricity.wsu.edu (Bob Lewis) Date: Wed, 21 Nov 2007 11:41:35 -0800 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3D Surface In-Reply-To: References: Message-ID: <474489EF.1020805@tricity.wsu.edu> On 11/20/07, Anne Archibald posted: > Subject: > Re: [Numpy-discussion] OT: A Way to Approximate and Compress a 3D Surface > From: > "Anne Archibald" > Date: > Tue, 20 Nov 2007 17:13:31 -0500 > To: > "Discussion of Numerical Python" > > To: > "Discussion of Numerical Python" > > > On 20/11/2007, Geoffrey Zhu wrote: > >> I have N tabulated data points { (x_i, y_i, z_i) } that describes a 3D >> surface. The surface is pretty "smooth." However, the number of data >> points is too large to be stored and manipulated efficiently. To make >> it easier to deal with, I am looking for an easy method to compress >> and approximate the data. Maybe the approximation can be described by >> far fewer number of coefficients. >> >> If you can give me some hints about possible numpy or non-numpy >> solutions or let me know where is better to ask this kind of question, >> I would really appreciate it. > > This is an important task in computer graphics, in particular, in the > field of multiresolution modelling. If you look up "surface > simplification" you'll find many references to articles. I don't know > of a library offhand that does it, let alone one that is accessible > from python, but you could try looking at a toolkit that does > isosurface visualization - these are surfaces that can often be > simplified enormously. In particular it looks like VTK might be able > to do what you want. Anne Archibald is correct that surface simplification may ultimately be of great help. Other place to look besides VTK are GTS, the GNU Triangulated Surface library, and CGAL, the Computational Geometry Algorithms Library, which has a Python binding. It occurs to me, though, that we should first verify that you do indeed have a surface in the first place. All you tell us is that you have a set of N points in 3-space. Are they connected? That is, does each point have well-defined neighbors? If so, do these vertices and connections form a mesh that defines a surface? - Bob Lewis From pecora at anvil.nrl.navy.mil Wed Nov 21 14:43:18 2007 From: pecora at anvil.nrl.navy.mil (Lou Pecora) Date: Wed, 21 Nov 2007 14:43:18 -0500 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: <6ce0ac130711210855j4dfe417cm895d88b2ff8684bb@mail.gmail.com> References: <114769.63299.qm@web34415.mail.mud.yahoo.com> <6ce0ac130711210855j4dfe417cm895d88b2ff8684bb@mail.gmail.com> Message-ID: <47448A56.40004@anvil.nrl.navy.mil> Brian Granger wrote: > On Nov 20, 2007 7:33 AM, Lou Pecora wrote: > > >> Lately, I've been coding up a package to solved >> Schrodinger's Equation for 2D arbitrarily shaped, >> infinite wall potentials. I've settled on a Boundary >> Element Approach to get the eigenfunctions in these >> systems. The goal is to study phenomena associated >> with quantum chaos in the semiclassical regime. These >> calculations tend to run on the order of 10s of >> minutes to an hour. I eventually will be writing C >> extensions for the slower running functions (bottle >> necks). I've done that before and it's a big help for >> speed. >> > > Very nice. Any plans to release that code? > > Brian > > No, not now. I'm just trying to get it up and running. Getting it ready for any release is way down the road. But if/when that happens, I would consider releasing the code. -- Cheers, Lou Pecora Code 6362 Naval Research Lab Washington, DC 20375, USA Ph: +202-767-6002 email: pecora at anvil.nrl.navy.mil From robert.kern at gmail.com Wed Nov 21 15:12:26 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 21 Nov 2007 14:12:26 -0600 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3DSurface In-Reply-To: <474489A8.4080309@astraw.com> References: <710F2847B0018641891D9A21602763600B6E39@ex3.envision.co.il> <47445F86.2090608@noaa.gov> <474489A8.4080309@astraw.com> Message-ID: <4744912A.8090102@gmail.com> Andrew Straw wrote: > Christopher Barker wrote: >>> For data interpolation: 2D-Delaunay triangulation based method (I think you can find one in the scipy cookbook). >> yup -- but then you need the decimation to remove the "unneeded" >> points. I don't think Scipy has that. > > The sandbox does, thanks to Robert Kern. (And I should really submit a > patch to move this into the main scipy.) > http://www.scipy.org/Cookbook/Matplotlib/Gridding_irregularly_spaced_data No, this does interpolation. It does not do surface simplification. Also, I'm moving the package over to scikits, instead. http://projects.scipy.org/scipy/scikits/browser/trunk/delaunay -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From curiousjan at gmail.com Wed Nov 21 17:37:31 2007 From: curiousjan at gmail.com (Jan Strube) Date: Wed, 21 Nov 2007 14:37:31 -0800 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <8CDECCD8-C7A6-43F8-A62C-7F279661E8CC@gmail.com> > >> a) Can you guys tell me briefly about the kind of problems you are >> tackling with numpy and scipy? > > I'm using python with numpy,scipy, pytables and matplotlib for data > analysis in the field of high energy particle physics. Most of the > work is histograming millions of events, fitting functions to the > distributions or applying cuts to yield optimized signal/background > ratios. I often use the random number and optimization facilities for > these purposes. > Cool! Me, too! In addition I am using Paida (http:// paida.sourceforge.net) for histogramming. I am interfacing that with matplotlib for the plotting, though. > Most of my colleagues use ROOT (root.cern.ch) which has also a python > binding, however, I love the simplicity of numpy's ufuncs and indexing > capabilities, which makes the code much denser and readable. In addition to more reliable in my experience. > In particular for the simulation yes, depending on the level of detail > of course. But only parts, eg. random number generation for certain > distributions had to be coded in C/C++. > Are you saying you extended the scipy/numpy tools for this? Do you think it would make sense to put some of that stuff on the wiki? > > Cheers! Bernhard Cheers, Jan From wfspotz at sandia.gov Wed Nov 21 18:54:40 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Wed, 21 Nov 2007 16:54:40 -0700 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <474465EE.20305@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <474465EE.20305@mur.at> Message-ID: On Nov 21, 2007, at 10:07 AM, Georg Holzmann wrote: > BTW: what is the difference between PyArray_SimpleNewFromData() and > PyArray_FromDimsAndData() ? > (I don't have this book ...) PyArray_SimpleNewFromData() is the new version and PyArray_FromDimsAndData() is the old version :-) Travis may be able to better explain. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From bernhard.voigt at gmail.com Thu Nov 22 05:21:37 2007 From: bernhard.voigt at gmail.com (bernhard.voigt at gmail.com) Date: Thu, 22 Nov 2007 02:21:37 -0800 (PST) Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: <8CDECCD8-C7A6-43F8-A62C-7F279661E8CC@gmail.com> References: <8CDECCD8-C7A6-43F8-A62C-7F279661E8CC@gmail.com> Message-ID: <9cad82db-8f27-405a-8d4d-ad892041be79@v4g2000hsf.googlegroups.com> > > In particular for the simulation yes, depending on the level of detail > > of course. But only parts, eg. random number generation for certain > > distributions had to be coded in C/C++. > > Are you saying you extended the scipy/numpy tools for this? > Do you think it would make sense to put some of that stuff on the wiki? No, this is very special to my application and not really numpy specific. I had to write a Metropolis-Hastings sampler, which worked in python but was too slow. I've coded this for a specific distribution in C++ and pass numpy arrays and python lists from and to C++ functions using boost::python. Bernhard From ptenconi at eco.uninsubria.it Thu Nov 22 09:13:08 2007 From: ptenconi at eco.uninsubria.it (Paolo Tenconi) Date: Thu, 22 Nov 2007 15:13:08 +0100 Subject: [Numpy-discussion] Numpy matrix and Blitz Matrix Message-ID: Hi Everyone, *) numpy arrays are automatically converted to blitz arrays. That's fine. *) I need to work with blitz Matrix objects, but I noticed that numpy matrix objects are not converted to them and I get a compilation error too 3) Does someone on the list succeeded doing that? Or is there a workaround to create quickly a blitz matrix object from a numpy 2D array or numpy-matrix without wasting time in casting operations? It would be nice having an automatic translation of numpy.matrix to blitz Matrix as matrices are used frequently in scientific computing. Any suggestion or help would be very glad. Paolo #------------------ EXAMPLE CODE ----------------------------------- import numpy import scipy #Dot product with arrays (it works) x=numpy.array([[1,2,],[3,4]]) y=numpy.zeros((2,2)) scipy.weave.inline("""y=x*x;""",['x','y'],type_converters=weave.converters.blitz,compiler='gcc',force=1) #Matrix multiplication with matrix (gives compilation error) X=numpy.matlib.mat('1 2; 3 4') Y=numpy.matlib.zeros((2,2)) scipy.weave.inline("""Y=X*X;""",['X','Y'],type_converters=weave.converters.blitz,compiler='gcc') #-------------------------------------------------------------------- From matthieu.brucher at gmail.com Thu Nov 22 13:02:17 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 22 Nov 2007 19:02:17 +0100 Subject: [Numpy-discussion] Adding ACML support Message-ID: Hi, I'm trying to get numpy work with AMD's MKL. The problem is that they not give a CBLAS interface, only the BLAS one. Can the numpy distutils check the existence of the cblas interface somewhere ? For the moment, it checks something with BLAS (but no actual testing takes place), but not CBLAS. Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Thu Nov 22 23:14:07 2007 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 22 Nov 2007 23:14:07 -0500 Subject: [Numpy-discussion] loadtxt bug? Message-ID: In numpy.core.numeric.py you will find loadtxt, which uses the following:: line = line[:line.find(comments)].strip() I believe there is a bug here (when a line has no comment). To illustrate:: >>> line = "12345" >>> comments = "#" >>> line[:line.find(comments)] '1234' So I propose this should be:: try: line = line[:line.index(comments)].strip() except ValueError: line = line.strip() Am I right? Cheers, Alan Isaac From gael.varoquaux at normalesup.org Fri Nov 23 02:14:19 2007 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 23 Nov 2007 08:14:19 +0100 Subject: [Numpy-discussion] loadtxt bug? In-Reply-To: References: Message-ID: <20071123071419.GA31662@clipper.ens.fr> On Thu, Nov 22, 2007 at 11:14:07PM -0500, Alan G Isaac wrote: > In numpy.core.numeric.py you will find loadtxt, which uses > the following:: > line = line[:line.find(comments)].strip() > I believe there is a bug here (when a line has no comment). > To illustrate:: > >>> line = "12345" > >>> comments = "#" > >>> line[:line.find(comments)] > '1234' Unless you are sure that line always ends with a "\n". This is the case in the code you are refering too. Unless I am missing some thing. Ga?l From konrad.hinsen at laposte.net Fri Nov 23 02:45:57 2007 From: konrad.hinsen at laposte.net (Konrad Hinsen) Date: Fri, 23 Nov 2007 08:45:57 +0100 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <647D88E3-9C72-4BE8-977A-E716A1D28237@laposte.net> On 17.11.2007, at 03:50, Rahul Garg wrote: > It would be awesome if you guys could respond to some of the following > questions : > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? For me, NumPy is an important building block in a set of computational Python libraries that form the basis of my daily work in molecular simulations. The main building block is my Molecular Modelling Toolkit (http://dirac.cnrs-orleans.fr/MMTK/), but NumPy provides the basic data structure (arrays) for much of what MMTK does, and even more so for interfacing with the rest of the world. I don't use SciPy at all. > b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? There is some C code (and an increasing amount of Pyrex code) in my Python environment, and it is essential for good performance. However, in terms of code quantity, it is negligible. > c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? All of them. I use threading (multicores and SMPs) in MMTK, and coarse-grained parallelization as implemented in ScientificPython for analyzing large data sets. ScientificPython has two parallel computing modules. The easiest to use implements a master-slave model in which a master process delegates computational tasks to an arbitrary (and possibly varying) number of slave processes: http://dirac.cnrs-orleans.fr/hg/ScientificPython/main/file/ 73cc270217fc/Examples/master_slave_demo.py The other parallelization package is based on the BSP model of parallel computing: http://dirac.cnrs-orleans.fr/hg/ScientificPython/main/file/ 73cc270217fc/Examples/BSP/ http://dirac.cnrs-orleans.fr/ScientificPython/ScientificPythonManual/ It probably has a steeper learning curve, but it is suitable for more complex parallel programs because it permits the construction of parallel libraries. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Centre de Biophysique Mol?culaire, CNRS Orl?ans Synchrotron Soleil - Division Exp?riences Saint Aubin - BP 48 91192 Gif sur Yvette Cedex, France Tel. +33-1 69 35 97 15 E-Mail: hinsen at cnrs-orleans.fr --------------------------------------------------------------------- From gael.varoquaux at normalesup.org Fri Nov 23 03:07:16 2007 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 23 Nov 2007 09:07:16 +0100 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <20071123080716.GD31662@clipper.ens.fr> On Fri, Nov 16, 2007 at 07:50:07PM -0700, Rahul Garg wrote: > It would be awesome if you guys could respond to some of the following > questions : OK, I'll take a bit of time to do this. > a) Can you guys tell me briefly about the kind of problems you are > tackling with numpy and scipy? My day-to-day use is data processing of an atomic physics experiment (Bose Einstein condensation). The data created by the experiment is retrived from a bunch of instruments (scopes, cameras, and sensors) and stored in an HDF5 file for each run, along with the parameters of the run. This is done in Matlab for historical reasons, and MatLab is an absolute pain for this. I have implemented a similar system in Python for another experiment (I wrote an article sumerizing the lessons learnt and the patterns to follow, hell this was the fourth experiment control system I built, each time using a different platform www.gael-varoquaux.info/computers/agile_computer_control_of_an_experiment.pdf ). The data is first stored on a windows box that runs the experiment, then it is downloaded by our Linux server that keeps a hash table of all the experiments ran and some figures of merit. The data processing is done on the Linux box (quite often connected remotely using Nomachine's NX). The data is accessed in a database-like way through a custom Python module (1000 LOC). Recently I have switched to a object-relationnal-mapping-like loading of the HDF5-files (hierarchical data trees) through a database-like interface. It rocks, IMHO. We do simple fitting, blurring, filter, averaging, fft, ... on the data, and plot the results. Another of my uses is for simple toy models. Recently I have been elaborating a statistical model of an experiment, so I was running Monte Carlo simulations, but I have ran dynamical model integrations, and linear algebra calculations always revolving around the physics of cold atomic gases. > b) Have you ever felt that numpy/scipy was slow and had to switch to > C/C++/Fortran? Never. However I really like the fact that I can if I need to. When controling instruments in Python I often have to use their C SDK. In which case I use Pyrex or Ctypes. > c) Do you use any form of parallel processing? Multicores? SMPs? > Clusters? If yes how did u utilize them? No. Most often I am limited by IO. I have the feeling that unlike most people on the list, my main work is not on a computer, but on an experiment. Ga?l From cournape at gmail.com Fri Nov 23 03:14:13 2007 From: cournape at gmail.com (David Cournapeau) Date: Fri, 23 Nov 2007 17:14:13 +0900 Subject: [Numpy-discussion] Adding ACML support In-Reply-To: References: Message-ID: <5b8d13220711230014x3cf9be8dxc90a8ef47b175b2a@mail.gmail.com> On Nov 23, 2007 3:02 AM, Matthieu Brucher wrote: > Hi, > > I'm trying to get numpy work with AMD's MKL. The problem is that they not > give a CBLAS interface, only the BLAS one. > Can the numpy distutils check the existence of the cblas interface somewhere > ? For the moment, it checks something with BLAS (but no actual testing takes > place), but not CBLAS. > My understanding, but that's only from having read the distutils sources, is that you don't check for cblas. You implement a acml checker, like MKL, but you have to define NO_ATLAS_INFO to 1 to say that cblas is not available (if you read the setup.py of numpy.core, which needs cblas and not blas, you will see that's how it is decide whether _dotblas should be built or not). David From haase at msg.ucsf.edu Fri Nov 23 03:25:37 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 23 Nov 2007 09:25:37 +0100 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() Message-ID: Hi, this question might habe been answered before: I have my own ndarray-derived class. I did this, so that I can add another "custom attribute" -- let's say arr.filename All works very except, except when I do arr2 = arr.transpose() my arr2 is still of the *type* of my derived class, however trying a arr2.filename now gives me an AtttributeError. What am I doing wrong? How can this be fixed? (I'm using numpy 1.0.1 - please let me know if this has been fixed since) Thanks, Sebastian Haase From yves.revaz at obspm.fr Fri Nov 23 04:19:06 2007 From: yves.revaz at obspm.fr (Yves Revaz) Date: Fri, 23 Nov 2007 10:19:06 +0100 Subject: [Numpy-discussion] numpy : your experiences? In-Reply-To: References: Message-ID: <47469B0A.1030302@obspm.fr> Rahul Garg wrote: >a) Can you guys tell me briefly about the kind of problems you are >tackling with numpy and scipy? > > Reduction of large N-body simulations of astrophysical gravitational systems (N up to 268 millions). See http://aramis.obspm.fr/~revaz/pNbody/. >b) Have you ever felt that numpy/scipy was slow and had to switch to >C/C++/Fortran? > > Some specific functions are missing in numpy and a pure python implementation is too slow. In that case, I implement the specific function in C, as a python module. >c) Do you use any form of parallel processing? Multicores? SMPs? >Clusters? If yes how did u utilize them? > > > > Yes, my toolbox is now nearly completely parallelized with mpi4py. Some parallelization parts are also implemented in C modules. It works fine on beowulf clusters, multicores or smps. -- (o o) --------------------------------------------oOO--(_)--OOo------- Yves Revaz Lerma Batiment A Tel : ++ 33 (0) 1 40 51 20 79 Observatoire de Paris Fax : ++ 33 (0) 1 40 51 20 02 77 av Denfert-Rochereau e-mail : yves.revaz at obspm.fr F-75014 Paris Web : http://obswww.unige.ch/~revaz/ FRANCE ---------------------------------------------------------------- From matthieu.brucher at gmail.com Fri Nov 23 04:24:07 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 23 Nov 2007 10:24:07 +0100 Subject: [Numpy-discussion] Patch for numpy.i (not C89 compliant) Message-ID: Hi, I submitted a ticket and then a patch for numpy.i (http://projects.scipy.org/scipy/numpy/ticket/620). the problem is that some typemaps use a C99 syntax whereas most C compiler still are only C89 compliant (mainly Visual Studio). Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher From aisaac at american.edu Fri Nov 23 07:58:13 2007 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 23 Nov 2007 07:58:13 -0500 Subject: [Numpy-discussion] loadtxt bug? In-Reply-To: <20071123071419.GA31662@clipper.ens.fr> References: <20071123071419.GA31662@clipper.ens.fr> Message-ID: > On Thu, Nov 22, 2007 at 11:14:07PM -0500, Alan G Isaac wrote: >> In numpy.core.numeric.py you will find loadtxt, which uses >> the following:: >> line = line[:line.find(comments)].strip() On Fri, 23 Nov 2007, Gael Varoquaux apparently wrote: > Unless you are sure that line always ends with a "\n". This is the case > in the code you are refering too. Unless I am missing some thing. I am not understanding your comment here. In the line of code above, is there a problem or not? Specifically, is it not the case that the last line of a text file is not guaranteed to have a terminator? E.g., Does this not raise the possibility that a digit will be clipped from the last line? Of course strip will afterwards strip any line terminators. Cheers, Alan Isaac From gael.varoquaux at normalesup.org Fri Nov 23 08:51:25 2007 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 23 Nov 2007 14:51:25 +0100 Subject: [Numpy-discussion] loadtxt bug? In-Reply-To: References: <20071123071419.GA31662@clipper.ens.fr> Message-ID: <20071123135125.GC12028@clipper.ens.fr> On Fri, Nov 23, 2007 at 07:58:13AM -0500, Alan G Isaac wrote: > > On Thu, Nov 22, 2007 at 11:14:07PM -0500, Alan G Isaac wrote: > >> In numpy.core.numeric.py you will find loadtxt, which uses > >> the following:: > >> line = line[:line.find(comments)].strip() > On Fri, 23 Nov 2007, Gael Varoquaux apparently wrote: > > Unless you are sure that line always ends with a "\n". This is the case > > in the code you are refering too. Unless I am missing some thing. > I am not understanding your comment here. In the line of > code above, is there a problem or not? I don't know. > Specifically, is it not the case that the last line of > a text file is not guaranteed to have a terminator? This is what I do not know, and I would appreciate to be enlightened. > Does this not raise the possibility that a digit will be > clipped from the last line? Yes. If the line does not end by a terminator you have a problem. Sorry if I am adding more confusion than help, I was just raising another question, or rather a comment on your question, rather than giving an answer. Ga?l From pgmdevlist at gmail.com Fri Nov 23 09:37:29 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 23 Nov 2007 09:37:29 -0500 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() In-Reply-To: References: Message-ID: <200711230937.29219.pgmdevlist@gmail.com> On Friday 23 November 2007 03:25:37 Sebastian Haase wrote: > Hi, > this question might habe been answered before: > I have my own ndarray-derived class. I did this, so that I can add > another "custom attribute" -- let's say > arr.filename Sebastian, Could you post the __new__ and __array_finalize__ for your class ? Are you sure you update the attribute in __array_finalize__ ? And this page may come handy: http://www.scipy.org/Subclasses From stefan at sun.ac.za Fri Nov 23 10:01:11 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 23 Nov 2007 17:01:11 +0200 Subject: [Numpy-discussion] Patch for numpy.i (not C89 compliant) In-Reply-To: References: Message-ID: <20071123150111.GA28925@mentat.za.net> On Fri, Nov 23, 2007 at 10:24:07AM +0100, Matthieu Brucher wrote: > I submitted a ticket and then a patch for numpy.i > (http://projects.scipy.org/scipy/numpy/ticket/620). the problem is > that some typemaps use a C99 syntax whereas most C compiler still are > only C89 compliant (mainly Visual Studio). Thanks, Matthieu. It is fixed in SVN. Regards St?fan From stefan at sun.ac.za Fri Nov 23 10:03:42 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 23 Nov 2007 17:03:42 +0200 Subject: [Numpy-discussion] freebsd patch Message-ID: <20071123150342.GB28925@mentat.za.net> Hi all, Ticket 618 [1] proposes a patch to make numpy pass all tests on FreeBSD. Would anyone familiar with FreeBSD please glance over the patch, apply it and make sure that it works as intended? Thanks St?fan [1] http://scipy.org/scipy/numpy/ticket/618 From stefan at sun.ac.za Fri Nov 23 10:05:54 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 23 Nov 2007 17:05:54 +0200 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() In-Reply-To: References: Message-ID: <20071123150554.GC28925@mentat.za.net> Hi Sebastian On Fri, Nov 23, 2007 at 09:25:37AM +0100, Sebastian Haase wrote: > Hi, > this question might habe been answered before: > I have my own ndarray-derived class. I did this, so that I can add > another "custom attribute" -- let's say > arr.filename > All works very except, except when I do > arr2 = arr.transpose() > my arr2 is still of the *type* of my derived class, however trying a > arr2.filename > now gives me an AtttributeError. > > What am I doing wrong? How can this be fixed? > (I'm using numpy 1.0.1 - please let me know if this has been fixed > since) Show us the code, then we can take a look. I assume you have read http://www.scipy.org/Subclasses Regards St?fan From wfspotz at sandia.gov Fri Nov 23 10:42:37 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Fri, 23 Nov 2007 08:42:37 -0700 Subject: [Numpy-discussion] Patch for numpy.i (not C89 compliant) In-Reply-To: References: Message-ID: <13749E99-031B-46DB-90E8-483B235DE178@sandia.gov> Thanks, Matthieu. On Nov 23, 2007, at 2:24 AM, Matthieu Brucher wrote: > Hi, > > I submitted a ticket and then a patch for numpy.i > (http://projects.scipy.org/scipy/numpy/ticket/620). the problem is > that some typemaps use a C99 syntax whereas most C compiler still are > only C89 compliant (mainly Visual Studio). > > > Matthieu > -- > French PhD student > Website : http://miles.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/? > blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From haase at msg.ucsf.edu Fri Nov 23 10:43:12 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 23 Nov 2007 16:43:12 +0100 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() In-Reply-To: <200711230937.29219.pgmdevlist@gmail.com> References: <200711230937.29219.pgmdevlist@gmail.com> Message-ID: On Nov 23, 2007 3:37 PM, Pierre GM wrote: > On Friday 23 November 2007 03:25:37 Sebastian Haase wrote: > > Hi, > > this question might habe been answered before: > > I have my own ndarray-derived class. I did this, so that I can add > > another "custom attribute" -- let's say > > arr.filename > > Sebastian, > Could you post the __new__ and __array_finalize__ for your class ? Are you > sure you update the attribute in __array_finalize__ ? > And this page may come handy: > http://www.scipy.org/Subclasses Aah - I guess I did not do my homework ;-) Here is my code: class ndarray_inMrcFile(N.memmap): pass data = self.data data.__class__ = ndarray_inMrcFile Two points: 1.) Please don't get distracted by the way how I change the class type "after the fact". I get the data as a memmaped slice from a file. Then I do some (potentially not entirely clean) acrobatic to have this slice change it's class-type so that I can attach the original memmap (+ other attributes, such as filename) onto the ndarray 2.) I guess the """class ndarray_inMrcFile(N.memmap): pass""" construct is to simplistic .... Could someone suggest a *minimal* definition, so that my attributes would be preserved ? A few comments about the wiki page: 1) The example does not show the printed info, such as "__new__ received %s", in the example session 2) I don't understand, why in the example the "info" attribute is set in "__new__()", even though the text above says: """However, we need to keep in mind that any attribute that we define in the __new__ method will be shared among all the instances. If we want instance-specific attributes, we still need some specific initialization. We cannot use the __init__ method, as it won't be called. That's where __array_finalize__ comes to play.""" 3) the text about "The __array_finalize__ method" should at least once say that it is defined as "def __array_finalize__(self,obj)" -- otherwise I could only guess where the "obj" comes from. 4) related to the text after: "In other terms, __array_finalize__ is called" How do I know if a function or method actually invokes __new__ ;-) ? Would I have to study the numpy source ? 5) For the "__array_finalize__ method" there a text that says: """Subclasses inherit a default implementation of this method that does nothing""" --- but how about "__new__()" : what happens if you don't define that (either) ? (Hope these comments are helpful) Thanks for the help, Sebastian From haase at msg.ucsf.edu Fri Nov 23 11:01:22 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 23 Nov 2007 17:01:22 +0100 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() In-Reply-To: References: <200711230937.29219.pgmdevlist@gmail.com> Message-ID: On Nov 23, 2007 4:43 PM, Sebastian Haase wrote: > On Nov 23, 2007 3:37 PM, Pierre GM wrote: > > On Friday 23 November 2007 03:25:37 Sebastian Haase wrote: > > > Hi, > > > this question might habe been answered before: > > > I have my own ndarray-derived class. I did this, so that I can add > > > another "custom attribute" -- let's say > > > arr.filename > > > > Sebastian, > > Could you post the __new__ and __array_finalize__ for your class ? Are you > > sure you update the attribute in __array_finalize__ ? > > And this page may come handy: > > http://www.scipy.org/Subclasses > > Aah - I guess I did not do my homework ;-) > Here is my code: > class ndarray_inMrcFile(N.memmap): > pass > > data = self.data > data.__class__ = ndarray_inMrcFile > > Two points: > 1.) Please don't get distracted by the way how I change the class type > "after the fact". I get the data as a memmaped slice from a file. Then > I do some (potentially not entirely clean) acrobatic to have this > slice change it's class-type so that I can attach the original memmap > (+ other attributes, such as filename) onto the ndarray > > 2.) I guess the """class ndarray_inMrcFile(N.memmap): > pass""" construct is to simplistic .... > Could someone suggest a *minimal* definition, so that my attributes > would be preserved ? > > > A few comments about the wiki page: > 1) The example does not show the printed info, such as "__new__ > received %s", in the example session > > 2) I don't understand, why in the example the "info" attribute is set > in "__new__()", even though the text above says: > """However, we need to keep in mind that any attribute that we define > in the __new__ method will be shared among all the instances. If we > want instance-specific attributes, we still need some specific > initialization. We cannot use the __init__ method, as it won't be > called. That's where __array_finalize__ comes to play.""" > > 3) the text about "The __array_finalize__ method" should at least > once say that it is defined as "def __array_finalize__(self,obj)" -- > otherwise I could only guess where the "obj" comes from. > > 4) related to the text after: "In other terms, __array_finalize__ is called" > How do I know if a function or method actually invokes __new__ ;-) > ? Would I have to study the numpy source ? > > 5) For the "__array_finalize__ method" there a text that says: > """Subclasses inherit a default implementation of this method that > does nothing""" --- but how about "__new__()" : what happens if you > don't define that (either) ? > > (Hope these comments are helpful) > > Thanks for the help, > Sebastian > First try seems to show that just changing my class def to: class ndarray_inMrcFile(N.memmap): def __array_finalize__(self,obj): self.Mrc = getattr(obj, 'Mrc', None) Seems to add the wanted attribute back into result of transpose(). However now I get (many!) exceptions like: Exception exceptions.AttributeError: "'ndarray_inMrcFile' object has no attribut e '_mmap'" in References: <200711230937.29219.pgmdevlist@gmail.com> Message-ID: On Nov 23, 2007 5:01 PM, Sebastian Haase wrote: > > On Nov 23, 2007 4:43 PM, Sebastian Haase wrote: > > On Nov 23, 2007 3:37 PM, Pierre GM wrote: > > > On Friday 23 November 2007 03:25:37 Sebastian Haase wrote: > > > > Hi, > > > > this question might habe been answered before: > > > > I have my own ndarray-derived class. I did this, so that I can add > > > > another "custom attribute" -- let's say > > > > arr.filename > > > > > > Sebastian, > > > Could you post the __new__ and __array_finalize__ for your class ? Are you > > > sure you update the attribute in __array_finalize__ ? > > > And this page may come handy: > > > http://www.scipy.org/Subclasses > > > > Aah - I guess I did not do my homework ;-) > > Here is my code: > > class ndarray_inMrcFile(N.memmap): > > pass > > > > data = self.data > > data.__class__ = ndarray_inMrcFile > > > > Two points: > > 1.) Please don't get distracted by the way how I change the class type > > "after the fact". I get the data as a memmaped slice from a file. Then > > I do some (potentially not entirely clean) acrobatic to have this > > slice change it's class-type so that I can attach the original memmap > > (+ other attributes, such as filename) onto the ndarray > > > > 2.) I guess the """class ndarray_inMrcFile(N.memmap): > > pass""" construct is to simplistic .... > > Could someone suggest a *minimal* definition, so that my attributes > > would be preserved ? > > > > > > A few comments about the wiki page: > > 1) The example does not show the printed info, such as "__new__ > > received %s", in the example session > > > > 2) I don't understand, why in the example the "info" attribute is set > > in "__new__()", even though the text above says: > > """However, we need to keep in mind that any attribute that we define > > in the __new__ method will be shared among all the instances. If we > > want instance-specific attributes, we still need some specific > > initialization. We cannot use the __init__ method, as it won't be > > called. That's where __array_finalize__ comes to play.""" > > > > 3) the text about "The __array_finalize__ method" should at least > > once say that it is defined as "def __array_finalize__(self,obj)" -- > > otherwise I could only guess where the "obj" comes from. > > > > 4) related to the text after: "In other terms, __array_finalize__ is called" > > How do I know if a function or method actually invokes __new__ ;-) > > ? Would I have to study the numpy source ? > > > > 5) For the "__array_finalize__ method" there a text that says: > > """Subclasses inherit a default implementation of this method that > > does nothing""" --- but how about "__new__()" : what happens if you > > don't define that (either) ? > > > > (Hope these comments are helpful) > > > > Thanks for the help, > > Sebastian > > > > > First try seems to show that just changing my class def to: > class ndarray_inMrcFile(N.memmap): > def __array_finalize__(self,obj): > self.Mrc = getattr(obj, 'Mrc', None) > > Seems to add the wanted attribute back into result of transpose(). > However now I get (many!) exceptions like: > Exception exceptions.AttributeError: "'ndarray_inMrcFile' object has no attribut > e '_mmap'" in ndarray_inMrcFile([ 6,.... > > Do I need to call super.__array_finalize__(obj) first ? > > -Sebastian > This seems to work without any problem now: class ndarray_inMrcFile(N.ndarray): def __array_finalize__(self,obj): self.Mrc = getattr(obj, 'Mrc', None) Comments? -Sebastian From pgmdevlist at gmail.com Fri Nov 23 11:05:08 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 23 Nov 2007 11:05:08 -0500 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() In-Reply-To: References: <200711230937.29219.pgmdevlist@gmail.com> Message-ID: <200711231105.08704.pgmdevlist@gmail.com> > 2.) I guess the """class ndarray_inMrcFile(N.memmap): > pass""" construct is to simplistic .... > Could someone suggest a *minimal* definition, so that my attributes > would be preserved ? Mmh, I would at least try to explicitly call the N.memmap.__new__ > A few comments about the wiki page: > 1) The example does not show the printed info, such as "__new__ > received %s", in the example session OK, I'll edit the example > 2) I don't understand, why in the example the "info" attribute is set > in "__new__()", even though the text above says: > """However, we need to keep in mind that any attribute that we define > in the __new__ method will be shared among all the instances. If we > want instance-specific attributes, we still need some specific > initialization. We cannot use the __init__ method, as it won't be > called. That's where __array_finalize__ comes to play.""" Yeah, that's not very clear. Well, look at the code of the example. In line 11, we call a view of the array, with our class as argument. That calls __array_finalize__, with "subarr" as the obj argument. __array_finalize__ sets a default "info" attribute: an empty dictionary, as the argumnet doesn't have an info attribute yet. In lines 14-18, we set the "info" argument of our object to the value we want. > 3) the text about "The __array_finalize__ method" should at least > once say that it is defined as "def __array_finalize__(self,obj)" -- > otherwise I could only guess where the "obj" comes from. OK, same as for 1. > 4) related to the text after: "In other terms, __array_finalize__ is > called" How do I know if a function or method actually invokes __new__ > ;-) ? Would I have to study the numpy source ? As a rule of thumb, __new__ is called each time you have a construction like MyClass(whatverargumentisneeded). When you rely on views (such as w/ the .view method, or ravel, or transpose...), you call __array_finalize__ . You don't really have to study the numpy code (even if it helps) > 5) For the "__array_finalize__ method" there a text that says: > """Subclasses inherit a default implementation of this method that > does nothing""" --- but how about "__new__()" : what happens if you > don't define that (either) ? Try. The ndarray doesn't have a **keywords option: you can't pass extra parameters to ndarray.__new__. In other terms, if you need to pass some specific parameter to your class (in our example, info), you'll choke ndarray. From pgmdevlist at gmail.com Fri Nov 23 11:07:21 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 23 Nov 2007 11:07:21 -0500 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() In-Reply-To: References: Message-ID: <200711231107.21867.pgmdevlist@gmail.com> > First try seems to show that just changing my class def to: > class ndarray_inMrcFile(N.memmap): > def __array_finalize__(self,obj): > self.Mrc = getattr(obj, 'Mrc', None) > > Seems to add the wanted attribute back into result of transpose(). Yep. Specific subclass attributes should be defined in __array_finalize__. > However now I get (many!) exceptions like: > Exception exceptions.AttributeError: "'ndarray_inMrcFile' object has no > attribut e '_mmap'" in ndarray_inMrcFile([ 6,.... > > Do I need to call super.__array_finalize__(obj) first ? Nope, you just need a __new__ that calls the __new__ of you parent class. Keep it minimal. From zyzhu2000 at gmail.com Fri Nov 23 11:19:47 2007 From: zyzhu2000 at gmail.com (Geoffrey Zhu) Date: Fri, 23 Nov 2007 10:19:47 -0600 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3D Surface In-Reply-To: <474489EF.1020805@tricity.wsu.edu> References: <474489EF.1020805@tricity.wsu.edu> Message-ID: Hi Bob, Anne, and everyone, On Nov 21, 2007 1:41 PM, Bob Lewis wrote: > On 11/20/07, Anne Archibald posted: > > > Subject: > > Re: [Numpy-discussion] OT: A Way to Approximate and Compress a 3D Surface > > From: > > "Anne Archibald" > > Date: > > Tue, 20 Nov 2007 17:13:31 -0500 > > To: > > "Discussion of Numerical Python" > > > > To: > > "Discussion of Numerical Python" > > > > > > On 20/11/2007, Geoffrey Zhu wrote: > > > >> I have N tabulated data points { (x_i, y_i, z_i) } that describes a 3D > >> surface. The surface is pretty "smooth." However, the number of data > >> points is too large to be stored and manipulated efficiently. To make > >> it easier to deal with, I am looking for an easy method to compress > >> and approximate the data. Maybe the approximation can be described by > >> far fewer number of coefficients. > >> > >> If you can give me some hints about possible numpy or non-numpy > >> solutions or let me know where is better to ask this kind of question, > >> I would really appreciate it. > > > > This is an important task in computer graphics, in particular, in the > > field of multiresolution modelling. If you look up "surface > > simplification" you'll find many references to articles. I don't know > > of a library offhand that does it, let alone one that is accessible > > from python, but you could try looking at a toolkit that does > > isosurface visualization - these are surfaces that can often be > > simplified enormously. In particular it looks like VTK might be able > > to do what you want. > > Anne Archibald is correct that surface simplification may ultimately > be of great help. Other place to look besides VTK are GTS, the GNU > Triangulated Surface library, and CGAL, the Computational Geometry > Algorithms Library, which has a Python binding. > > It occurs to me, though, that we should first verify that you do > indeed have a surface in the first place. All you tell us is that you > have a set of N points in 3-space. Are they connected? That is, does > each point have well-defined neighbors? If so, do these vertices and > connections form a mesh that defines a surface? > > - Bob Lewis > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > First I'd like to thank everyone for helping me on this. It does look like surface simplification will greatly help my problem. Yes, I indeed have a surface and the points are connected. In fact, I have a function z=f(x,y). Except at certain points, it is continuous and smooth. It is very computationally intensive to calculate f(x,y), so I have to calculate certain points {(x_i,y_i_,z_i)} and store the data in a database. However, I have thousands of such functions and therefore a lot of points to deal with and that makes manipulating and storing them difficult and slow. One thing about triangulation I haven't figured out is how to add multiple such functions together. So if I have a set of triangles that represent f1(x,y) and another set of triangles that represent f2(x,y), is there any quick way to get f1(x,y)+f2(x,y) from the triangulation results of the parts? Thanks again for all your help, Geoffrey From pgmdevlist at gmail.com Fri Nov 23 11:18:53 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 23 Nov 2007 11:18:53 -0500 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() In-Reply-To: References: Message-ID: <200711231118.53694.pgmdevlist@gmail.com> > This seems to work without any problem now: > class ndarray_inMrcFile(N.ndarray): > def __array_finalize__(self,obj): > self.Mrc = getattr(obj, 'Mrc', None) > > Comments? That should work if you want a subclass of ndarray. That probably won't if you want a subclass of memmap. Once again, I'd do a def __new__(**options) N.memmap.__new__(**options) or something to this effect... From stefan at sun.ac.za Fri Nov 23 12:32:06 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 23 Nov 2007 19:32:06 +0200 Subject: [Numpy-discussion] OT: A Way to Approximate and Compress a 3D Surface In-Reply-To: References: <474489EF.1020805@tricity.wsu.edu> Message-ID: <20071123173206.GE28925@mentat.za.net> On Fri, Nov 23, 2007 at 10:19:47AM -0600, Geoffrey Zhu wrote: > One thing about triangulation I haven't figured out is how to add > multiple such functions together. So if I have a set of triangles that > represent f1(x,y) and another set of triangles that represent f2(x,y), > is there any quick way to get f1(x,y)+f2(x,y) from the triangulation > results of the parts? As a simple, and probably na?eve, first guess: if f1 is defined at a point but f2 is not, interpolate f2 to that point and add to f1. Thus, the result of f1+f2 will have points in the locations where either f1 or f2 have them. Thereafter, you can again simplify the mesh. Regards St?fan From aisaac at american.edu Fri Nov 23 23:03:21 2007 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 23 Nov 2007 23:03:21 -0500 Subject: [Numpy-discussion] loadtxt bug? In-Reply-To: <20071123135125.GC12028@clipper.ens.fr> References: <20071123071419.GA31662@clipper.ens.fr><20071123135125.GC12028@clipper.ens.fr> Message-ID: > On Fri, Nov 23, 2007 at 07:58:13AM -0500, Alan G Isaac wrote: >> Specifically, is it not the case that the last line of >> a text file is not guaranteed to have a terminator? Does >> this not raise the possibility that a digit will be >> clipped from the last line? On Fri, 23 Nov 2007, Gael Varoquaux apparently wrote: > Yes. If the line does not end by a terminator you have a problem. I do not know how common this situation is, but: - it is common enough that some editors address it explicitly (e.g., Vim, Epsilon) - Java's TextReader addresses it explicitly Based on these considerations an unterminated final line looks like a possibility. If so, perhaps that should be addressed by changing that line I pointed out in `loadtxt`. Cheers, Alan Isaac From markbak at gmail.com Sat Nov 24 08:06:02 2007 From: markbak at gmail.com (mark) Date: Sat, 24 Nov 2007 05:06:02 -0800 (PST) Subject: [Numpy-discussion] Installation of Python/mpl/numpy/scipy/Ipython for students Message-ID: A question was posted a week or so ago on the scipy list about installing Python/mpl/numpy/scipy/Ipython by David Arnold (both Windows and Mac). He was interested in a simple install for students (and others that are less computer-savy). We had a little off-list correspondence, but thought to bring it back to the list, also posting on mpl (cause I read that list and it seems pretty responsive) and the numpy Google discussion group (they respond VERY quickly). On the scipy list it was suggested to use Sage reply, but that sounds like a little overkill to me. I have used the enthought package on windows successfully in class in the past, but I understand that they have now switched to eggs, which is not convenient (yet). I also don't like all the other things I have to set myself. A switch on the IDLE GUI to get mpl to work interactively (which is now seems impossible to do), and setting my PYTHONPATH with environment variables. It is all easy to do, but not for students, especially students without much knowledge of how computers work let alone access to set these variables and options. At one point I was reading about making a superpack (or a name like that) installation for Python with our favorite packages as mentioned above. I am not sure where that is going. Are others dreaming about such a superpack? What needs to be done to get it going? Mark From haase at msg.ucsf.edu Sat Nov 24 08:17:06 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 24 Nov 2007 14:17:06 +0100 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() In-Reply-To: <200711231118.53694.pgmdevlist@gmail.com> References: <200711231118.53694.pgmdevlist@gmail.com> Message-ID: On Nov 23, 2007 5:18 PM, Pierre GM wrote: > > > This seems to work without any problem now: > > class ndarray_inMrcFile(N.ndarray): > > def __array_finalize__(self,obj): > > self.Mrc = getattr(obj, 'Mrc', None) > > > > Comments? > > That should work if you want a subclass of ndarray. That probably won't if you > want a subclass of memmap. Once again, I'd do a > def __new__(**options) > N.memmap.__new__(**options) > or something to this effect... > One more question on this: If I really don't need the memmap features explicitely anymore, and decide to derive from N.ndarray: Is it correct to assume that N.ndarray does *not* do it's own N.ndarray.__new__(**options) or so, that I would have to call ? (In other words: If I derive from N.ndarray I do not need to do a def __new__(**options): N.ndarray.__new__(**options) , right?) -Sebastian From pgmdevlist at gmail.com Sat Nov 24 11:50:36 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 24 Nov 2007 11:50:36 -0500 Subject: [Numpy-discussion] my derived ndarray class object loses its attribute after a transpose() In-Reply-To: References: <200711231118.53694.pgmdevlist@gmail.com> Message-ID: <200711241150.36732.pgmdevlist@gmail.com> > Is it correct to assume that N.ndarray does *not* do it's own > N.ndarray.__new__(**options) or so, that I would have to call ? > (In other words: If I derive from N.ndarray I do not need to do a > def __new__(**options): > N.ndarray.__new__(**options) > , right?) That'd work if ndarray.__new__ were accepting an extra dictionary of parameters, a la **options. This is not the case. So, following the example in the wiki, if you want to build an InfoArray with InfoArray(x,info={}), you will need a __new__method to process the argument "info" that ndarray does not recognize. True, you could first create your object, then fill in as needed, such as: >>>x=InfoArray([1,2,3]) >>>x.info = {'name':'x'} It looks less clear than >>>InfoArray([1,2,3], info= {'name':'x'} don't you think ? From perrygeo at gmail.com Sun Nov 25 20:00:48 2007 From: perrygeo at gmail.com (Matthew Perry) Date: Sun, 25 Nov 2007 17:00:48 -0800 Subject: [Numpy-discussion] "moving window" in numpy Message-ID: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> Hi all, I'm not sure if my terminology is familiar but I'm trying to do a "moving window" analysis (ie a spatial filter or kernel) on a 2-D array representing elevation. For example, a 3x3 window centered on each cell is used to calculate the derivate slope of that cell. Can this easily be implemented using numpy? Currently I have tried implementing in pure python loops (too slow) and c++ (fast but more difficult to compile, distribute, wrap in python calls, etc). I think a good solution would be to leverage numpy which is both fast and and easy package for end users to install. An example of the C++ code I'm trying to emulate is at http://perrygeo.net/download/hillshade.html . Does anyone have any tips or examples out there? Where should I start researching this? -- Matthew T. Perry http://www.perrygeo.net From bradford.n.cross at gmail.com Sun Nov 25 20:14:25 2007 From: bradford.n.cross at gmail.com (Bradford Cross) Date: Sun, 25 Nov 2007 17:14:25 -0800 Subject: [Numpy-discussion] "moving window" in numpy In-Reply-To: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> References: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> Message-ID: Hi Matthew, I've been working on "rolling/windowed" libraries for quite a while. I'm currently working on implementations for basic moment and rank statistics - they are pretty much done and I am trying to maneuver them into apache commons math for java. I am also interested in implementing these statistics for python/numpy and R. I have seen a little bit for R, but nothing yet for numpy. /brad On Nov 25, 2007 5:00 PM, Matthew Perry wrote: > Hi all, > > I'm not sure if my terminology is familiar but I'm trying to do a > "moving window" analysis (ie a spatial filter or kernel) on a 2-D > array representing elevation. For example, a 3x3 window centered on > each cell is used to calculate the derivate slope of that cell. > > Can this easily be implemented using numpy? > > Currently I have tried implementing in pure python loops (too slow) > and c++ (fast but more difficult to compile, distribute, wrap in > python calls, etc). I think a good solution would be to leverage numpy > which is both fast and and easy package for end users to install. > > An example of the C++ code I'm trying to emulate is at > http://perrygeo.net/download/hillshade.html . Does anyone have any > tips or examples out there? Where should I start researching this? > > -- > Matthew T. Perry > http://www.perrygeo.net > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wfspotz at sandia.gov Sun Nov 25 22:55:56 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Sun, 25 Nov 2007 20:55:56 -0700 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <474465EE.20305@mur.at> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <474465EE.20305@mur.at> Message-ID: <538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov> OK, I'm going to try to get to this soon, but I want to discuss the interface some before committing to development. First, my plan is to add to numpy.i, typemaps for signatures like the following: %typemap(argout) (double** ARGOUT_ARRAY1, int* DIM1) It is important to note that even though the same argument *names* are used, this is a different typemap signature than %typemap(argout) (double* ARGOUT_ARRAY1, int DIM1) and thus can have (logically) different implementations. For the latter, the typemap allocates a new numpy array and passes the buffer pointer and dimension in; for the former, the buffer pointer and dimension will be obtained from the wrapped function and used to build a new numpy array. As for having a COPY version of the first typemap signature (in addition to the non-copy, or "view" version), I currently do not plan to do this. Here's why: the only case I can conceive of for needing it is when the class provides a "view" method but not a "copy" method. Since there is no copy method to apply a typemap to, the swig user will have to add a new one with %extend, in which case the swig user can provide the copy logic. The example Georg gives in his link below is not a counter-example to this. He provides two methods, void getBigData(double **mtx, int *rows, int *cols) void getBigDataCopy(double **cpmtx, int *cprows, int *cpcols) but in addition to the same argument types, they have the exact same implementation. The only thing that is different is the names of the arguments, which is clearly so that we can apply different swig typemaps to the two methods. I submit that you will not find a class written by someone else that will be designed this way. You will only find it if you have control over the class, in which case you should put the copy logic inside the second method. (It is important to understand that if you have a class with functioning view and copy methods, then the view version of the typemap will work for both.) Maybe I'm wrong. But that is why I want to discuss this before committing to code. On Nov 21, 2007, at 10:07 AM, Georg Holzmann wrote: > Hallo! > > As chris said, I need to make an example: > http://grh.mur.at/software/numpy2carray.tar.gz > > I added the following class-example: > class_example.h: the C++ code > class_example.i: the SWIG interface file > class_example_usage.py: example usage in python > > > And some comments: > > Bill Spotz schrieb: >> Here is what I am proposing you do: in your interface file, add >> something like >> PyObject * getMatrix() >> { >> npy_intp dims[2] = { /* Obtain the dimensions to your >> internal matrix */ }; >> double * data = /* Obtain the pointer to you internal >> matrix */; >> return PyArray_SimpleNewFromData(2, dims, NPY_DOUBLE, >> (void*)data); >> } >> For your function, use the INPLACE_ARRAY2 typemap: >> %apply (double* INPLACE_ARRAY2, int DIM1, int DIM2) {(double* >> matrix, int rows, int cols)}; >> void myFunction(double* matrix, int rows, int cols, double >> parameter, ...); >> And then in python you can do this: >> m = getMatrix() >> myFunction(m, 3.14, ...) > > I see ... but what I mean with output of a matrix without copy is > actually only your getMatrix() funtion I guess - this is basically > what getBigData() does in class_example.h together with > class_example.i. > > BTW: what is the difference between PyArray_SimpleNewFromData() and > PyArray_FromDimsAndData() ? > (I don't have this book ...) > >>> Again I do not see the problem - see e.g. ARRAY2_OUT_COPY in >>> numpy2carray.i, shouldn't this be the same ? >> I do not understand the use case for which that typemap works. >> You create "ttype t1", then pass its address to the function as >> the pointer to the data. This has to be useless to the function. >> After the function returns, you access this pointer as if it >> points to meaningful data. How is this possible? The address of >> t1 isn't going to change, and only a single element is allocated. > > See also the class_example_usage.py - there I used the > ARRAY2_OUT_COPY for the getBigDataCopy() method. > > LG > Georg ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From grh at mur.at Mon Nov 26 02:59:53 2007 From: grh at mur.at (Georg Holzmann) Date: Mon, 26 Nov 2007 08:59:53 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <474465EE.20305@mur.at> <538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov> Message-ID: <474A7CF9.1060107@mur.at> Hallo! > First, my plan is to add to numpy.i, typemaps for signatures like the > following: > > %typemap(argout) (double** ARGOUT_ARRAY1, int* DIM1) > > It is important to note that even though the same argument *names* are > used, this is a different typemap signature than > > %typemap(argout) (double* ARGOUT_ARRAY1, int DIM1) > > and thus can have (logically) different implementations. For the > latter, the typemap allocates a new numpy array and passes the buffer > pointer and dimension in; for the former, the buffer pointer and > dimension will be obtained from the wrapped function and used to build a > new numpy array. Hm ... maybe it is more clear for users then, if the new typemap has a different name ? In example ARRAY_VIEW1, or something else with "VIEW" (although it's not really a view ?) ? > As for having a COPY version of the first typemap signature (in addition > to the non-copy, or "view" version), I currently do not plan to do [...] > The example Georg gives in his link below is not a counter-example to > this. He provides two methods, > > void getBigData(double **mtx, int *rows, int *cols) > void getBigDataCopy(double **cpmtx, int *cprows, int *cpcols) > > but in addition to the same argument types, they have the exact same > implementation. The only thing that is different is the names of the > arguments, which is clearly so that we can apply different swig typemaps > to the two methods. I submit that you will not find a class written by > someone else that will be designed this way. You will only find it if Yes this copy method was only for demonstration - I also did not use this method for my "real" code. One can simply call .copy() in python ... LG Georg From schut at sarvision.nl Mon Nov 26 04:59:47 2007 From: schut at sarvision.nl (Vincent Schut) Date: Mon, 26 Nov 2007 10:59:47 +0100 Subject: [Numpy-discussion] "moving window" in numpy In-Reply-To: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> References: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> Message-ID: <474A9913.70709@sarvision.nl> Matthew Perry wrote: > Hi all, > > I'm not sure if my terminology is familiar but I'm trying to do a > "moving window" analysis (ie a spatial filter or kernel) on a 2-D > array representing elevation. For example, a 3x3 window centered on > each cell is used to calculate the derivate slope of that cell. > > Can this easily be implemented using numpy? > If I understand you correctly, scipy.ndimage does exactly what you ask for. See http://www.scipy.org/SciPyPackages/Ndimage. Cheers, VS. From schut at sarvision.nl Mon Nov 26 04:59:47 2007 From: schut at sarvision.nl (Vincent Schut) Date: Mon, 26 Nov 2007 10:59:47 +0100 Subject: [Numpy-discussion] "moving window" in numpy In-Reply-To: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> References: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> Message-ID: <474A9913.70709@sarvision.nl> Matthew Perry wrote: > Hi all, > > I'm not sure if my terminology is familiar but I'm trying to do a > "moving window" analysis (ie a spatial filter or kernel) on a 2-D > array representing elevation. For example, a 3x3 window centered on > each cell is used to calculate the derivate slope of that cell. > > Can this easily be implemented using numpy? > If I understand you correctly, scipy.ndimage does exactly what you ask for. See http://www.scipy.org/SciPyPackages/Ndimage. Cheers, VS. From meine at informatik.uni-hamburg.de Mon Nov 26 03:04:38 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Mon, 26 Nov 2007 09:04:38 +0100 Subject: [Numpy-discussion] "moving window" in numpy In-Reply-To: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> References: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> Message-ID: <200711260904.47409.hans_meine@gmx.net> On Montag 26 November 2007, Matthew Perry wrote: > Hi all, > > I'm not sure if my terminology is familiar but I'm trying to do a > "moving window" analysis (ie a spatial filter or kernel) on a 2-D > array representing elevation. For example, a 3x3 window centered on > each cell is used to calculate the derivate slope of that cell. Did you have a look at ndimage? http://www.scipy.org/SciPyPackages/Ndimage > Currently I have tried implementing in pure python loops (too slow) You could probably have a faster, pure python version with some indexing tricks. E.g. if your gradient filter is static (i.e. the weights do not adapt to the local image content), you can apply it to whole images that contain shifted image contents. This way, you're using the fast C loops of NumPy. -- Ciao, / / .o. /--/ ..o / / ANS ooo -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From stefan at sun.ac.za Mon Nov 26 08:34:56 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 26 Nov 2007 15:34:56 +0200 Subject: [Numpy-discussion] "moving window" in numpy In-Reply-To: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> References: <5383fa5e0711251700m2eb52cb9tde647af8a88c3eef@mail.gmail.com> Message-ID: <20071126133456.GD12329@mentat.za.net> Hi Matthew On Sun, Nov 25, 2007 at 05:00:48PM -0800, Matthew Perry wrote: > I'm not sure if my terminology is familiar but I'm trying to do a > "moving window" analysis (ie a spatial filter or kernel) on a 2-D > array representing elevation. For example, a 3x3 window centered on > each cell is used to calculate the derivate slope of that cell. Take a look at Anne Archibald's post: http://projects.scipy.org/pipermail/numpy-discussion/2006-November/024760.html and its attachment here: http://projects.scipy.org/pipermail/numpy-discussion/attachments/20061120/d057cb0b/attachment.py Regards St?fan From darren.dale at cornell.edu Mon Nov 26 14:47:39 2007 From: darren.dale at cornell.edu (Darren Dale) Date: Mon, 26 Nov 2007 14:47:39 -0500 Subject: [Numpy-discussion] is numpy.typeNA out of date? Message-ID: <200711261447.39282.darren.dale@cornell.edu> Is the numpy.typeNA dictionary out of date? All of the verbose names for datatypes are captialized, and can't be found in the numpy namespace. Thanks, Darren From engel at physics.harvard.edu Mon Nov 26 16:30:11 2007 From: engel at physics.harvard.edu (Hans-Andreas Engel) Date: Mon, 26 Nov 2007 21:30:11 +0000 (UTC) Subject: [Numpy-discussion] How to remove loops over inner() Message-ID: Dear all: After using numpy for several weeks, I am very happy about it and deeply impressed about the performance improvements it brings in my python code. Now I have stumbled upon a problem, where I cannot use numpy to eliminate all my loops in python. Currently the return value of inner(a, b) is defined as inner(a, b)[I, J] = sum_k a[I, k] * b[J, k], for some super indices I and J. Somewhat more general is the tensordot() function that allows to specify over which axes K is summed over. However, if I understand numpy correctly, the following more general version is currently missing: inner(a, b, keep_axis=0)[H, I, J] = sum_k a[H, I, k] * b[H, J, k]. Here H would be an additional super index (specified via the keep_axis keyword), on which no outer product is taken, i.e., the same index is used for a[] and b[]. This more general definition would allow elimination of an extra level of loops. For example, I wish to calculate the following a = rand(200, 5, 2) b = rand(200, 4, 2) r = empty(a.shape[:-1] + b.shape[1:-1]) for h in range(a.shape[0]): r[h] = inner(a[h], b[h]) How could I eliminate the loop? It would be great if there would be the mentioned generalized version of the inner() [or tensordot()] function, since it would eliminate this loop and make my code much faster. What are your opinions? Would such a feature be desirable (or is it already implemented)? Thank you, Best, Hansres From Chris.Barker at noaa.gov Mon Nov 26 17:52:36 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 26 Nov 2007 14:52:36 -0800 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <474465EE.20305@mur.at> <538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov> Message-ID: <474B4E34.80900@noaa.gov> Bill Spotz wrote: > First, my plan is to add to numpy.i, typemaps for signatures like the > following: > > %typemap(argout) (double** ARGOUT_ARRAY1, int* DIM1) > > It is important to note that even though the same argument *names* > are used, this is a different typemap signature than > > %typemap(argout) (double* ARGOUT_ARRAY1, int DIM1) > > and thus can have (logically) different implementations. maybe it's a good idea to give the arguments different names, though, just for clarity. > As for having a COPY version of the first typemap signature (in > addition to the non-copy, or "view" version), I currently do not plan > to do this. I think you've got it right. Thanks for all this! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ivilata at carabos.com Tue Nov 27 12:04:34 2007 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Tue, 27 Nov 2007 18:04:34 +0100 Subject: [Numpy-discussion] ANN: PyTables & PyTables Pro 2.0.2 are out! Message-ID: <20071127170434.GE10078@tardis.terramar.selidor.net> Hi everyone, We at Carabos are happy to announce the simultaneous release of the new 2.0.2 versions of both PyTables and PyTables Pro. They are mainly bugfix releases, and users of previous versions are encouraged to upgrade. And now the official announce: ============================================ Announcing PyTables and PyTables Pro 2.0.2 ============================================ PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables Pro adds OPSI, a powerful indexing engine for executing very fast queries in large tables. In this version, some bugs have been fixed, being the most important a problem when moving or renaming a group. Some small improvements have been added as well. Besides, a *critical* bug has been fixed in the Pro version (the problem arose when doing repeated queries using the same index). Because of this, an upgrade is strongly recommended. In case you want to know more in detail what has changed in this version, have a look at ``RELEASE_NOTES.txt``. Find the HTML version for this document at: http://www.pytables.org/moin/ReleaseNotes/Release_2.0.2 You can download a source package of the version 2.0.2 with generated PDF and HTML docs and binaries for Windows from http://www.pytables.org/download/stable/ For an on-line version of the manual, visit: http://www.pytables.org/docs/manual-2.0.2 Migration Notes for PyTables 1.x users ====================================== If you are a user of PyTables 1.x, probably it is worth for you to look at ``MIGRATING_TO_2.x.txt`` file where you will find directions on how to migrate your existing PyTables 1.x apps to the 2.x versions. You can find an HTML version of this document at http://www.pytables.org/moin/ReleaseNotes/Migrating_To_2.x Resources ========= Go to the PyTables web site for more details: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ To know more about the company behind the development of PyTables, see: http://www.carabos.com/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last, but not least thanks a lot to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 307 bytes Desc: Digital signature URL: From Andy.cheesman at bristol.ac.uk Tue Nov 27 12:56:05 2007 From: Andy.cheesman at bristol.ac.uk (Andy Cheesman) Date: Tue, 27 Nov 2007 17:56:05 +0000 Subject: [Numpy-discussion] Appending a numpy array to existing text file Message-ID: <474C5A35.2020603@bristol.ac.uk> Hi people, Just a quick question, how do I append a numpy array to an existing ascii output file? I've looked at the numpy.savetxt function and it performs all the formating functions I desire for my output file but it doesn't let me specify how I save my array type Is there a clever work around for this or does anyone have any suggestions? Many Thanks for your help Andy From berthe.loic at gmail.com Tue Nov 27 13:30:43 2007 From: berthe.loic at gmail.com (LB) Date: Tue, 27 Nov 2007 10:30:43 -0800 (PST) Subject: [Numpy-discussion] Appending a numpy array to existing text file In-Reply-To: <474C5A35.2020603@bristol.ac.uk> References: <474C5A35.2020603@bristol.ac.uk> Message-ID: If you just want to add your matrix to an existing ascii file, you can open this file in append mode and give the file handle to numpy.savetxt : f_handle = file('my_file.dat', 'a') savetxt(f_handle, my_matrix) f_handle.close() HTH -- LB From giorgio at gilestro.tk Tue Nov 27 13:38:27 2007 From: giorgio at gilestro.tk (Giorgio F. Gilestro) Date: Tue, 27 Nov 2007 12:38:27 -0600 Subject: [Numpy-discussion] how do I speed up this? Message-ID: <474C6423.5050104@gilestro.tk> Hello everyone, ma and new_ma are bi-dimensional array with shape (a1, a2) on which I am performing the following iteration: for fd in range(a1): new_ma[fd] = [( ma[fd][i-5:i+5].sum() == 0 )*1 for i in range (a2)] Is there any faster more elegant way to do that with numpy? Thanks a lot! Giorgio -- giorgio at gilestro.tk http://www.cafelamarck.it From tim.hochberg at ieee.org Tue Nov 27 14:26:30 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 27 Nov 2007 12:26:30 -0700 Subject: [Numpy-discussion] how do I speed up this? In-Reply-To: <474C6423.5050104@gilestro.tk> References: <474C6423.5050104@gilestro.tk> Message-ID: On Nov 27, 2007 11:38 AM, Giorgio F. Gilestro wrote: > Hello everyone, > > ma and new_ma are bi-dimensional array with shape (a1, a2) on which I am > performing the following iteration: > > for fd in range(a1): > new_ma[fd] = [( ma[fd][i-5:i+5].sum() == 0 )*1 for i in range (a2)] This looks like it's probably buggy; when, for example,, a2 == 0, I don't think ma[fd][-5:5] is going to return what you want. > > Is there any faster more elegant way to do that with numpy? There are faster ways, I'm not sure that they are more elegant. It looks like what you want is essentially a moving average, possible with periodic boundary conditions. There are a few different tricks that can make this fast; which is appropriate depends on what the shape of ma is expected to be. If 'a1' if large, then you can profitably remove the outer loop; something vaguely like: for i in range(a2): new_ma[:, i] = ( ma[fd][i-5:i+5].sum() == 0) (This is still buggy as above, since I'm not sure what kind of boundary conditions you need). If that's not applicable, another approach is to loop over the offset, in this case -5 to 5, and remove the loop over a2. That method is more complicated and I'm going to avoid describing it in detail unless needed. One might even be able to do both, but that's probably overkill. HTH -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From giorgio at gilestro.tk Tue Nov 27 14:41:57 2007 From: giorgio at gilestro.tk (Giorgio F. Gilestro) Date: Tue, 27 Nov 2007 13:41:57 -0600 Subject: [Numpy-discussion] how do I speed up this? In-Reply-To: References: <474C6423.5050104@gilestro.tk> Message-ID: <474C7305.5000505@gilestro.tk> Thanks Tim, shape of the array is variable but follows the rule (x, y*1440) with both x an y being values greater than 1 (usually around 10 or 20). So as result second dimension is always much bigger. (and the bug you foresaw is actually taken care of) I figured out that somehow removing the unnecessary *1 multiplication after the boolean verification will increase speed by almost 100% (wow!). I guess if I can get rid of the iteration too and use some internal function I'd probably get even faster. Could .flat() help me somehow? Timothy Hochberg wrote: > > > On Nov 27, 2007 11:38 AM, Giorgio F. Gilestro > wrote: > > Hello everyone, > > ma and new_ma are bi-dimensional array with shape (a1, a2) on > which I am > performing the following iteration: > > for fd in range(a1): > new_ma[fd] = [( ma[fd][i-5:i+5].sum() == 0 )*1 for i in range > (a2)] > > > This looks like it's probably buggy; when, for example,, a2 == 0, I > don't think ma[fd][-5:5] is going to return what you want. > > > Is there any faster more elegant way to do that with numpy? > > > There are faster ways, I'm not sure that they are more elegant. It > looks like what you want is essentially a moving average, possible > with periodic boundary conditions. There are a few different tricks > that can make this fast; which is appropriate depends on what the > shape of ma is expected to be. If 'a1' if large, then you can > profitably remove the outer loop; something vaguely like: > > for i in range(a2): > new_ma[:, i] = ( ma[fd][i-5:i+5].sum() == 0) > > (This is still buggy as above, since I'm not sure what kind of > boundary conditions you need). > > If that's not applicable, another approach is to loop over the offset, > in this case -5 to 5, and remove the loop over a2. That method is more > complicated and I'm going to avoid describing it in detail unless > needed. One might even be able to do both, but that's probably overkill. > > HTH > > -- > . __ > . |-\ > . > . tim.hochberg at ieee.org > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- giorgio at gilestro.tk http://www.cafelamarck.it From tim.hochberg at ieee.org Tue Nov 27 14:45:36 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 27 Nov 2007 12:45:36 -0700 Subject: [Numpy-discussion] How to remove loops over inner() In-Reply-To: References: Message-ID: On Nov 26, 2007 2:30 PM, Hans-Andreas Engel wrote: > Dear all: > > After using numpy for several weeks, I am very happy about it and > deeply impressed about the performance improvements it brings in my > python code. Now I have stumbled upon a problem, where I cannot use > numpy to eliminate all my loops in python. > > Currently the return value of inner(a, b) is defined as > inner(a, b)[I, J] = sum_k a[I, k] * b[J, k], > for some super indices I and J. Somewhat more general is the > tensordot() function that allows to specify over which axes K is > summed over. > > However, if I understand numpy correctly, the following more general > version is currently missing: > inner(a, b, keep_axis=0)[H, I, J] = sum_k a[H, I, k] * b[H, J, k]. > Here H would be an additional super index (specified via the keep_axis > keyword), on which no outer product is taken, i.e., the same index is > used for a[] and b[]. > > This more general definition would allow elimination of an extra level > of loops. For example, I wish to calculate the following > a = rand(200, 5, 2) > b = rand(200, 4, 2) > r = empty(a.shape[:-1] + b.shape[1:-1]) > for h in range(a.shape[0]): > r[h] = inner(a[h], b[h]) > How could I eliminate the loop? It would be great if there would be > the mentioned generalized version of the inner() [or tensordot()] > function, since it would eliminate this loop and make my code much > faster. > > What are your opinions? Would such a feature be desirable (or is it > already implemented)? Essentially, you want to operate on a stack of two dimensional arrays, correct? I'd be mildly supportive of something like this for tensordot; I'd prefer more descriptive name for keep_axis, but I don't know what it would be off the top of my head. In any event it should be XXX_axes and optionally take a sequence of axes so that more than one can be ignored. You could trivially build more specific functions on top of tensordot, so I don't see that inner needs to be changed as it's basically a convenience function anyway. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Tue Nov 27 14:50:55 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 27 Nov 2007 12:50:55 -0700 Subject: [Numpy-discussion] how do I speed up this? In-Reply-To: <474C7305.5000505@gilestro.tk> References: <474C6423.5050104@gilestro.tk> <474C7305.5000505@gilestro.tk> Message-ID: On Nov 27, 2007 12:41 PM, Giorgio F. Gilestro wrote: > Thanks Tim, > shape of the array is variable but follows the rule (x, y*1440) with > both x an y being values greater than 1 (usually around 10 or 20). OK. That's helpful. > So as result second dimension is always much bigger. (and the bug you > foresaw is actually taken care of) Are you sure? This still looks problematic. > > I figured out that somehow removing the unnecessary *1 multiplication > after the boolean verification will increase speed by almost 100% (wow!). > I guess if I can get rid of the iteration too and use some internal > function I'd probably get even faster. > Could .flat() help me somehow? I doubt it. I'll look at this a little later if I can find some time and see what I can come up with. > > > Timothy Hochberg wrote: > > > > > > On Nov 27, 2007 11:38 AM, Giorgio F. Gilestro > > wrote: > > > > Hello everyone, > > > > ma and new_ma are bi-dimensional array with shape (a1, a2) on > > which I am > > performing the following iteration: > > > > for fd in range(a1): > > new_ma[fd] = [( ma[fd][i-5:i+5].sum() == 0 )*1 for i in range > > (a2)] > > > > > > This looks like it's probably buggy; when, for example,, a2 == 0, I > > don't think ma[fd][-5:5] is going to return what you want. > > > > > > Is there any faster more elegant way to do that with numpy? > > > > > > There are faster ways, I'm not sure that they are more elegant. It > > looks like what you want is essentially a moving average, possible > > with periodic boundary conditions. There are a few different tricks > > that can make this fast; which is appropriate depends on what the > > shape of ma is expected to be. If 'a1' if large, then you can > > profitably remove the outer loop; something vaguely like: > > > > for i in range(a2): > > new_ma[:, i] = ( ma[fd][i-5:i+5].sum() == 0) > > > > (This is still buggy as above, since I'm not sure what kind of > > boundary conditions you need). > > > > If that's not applicable, another approach is to loop over the offset, > > in this case -5 to 5, and remove the loop over a2. That method is more > > complicated and I'm going to avoid describing it in detail unless > > needed. One might even be able to do both, but that's probably overkill. > > > > HTH > > > > -- > > . __ > > . |-\ > > . > > . tim.hochberg at ieee.org > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > giorgio at gilestro.tk > http://www.cafelamarck.it > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Schmucker at 4DTechnology.com Tue Nov 27 23:51:06 2007 From: Mark.Schmucker at 4DTechnology.com (Mark Schmucker) Date: Tue, 27 Nov 2007 21:51:06 -0700 Subject: [Numpy-discussion] How can I constrain linear_least_squares to integer solutions? Message-ID: Hi, I have successfully used LinearAlgebra.linear_least_squares to estimate solutions to continuous functions. The coefficients returned in that case are of course floating-point values. Now I have a problem where some of the terms are continuous but some must be constrained to integer multiples. As a simple example, y = c1 * x^2 + c0 * x where c1 is floating-point in the usual way, but c0 can only be integer-valued. (The actual problem has many more terms.) Can anyone tell me how to constrain some of the coefficients to integer values? Thank you for any insight. Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Nov 28 01:07:30 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 27 Nov 2007 23:07:30 -0700 Subject: [Numpy-discussion] How can I constrain linear_least_squares to integer solutions? In-Reply-To: References: Message-ID: On Nov 27, 2007 9:51 PM, Mark Schmucker wrote: > Hi, > > > > I have successfully used LinearAlgebra.linear_least_squares to estimate > solutions to continuous functions. The coefficients returned in that case > are of course floating-point values. > > > > Now I have a problem where some of the terms are continuous but some must > be constrained to integer multiples. As a simple example, > > > > y = c1 * x^2 + c0 * x > > > > where c1 is floating-point in the usual way, but c0 can only be > integer-valued. (The actual problem has many more terms.) > > > > Can anyone tell me how to constrain some of the coefficients to integer > values? > > This is not a trivial problem, as you can see by googling mixed integer least squares (MILS). Much will depend on the nature of the parameters, the number of variables you are using in the fit, and how exact the solution needs to be. One approach would be to start by rounding the coefficients that must be integer and improve the solution using annealing or genetic algorithms to jig the integer coefficients while fitting the remainder in the usual least square way, but that wouldn't have the elegance of some of the specific methods used for this sort of problem. However, I don't know of a package in scipy that implements those more sophisticated algorithms, perhaps someone else on this list who knows more about these things than I can point you in the right direction. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Nov 28 02:59:37 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 28 Nov 2007 09:59:37 +0200 Subject: [Numpy-discussion] How can I constrain linear_least_squares to integer solutions? In-Reply-To: References: Message-ID: <20071128075937.GC23287@mentat.za.net> On Tue, Nov 27, 2007 at 11:07:30PM -0700, Charles R Harris wrote: > This is not a trivial problem, as you can see by googling mixed integer least > squares (MILS). Much will depend on the nature of the parameters, the number of > variables you are using in the fit, and how exact the solution needs to be. One > approach would be to start by rounding the coefficients that must be integer > and improve the solution using annealing or genetic algorithms to jig the > integer coefficients while fitting the remainder in the usual least square way, > but that wouldn't have the elegance of some of the specific methods used for > this sort of problem. However, I don't know of a package in scipy that > implements those more sophisticated algorithms, perhaps someone else on this > list who knows more about these things than I can point you in the right > direction. Would this be a good candidate for a genetic algorithm? I haven't used GA before, so I don't know the typical rate of convergence or its applicability to optimization problems. Regards St?fan From wright at esrf.fr Wed Nov 28 03:17:31 2007 From: wright at esrf.fr (Jon Wright) Date: Wed, 28 Nov 2007 09:17:31 +0100 Subject: [Numpy-discussion] numpy.put as increment? In-Reply-To: <20071128075937.GC23287@mentat.za.net> References: <20071128075937.GC23287@mentat.za.net> Message-ID: <474D241B.6000500@esrf.fr> Hello everyone, I am trying to sum points onto a 3d grid using "put" with a list of computed array indices. I am looking for a version of "put" which increments, e.g. replace: for i in ind: a.flat[i] = v[i] with "+=" to increment. Other operations might also be useful? I tried using "take", then incrementing the result, but still have problems if the indices are not unique. Thanks for any suggestions, Jon From Andy.cheesman at bristol.ac.uk Wed Nov 28 05:09:56 2007 From: Andy.cheesman at bristol.ac.uk (Andy Cheesman) Date: Wed, 28 Nov 2007 10:09:56 +0000 Subject: [Numpy-discussion] Appending a numpy array to existing text file In-Reply-To: References: <474C5A35.2020603@bristol.ac.uk> Message-ID: <474D3E74.6070909@bristol.ac.uk> It does indeed work Thanks for the help Andy LB wrote: > If you just want to add your matrix to an existing ascii file, you can > open this file in append mode and give the file handle to > numpy.savetxt : > > f_handle = file('my_file.dat', 'a') > savetxt(f_handle, my_matrix) > f_handle.close() > > HTH > > -- > LB > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From cimrman3 at ntc.zcu.cz Wed Nov 28 05:29:20 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 28 Nov 2007 11:29:20 +0100 Subject: [Numpy-discussion] documentation generator based on pyparsing Message-ID: <474D4300.6020904@ntc.zcu.cz> Hi, At http://scipy.org/Generate_Documentation you can find a very small documentation generator for NumPy/SciPy modules based on pyparsing package (by Paul McGuire). I am not sure if this belongs to where I put it, so feel free to (re)move the page as needed. I hope it might be interesting for you. r. From tim.hochberg at ieee.org Wed Nov 28 12:03:10 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 28 Nov 2007 10:03:10 -0700 Subject: [Numpy-discussion] How can I constrain linear_least_squares to integer solutions? In-Reply-To: <20071128075937.GC23287@mentat.za.net> References: <20071128075937.GC23287@mentat.za.net> Message-ID: On Nov 28, 2007 12:59 AM, Stefan van der Walt wrote: > On Tue, Nov 27, 2007 at 11:07:30PM -0700, Charles R Harris wrote: > > This is not a trivial problem, as you can see by googling mixed integer > least > > squares (MILS). Much will depend on the nature of the parameters, the > number of > > variables you are using in the fit, and how exact the solution needs to > be. One > > approach would be to start by rounding the coefficients that must be > integer > > and improve the solution using annealing or genetic algorithms to jig > the > > integer coefficients while fitting the remainder in the usual least > square way, > > but that wouldn't have the elegance of some of the specific methods used > for > > this sort of problem. However, I don't know of a package in scipy that > > implements those more sophisticated algorithms, perhaps someone else on > this > > list who knows more about these things than I can point you in the right > > direction. > > Would this be a good candidate for a genetic algorithm? I haven't > used GA before, so I don't know the typical rate of convergence or its > applicability to optimization problems. > > Regards > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > If the number of terms is not huge and the function is well behaved; it might be worth trying the following simple and stupid approach: 1. Find the floating point minimum. 2. for each set of possible set of integer coefficients near the FP minimum: 1. Solve for the floating point coefficients with the integer coefficients fixed. 2. If the minimum is the best so far, stash it somewhere for later. 3. Return the best set of coefficients. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From sameerslists at gmail.com Wed Nov 28 13:21:29 2007 From: sameerslists at gmail.com (Sameer DCosta) Date: Wed, 28 Nov 2007 12:21:29 -0600 Subject: [Numpy-discussion] Converting char array to float Message-ID: <8fb8cc060711281021r19eae508o1328be4e3a69b552@mail.gmail.com> I'm trying to convert a character array to a floating point array. I'm using one of the recent svn builds. It is surprising that astype does not do the job. However if I first convert the char array to an array and then use astype everything works fine. Is this a bug? import numpy as N print N.__version__ # Output = '1.0.5.dev4426' a = N.char.array(['123.45', '234.56']) b = N.array(a).astype(N.float) # This works. print b b = a.astype(N.float) # This does not work and raises an exception ValueError: Can only create a chararray from string data. Thanks. Sameer From matthieu.brucher at gmail.com Wed Nov 28 13:39:45 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 28 Nov 2007 19:39:45 +0100 Subject: [Numpy-discussion] Converting char array to float In-Reply-To: <8fb8cc060711281021r19eae508o1328be4e3a69b552@mail.gmail.com> References: <8fb8cc060711281021r19eae508o1328be4e3a69b552@mail.gmail.com> Message-ID: a does not seem to be an array, so it is not surprising that you need to convert it to an array first. Matthieu 2007/11/28, Sameer DCosta : > > I'm trying to convert a character array to a floating point array. I'm > using one of the recent svn builds. It is surprising that astype does > not do the job. However if I first convert the char array to an array > and then use astype everything works fine. Is this a bug? > > import numpy as N > print N.__version__ # Output = '1.0.5.dev4426' > a = N.char.array(['123.45', '234.56']) > b = N.array(a).astype(N.float) # This works. > print b > b = a.astype(N.float) # This does not work and raises an exception > > ValueError: Can only create a chararray from string data. > > > Thanks. > Sameer > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Wed Nov 28 13:37:20 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 28 Nov 2007 13:37:20 -0500 Subject: [Numpy-discussion] Converting char array to float In-Reply-To: <8fb8cc060711281021r19eae508o1328be4e3a69b552@mail.gmail.com> References: <8fb8cc060711281021r19eae508o1328be4e3a69b552@mail.gmail.com> Message-ID: <200711281337.21457.pgmdevlist@gmail.com> Sameer, I can't tell whether it's a bug or a feature, but I can give you some explanation: when you call .astype on your chararray, you call the __array_finalize__ of the chararray, which requires the dtype to be string like. Obviously, that won't work in your case. Transforming the chararray to a regular array of strings bypass this problem. That's what you're doing with the N.array(a) statement in your example. Two comments, however: * Try to use N.asarray() instead, as you won't copy the data (or use N.array(a,copy=False)) * You can also view your charray as a regular ndarray, and then use the astype method: a.view(N.ndarray).astype(float_) From pgmdevlist at gmail.com Wed Nov 28 13:51:58 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 28 Nov 2007 13:51:58 -0500 Subject: [Numpy-discussion] Converting char array to float In-Reply-To: References: <8fb8cc060711281021r19eae508o1328be4e3a69b552@mail.gmail.com> Message-ID: <200711281351.58732.pgmdevlist@gmail.com> On Wednesday 28 November 2007 13:39:45 Matthieu Brucher wrote: > a does not seem to be an array, so it is not surprising that you need to > convert it to an array first. Well, a *IS* a regular chararray, and therefore a subclass of ndarray (try isinstance). The problem isn't here, it's that the subclass doesn't have its own .astype() method. Instead, we use the standard ndarray.astype, which calls the __array_finalize__ method from the subclass, and this one requires a chararray. Maybe we could implement a simple method: def astype(self, newdtype): self.view(N.ndarray).astype(newdtype) But wouldn't that break something down the line ? From nwagner at iam.uni-stuttgart.de Wed Nov 28 14:25:31 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Wed, 28 Nov 2007 20:25:31 +0100 Subject: [Numpy-discussion] documentation generator based on pyparsing In-Reply-To: <474D4300.6020904@ntc.zcu.cz> References: <474D4300.6020904@ntc.zcu.cz> Message-ID: On Wed, 28 Nov 2007 11:29:20 +0100 Robert Cimrman wrote: > Hi, > > At http://scipy.org/Generate_Documentation you can find >a very small > documentation generator for NumPy/SciPy modules based on >pyparsing > package (by Paul McGuire). I am not sure if this belongs >to where I put > it, so feel free to (re)move the page as needed. I hope >it might be > interesting for you. > > r. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion Hi Robert, The output of ./gendocs.py -m 'scipy.linsolve.umfpack' differs from your example output (available at http://scipy.org/Generate_Documentation) ./gendocs.py -m 'scipy.linsolve.umfpack' generating docs for "scipy.linsolve.umfpack"... output LaTeX source file: ./scipy.linsolve.umfpack.tex ['Contains'] ['Description', '-------------'] ['Installation', '--------------'] ['Examples', '----------'] ['Arguments of UmfpackContext solution methods', '----------------------------------------------'] ['Setting control parameters', '----------------------------'] ['Author'] ['Other contributors'] ['UmfpackContext class'] This is pdfeTeX, Version 3.141592-1.21a-2.2 (Web2C 7.5.4) entering extended mode (./scipy.linsolve.umfpack.tex LaTeX2e <2003/12/01> Babel and hyphenation patterns for american, french, german, ngerman, b ahasa, basque, bulgarian, catalan, croatian, czech, danish, dutch, esperanto, e stonian, finnish, greek, icelandic, irish, italian, latin, magyar, norsk, polis h, portuges, romanian, russian, serbian, slovak, slovene, spanish, swedish, tur kish, ukrainian, nohyphenation, loaded. (/usr/share/texmf/tex/latex/base/article.cls Document Class: article 2004/02/16 v1.4f Standard LaTeX document class (/usr/share/texmf/tex/latex/base/size10.clo)) (/usr/share/texmf/tex/latex/tools/bm.sty) (/usr/share/texmf/tex/latex/a4wide/a4wide.sty (/usr/share/texmf/tex/latex/ntgclass/a4.sty)) ! Undefined control sequence. l.12 \set (/usr/share/texmf/tex/latex/graphics/graphicx.sty (/usr/share/texmf/tex/latex/graphics/keyval.sty) (/usr/share/texmf/tex/latex/graphics/graphics.sty (/usr/share/texmf/tex/latex/graphics/trig.sty) (/usr/share/texmf/tex/latex/graphics/graphics.cfg) (/usr/share/texmf/tex/latex/graphics/pdftex.def))) (./scipy.linsolve.umfpack.aux) [1{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex .map}] (./scipy.linsolve.umfpack.toc) [1] Overfull \hbox (32.25606pt too wide) in paragraph at lines 66--69 \OT1/cmr/m/n/10 A. Davis. All Rights Re-served. UMF-PACK home-page: http://www. cise.ufl.edu/research/sparse/umfpack Overfull \hbox (68.30923pt too wide) in paragraph at lines 84--88 []\OT1/cmr/m/n/10 [umfpack] li-brary[]dirs = /UMFPACK/UMFPACK/Lib in-clude []dirs = /UMFPACK/UMFPACK/Include Overfull \hbox (90.36482pt too wide) in paragraph at lines 91--95 []\OT1/cmr/m/n/10 [amd] li-brary[]dirs = /UFsparse/AMD/Lib in-clude[]dirs = /UFsparse/AMD/Include, /UFsparse/UFconfig Overfull \hbox (49.97585pt too wide) in paragraph at lines 96--100 []\OT1/cmr/m/n/10 [umfpack] li-brary[]dirs = /UFsparse/UMFPACK/Lib in-clud e[]dirs = /UFsparse/UMFPACK/Include, ! You can't use `macro parameter character #' in vertical mode. l.109 # Contruct the solver. ! You can't use `macro parameter character #' in horizontal mode. l.110 umfpack = um.UmfpackContext() # Use default 'di' family of UMFPACK rou... ! You can't use `macro parameter character #' in vertical mode. l.112 # One-shot solution. ! You can't use `macro parameter character #' in horizontal mode. l.114 # same as: ! You can't use `macro parameter character #' in vertical mode. l.119 # Make LU decomposition. ! You can't use `macro parameter character #' in horizontal mode. l.122 # Use already LU-decomposed matrix. ! You can't use `macro parameter character #' in horizontal mode. l.125 # same as: Overfull \hbox (6.97574pt too wide) in paragraph at lines 119--128 \OT1/cmr/m/n/10 umf-pack( um.UMFPACK[]A, mtx, rhs1, au-to-Trans-pose = True ) s ol2 = umf-pack( um.UMFPACK[]A, ! You can't use `macro parameter character #' in vertical mode. l.131 # Make symbolic decomposition. ! You can't use `macro parameter character #' in horizontal mode. l.133 # Print statistics. Overfull \hbox (14.16289pt too wide) in paragraph at lines 131--135 []\OT1/cmr/m/n/10 Make sym-bolic de-com-po-si-tion. umf-pack.symbolic( mtx0 ) Print statis-tics. umf-pack.report[]symbolic() ! You can't use `macro parameter character #' in vertical mode. l.138 # Make LU decomposition of mtx1 which has same structure as mtx0. ! You can't use `macro parameter character #' in horizontal mode. l.140 # Print statistics. ! You can't use `macro parameter character #' in vertical mode. l.143 # Use already LU-decomposed matrix. ! You can't use `macro parameter character #' in vertical mode. l.148 # Make LU decomposition of mtx2 which has same structure as mtx0. ! You can't use `macro parameter character #' in vertical mode. l.152 # Print all statistics. ! You can't use `macro parameter character #' in vertical mode. l.157 # Get LU factors and permutation matrices of a matrix. [2] ! You can't use `macro parameter character #' in horizontal mode. l.197 umfpack.control[um.UMFPACK\_PRL] = 4 # Let's be more verbose. [3] (./scipy.linsolve.umfpack.aux) ) (see the transcript file for additional information) Output written on scipy.linsolve.umfpack.pdf (4 pages, 49489 bytes). Transcript written on scipy.linsolve.umfpack.log. This is pdfeTeX, Version 3.141592-1.21a-2.2 (Web2C 7.5.4) entering extended mode (./scipy.linsolve.umfpack.tex LaTeX2e <2003/12/01> Babel and hyphenation patterns for american, french, german, ngerman, b ahasa, basque, bulgarian, catalan, croatian, czech, danish, dutch, esperanto, e stonian, finnish, greek, icelandic, irish, italian, latin, magyar, norsk, polis h, portuges, romanian, russian, serbian, slovak, slovene, spanish, swedish, tur kish, ukrainian, nohyphenation, loaded. (/usr/share/texmf/tex/latex/base/article.cls Document Class: article 2004/02/16 v1.4f Standard LaTeX document class (/usr/share/texmf/tex/latex/base/size10.clo)) (/usr/share/texmf/tex/latex/tools/bm.sty) (/usr/share/texmf/tex/latex/a4wide/a4wide.sty (/usr/share/texmf/tex/latex/ntgclass/a4.sty)) ! Undefined control sequence. l.12 \set (/usr/share/texmf/tex/latex/graphics/graphicx.sty (/usr/share/texmf/tex/latex/graphics/keyval.sty) (/usr/share/texmf/tex/latex/graphics/graphics.sty (/usr/share/texmf/tex/latex/graphics/trig.sty) (/usr/share/texmf/tex/latex/graphics/graphics.cfg) (/usr/share/texmf/tex/latex/graphics/pdftex.def))) (./scipy.linsolve.umfpack.aux) [1{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex .map}] (./scipy.linsolve.umfpack.toc) [1] Overfull \hbox (32.25606pt too wide) in paragraph at lines 66--69 \OT1/cmr/m/n/10 A. Davis. All Rights Re-served. UMF-PACK home-page: http://www. cise.ufl.edu/research/sparse/umfpack Overfull \hbox (68.30923pt too wide) in paragraph at lines 84--88 []\OT1/cmr/m/n/10 [umfpack] li-brary[]dirs = /UMFPACK/UMFPACK/Lib in-clude []dirs = /UMFPACK/UMFPACK/Include Overfull \hbox (90.36482pt too wide) in paragraph at lines 91--95 []\OT1/cmr/m/n/10 [amd] li-brary[]dirs = /UFsparse/AMD/Lib in-clude[]dirs = /UFsparse/AMD/Include, /UFsparse/UFconfig Overfull \hbox (49.97585pt too wide) in paragraph at lines 96--100 []\OT1/cmr/m/n/10 [umfpack] li-brary[]dirs = /UFsparse/UMFPACK/Lib in-clud e[]dirs = /UFsparse/UMFPACK/Include, ! You can't use `macro parameter character #' in vertical mode. l.109 # Contruct the solver. ! You can't use `macro parameter character #' in horizontal mode. l.110 umfpack = um.UmfpackContext() # Use default 'di' family of UMFPACK rou... ! You can't use `macro parameter character #' in vertical mode. l.112 # One-shot solution. ! You can't use `macro parameter character #' in horizontal mode. l.114 # same as: ! You can't use `macro parameter character #' in vertical mode. l.119 # Make LU decomposition. ! You can't use `macro parameter character #' in horizontal mode. l.122 # Use already LU-decomposed matrix. ! You can't use `macro parameter character #' in horizontal mode. l.125 # same as: Overfull \hbox (6.97574pt too wide) in paragraph at lines 119--128 \OT1/cmr/m/n/10 umf-pack( um.UMFPACK[]A, mtx, rhs1, au-to-Trans-pose = True ) s ol2 = umf-pack( um.UMFPACK[]A, ! You can't use `macro parameter character #' in vertical mode. l.131 # Make symbolic decomposition. ! You can't use `macro parameter character #' in horizontal mode. l.133 # Print statistics. Overfull \hbox (14.16289pt too wide) in paragraph at lines 131--135 []\OT1/cmr/m/n/10 Make sym-bolic de-com-po-si-tion. umf-pack.symbolic( mtx0 ) Print statis-tics. umf-pack.report[]symbolic() ! You can't use `macro parameter character #' in vertical mode. l.138 # Make LU decomposition of mtx1 which has same structure as mtx0. ! You can't use `macro parameter character #' in horizontal mode. l.140 # Print statistics. ! You can't use `macro parameter character #' in vertical mode. l.143 # Use already LU-decomposed matrix. ! You can't use `macro parameter character #' in vertical mode. l.148 # Make LU decomposition of mtx2 which has same structure as mtx0. ! You can't use `macro parameter character #' in vertical mode. l.152 # Print all statistics. ! You can't use `macro parameter character #' in vertical mode. l.157 # Get LU factors and permutation matrices of a matrix. [2] ! You can't use `macro parameter character #' in horizontal mode. l.197 umfpack.control[um.UMFPACK\_PRL] = 4 # Let's be more verbose. [3] (./scipy.linsolve.umfpack.aux) ) (see the transcript file for additional information) Output written on scipy.linsolve.umfpack.pdf (4 pages, 49489 bytes). Transcript written on scipy.linsolve.umfpack.log. This is pdfeTeX, Version 3.141592-1.21a-2.2 (Web2C 7.5.4) entering extended mode (./scipy.linsolve.umfpack.tex LaTeX2e <2003/12/01> Babel and hyphenation patterns for american, french, german, ngerman, b ahasa, basque, bulgarian, catalan, croatian, czech, danish, dutch, esperanto, e stonian, finnish, greek, icelandic, irish, italian, latin, magyar, norsk, polis h, portuges, romanian, russian, serbian, slovak, slovene, spanish, swedish, tur kish, ukrainian, nohyphenation, loaded. (/usr/share/texmf/tex/latex/base/article.cls Document Class: article 2004/02/16 v1.4f Standard LaTeX document class (/usr/share/texmf/tex/latex/base/size10.clo)) (/usr/share/texmf/tex/latex/tools/bm.sty) (/usr/share/texmf/tex/latex/a4wide/a4wide.sty (/usr/share/texmf/tex/latex/ntgclass/a4.sty)) ! Undefined control sequence. l.12 \set (/usr/share/texmf/tex/latex/graphics/graphicx.sty (/usr/share/texmf/tex/latex/graphics/keyval.sty) (/usr/share/texmf/tex/latex/graphics/graphics.sty (/usr/share/texmf/tex/latex/graphics/trig.sty) (/usr/share/texmf/tex/latex/graphics/graphics.cfg) (/usr/share/texmf/tex/latex/graphics/pdftex.def))) (./scipy.linsolve.umfpack.aux) [1{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex .map}] (./scipy.linsolve.umfpack.toc) [1] Overfull \hbox (32.25606pt too wide) in paragraph at lines 66--69 \OT1/cmr/m/n/10 A. Davis. All Rights Re-served. UMF-PACK home-page: http://www. cise.ufl.edu/research/sparse/umfpack Overfull \hbox (68.30923pt too wide) in paragraph at lines 84--88 []\OT1/cmr/m/n/10 [umfpack] li-brary[]dirs = /UMFPACK/UMFPACK/Lib in-clude []dirs = /UMFPACK/UMFPACK/Include Overfull \hbox (90.36482pt too wide) in paragraph at lines 91--95 []\OT1/cmr/m/n/10 [amd] li-brary[]dirs = /UFsparse/AMD/Lib in-clude[]dirs = /UFsparse/AMD/Include, /UFsparse/UFconfig Overfull \hbox (49.97585pt too wide) in paragraph at lines 96--100 []\OT1/cmr/m/n/10 [umfpack] li-brary[]dirs = /UFsparse/UMFPACK/Lib in-clud e[]dirs = /UFsparse/UMFPACK/Include, ! You can't use `macro parameter character #' in vertical mode. l.109 # Contruct the solver. ! You can't use `macro parameter character #' in horizontal mode. l.110 umfpack = um.UmfpackContext() # Use default 'di' family of UMFPACK rou... ! You can't use `macro parameter character #' in vertical mode. l.112 # One-shot solution. ! You can't use `macro parameter character #' in horizontal mode. l.114 # same as: ! You can't use `macro parameter character #' in vertical mode. l.119 # Make LU decomposition. ! You can't use `macro parameter character #' in horizontal mode. l.122 # Use already LU-decomposed matrix. ! You can't use `macro parameter character #' in horizontal mode. l.125 # same as: Overfull \hbox (6.97574pt too wide) in paragraph at lines 119--128 \OT1/cmr/m/n/10 umf-pack( um.UMFPACK[]A, mtx, rhs1, au-to-Trans-pose = True ) s ol2 = umf-pack( um.UMFPACK[]A, ! You can't use `macro parameter character #' in vertical mode. l.131 # Make symbolic decomposition. ! You can't use `macro parameter character #' in horizontal mode. l.133 # Print statistics. Overfull \hbox (14.16289pt too wide) in paragraph at lines 131--135 []\OT1/cmr/m/n/10 Make sym-bolic de-com-po-si-tion. umf-pack.symbolic( mtx0 ) Print statis-tics. umf-pack.report[]symbolic() ! You can't use `macro parameter character #' in vertical mode. l.138 # Make LU decomposition of mtx1 which has same structure as mtx0. ! You can't use `macro parameter character #' in horizontal mode. l.140 # Print statistics. ! You can't use `macro parameter character #' in vertical mode. l.143 # Use already LU-decomposed matrix. ! You can't use `macro parameter character #' in vertical mode. l.148 # Make LU decomposition of mtx2 which has same structure as mtx0. ! You can't use `macro parameter character #' in vertical mode. l.152 # Print all statistics. ! You can't use `macro parameter character #' in vertical mode. l.157 # Get LU factors and permutation matrices of a matrix. [2] ! You can't use `macro parameter character #' in horizontal mode. l.197 umfpack.control[um.UMFPACK\_PRL] = 4 # Let's be more verbose. [3] (./scipy.linsolve.umfpack.aux) ) (see the transcript file for additional information) Output written on scipy.linsolve.umfpack.pdf (4 pages, 49489 bytes). Transcript written on scipy.linsolve.umfpack.log. How can I fix the problem ? Nils From oliphant at enthought.com Wed Nov 28 19:05:33 2007 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 28 Nov 2007 18:05:33 -0600 Subject: [Numpy-discussion] Converting char array to float In-Reply-To: <8fb8cc060711281021r19eae508o1328be4e3a69b552@mail.gmail.com> References: <8fb8cc060711281021r19eae508o1328be4e3a69b552@mail.gmail.com> Message-ID: <474E024D.9010306@enthought.com> Sameer DCosta wrote: > I'm trying to convert a character array to a floating point array. I'm > using one of the recent svn builds. It is surprising that astype does > not do the job. However if I first convert the char array to an array > and then use astype everything works fine. Is this a bug? > > import numpy as N > print N.__version__ # Output = '1.0.5.dev4426' > a = N.char.array(['123.45', '234.56']) > b = N.array(a).astype(N.float) # This works. > print b > b = a.astype(N.float) # This does not work and raises an exception > > ValueError: Can only create a chararray from string data. > > The problem is that astype for a chararray will by default try to create a class of chararray. This sub-class only allows string data and so the base-class astype(float) will fail. The astype method could be over-ridden in order to support automatic conversion to other kinds of arrays, but I lean towards asking "why?" because "explicit is better than implicit" (although it is admittedly arguable which is which in this case...) -Travis > Thanks. > Sameer > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Wed Nov 28 19:27:01 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 28 Nov 2007 17:27:01 -0700 Subject: [Numpy-discussion] How can I constrain linear_least_squares to integer solutions? In-Reply-To: <20071128075937.GC23287@mentat.za.net> References: <20071128075937.GC23287@mentat.za.net> Message-ID: On Nov 28, 2007 12:59 AM, Stefan van der Walt wrote: > On Tue, Nov 27, 2007 at 11:07:30PM -0700, Charles R Harris wrote: > > This is not a trivial problem, as you can see by googling mixed integer > least > > squares (MILS). Much will depend on the nature of the parameters, the > number of > > variables you are using in the fit, and how exact the solution needs to > be. One > > approach would be to start by rounding the coefficients that must be > integer > > and improve the solution using annealing or genetic algorithms to jig > the > > integer coefficients while fitting the remainder in the usual least > square way, > > but that wouldn't have the elegance of some of the specific methods used > for > > this sort of problem. However, I don't know of a package in scipy that > > implements those more sophisticated algorithms, perhaps someone else on > this > > list who knows more about these things than I can point you in the right > > direction. > > Would this be a good candidate for a genetic algorithm? I haven't > used GA before, so I don't know the typical rate of convergence or its > applicability to optimization problems. > It depends. Just to show the sort of problems involved, suppose you have 32 integer variables and are looking for the last bit of optimization. If the floating point optimum is at (.5, .5, ...., .5) and the error is symmetrical, then each vertex of the surrounding integer cube is a solution and there are 2**32 of them. If the error isn't symmetrical, and chances are that with that many variables it is very far from that, then you have to search a larger region. That's a lot of points. The more sophisticated algorithms try to eliminate whole regions of points and keep narrowing things down, but even so the problem can easily get out of hand. If you just need a good solution, a genetic algorithm is a good bet to find one without too much hassle. I had a similar problem in designing a digital min/max FIR filter where I needed 15 bit integer coefficients for hardware implementation. There was a narrow high rejection band in the filter and simply rounding the coefficients left spikes in the response through that band. With a GA I was able to eliminate the spikes in about 30 minutes of evolution using a python/Numeric program. In that case the performance of annealing was quite dependent on choosing the right parameters for cooling rate, etc., while the GA was quite robust and straight forward. There was no guarantee the I ended up with the best solution, but what I got was good enough. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From discerptor at gmail.com Thu Nov 29 01:21:58 2007 From: discerptor at gmail.com (Joshua Lippai) Date: Wed, 28 Nov 2007 22:21:58 -0800 Subject: [Numpy-discussion] Building NumPy on Mac OS X without Apple GCC Message-ID: <9911419a0711282221s14ea24ddy558d8d3cf6a6681b@mail.gmail.com> I updated my GCC to a more recent version a day ago, since Apple's Xcode Tools only provide GCC 4.0 and the current release of GNU's GCC is 4.2. I successfully achieved this, but now I run into a problem when trying to build NumPy: gcc: unrecognized option '-no-cpp-precomp' cc1: error: unrecognized command line option "-arch" cc1: error: unrecognized command line option "-arch" cc1: error: unrecognized command line option "-Wno-long-double" gcc: unrecognized option '-no-cpp-precomp' cc1: error: unrecognized command line option "-arch" cc1: error: unrecognized command line option "-arch" cc1: error: unrecognized command line option "-Wno-long-double" Upon investigation into the matter, I found out that these options (no-cpp-precomp and Wno-long-double) are only valid in Apple's GCC and not the regular GNU release. Yet it seems NumPy automatically assumes Apple's GCC is being used when it realizes the target is OS X. Is there a way around this, or at least some way to specify Apple's GCC? NumPy is the only package I've tried building so far that has a problem with this. From robert.kern at gmail.com Thu Nov 29 01:42:40 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 29 Nov 2007 00:42:40 -0600 Subject: [Numpy-discussion] Building NumPy on Mac OS X without Apple GCC In-Reply-To: <9911419a0711282221s14ea24ddy558d8d3cf6a6681b@mail.gmail.com> References: <9911419a0711282221s14ea24ddy558d8d3cf6a6681b@mail.gmail.com> Message-ID: <474E5F60.2060507@gmail.com> Joshua Lippai wrote: > I updated my GCC to a more recent version a day ago, since Apple's > Xcode Tools only provide GCC 4.0 and the current release of GNU's GCC > is 4.2. I successfully achieved this, but now I run into a problem > when trying to build NumPy: > > gcc: unrecognized option '-no-cpp-precomp' > cc1: error: unrecognized command line option "-arch" > cc1: error: unrecognized command line option "-arch" > cc1: error: unrecognized command line option "-Wno-long-double" > gcc: unrecognized option '-no-cpp-precomp' > cc1: error: unrecognized command line option "-arch" > cc1: error: unrecognized command line option "-arch" > cc1: error: unrecognized command line option "-Wno-long-double" > > > Upon investigation into the matter, I found out that these options > (no-cpp-precomp and Wno-long-double) are only valid in Apple's GCC and > not the regular GNU release. Yet it seems NumPy automatically assumes > Apple's GCC is being used when it realizes the target is OS X. Is > there a way around this, or at least some way to specify Apple's GCC? > NumPy is the only package I've tried building so far that has a > problem with this. I'm surprised that you've built other Python extension modules because numpy does not add these flags; Python does. Python extensions should be built with the same compiler that Python itself was built with. If you are using the binary distribution from www.python.org, you should use Apple's gcc, not a different one. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From discerptor at gmail.com Thu Nov 29 01:56:40 2007 From: discerptor at gmail.com (Joshua Lippai) Date: Thu, 29 Nov 2007 06:56:40 +0000 Subject: [Numpy-discussion] Building NumPy on Mac OS X without Apple GCC In-Reply-To: <474E5F60.2060507@gmail.com> References: <9911419a0711282221s14ea24ddy558d8d3cf6a6681b@mail.gmail.com> <474E5F60.2060507@gmail.com> Message-ID: <9911419a0711282256p5ff4f9ecvaa90837dfec8b60b@mail.gmail.com> > Joshua Lippai wrote: > > I updated my GCC to a more recent version a day ago, since Apple's > > Xcode Tools only provide GCC 4.0 and the current release of GNU's GCC > > is 4.2. I successfully achieved this, but now I run into a problem > > when trying to build NumPy: > > > > gcc: unrecognized option '-no-cpp-precomp' > > cc1: error: unrecognized command line option "-arch" > > cc1: error: unrecognized command line option "-arch" > > cc1: error: unrecognized command line option "-Wno-long-double" > > gcc: unrecognized option '-no-cpp-precomp' > > cc1: error: unrecognized command line option "-arch" > > cc1: error: unrecognized command line option "-arch" > > cc1: error: unrecognized command line option "-Wno-long-double" > > > > > > Upon investigation into the matter, I found out that these options > > (no-cpp-precomp and Wno-long-double) are only valid in Apple's GCC and > > not the regular GNU release. Yet it seems NumPy automatically assumes > > Apple's GCC is being used when it realizes the target is OS X. Is > > there a way around this, or at least some way to specify Apple's GCC? > > NumPy is the only package I've tried building so far that has a > > problem with this. > > I'm surprised that you've built other Python extension modules because numpy > does not add these flags; Python does. Python extensions should be built with > the same compiler that Python itself was built with. If you are using the binary > distribution from www.python.org, you should use Apple's gcc, not a different one. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Thanks for the reply. Well, I built my Python stuff, including NumPy previously, before I changed to the higher GCC version. Do you know if there's an option I can toggle that will specify Apple's GCC to be used? From robert.kern at gmail.com Thu Nov 29 02:06:29 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 29 Nov 2007 01:06:29 -0600 Subject: [Numpy-discussion] Building NumPy on Mac OS X without Apple GCC In-Reply-To: <9911419a0711282256p5ff4f9ecvaa90837dfec8b60b@mail.gmail.com> References: <9911419a0711282221s14ea24ddy558d8d3cf6a6681b@mail.gmail.com> <474E5F60.2060507@gmail.com> <9911419a0711282256p5ff4f9ecvaa90837dfec8b60b@mail.gmail.com> Message-ID: <474E64F5.5090306@gmail.com> Joshua Lippai wrote: > Thanks for the reply. Well, I built my Python stuff, including NumPy > previously, before I changed to the higher GCC version. Do you know if > there's an option I can toggle that will specify Apple's GCC to be > used? $ CC=/usr/bin/gcc python setup.py build -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From discerptor at gmail.com Thu Nov 29 03:51:56 2007 From: discerptor at gmail.com (Joshua Lippai) Date: Thu, 29 Nov 2007 00:51:56 -0800 Subject: [Numpy-discussion] Building NumPy on Mac OS X without Apple GCC In-Reply-To: <474E64F5.5090306@gmail.com> References: <9911419a0711282221s14ea24ddy558d8d3cf6a6681b@mail.gmail.com> <474E5F60.2060507@gmail.com> <9911419a0711282256p5ff4f9ecvaa90837dfec8b60b@mail.gmail.com> <474E64F5.5090306@gmail.com> Message-ID: <9911419a0711290051y4f14af60v9eeaa9e3d1138eb2@mail.gmail.com> > > > Thanks for the reply. Well, I built my Python stuff, including NumPy > > previously, before I changed to the higher GCC version. Do you know if > > there's an option I can toggle that will specify Apple's GCC to be > > used? > > $ CC=/usr/bin/gcc python setup.py build > > -- > > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Thanks, it worked perfectly. I'd hate to be a little off topic in the NumPy discussion and bug you, but when I try to compile the latest scipy, I get: g++: unrecognized option '-no-cpp-precomp' cc1plus: error: unrecognized command line option "-arch" cc1plus: error: unrecognized command line option "-arch" cc1plus: error: unrecognized command line option "-Wno-long-double" g++: unrecognized option '-no-cpp-precomp' cc1plus: error: unrecognized command line option "-arch" cc1plus: error: unrecognized command line option "-arch" cc1plus: error: unrecognized command line option "-Wno-long-double" And I assume from the way it looks that it's due to a similar reason as my previous compiling problem. I tried adding in CCplus=/usr/bin/g++ but it doesn't seem to do the trick. Again, sorry to keep bugging on an issue I created in my compiling environment, but is there something different I should be typing? From cimrman3 at ntc.zcu.cz Thu Nov 29 04:01:26 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Thu, 29 Nov 2007 10:01:26 +0100 Subject: [Numpy-discussion] documentation generator based on pyparsing In-Reply-To: References: <474D4300.6020904@ntc.zcu.cz> Message-ID: <474E7FE6.7040604@ntc.zcu.cz> Hi Nils, Nils Wagner wrote: > The output of > > ./gendocs.py -m 'scipy.linsolve.umfpack' > > differs from your example output (available at > http://scipy.org/Generate_Documentation) I had to update the umfpack info.py file (where the module docstring is) to conform the documentation standards. The gendocs.py relies on that, so use, please, the newest SVN version of scipy - it should work with rev. 3601 and later. r. From david at ar.media.kyoto-u.ac.jp Thu Nov 29 03:53:16 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 29 Nov 2007 17:53:16 +0900 Subject: [Numpy-discussion] Building NumPy on Mac OS X without Apple GCC In-Reply-To: <9911419a0711290051y4f14af60v9eeaa9e3d1138eb2@mail.gmail.com> References: <9911419a0711282221s14ea24ddy558d8d3cf6a6681b@mail.gmail.com> <474E5F60.2060507@gmail.com> <9911419a0711282256p5ff4f9ecvaa90837dfec8b60b@mail.gmail.com> <474E64F5.5090306@gmail.com> <9911419a0711290051y4f14af60v9eeaa9e3d1138eb2@mail.gmail.com> Message-ID: <474E7DFC.2010602@ar.media.kyoto-u.ac.jp> Joshua Lippai wrote: > > Thanks, it worked perfectly. I'd hate to be a little off topic in the > NumPy discussion and bug you, but when I try to compile the latest > scipy, I get: > > g++: unrecognized option '-no-cpp-precomp' > cc1plus: error: unrecognized command line option "-arch" > cc1plus: error: unrecognized command line option "-arch" > cc1plus: error: unrecognized command line option "-Wno-long-double" > g++: unrecognized option '-no-cpp-precomp' > cc1plus: error: unrecognized command line option "-arch" > cc1plus: error: unrecognized command line option "-arch" > cc1plus: error: unrecognized command line option "-Wno-long-double" > > And I assume from the way it looks that it's due to a similar reason > as my previous compiling problem. I tried adding in > CCplus=/usr/bin/g++ but it doesn't seem to do the trick. Again, sorry > to keep bugging on an issue I created in my compiling environment, but > is there something different I should be typing? > The usual makefile variable for c++ makefiles is CXX, not CCplus, so I would suggest trying that first. David From discerptor at gmail.com Thu Nov 29 04:34:27 2007 From: discerptor at gmail.com (Joshua Lippai) Date: Thu, 29 Nov 2007 01:34:27 -0800 Subject: [Numpy-discussion] Building NumPy on Mac OS X without Apple GCC In-Reply-To: <474E7DFC.2010602@ar.media.kyoto-u.ac.jp> References: <9911419a0711282221s14ea24ddy558d8d3cf6a6681b@mail.gmail.com> <474E5F60.2060507@gmail.com> <9911419a0711282256p5ff4f9ecvaa90837dfec8b60b@mail.gmail.com> <474E64F5.5090306@gmail.com> <9911419a0711290051y4f14af60v9eeaa9e3d1138eb2@mail.gmail.com> <474E7DFC.2010602@ar.media.kyoto-u.ac.jp> Message-ID: <9911419a0711290134g464247c8s6c0a2befc0a6c1a4@mail.gmail.com> > Joshua Lippai wrote: > > > > Thanks, it worked perfectly. I'd hate to be a little off topic in the > > NumPy discussion and bug you, but when I try to compile the latest > > scipy, I get: > > > > g++: unrecognized option '-no-cpp-precomp' > > cc1plus: error: unrecognized command line option "-arch" > > cc1plus: error: unrecognized command line option "-arch" > > cc1plus: error: unrecognized command line option "-Wno-long-double" > > g++: unrecognized option '-no-cpp-precomp' > > cc1plus: error: unrecognized command line option "-arch" > > cc1plus: error: unrecognized command line option "-arch" > > cc1plus: error: unrecognized command line option "-Wno-long-double" > > > > And I assume from the way it looks that it's due to a similar reason > > as my previous compiling problem. I tried adding in > > CCplus=/usr/bin/g++ but it doesn't seem to do the trick. Again, sorry > > to keep bugging on an issue I created in my compiling environment, but > > is there something different I should be typing? > > > The usual makefile variable for c++ makefiles is CXX, not CCplus, so I > would suggest trying that first. > > David > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Thanks, that did it. Everything's building correctly again. From engel at physics.harvard.edu Thu Nov 29 09:48:23 2007 From: engel at physics.harvard.edu (Hans-Andreas Engel) Date: Thu, 29 Nov 2007 14:48:23 +0000 (UTC) Subject: [Numpy-discussion] How to remove loops over inner() References: Message-ID: > Essentially, you want to operate on a stack of two dimensional arrays, correct? Yes, this is correct -- and I also think that one should be able to provide a list of axes to be ignored. > I'd be mildly supportive of something like this for tensordot; I'd prefer > more descriptive name for keep_axis, but I don't know what it would be off > the top of my head. In any event it should be XXX_axes and optionally take a > sequence of axes so that more than one can be ignored. You could trivially > build more specific functions on top of tensordot, so I don't see that inner > needs to be changed as it's basically a convenience function anyway. Good point; indeed a generalization of |tensordot()| would be sufficient for me! I don't know the best naming here, |ignore_axes| might be a bit more descriptive. Would this feature only require a few lines of code? Would this be easy to implement for an expert on the intricacies of the numpy internals? Best, Hansres From giorgio at gilestro.tk Thu Nov 29 16:54:26 2007 From: giorgio at gilestro.tk (Giorgio F. Gilestro) Date: Thu, 29 Nov 2007 15:54:26 -0600 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc Message-ID: <474F3512.7000908@gilestro.tk> Hi guys, does anyone of you happen to have sitting somewhere a DMG of a recent version of SciPy compiled for MacOSX 10.4? The SciPy webpage does not carry official releases and it is sending me to the Scipy Superpack by Chris Fonnesbeck but that superpack seems to be for intel cpu only. I didn't think making SciPy working in mac would have been such a pain in the bum. Linux and Win worked without problem but mac is driving me crazy (yes, compiling is a problem too.) Thanks! -- giorgio at gilestro.tk http://www.cafelamarck.it From zpincus at stanford.edu Thu Nov 29 17:32:52 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Thu, 29 Nov 2007 17:32:52 -0500 Subject: [Numpy-discussion] display numpy array as image Message-ID: Hello all, I'm curious if people have experience with / preferences for how to display a numpy array onscreen as an image. Pyglet looks relatively easy -- you can feed an image buffer object with a string or a ctypes pointer. I presume getting a string from an array is plenty fast, but the ctypes pointer option is intriguing as it allows for dealing with simple strided arrays (the image objects allow for an arbitrary number of bytes between rows). Is it possible to get a ctypes pointer to the beginning of the array buffer from numpy without too much ugliness? wxPython looks pretty easy too, as there are facilities for getting pixels from a buffer. Does anyone have any experience with these? Are there ways of allowing a numpy array and a wxPython image to point to the same memory? Anyhow, these are specific questions, but I'd also appreciate any general thoughts about good approaches for getting pixels from numpy arrays onscreen. Thanks, Zach From giorgio at gilestro.tk Thu Nov 29 17:48:02 2007 From: giorgio at gilestro.tk (Giorgio F. Gilestro) Date: Thu, 29 Nov 2007 16:48:02 -0600 Subject: [Numpy-discussion] display numpy array as image In-Reply-To: References: Message-ID: <474F41A2.20105@gilestro.tk> I am not sure I got what you mean but I am using PIL to convert arrays to images and viceversa see http://mail.python.org/pipermail/image-sig/2006-September/004099.html I embed bmps using wxpython. Zachary Pincus wrote: > Hello all, > > I'm curious if people have experience with / preferences for how to > display a numpy array onscreen as an image. > > Pyglet looks relatively easy -- you can feed an image buffer object > with a string or a ctypes pointer. I presume getting a string from an > array is plenty fast, but the ctypes pointer option is intriguing as > it allows for dealing with simple strided arrays (the image objects > allow for an arbitrary number of bytes between rows). Is it possible > to get a ctypes pointer to the beginning of the array buffer from > numpy without too much ugliness? > > wxPython looks pretty easy too, as there are facilities for getting > pixels from a buffer. Does anyone have any experience with these? Are > there ways of allowing a numpy array and a wxPython image to point to > the same memory? > > Anyhow, these are specific questions, but I'd also appreciate any > general thoughts about good approaches for getting pixels from numpy > arrays onscreen. > > Thanks, > > Zach > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- giorgio at gilestro.tk http://www.cafelamarck.it From discerptor at gmail.com Thu Nov 29 18:05:06 2007 From: discerptor at gmail.com (Joshua Lippai) Date: Thu, 29 Nov 2007 15:05:06 -0800 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <474F3512.7000908@gilestro.tk> References: <474F3512.7000908@gilestro.tk> Message-ID: <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> It's actually pretty simple to compile it yourself once you've installed the latest Xcode from http://developer.apple.com and X11 from the OS X Tiger install disc. The instructions on Scipy's official OS X installation page ( http://scipy.org/Installing_SciPy/Mac_OS_X ) are great for that. That said, it is true that the Scipy Superpack is now Intel-only. You should probably email the guy and ask him to make older versions of the superpack available, or at least the last PPC one he made. Best of luck. Josh On Nov 29, 2007 1:54 PM, Giorgio F. Gilestro wrote: > Hi guys, > does anyone of you happen to have sitting somewhere a DMG of a recent > version of SciPy compiled for MacOSX 10.4? > The SciPy webpage does not carry official releases and it is sending me > to the Scipy Superpack by Chris Fonnesbeck but that superpack seems to > be for intel cpu only. I didn't think making SciPy working in mac would > have been such a pain in the bum. Linux and Win worked without problem > but mac is driving me crazy (yes, compiling is a problem too.) > Thanks! > > > -- > giorgio at gilestro.tk > http://www.cafelamarck.it > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From amcmorl at gmail.com Thu Nov 29 18:08:00 2007 From: amcmorl at gmail.com (Angus McMorland) Date: Fri, 30 Nov 2007 12:08:00 +1300 Subject: [Numpy-discussion] display numpy array as image In-Reply-To: References: Message-ID: On 30/11/2007, Zachary Pincus wrote: > Hello all, > > I'm curious if people have experience with / preferences for how to > display a numpy array onscreen as an image. I'm not sure if you're after anything specific, but a very convenient way to show 2-D arrays on screen is matplotlib (mpl), which is the recommended graphical interface to numpy. With the pylab interface to mpl, the command is as simple as >>>pylab.imshow(arr) or >>>pylab.matshow(arr) for slightly different axis behaviour. Angus. -- AJC McMorland, PhD Student Physiology, University of Auckland From discerptor at gmail.com Thu Nov 29 18:13:58 2007 From: discerptor at gmail.com (Joshua Lippai) Date: Thu, 29 Nov 2007 15:13:58 -0800 Subject: [Numpy-discussion] display numpy array as image In-Reply-To: References: Message-ID: <9911419a0711291513x25ce5ab0p7e1ded75417d6a2b@mail.gmail.com> On Nov 29, 2007 2:32 PM, Zachary Pincus wrote: > Hello all, > > I'm curious if people have experience with / preferences for how to > display a numpy array onscreen as an image. > > Pyglet looks relatively easy -- you can feed an image buffer object > with a string or a ctypes pointer. I presume getting a string from an > array is plenty fast, but the ctypes pointer option is intriguing as > it allows for dealing with simple strided arrays (the image objects > allow for an arbitrary number of bytes between rows). Is it possible > to get a ctypes pointer to the beginning of the array buffer from > numpy without too much ugliness? > > wxPython looks pretty easy too, as there are facilities for getting > pixels from a buffer. Does anyone have any experience with these? Are > there ways of allowing a numpy array and a wxPython image to point to > the same memory? > > Anyhow, these are specific questions, but I'd also appreciate any > general thoughts about good approaches for getting pixels from numpy > arrays onscreen. > > Thanks, > > Zach > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Have you tried the matplotlib/pylab module? It has pretty simple ways of doing what you're thinking of if I'm understanding your intent correctly.. Josh From Chris.Barker at noaa.gov Thu Nov 29 18:19:03 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 29 Nov 2007 15:19:03 -0800 Subject: [Numpy-discussion] display numpy array as image In-Reply-To: References: Message-ID: <474F48E7.9050301@noaa.gov> Zachary Pincus wrote: > wxPython looks pretty easy too, as there are facilities for getting > pixels from a buffer. Does anyone have any experience with these? some. > Are > there ways of allowing a numpy array and a wxPython image to point to > the same memory? yup. You can build a wxImage from a buffer, and numpy provides a buffer interface, so they end up with them sharing the same memory, as long as your numpy array is contiguous 24 rgb. I've enclosed a sample that generates a wx.Image from a numpy array, then every time you push the button, the array is altered in-place, and you can see the image change. ( think you need at least wxPython 2.8 for this to work) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: ImageNumpy.py Type: text/x-python Size: 3823 bytes Desc: not available URL: From Chris.Barker at noaa.gov Thu Nov 29 18:21:08 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 29 Nov 2007 15:21:08 -0800 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> Message-ID: <474F4964.9030700@noaa.gov> Joshua Lippai wrote: > You should probably email the guy and ask him to make > older versions of the superpack available, or at least the last PPC > one he made. Best of luck. Even better would be Universal (fat) binaries -- I'm pretty sure this is now possible, but haven't figured it out myself yet. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From discerptor at gmail.com Thu Nov 29 18:23:39 2007 From: discerptor at gmail.com (Joshua Lippai) Date: Thu, 29 Nov 2007 15:23:39 -0800 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <474F4964.9030700@noaa.gov> References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> Message-ID: <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> On Nov 29, 2007 3:21 PM, Christopher Barker wrote: > Joshua Lippai wrote: > > You should probably email the guy and ask him to make > > older versions of the superpack available, or at least the last PPC > > one he made. Best of luck. > > Even better would be Universal (fat) binaries -- I'm pretty sure this is > now possible, but haven't figured it out myself yet. > > -Chris > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Chris Fonnesbeck used to distribute both PowerPC and Intel binaries, but he can't compile them for PowerPC at all anymore because he no longer has access to a PowerPC machine. That's why now the site's manager is only distributing Intel binaries; Universal builds won't be possible for anything he does from the present time forward. Someone else would have to compile on the PPC side for him. Josh From giorgio at gilestro.tk Thu Nov 29 18:29:24 2007 From: giorgio at gilestro.tk (Giorgio F. Gilestro) Date: Thu, 29 Nov 2007 17:29:24 -0600 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> Message-ID: <474F4B54.8050902@gilestro.tk> Hi Josh, you're right I should probably email the guy directly. I eventually managed to compile it installing the latest xcode, the latest cctool to fix the __dso_handle bug, the gfortran and a couple of other things. It took me an entire afternoon and now I don't have errors on the the scipy.test (but I still have failures :( ) Now I just have to figure out why mac version of matplotlib does not seem to have matplotlib.dates and where did drange go! Thanks Joshua Lippai wrote: > It's actually pretty simple to compile it yourself once you've > installed the latest Xcode from http://developer.apple.com and X11 > from the OS X Tiger install disc. The instructions on Scipy's official > OS X installation page ( http://scipy.org/Installing_SciPy/Mac_OS_X ) > are great for that. That said, it is true that the Scipy Superpack is > now Intel-only. You should probably email the guy and ask him to make > older versions of the superpack available, or at least the last PPC > one he made. Best of luck. > > Josh > > On Nov 29, 2007 1:54 PM, Giorgio F. Gilestro wrote: > >> Hi guys, >> does anyone of you happen to have sitting somewhere a DMG of a recent >> version of SciPy compiled for MacOSX 10.4? >> The SciPy webpage does not carry official releases and it is sending me >> to the Scipy Superpack by Chris Fonnesbeck but that superpack seems to >> be for intel cpu only. I didn't think making SciPy working in mac would >> have been such a pain in the bum. Linux and Win worked without problem >> but mac is driving me crazy (yes, compiling is a problem too.) >> Thanks! >> >> >> -- >> giorgio at gilestro.tk >> http://www.cafelamarck.it >> >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- giorgio at gilestro.tk http://www.cafelamarck.it From robert.kern at gmail.com Thu Nov 29 18:33:12 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 29 Nov 2007 17:33:12 -0600 Subject: [Numpy-discussion] display numpy array as image In-Reply-To: References: Message-ID: <474F4C38.6060203@gmail.com> Zachary Pincus wrote: > Hello all, > > I'm curious if people have experience with / preferences for how to > display a numpy array onscreen as an image. > > Pyglet looks relatively easy -- you can feed an image buffer object > with a string or a ctypes pointer. I presume getting a string from an > array is plenty fast, but the ctypes pointer option is intriguing as > it allows for dealing with simple strided arrays (the image objects > allow for an arbitrary number of bytes between rows). Is it possible > to get a ctypes pointer to the beginning of the array buffer from > numpy without too much ugliness? In [16]: from numpy import * In [17]: a = arange(10) In [18]: dir(a.ctypes) Out[18]: ['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', '_arr', '_as_parameter_', '_ctypes', '_data', '_zerod', 'data', 'data_as', 'get_as_parameter', 'get_data', 'get_shape', 'get_strides', 'shape', 'shape_as', 'strides', 'strides_as'] In [22]: import ctypes In [24]: a.ctypes.data_as(ctypes.POINTER(ctypes.c_long)) Out[24]: In [25]: a.ctypes.get_shape() Out[25]: In [26]: a.ctypes.get_strides() Out[26]: In [27]: a.ctypes.get_as_parameter() Out[27]: c_void_p(27442576) You might want to use the new ctypes-based OpenGL 3.0+ package. It has numpy support a bit more directly. You can use Pyglet for its windowing and all of the other surrounding infrastructure and use OpenGL directly for the drawing. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Thu Nov 29 18:43:37 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 29 Nov 2007 15:43:37 -0800 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> Message-ID: <474F4EA9.407@noaa.gov> Joshua Lippai wrote: > Chris Fonnesbeck used to distribute both PowerPC and Intel binaries, > but he can't compile them for PowerPC at all anymore because he no > longer has access to a PowerPC machine. You can build fat binaries with a single machine -- either one. Whether Chris wants to spend his time figuring all that out is up to him, but it's not a hardware limitation. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From barrywark at gmail.com Thu Nov 29 20:05:27 2007 From: barrywark at gmail.com (Barry Wark) Date: Thu, 29 Nov 2007 17:05:27 -0800 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <474F4EA9.407@noaa.gov> References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> <474F4EA9.407@noaa.gov> Message-ID: Using the gfortran from http://r.research.att.com/tools/, it's trivial to build a universal build from source. The instructions on scipy.org won't lead you astray. I will ask around at work. Perhaps we can start building universal scipy builds for distribution. Can anyone from the scipy devs email me off list if you'd like to pursue this?... triggering a build automatically from SVN commits or such would be good. Barry On Nov 29, 2007 3:43 PM, Christopher Barker wrote: > Joshua Lippai wrote: > > Chris Fonnesbeck used to distribute both PowerPC and Intel binaries, > > but he can't compile them for PowerPC at all anymore because he no > > longer has access to a PowerPC machine. > > You can build fat binaries with a single machine -- either one. Whether > Chris wants to spend his time figuring all that out is up to him, but > it's not a hardware limitation. > > > -Chris > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From jh at physics.ucf.edu Thu Nov 29 21:03:39 2007 From: jh at physics.ucf.edu (Joe Harrington) Date: Thu, 29 Nov 2007 21:03:39 -0500 Subject: [Numpy-discussion] display numpy array as image Message-ID: <1196388219.6047.215.camel@glup.physics.ucf.edu> If you want to explore the array interactively, blink images, mess with colormaps using the mouse, rescale the image values, mark regions, add labels, look at dynamic plots of rows and columns, etc., get the ds9 image viewer and the xpa programs that come with it that allow it to communicate with other programs: ftp://sao-ftp.harvard.edu/pub/rd/ds9 http://hea-www.harvard.edu/RD/ds9/index.html Then get the Python numdisplay package, which uses xpa. You have to get numdisplay from inside the stsci_python package: http://www.stsci.edu/resources/software_hardware/pyraf/stsci_python/current/download Just grab the numdisplay directory from within that. Older versions of numdisplay are standalone but don't work perfectly. Beware, there are outdated web sites about numdisplay on the stsci site. Don't google! Run ds9 before you load numdisplay. Then you can send your python arrays to a real interactive data viewer at will. There are even mechanisms to define physical coordinates mapped from the image coordinates. --jh-- From david at ar.media.kyoto-u.ac.jp Thu Nov 29 22:17:36 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 30 Nov 2007 12:17:36 +0900 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> <474F4EA9.407@noaa.gov> Message-ID: <474F80D0.2010500@ar.media.kyoto-u.ac.jp> Barry Wark wrote: > Using the gfortran from http://r.research.att.com/tools/, it's trivial > to build a universal build from source. The instructions on scipy.org > won't lead you astray. > > I will ask around at work. Perhaps we can start building universal > scipy builds for distribution. Can anyone from the scipy devs email me > off list if you'd like to pursue this?... triggering a build > automatically from SVN commits or such would be good. That would be cool, but where to compile the binary ? scipy does not have build farm. Having a 100 % automatic way to build a binary would certainly be a step in this direction, though. There is also the need to test the compiled binary, which is not trivial with fat binaries. To sum up: - we need at least one x86 mac os X machine to build/test (I don't know if you can test python softwares through rosetta: I have only experience testing pure C softwares built as fat binaries) - we need a way to automate the build entirely - we need a way to automate the packaging (distutils can do only part of it or all of it ? Building a basic .pkg inside a dmg is not hard from the command line, but I don't know how it scales for non trivial packages). cheers, David From wfspotz at sandia.gov Thu Nov 29 23:16:42 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 29 Nov 2007 21:16:42 -0700 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <474B4E34.80900@noaa.gov> References: <473DFCC2.1050704@mur.at> <47421469.3050408@noaa.gov> <474296D8.9000405@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <474465EE.20305@mur.at> <538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov> <474B4E34.80900@noaa.gov> Message-ID: <9AC12CB0-D618-4322-A740-CD5C1EAD5AC8@sandia.gov> I have just committed the latest version of numpy.i (a swig interface file for bridging between C arrays and numerical python) to the numpy svn repository. There are three relatively new features that are now supported: * It is now possible to wrap functions that expect integer arguments and have swig generate wrappers that allow you to provide numpy array scalars. * A new ARGOUTVIEW suite of typemaps is provided that allows your wrapped function to provide a pointer to internal data and that returns a numpy array encapsulating it. This is potentially dangerous, but necessary in some situations for very large data buffers. * New typemaps are provided that correctly handle FORTRAN ordered 2D and 3D arrays. Tests for the ARGOUTVIEW and FORTRAN ordered arrays have also been added, and the documentation (doc/numpy_swig.*) has been updated to reflect all of these changes. On Nov 26, 2007, at 3:52 PM, Christopher Barker wrote: > Bill Spotz wrote: >> First, my plan is to add to numpy.i, typemaps for signatures like the >> following: >> >> %typemap(argout) (double** ARGOUT_ARRAY1, int* DIM1) >> >> It is important to note that even though the same argument *names* >> are used, this is a different typemap signature than >> >> %typemap(argout) (double* ARGOUT_ARRAY1, int DIM1) >> >> and thus can have (logically) different implementations. > > maybe it's a good idea to give the arguments different names, though, > just for clarity. > >> As for having a COPY version of the first typemap signature (in >> addition to the non-copy, or "view" version), I currently do not plan >> to do this. > > I think you've got it right. > > Thanks for all this! > > -Chris > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From zpincus at stanford.edu Thu Nov 29 23:59:05 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Thu, 29 Nov 2007 23:59:05 -0500 Subject: [Numpy-discussion] display numpy array as image In-Reply-To: <1196388219.6047.215.camel@glup.physics.ucf.edu> References: <1196388219.6047.215.camel@glup.physics.ucf.edu> Message-ID: Thanks for the suggestions, everyone! All very informative and most helpful. For what it's worth, here's my application: I'm building a tool for image processing which needs some manual input in a few places (e.g. user draws a few lines). The images are greyscale images with 12-14 bits of dynamic range (from a microscope), so I need to have some basic brightness/contrast/gamma controls, as well as allowing basic drawing on the image to get the needed user input. It looks like GL or wx will be best suited here, I think? (I presume that python/numpy/ [GL|wx] can keep up with things like dragging a slider to change brightness/contrast/other LUT changes, as long as I code reasonably.) Anyhow, thanks for all the input, Zach On Nov 29, 2007, at 9:03 PM, Joe Harrington wrote: > If you want to explore the array interactively, blink images, mess > with > colormaps using the mouse, rescale the image values, mark regions, add > labels, look at dynamic plots of rows and columns, etc., get the ds9 > image viewer and the xpa programs that come with it that allow it to > communicate with other programs: > > ftp://sao-ftp.harvard.edu/pub/rd/ds9 > http://hea-www.harvard.edu/RD/ds9/index.html > > Then get the Python numdisplay package, which uses xpa. You have > to get > numdisplay from inside the stsci_python package: > > http://www.stsci.edu/resources/software_hardware/pyraf/stsci_python/ > current/download > > Just grab the numdisplay directory from within that. Older > versions of > numdisplay are standalone but don't work perfectly. Beware, there are > outdated web sites about numdisplay on the stsci site. Don't google! > > Run ds9 before you load numdisplay. Then you can send your python > arrays to a real interactive data viewer at will. There are even > mechanisms to define physical coordinates mapped from the image > coordinates. > > --jh-- > > From nwagner at iam.uni-stuttgart.de Fri Nov 30 02:36:11 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 30 Nov 2007 08:36:11 +0100 Subject: [Numpy-discussion] documentation generator based on pyparsing In-Reply-To: <474E7FE6.7040604@ntc.zcu.cz> References: <474D4300.6020904@ntc.zcu.cz> <474E7FE6.7040604@ntc.zcu.cz> Message-ID: On Thu, 29 Nov 2007 10:01:26 +0100 Robert Cimrman wrote: > Hi Nils, > > Nils Wagner wrote: >> The output of >> >> ./gendocs.py -m 'scipy.linsolve.umfpack' >> >> differs from your example output (available at >> http://scipy.org/Generate_Documentation) > > I had to update the umfpack info.py file (where the >module docstring is) > to conform the documentation standards. The gendocs.py >relies on that, > so use, please, the newest SVN version of scipy - it >should work with > rev. 3601 and later. Hi Robert, Thank you for your note. It works fine for me with python2.5. However python2.3 results in ./gendocs.py -m 'scipy.linsolve.umfpack' Traceback (most recent call last): File "./gendocs.py", line 261, in ? main() File "./gendocs.py", line 207, in main default = 1, help = help['page'] ) File "/usr/lib64/python2.3/optparse.py", line 820, in add_option option = self.option_class(*args, **kwargs) File "/usr/lib64/python2.3/optparse.py", line 430, in __init__ checker(self) File "/usr/lib64/python2.3/optparse.py", line 499, in _check_type raise OptionError("invalid option type: %r" % self.type, self) optparse.OptionError: option -p/--page: invalid option type: How can I resolve this problem ? Nils From numpy at mspacek.mm.st Fri Nov 30 03:47:41 2007 From: numpy at mspacek.mm.st (Martin Spacek) Date: Fri, 30 Nov 2007 00:47:41 -0800 Subject: [Numpy-discussion] Loading a > GB file into array Message-ID: <474FCE2D.2000908@mspacek.mm.st> I need to load a 1.3GB binary file entirely into a single numpy.uint8 array. I've been using numpy.fromfile(), but for files > 1.2GB on my win32 machine, I get a memory error. Actually, since I have several other python modules imported at the same time, including pygame, I get a "pygame parachute" and a segfault that dumps me out of python: data = numpy.fromfile(f, numpy.uint8) # where f is the open file 1382400000 items requested but only 0 read Fatal Python error: (pygame parachute) Segmentation Fault If I stick to just doing it at the interpreter with only numpy imported, I can open up files that are roughly 100MB bigger, but any more than that and I get a clean MemoryError. This machine has 2GB of RAM. I've tried setting the /3GB switch on winxp bootup, as well as all the registry suggestions at http://www.msfn.org/board/storage-process-command-t62001.html. No luck. I get the same error in (32bit) ubuntu for a sufficiently big file. I find that if I load the file in two pieces into two arrays, say 1GB and 0.3GB respectively, I can avoid the memory error. So it seems that it's not that windows can't allocate the memory, just that it can't allocate enough contiguous memory. I'm OK with this, but for indexing convenience, I'd like to be able to treat the two arrays as if they were one. Specifically, this file is movie data, and the array I'd like to get out of this is of shape (nframes, height, width). Right now I'm getting two arrays that are something like (0.8*nframes, height, width) and (0.2*nframes, height, width). Later in the code, I only need to index over the 0th dimension, i.e. the frame index. I'd like to access all the data using a single range of frame indices. Is there any way to combine these two arrays into what looks like a single array, without having to do any copying within memory? I've tried using numpy.concatenate(), but that gives me a MemoryError because, I presume, it's doing a copy. Would it be better to load the file one frame at a time, generating nframes arrays of shape (height, width), and sticking them consecutively in a python list? I'm using numpy 1.0.4 (compiled from source tarball with Intel's MKL library) on python 2.5.1 in winxp. Thanks for any advice, Martin From cimrman3 at ntc.zcu.cz Fri Nov 30 03:48:09 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 30 Nov 2007 09:48:09 +0100 Subject: [Numpy-discussion] documentation generator based on pyparsing In-Reply-To: References: <474D4300.6020904@ntc.zcu.cz> <474E7FE6.7040604@ntc.zcu.cz> Message-ID: <474FCE49.4000909@ntc.zcu.cz> Nils Wagner wrote: > Thank you for your note. It works fine for me with > python2.5. However python2.3 results in > > ./gendocs.py -m 'scipy.linsolve.umfpack' > Traceback (most recent call last): > File "./gendocs.py", line 261, in ? > main() > File "./gendocs.py", line 207, in main > default = 1, help = help['page'] ) > File "/usr/lib64/python2.3/optparse.py", line 820, in > add_option > option = self.option_class(*args, **kwargs) > File "/usr/lib64/python2.3/optparse.py", line 430, in > __init__ > checker(self) > File "/usr/lib64/python2.3/optparse.py", line 499, in > _check_type > raise OptionError("invalid option type: %r" % > self.type, self) > optparse.OptionError: option -p/--page: invalid option > type: > > How can I resolve this problem ? > In gendocs.py, line 205, try to replace 'type = int' with 'type = "int"': parser.add_option( "-p", "--page", metavar = 'page', type = "int", action = "store", dest = "page", default = 1, help = help['page'] ) thanks for reporting bugs, r. From nwagner at iam.uni-stuttgart.de Fri Nov 30 04:19:54 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 30 Nov 2007 10:19:54 +0100 Subject: [Numpy-discussion] documentation generator based on pyparsing In-Reply-To: <474FCE49.4000909@ntc.zcu.cz> References: <474D4300.6020904@ntc.zcu.cz> <474E7FE6.7040604@ntc.zcu.cz> <474FCE49.4000909@ntc.zcu.cz> Message-ID: On Fri, 30 Nov 2007 09:48:09 +0100 Robert Cimrman wrote: > Nils Wagner wrote: >> Thank you for your note. It works fine for me with >> python2.5. However python2.3 results in >> >> ./gendocs.py -m 'scipy.linsolve.umfpack' >> Traceback (most recent call last): >> File "./gendocs.py", line 261, in ? >> main() >> File "./gendocs.py", line 207, in main >> default = 1, help = help['page'] ) >> File "/usr/lib64/python2.3/optparse.py", line 820, in >> add_option >> option = self.option_class(*args, **kwargs) >> File "/usr/lib64/python2.3/optparse.py", line 430, in >> __init__ >> checker(self) >> File "/usr/lib64/python2.3/optparse.py", line 499, in >> _check_type >> raise OptionError("invalid option type: %r" % >> self.type, self) >> optparse.OptionError: option -p/--page: invalid option >> type: >> >> How can I resolve this problem ? >> > > In gendocs.py, line 205, try to replace 'type = int' >with 'type = "int"': > > parser.add_option( "-p", "--page", metavar = 'page', >type = "int", > action = "store", dest = "page", > default = 1, help = help['page'] >) > > thanks for reporting bugs, > r. Great ! Thank you very much ! It works for me. Cheers, Nils From haase at msg.ucsf.edu Fri Nov 30 05:55:36 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 30 Nov 2007 11:55:36 +0100 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: <9AC12CB0-D618-4322-A740-CD5C1EAD5AC8@sandia.gov> References: <473DFCC2.1050704@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <474465EE.20305@mur.at> <538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov> <474B4E34.80900@noaa.gov> <9AC12CB0-D618-4322-A740-CD5C1EAD5AC8@sandia.gov> Message-ID: On Nov 30, 2007 5:16 AM, Bill Spotz wrote: > I have just committed the latest version of numpy.i (a swig interface > file for bridging between C arrays and numerical python) to the numpy > svn repository. There are three relatively new features that are now > supported: > > * It is now possible to wrap functions that expect integer arguments > and have swig > generate wrappers that allow you to provide numpy array scalars. > > * A new ARGOUTVIEW suite of typemaps is provided that allows your > wrapped function > to provide a pointer to internal data and that returns a numpy > array encapsulating > it. This is potentially dangerous, but necessary in some > situations for very > large data buffers. > Hi, This sounds great !! Could you elaborate on what you mean by "potentially dangerous" !? Is it important *how* the internal data memory was allocated ? Using "new" or using "malloc" ? This would require (following the standard) that the correct, i.e. corresponding, delete[] or free() [respectively], de-allocation function is called. When is the memory being freed ? Is (or can !) python taking ownership of the memory and calls the correct "free"/"delete[]" function ? Thanks, Sebastian Haase From giorgio at gilestro.tk Fri Nov 30 10:20:12 2007 From: giorgio at gilestro.tk (Giorgio F. Gilestro) Date: Fri, 30 Nov 2007 09:20:12 -0600 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> <474F4EA9.407@noaa.gov> Message-ID: <47502A2C.5090307@gilestro.tk> Even just a build of the last stable version will do it. Most people (especially those who don't want to go through the hassle of compiling) are going to be perfectly happy with a binary of the latest release. Thanks! Barry Wark wrote: > Using the gfortran from http://r.research.att.com/tools/, it's trivial > to build a universal build from source. The instructions on scipy.org > won't lead you astray. > > I will ask around at work. Perhaps we can start building universal > scipy builds for distribution. Can anyone from the scipy devs email me > off list if you'd like to pursue this?... triggering a build > automatically from SVN commits or such would be good. > > Barry > > On Nov 29, 2007 3:43 PM, Christopher Barker wrote: >> Joshua Lippai wrote: >>> Chris Fonnesbeck used to distribute both PowerPC and Intel binaries, >>> but he can't compile them for PowerPC at all anymore because he no >>> longer has access to a PowerPC machine. >> You can build fat binaries with a single machine -- either one. Whether >> Chris wants to spend his time figuring all that out is up to him, but >> it's not a hardware limitation. >> >> >> -Chris >> >> >> >> -- >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From wfspotz at sandia.gov Fri Nov 30 11:26:26 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Fri, 30 Nov 2007 09:26:26 -0700 Subject: [Numpy-discussion] swig numpy2carray converters In-Reply-To: References: <473DFCC2.1050704@mur.at> <4742EE16.8030307@mur.at> <8710782A-E850-405C-8517-07F8F025E696@sandia.gov> <474345DF.8040202@mur.at> <91B30456-5988-4FC4-8E82-E3AF2CDF6E79@sandia.gov> <474465EE.20305@mur.at> <538EB43E-7DA1-451C-AA4B-503E131E83C9@sandia.gov> <474B4E34.80900@noaa.gov> <9AC12CB0-D618-4322-A740-CD5C1EAD5AC8@sandia.gov> Message-ID: <32E00A2A-33B9-4A73-A5AD-C1EA83D28076@sandia.gov> The numpy array is created using PyArray_SimpleNewFromData(). From the Guide to NumPy, "[y]ou should ensure that the provided memory is not freed while the returned array is in existence." That is, numpy does not try to deallocate the memory when the ndarray object is destroyed, but it also *assumes* that you do not deallocate it yourself. Since numpy.i is dealing with essentially non-reference- counted pointers to raw buffers, I have no way to guarantee this. I think the typical use case will be that you wrap a class that allocates a chunk of memory (that it deallocates when it gets destroyed). It gives you a view to that data, which we then encapsulate with a numpy array. If you create such an object, extract such an array, and then destroy the object but not the array, then the numpy array will point to garbage, possibly resulting in a segmentation fault when you try to access it. The only answer I know of is to not destroy the object before the array. There are situations where you just can't get around it (based on how the C++ class was designed). But if you can get around it, you should. On Nov 30, 2007, at 3:55 AM, Sebastian Haase wrote: > Could you elaborate on what you mean by "potentially dangerous" !? > Is it important *how* the internal data memory was allocated ? Using > "new" or using "malloc" ? > This would require (following the standard) that the correct, i.e. > corresponding, delete[] or free() [respectively], de-allocation > function is called. > > When is the memory being freed ? > Is (or can !) python taking ownership of the memory and calls the > correct "free"/"delete[]" function ? ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From barrywark at gmail.com Fri Nov 30 11:39:49 2007 From: barrywark at gmail.com (Barry Wark) Date: Fri, 30 Nov 2007 08:39:49 -0800 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <474F80D0.2010500@ar.media.kyoto-u.ac.jp> References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> <474F4EA9.407@noaa.gov> <474F80D0.2010500@ar.media.kyoto-u.ac.jp> Message-ID: On 11/29/07, David Cournapeau wrote: > Barry Wark wrote: > > Using the gfortran from http://r.research.att.com/tools/, it's trivial > > to build a universal build from source. The instructions on scipy.org > > won't lead you astray. > > > > I will ask around at work. Perhaps we can start building universal > > scipy builds for distribution. Can anyone from the scipy devs email me > > off list if you'd like to pursue this?... triggering a build > > automatically from SVN commits or such would be good. > That would be cool, but where to compile the binary ? scipy does not > have build farm. Having a 100 % automatic way to build a binary would > certainly be a step in this direction, though. There is also the need to > test the compiled binary, which is not trivial with fat binaries. To sum up: > - we need at least one x86 mac os X machine to build/test (I don't > know if you can test python softwares through rosetta: I have only > experience testing pure C softwares built as fat binaries) One of our machines (OSX Intel) is already the buildbot slave for the numpy buildbot. I think I can get us OK'd to build scipy on there as well. We generate our own scipy builds so it shouldn't be too big a problem. Unless there's an automated system, I'd prefer to to just release builds. Some remaining issues: - which SDK to build against. Leopard ships with a Python build against the 10.5 SDK. It would be much easier, at least initially, for us to produce binaries against the Leopard Python 2.5. - how to deal with library dependencies such as fftw2. We currently use MacPorts but I suppose it needs to be compiled statically or do we just require that users install MacPort's fftw2? > - we need a way to automate the build entirely It seems like scipy needs a buildbot. > - we need a way to automate the packaging (distutils can do only > part of it or all of it ? Building a basic .pkg inside a dmg is not hard > from the command line, but I don't know how it scales for non trivial > packages). It's relatively easy to build a single pkg for the entire scipy distribution (that's what we do now) and put that on a DMG (as you said). When scipy becomes a less monolithic package (i.e. using setuptools namespace packages), we can create a pkg for each package and a mpkg combining them all. ~barry From kwmsmith at gmail.com Fri Nov 30 13:00:36 2007 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 30 Nov 2007 12:00:36 -0600 Subject: [Numpy-discussion] Loading a > GB file into array In-Reply-To: <474FCE2D.2000908@mspacek.mm.st> References: <474FCE2D.2000908@mspacek.mm.st> Message-ID: On Nov 30, 2007 2:47 AM, Martin Spacek wrote: > I need to load a 1.3GB binary file entirely into a single numpy.uint8 > array. I've been using numpy.fromfile(), but for files > 1.2GB on my > win32 machine, I get a memory error. Actually, since I have several > other python modules imported at the same time, including pygame, I get > a "pygame parachute" and a segfault that dumps me out of python: > > data = numpy.fromfile(f, numpy.uint8) # where f is the open file > > 1382400000 items requested but only 0 read > Fatal Python error: (pygame parachute) Segmentation Fault You might try numpy.memmap -- others have had success with it for large files (32 bit should be able to handle a 1.3 GB file, AFAIK). See for example: http://www.thescripts.com/forum/thread654599.html Kurt > > If I stick to just doing it at the interpreter with only numpy imported, > I can open up files that are roughly 100MB bigger, but any more than > that and I get a clean MemoryError. This machine has 2GB of RAM. I've > tried setting the /3GB switch on winxp bootup, as well as all the > registry suggestions at > http://www.msfn.org/board/storage-process-command-t62001.html. No luck. > I get the same error in (32bit) ubuntu for a sufficiently big file. > > I find that if I load the file in two pieces into two arrays, say 1GB > and 0.3GB respectively, I can avoid the memory error. So it seems that > it's not that windows can't allocate the memory, just that it can't > allocate enough contiguous memory. I'm OK with this, but for indexing > convenience, I'd like to be able to treat the two arrays as if they were > one. Specifically, this file is movie data, and the array I'd like to > get out of this is of shape (nframes, height, width). Right now I'm > getting two arrays that are something like (0.8*nframes, height, width) > and (0.2*nframes, height, width). Later in the code, I only need to > index over the 0th dimension, i.e. the frame index. > > I'd like to access all the data using a single range of frame indices. > Is there any way to combine these two arrays into what looks like a > single array, without having to do any copying within memory? I've tried > using numpy.concatenate(), but that gives me a MemoryError because, I > presume, it's doing a copy. Would it be better to load the file one > frame at a time, generating nframes arrays of shape (height, width), and > sticking them consecutively in a python list? > > I'm using numpy 1.0.4 (compiled from source tarball with Intel's MKL > library) on python 2.5.1 in winxp. > > Thanks for any advice, > > Martin > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From ivilata at carabos.com Fri Nov 30 13:19:38 2007 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Fri, 30 Nov 2007 19:19:38 +0100 Subject: [Numpy-discussion] Loading a > GB file into array In-Reply-To: <474FCE2D.2000908@mspacek.mm.st> References: <474FCE2D.2000908@mspacek.mm.st> Message-ID: <20071130181938.GA6080@tardis.terramar.selidor.net> Martin Spacek (el 2007-11-30 a les 00:47:41 -0800) va dir:: >[...] > I find that if I load the file in two pieces into two arrays, say 1GB > and 0.3GB respectively, I can avoid the memory error. So it seems that > it's not that windows can't allocate the memory, just that it can't > allocate enough contiguous memory. I'm OK with this, but for indexing > convenience, I'd like to be able to treat the two arrays as if they were > one. Specifically, this file is movie data, and the array I'd like to > get out of this is of shape (nframes, height, width). >[...] Well, one thing you could do is dump your data into a PyTables_ ``CArray`` dataset, which you may afterwards access as if its was a NumPy array to get slices which are actually NumPy arrays. PyTables datasets have no problem in working with datasets exceeding memory size. For instance:: h5f = tables.openFile('foo.h5', 'w') carray = h5f.createCArray( '/', 'bar', atom=tables.UInt8Atom(), shape=(TOTAL_NROWS, 3) ) base = 0 for array in your_list_of_partial_arrays: carray[base:base+len(array)] = array base += len(array) carray.flush() # Now you can access ``carray`` as a NumPy array. carray[42] --> a (3,) uint8 NumPy array carray[10:20] --> a (10, 3) uint8 NumPy array carray[42,2] --> a NumPy uint8 scalar, "width" for row 42 (You may use an ``EArray`` dataset if you want to enlarge it with new rows afterwards, or a ``Table`` if you want a different type for each field.) .. _PyTables: http://www.pytables.org/ HTH, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 307 bytes Desc: Digital signature URL: From jh at physics.ucf.edu Fri Nov 30 14:48:48 2007 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 30 Nov 2007 14:48:48 -0500 Subject: [Numpy-discussion] display numpy array as image (Giorgio F. Gilestro) Message-ID: <1196452128.6047.374.camel@glup.physics.ucf.edu> I was misinformed about the status of numdisplay's pages. The package is available as both part of stsci_python and independently, and its (up-to-date) home page is here: http://stsdas.stsci.edu/numdisplay/ Googling numdisplay finds that page. My apologies to those inconvenienced by my prior post. Note that there is practical documentation and examples of how to use it for astronomical image display in the Data Analysis Tutorial: http://www.scipy.org/wikis/topical_software/Tutorial --jh-- From bryan at cole.uklinux.net Fri Nov 30 14:51:23 2007 From: bryan at cole.uklinux.net (Bryan Cole) Date: Fri, 30 Nov 2007 19:51:23 +0000 Subject: [Numpy-discussion] Loading a > GB file into array In-Reply-To: <20071130181938.GA6080@tardis.terramar.selidor.net> References: <474FCE2D.2000908@mspacek.mm.st> <20071130181938.GA6080@tardis.terramar.selidor.net> Message-ID: <1196452282.26940.11.camel@pc1.cole.uklinux.net> > > Well, one thing you could do is dump your data into a PyTables_ > ``CArray`` dataset, which you may afterwards access as if its was a > NumPy array to get slices which are actually NumPy arrays. PyTables > datasets have no problem in working with datasets exceeding memory size. > For instance:: I've recently started using PyTables for storing large datasets and I'd give it 10/10! Access is fast enough you can just access the data you need and leave the full array on disk. BC From robert.kern at gmail.com Fri Nov 30 15:16:15 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 30 Nov 2007 14:16:15 -0600 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> <474F4EA9.407@noaa.gov> <474F80D0.2010500@ar.media.kyoto-u.ac.jp> Message-ID: <47506F8F.8070207@gmail.com> Barry Wark wrote: > Some remaining issues: > - which SDK to build against. Leopard ships with a Python build > against the 10.5 SDK. It would be much easier, at least initially, for > us to produce binaries against the Leopard Python 2.5. I would prefer that we use the Python binary from www.python.org. That should work on 10.3.9+. > - how to deal with library dependencies such as fftw2. We currently > use MacPorts but I suppose it needs to be compiled statically or do we > just require that users install MacPort's fftw2? "Official" binaries intended for distribution from scipy.org or scipy.sf.net should not be linked against FFTW or UMFPACK since they are GPLed. The real problem is the Fortran runtime. It is possible to link against the Fortran runtime libraries statically so as to avoid needing the user to also install a Fortran compiler just to run scipy. Use the gfortran compiler built for R's use (and do not use the one from hpc.sf.net): http://r.research.att.com/tools/ Basically, copy the libgfortran.a library to a separate location, say ~/staticlibs/. Then do the following: $ export LDFLAGS="-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first" $ python setup.py config_fc --fcompiler=gnu95 --arch="-arch i386 -arch ppc" build_ext -L ~/staticlibs/ build -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From barrywark at gmail.com Fri Nov 30 16:20:54 2007 From: barrywark at gmail.com (Barry Wark) Date: Fri, 30 Nov 2007 13:20:54 -0800 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <47506F8F.8070207@gmail.com> References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> <474F4EA9.407@noaa.gov> <474F80D0.2010500@ar.media.kyoto-u.ac.jp> <47506F8F.8070207@gmail.com> Message-ID: On 11/30/07, Robert Kern wrote: > Barry Wark wrote: > > > Some remaining issues: > > - which SDK to build against. Leopard ships with a Python build > > against the 10.5 SDK. It would be much easier, at least initially, for > > us to produce binaries against the Leopard Python 2.5. > > I would prefer that we use the Python binary from www.python.org. That should > work on 10.3.9+. OK. > > > - how to deal with library dependencies such as fftw2. We currently > > use MacPorts but I suppose it needs to be compiled statically or do we > > just require that users install MacPort's fftw2? > > "Official" binaries intended for distribution from scipy.org or scipy.sf.net > should not be linked against FFTW or UMFPACK since they are GPLed. > Got it. > The real problem is the Fortran runtime. It is possible to link against the > Fortran runtime libraries statically so as to avoid needing the user to also > install a Fortran compiler just to run scipy. Use the gfortran compiler built > for R's use (and do not use the one from hpc.sf.net): > > http://r.research.att.com/tools/ > > Basically, copy the libgfortran.a library to a separate location, say > ~/staticlibs/. Then do the following: > > $ export LDFLAGS="-undefined dynamic_lookup -bundle -arch i386 -arch ppc > -Wl,-search_paths_first" > $ python setup.py config_fc --fcompiler=gnu95 --arch="-arch i386 -arch ppc" > build_ext -L ~/staticlibs/ build Thanks. I'll give it a try. From numpy at mspacek.mm.st Fri Nov 30 18:09:16 2007 From: numpy at mspacek.mm.st (Martin Spacek) Date: Fri, 30 Nov 2007 15:09:16 -0800 Subject: [Numpy-discussion] Loading a > GB file into array In-Reply-To: References: <474FCE2D.2000908@mspacek.mm.st> Message-ID: <4750981C.9060409@mspacek.mm.st> Kurt Smith wrote: > You might try numpy.memmap -- others have had success with it for > large files (32 bit should be able to handle a 1.3 GB file, AFAIK). Yeah, I looked into numpy.memmap. Two issues with that. I need to eliminate as much disk access as possible while my app is running. I'm displaying stimuli on a screen at 200Hz, so I have up to 5ms for each movie frame to load before it's too late and it drops a frame. I'm sort of faking a realtime OS on windows by setting the process priority really high. Disk access in the middle of that causes frames to drop. So I need to load the whole file into physical RAM, although it need not be contiguous. memmap doesn't do that, it loads on the fly as you index into the array, which drops frames, so that doesn't work for me. The 2nd problem I had with memmap was that I was getting a WindowsError related to memory: >>> data = np.memmap(1.3GBfname, dtype=np.uint8, mode='r') Traceback (most recent call last): File "", line 1, in File "C:\bin\Python25\Lib\site-packages\numpy\core\memmap.py", line 67, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc) WindowsError: [Error 8] Not enough storage is available to process this command This was for the same 1.3GB file. This is different from previous memory errors I mentioned. I don't get this on ubuntu. I can memmap a file up to 2GB on ubuntu no problem, but any larger than that and I get this: >>> data = np.memmap(2.1GBfname, dtype=np.uint8, mode='r') Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.5/site-packages/numpy/core/memmap.py", line 67, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc) OverflowError: cannot fit 'long' into an index-sized integer The OverflowError is on the bytes argument. If I try doing the mmap.mmap directly in Python, I get the same error. So I guess it's due to me running 32bit ubuntu. Martin From Chris.Barker at noaa.gov Fri Nov 30 18:34:14 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 30 Nov 2007 15:34:14 -0800 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <47506F8F.8070207@gmail.com> References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> <474F4EA9.407@noaa.gov> <474F80D0.2010500@ar.media.kyoto-u.ac.jp> <47506F8F.8070207@gmail.com> Message-ID: <47509DF6.2090104@noaa.gov> Robert Kern wrote: > I would prefer that we use the Python binary from www.python.org. That should > work on 10.3.9+. +1 -- there have always been enough issue with Apple;'s python that it's best to just use another version -- one binary for > 10.3.9 is the way to go. > "Official" binaries intended for distribution from scipy.org or scipy.sf.net > should not be linked against FFTW or UMFPACK since they are GPLed. Does that apply to binaries put up on pythonmac.org? It would be nice to have a "complete" version somewhere standard. > The real problem is the Fortran runtime. It is possible to link against the > Fortran runtime libraries statically so as to avoid needing the user to also > install a Fortran compiler just to run scipy. That would be great, too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Fri Nov 30 18:47:34 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 30 Nov 2007 17:47:34 -0600 Subject: [Numpy-discussion] scipy binary for macosx tiger on ppc In-Reply-To: <47509DF6.2090104@noaa.gov> References: <474F3512.7000908@gilestro.tk> <9911419a0711291505h79b5907ds31fec6be1b8a2a6d@mail.gmail.com> <474F4964.9030700@noaa.gov> <9911419a0711291523u2d0e28d2u772de17489ca3e1f@mail.gmail.com> <474F4EA9.407@noaa.gov> <474F80D0.2010500@ar.media.kyoto-u.ac.jp> <47506F8F.8070207@gmail.com> <47509DF6.2090104@noaa.gov> Message-ID: <4750A116.1080207@gmail.com> Christopher Barker wrote: > Robert Kern wrote: >> "Official" binaries intended for distribution from scipy.org or scipy.sf.net >> should not be linked against FFTW or UMFPACK since they are GPLed. > > Does that apply to binaries put up on pythonmac.org? It would be nice to > have a "complete" version somewhere standard. I would argue "yes, it does apply to pythonmac.org, too." scipy is a BSD-licensed package. Putting GPLed components into the binaries which are distributed at "standard" locations like our website, PyPI, or pythonmac.org confuses that picture. I would be mollified if the download link had an explanation attached, but unfortunately, pythonmac.org/packages does not have any space for that. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From numpy at mspacek.mm.st Fri Nov 30 20:00:47 2007 From: numpy at mspacek.mm.st (Martin Spacek) Date: Fri, 30 Nov 2007 17:00:47 -0800 Subject: [Numpy-discussion] Loading a > GB file into array In-Reply-To: <474FCE2D.2000908@mspacek.mm.st> References: <474FCE2D.2000908@mspacek.mm.st> Message-ID: <4750B23F.5090608@mspacek.mm.st> Martin Spacek wrote: > Would it be better to load the file one > frame at a time, generating nframes arrays of shape (height, width), > and sticking them consecutively in a python list? I just tried this, and it works. Looks like it's all in physical RAM (no disk thrashing on the 2GB machine), *and* it's easy to index into. I guess I should of thought of this a while ago, since each entry in a python list can point to anywhere in memory. Here's roughly what the code looks like: import numpy as np f = file(fname, 'rb') # 1.3GB file frames = [None] * nframes # init a list to hold all frames for framei in xrange(nframes): # one frame at a time... frame = np.fromfile(f, np.uint8, count=framesize) # load next frame frame.shape = (height, width) frames[framei] = frame # save it in the list -- Martin