From perry at stsci.edu Wed Sep 1 15:36:12 2010 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 1 Sep 2010 15:36:12 -0400 Subject: [SciPy-User] Developer Job Openings at Space Telescope Science Institute Message-ID: <63BE5DE9-4882-4B2E-B03A-F28349DB39C0@stsci.edu> We are advertising for two different positions at the Space Telescope Science Institute (located on the Johns Hopkins University Campus in Baltimore, Md). STScI is seeking Senior Systems Software Engineers to develop applications to calibrate and analyze data from the Hubble and the James Webb Space Telescopes. **************************************************************************** First position: Description The developer will work with other members of the Science Software Branch and instrument scientists to develop applications to calibrate and analyze data from the Hubble Space Telescope and its successor, the James Webb Space Telescope, as well as other astronomy-related projects. The individual will be developing open source applications primarily in Python and C, using open source tools developed at STScI, such as PyRAF and PyFITS, and elsewhere. Some projects may involve developing software applications or libraries as part of a team, or leading a team. Requirements Candidates should have experience writing applications to calibrate, reduce, and analyze scientific data, preferably astronomical data. A background in astronomy and experience with astronomical data reduction is highly desirable. Candidates should have experience writing large programs in a compiled language as well as experience with an interpreted language such as IDL, Matlab, or Python. Experience using array manipulations facilities such as are available in IDL, Matlab, numpy/numarray, or APL is a plus. Experience using software engineering tools such as debuggers, CVS or subversion, and bug trackers is strongly desired. Strong analytical, problem-solving, planning, and organizational skills are needed, and excellent written and verbal communication skills are essential. Prior experience in developing medium or large projects sufficient to demonstrate the specified knowledge, skills and abilities is required. Qualified candidates should possess a Bachelor's degree in a science- related field such as Physics, Astronomy, or Mathematics. A Master's or Ph.D degree is desirable. Substitution of additional relevant education or experience for the stated qualifications may be considered. Apply through the following link: https://www.ultirecruit.com/SPA1004/jobboard/JobDetails.aspx?__ID=*6D48E0EFCC47915A **************************************************************************** Second position: Description The developer will work with other members of the Science Software Branch to help in enhancing and maintaining our Python-based framework for developing astronomical data analysis and calibration applications. STScI has pioneered in the generation of tools for using Python for scientific analysis and programming through its development of PyRAF, numarray, PyFITS, and contributions to other Python Open Source projects. The individual being sought will help STScI maintain its leadership in this area by developing leading-edge capabilities by enhancing existing tools such as PyRAF and PyFITS, contributing to scipy, numpy, and matplotlib, and developing new libraries to meet the needs of future astronomical processing. Some projects may involve developing software tools as part of a team, or leading a team. Work will also require working with an external community on Open Source software projects. Requirements Candidates should be experienced with systems-level programming, preferably with C or C++ and familiar with variances in processor and operating system architectures (preferably Linux, OS X, and MS Windows) with regard to file systems, memory, data types and efficiency, as well as modern software development techniques including Object-Oriented design and programming. Experience with Python and writing C extensions for Python is highly desirable. A working knowledge of any of the following would be a plus: parsers, code generation, numerical techniques, image processing and data analysis, web and network protocols, or parallel processing. Experience using software engineering tools such as debuggers, version control systems (e.g., subversion), and bug trackers is strongly desired. Strong analytical, problem-solving, planning, and organizational skills are needed, and excellent written and verbal communication skills are essential. Prior experience in developing medium or large projects sufficient to demonstrate the specified knowledge, skills and abilities is required. Qualified candidates should possess a Bachelor's Degree in Computer Science, Physics, Math, or technically related field. Master's degree preferred. Substitution of additional relevant education or experience for the stated qualifications may be considered. Apply through the following link: https://www.ultirecruit.com/SPA1004/jobboard/JobDetails.aspx?__ID=*85D01A9E3BE42CFD **************************************************************************** STScI offers an excellent benefits package, tuition reimbursement, competitive salaries, and a stimulating work environment. Interested candidates are requested to complete an on-line application, attach a resume in the "Resume Upload Section." Please include job #10-0083 in the filename. Resumes received by October 15, 2010 will receive full consideration. Committed to the benefits of diversity, we strongly encourage qualified women and minority candidates to apply. EOE/AA/M/F/D/V. From jeremy at jeremysanders.net Wed Sep 1 17:13:10 2010 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Wed, 01 Sep 2010 22:13:10 +0100 Subject: [SciPy-User] ANN: Veusz 1.9 Message-ID: Veusz 1.9 --------- Velvet Ember Under Sky Zenith ----------------------------- http://home.gna.org/veusz/ Veusz is Copyright (C) 2003-2010 Jeremy Sanders Licenced under the GPL (version 2 or greater). Veusz is a Qt4 based scientific plotting package. It is written in Python, using PyQt4 for display and user-interfaces, and numpy for handling the numeric data. Veusz is designed to produce publication-ready Postscript/PDF/SVG output. The user interface aims to be simple, consistent and powerful. Veusz provides a GUI, command line, embedding and scripting interface (based on Python) to its plotting facilities. It also allows for manipulation and editing of datasets. Data can be captured from external sources such as internet sockets or other programs. Changes in 1.9: * Added "Dataset plugins" to easily allow the user to write or load Python code to make new datasets, optionally linked to existing datasets. Includes plugins as standard for - Adding and multiplying constants or datasets - Dividing datasets - Computing means of datasets - Computing extremes of datasets - Joining or combining datasets as parts or element by element - Thinning points from datasets - Converting from polar to Cartesian coordinates - Filtering datasets using expressions See here for details: http://barmag.net/veusz-wiki/DatasetPlugins * Added "Tools plugins" which allow the user to write Veusz macros in Python which can take parameters and be added to the menu. Includes plugins for - Randomizing or replacing colors - Finding and replacing text - Increasing / decreasing font sizes - Cloning widgets for different datasets See here for details: http://barmag.net/veusz-wiki/ToolsPlugins * C++ code added to speed up data to image conversion * Add vector field plot type * Add text dataset support for Import plugins * Add "Node" embedding interface to easily allow manipulation of the widget/setting tree. See http://barmag.net/veusz-wiki/EmbeddingPython * Veusz now starts in home directory or previous directory used. This can be switched off in the preferences dialog to use current directory. * Distances specified in points now show drop down box with common point sizes to make increasing/decreasing sizes much easier. Minor features added: * Color selectors are updated if new colors added to other color selectors * Exception dialog box uses web to send data rather than email, so no mail transfer agent is required. * Text entry box brought up with '..' goes away when clicking outside it * Improvements to appearance of buttons in Mac OS X Bug fixes: * Fixed dataset expressions for zero-dimension expressions * Dialog boxes are properly destroyed on closing to stop memory leaks * Do not give error if same dataset defined in dataset dialog and descriptor in data file * Fix license on some C++ files * Use correct web browser when clicking on links in menus * Exceptions are caught and user notified when loading plugins * Missing recent files are removed from recent files menu * Non finite values ignored when fitting data * When fitting data min and max properties used if set Features of package: * X-Y plots (with errorbars) * Line and function plots * Contour plots * Images (with colour mappings and colorbars) * Stepped plots (for histograms) * Bar graphs * Vector field plots * Plotting dates * Fitting functions to data * Stacked plots and arrays of plots * Plot keys * Plot labels * Shapes and arrows on plots * LaTeX-like formatting for text * EPS/PDF/PNG/SVG/EMF export * Scripting interface * Dataset creation/manipulation * Embed Veusz within other programs * Text, CSV, FITS and user-plugin importing * Data can be captured from external sources * User defined functions, constants and can import external Python functions * Plugin interface to allow user to write or load code to - import data using new formats - make new datasets, optionally linked to existing datasets - arbitrarily manipulate the document Requirements for source install: Python (2.4 or greater required) http://www.python.org/ Qt >= 4.3 (free edition) http://www.trolltech.com/products/qt/ PyQt >= 4.3 (SIP is required to be installed first) http://www.riverbankcomputing.co.uk/pyqt/ http://www.riverbankcomputing.co.uk/sip/ numpy >= 1.0 http://numpy.scipy.org/ Optional: Microsoft Core Fonts (recommended for nice output) http://corefonts.sourceforge.net/ PyFITS >= 1.1 (optional for FITS import) http://www.stsci.edu/resources/software_hardware/pyfits pyemf >= 2.0.0 (optional for EMF export) http://pyemf.sourceforge.net/ For EMF and better SVG export, PyQt >= 4.6 or better is required, to fix a bug in the C++ wrapping For documentation on using Veusz, see the "Documents" directory. The manual is in PDF, HTML and text format (generated from docbook). The examples are also useful documentation. Please also see and contribute to the Veusz wiki: http://barmag.net/veusz-wiki/ Issues with the current version: * Plots can sometimes be slow using antialiasing. Go to the preferences dialog or right click on the plot to disable antialiasing. * Some recent versions of PyQt/SIP will causes crashes when exporting SVG files. Update to 4.7.4 (if released) or a recent snapshot to solve this problem. If you enjoy using Veusz, I would love to hear from you. Please join the mailing lists at https://gna.org/mail/?group=veusz to discuss new features or if you'd like to contribute code. The latest code can always be found in the SVN repository. Jeremy Sanders From tcf4 at njit.edu Thu Sep 2 19:02:55 2010 From: tcf4 at njit.edu (Thomas Fitzpatrick) Date: Thu, 2 Sep 2010 23:02:55 +0000 (UTC) Subject: [SciPy-User] Discrete integration with delays Message-ID: Can anybody help me with the following equation? It is similar to a correlation, but I am not sure how this done as scipy's correlation packages are compiled into a pyd sigtools. I would like to get an array that is the same size as the input. Here is the equation I am trying to code: F(T) = integration[{I(t) + I(t - T)}*E(t)*Econj(t - T)]dt from -inf to +inf (The code be for a discrete function) I tried to do it by adapting this autocorrelation function: N = len(x) Rxx = np.zeros(N) for m in range(N): for n in range(N-m): Rxx[m] = Rxx[m] + x[n]*x[n+m-1] Here is my attempt: f1 = np.zeros(N) for m in range(N): for n in range(0, N - m): f1[m] = f1[m] + .5*(it[n] + it[n + m])*et[n]*et[n + m] This works pretty well, but it doesn't return an array of the same size as the input which is an option for the scipy function 'correlate.' From josef.pktd at gmail.com Thu Sep 2 21:47:06 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 2 Sep 2010 21:47:06 -0400 Subject: [SciPy-User] Discrete integration with delays In-Reply-To: References: Message-ID: On Thu, Sep 2, 2010 at 7:02 PM, Thomas Fitzpatrick wrote: > Can anybody help me with the following equation? It is similar to a correlation, > but I am not sure how this done as scipy's correlation packages are compiled > into a pyd sigtools. I would like to get an array that is the same size as the > input. > > Here is the equation I am trying to code: > > F(T) = integration[{I(t) + I(t - T)}*E(t)*Econj(t - T)]dt from -inf to +inf > (The code be for a discrete function) if you multiply out the terms integral (I(t)*E(t)) * Econj(t-T) + integral E*(t) * (I(t-T)*Econj(t-T)) dt , then this looks to me like two convolutions that any of the numpy scipy convolve should be able to handle (much faster than a python loop). Unless I'm interpreting something incorrectly, or the integrals are not finite. > > > I tried to do it by adapting this autocorrelation function: > > N = len(x) > Rxx = np.zeros(N) > > for m in range(N): > ? ?for n in range(N-m): > ? ? ? ?Rxx[m] = Rxx[m] + x[n]*x[n+m-1] > > > Here is my attempt: > > f1 = np.zeros(N) > for m in range(N): > ? ?for n in range(0, N - m): > ? ? ? ?f1[m] = f1[m] + .5*(it[n] + it[n + m])*et[n]*et[n + m] > > This works pretty well, but it doesn't return an array of the same size as the > input which is an option for the scipy function 'correlate.' padding the array with m (?) zeros might help, or padding with any other values to continue at the end. The convolution in ndimage have for example several options, reflect, same, ... not sure this helps Josef > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From Pierre.RAYBAUT at CEA.FR Fri Sep 3 12:03:02 2010 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Fri, 3 Sep 2010 18:03:02 +0200 Subject: [SciPy-User] [ANN] guidata v1.2.1 Message-ID: Hi all, I am pleased to announce that `guidata` v1.2.1 has been released. This library was initially written to help me to develop software for internal use only, but after a long process I've just been able to share this work with the Python community. Based on the Qt Python binding module PyQt4, guidata is a Python library for easy data set edition (and display) thanks to automatically generated graphical user interfaces - it also provides helpers and application development tools for PyQt4. guidata also provides the following features: * guidata.qthelpers: PyQt4 helpers * guidata.disthelpers: py2exe helpers * guidata.userconfig: .ini configuration management helpers (based on Python standard module ConfigParser) * guidata.configtools: library/application data management * guidata.gettext_helpers: translation helpers (based on the GNU tool gettext) * guidata.guitest: automatic GUI-based test launcher * guidata.utils: miscelleneous utilities guidata has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guidata/ Documentation, screenshots: http://packages.python.org/guidata/ Downloads (source + Python(x,y) plugin): http://sourceforge.net/projects/guidata/ Cheers, Pierre --- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From Pierre.RAYBAUT at CEA.FR Fri Sep 3 12:03:07 2010 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Fri, 3 Sep 2010 18:03:07 +0200 Subject: [SciPy-User] [ANN] guiqwt v2.0.3 Message-ID: Hi all, I am pleased to announce that `guiqwt` v2.0.3 has been released. This library was initially written to help me to develop signal/image processing software for internal use only, but after a long process I've just been able to share this work with the scientific Python community. Based on PyQwt (plot widgets for PyQt4 graphical user interfaces) and on scientific modules NumPy and SciPy, guiqwt is a Python library for efficient 2-D data plotting (curves, 1-D and 2-D histograms, images) and signal/image processing application development. As you certainly know, the most popular Python module for data plotting is currently matplotlib, an open-source library providing a lot of plot types and an API (the pylab interface) which is very close to MATLAB's plotting interface. guiqwt plotting features are quite limited in terms of plot types compared to matplotlib. However the currently implemented plot types are much more efficient. For example, the guiqwt image showing function (imshow) do not make any copy of the displayed data, hence allowing to show images much larger than with its matplotlib's counterpart. In other terms, when showing a 30-MB image (16-bits unsigned integers for example) with guiqwt, no additional memory is wasted to display the image (except for the offscreen image of course which depends on the window size) whereas matplotlib takes more than 600-MB of additional memory (the original array is duplicated four times using 64-bits float data types). guiqwt also provides the following features: guiqwt.pyplot: equivalent to matplotlib's pyplot module (pylab) supported plot items: * curves, error bar curves and 1-D histograms * images (RGB images are not supported), images with non-linear x/y scales, images with specified pixel size (e.g. loaded from DICOM files), 2-D histograms, pseudo-color images (pcolor) * labels, curve plot legends * shapes: polygon, polylines, rectangle, circle, ellipse and segment * annotated shapes (shapes with labels showing position and dimensions): rectangle with center position and size, circle with center position and diameter, ellipse with center position and diameters (these items are very useful to measure things directly on displayed images) curves, images and shapes: * multiple object selection for moving objects or editing their properties through automatically generated dialog boxes (guidata) * item list panel: move objects from foreground to background, show/hide objects, remove objects, ... * customizable aspect ratio * a lot of ready-to-use tools: plot canvas export to image file, image snapshot, image rectangular filter, etc. curves: * interval selection tools with labels showing results of computing on selected area * curve fitting tool with automatic fit, manual fit with sliders, ... images: * contrast adjustment panel: select the LUT by moving a range selection object on the image levels histogram, eliminate outliers, ... * X-axis and Y-axis cross-sections: support for multiple images, average cross-section tool on a rectangular area, ... * apply any affine transform to displayed images in real-time (rotation, magnification, translation, horizontal/vertical flip, ...) application development helpers: * ready-to-use curve and image plot widgets and dialog boxes * load/save graphical objects (curves, images, shapes) * a lot of test scripts which demonstrate guiqwt features guiqwt has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guiqwt/ Documentation, screenshots: http://packages.python.org/guiqwt/ Downloads (source + Python(x,y) plugin): http://sourceforge.net/projects/guiqwt/ Cheers, Pierre --- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From Pierre.RAYBAUT at CEA.FR Mon Sep 6 08:01:45 2010 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Mon, 6 Sep 2010 14:01:45 +0200 Subject: [SciPy-User] [ANN] guiqwt v2.0.3 In-Reply-To: References: Message-ID: Hi St?fan, That's an interesting question indeed (I'm sending this message to the scipy-user mailing list too, as I told you in private). Actually it was a hard choice to make. The Enthought Tool Suite has wonderful similar features thanks to Traits and Chaco but -especially at the time I made this decision- this is a too big package with too much dependencies, not as much documentation as needed (this has changed since then I think) and far too much updates and API compatibility breaks. In other words, after a careful analysis in 2008, ETS was not mature enough for our applications. We needed (and we still do) to rely on stable libraries for which major API changes do not occur too frequently. From a certain point of view, perhaps ETS is too ambitious with respect to the number of developers behind it, hence its lack of stability. So the idea was to develop a library much more simple and low-level than Traits and Chaco (less ambitious). The result is two tiny libraries with powerful features (tiny = easy to maintain = durability) and a low number of dependencies. Note that `guiqwt` has exclusive features like an image display engine (written in C++) which is very efficient (CPU + memory): real-time affine transforms (translation, rotation, resize, etc.) of possibly very large images on an average computer configuration *without* making any copy of the displayed array in memory. Cheers, Pierre > -----Message d'origine----- > De?: sjvdwalt at gmail.com [mailto:sjvdwalt at gmail.com] De la part de St?fan > van der Walt > Envoy??: samedi 4 septembre 2010 11:58 > ??: RAYBAUT Pierre 602007 > Objet?: Re: [SciPy-User] [ANN] guiqwt v2.0.3 > > Hi Pierre > > Wow, this is a massive amount of work! Out of curiosity, why did you > choose not leverage Traits and Chaco? Thank you for taking the > trouble to open source your work; the scientific community is indebted > (already for Python(x,y) that, e.g., our university uses, but also for > this). > > Kind regards > St?fan > > On Fri, Sep 3, 2010 at 6:03 PM, wrote: > > Hi all, > > > > I am pleased to announce that `guiqwt` v2.0.3 has been released. This > library was initially written to help me to develop signal/image > processing software for internal use only, but after a long process I've > just been able to share this work with the scientific Python community. > > > > Based on PyQwt (plot widgets for PyQt4 graphical user interfaces) and on > scientific modules NumPy and SciPy, guiqwt is a Python library for > efficient 2-D data plotting (curves, 1-D and 2-D histograms, images) and > signal/image processing application development. > > > > As you certainly know, the most popular Python module for data plotting > is currently matplotlib, an open-source library providing a lot of plot > types and an API (the pylab interface) which is very close to MATLAB's > plotting interface. > > > > guiqwt plotting features are quite limited in terms of plot types > compared to matplotlib. However the currently implemented plot types are > much more efficient. For example, the guiqwt image showing function > (imshow) do not make any copy of the displayed data, hence allowing to > show images much larger than with its matplotlib's counterpart. In other > terms, when showing a 30-MB image (16-bits unsigned integers for example) > with guiqwt, no additional memory is wasted to display the image (except > for the offscreen image of course which depends on the window size) > whereas matplotlib takes more than 600-MB of additional memory (the > original array is duplicated four times using 64-bits float data types). > > > > guiqwt also provides the following features: > > > > ? ?guiqwt.pyplot: equivalent to matplotlib's pyplot module (pylab) > > > > ? ?supported plot items: > > > > ? ? ? ?* curves, error bar curves and 1-D histograms > > ? ? ? ?* images (RGB images are not supported), images with non-linear > x/y scales, images with specified pixel size (e.g. loaded from DICOM > files), 2-D histograms, pseudo-color images (pcolor) > > ? ? ? ?* labels, curve plot legends > > ? ? ? ?* shapes: polygon, polylines, rectangle, circle, ellipse and > segment > > ? ? ? ?* annotated shapes (shapes with labels showing position and > dimensions): rectangle with center position and size, circle with center > position and diameter, ellipse with center position and diameters (these > items are very useful to measure things directly on displayed images) > > > > ? ?curves, images and shapes: > > > > ? ? ? ?* multiple object selection for moving objects or editing their > properties through automatically generated dialog boxes (guidata) > > ? ? ? ?* item list panel: move objects from foreground to background, > show/hide objects, remove objects, ... > > ? ? ? ?* customizable aspect ratio > > ? ? ? ?* a lot of ready-to-use tools: plot canvas export to image file, > image snapshot, image rectangular filter, etc. > > > > ? ?curves: > > > > ? ? ? ?* interval selection tools with labels showing results of > computing on selected area > > ? ? ? ?* curve fitting tool with automatic fit, manual fit with sliders, > ... > > > > ? ?images: > > > > ? ? ? ?* contrast adjustment panel: select the LUT by moving a range > selection object on the image levels histogram, eliminate outliers, ... > > ? ? ? ?* X-axis and Y-axis cross-sections: support for multiple images, > average cross-section tool on a rectangular area, ... > > ? ? ? ?* apply any affine transform to displayed images in real-time > (rotation, magnification, translation, horizontal/vertical flip, ...) > > > > ? ?application development helpers: > > > > ? ? ? ?* ready-to-use curve and image plot widgets and dialog boxes > > ? ? ? ?* load/save graphical objects (curves, images, shapes) > > ? ? ? ?* a lot of test scripts which demonstrate guiqwt features > > > > > > guiqwt has been successfully tested on GNU/Linux and Windows platforms. > > > > Python package index page: > > http://pypi.python.org/pypi/guiqwt/ > > > > Documentation, screenshots: > > http://packages.python.org/guiqwt/ > > > > Downloads (source + Python(x,y) plugin): > > http://sourceforge.net/projects/guiqwt/ > > > > Cheers, > > Pierre > > > > --- > > > > Dr. Pierre Raybaut > > CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > From amenity at enthought.com Tue Sep 7 14:26:04 2010 From: amenity at enthought.com (Amenity Applewhite) Date: Tue, 7 Sep 2010 13:26:04 -0500 Subject: [SciPy-User] EPD Webinar: How to solve ODEs with SciPy References: Message-ID: <7586F6F9-0CFF-471D-9FB6-94253826705C@enthought.com> Enthought Python Distribution Webinar September 10 This Friday,Warren Weckesser will host the first of three webinars in a series on solving differential equations in Python. We will take a close look at the two tools available for solving ordinary differential equations in SciPy: the "odeint" function and the "ode" class. Two examples will be discussed: (1) the famous Lorenz equations that exhibit chaos, and (2) the Gray-Scott reaction-diffusion equations in 1D, from which we will obtain a system of ordinary differential equations by using the "Method of Lines". The demonstrations also include 3D plots and animation using Matplotlib. Enthought Python Distribution Webinar How do I...solve differential equations with Python? Part I: SciPy Tools Friday, September 10: 1pm CST/6pm UTC Wait list (for non EPD subscribers): email:amenity at enthought.com Early in 2011, Warren will host Part II: boundary value problems, and in the spring he'll follow up with a third installment to the series. Have a fantastic September, The Enthought Team From jjstickel at vcn.com Tue Sep 7 14:49:20 2010 From: jjstickel at vcn.com (Jonathan Stickel) Date: Tue, 07 Sep 2010 12:49:20 -0600 Subject: [SciPy-User] numpy.diff of sparse matrix Message-ID: <4C868930.1060309@vcn.com> I'd like to use numpy.diff on a sparse matrix, but I get the following: In [76]: np.diff(np.identity(3),axis=0) Out[76]: array([[-1., 1., 0.], [ 0., -1., 1.]]) In [77]: np.diff(sparse.identity(3),axis=0) --------------------------------------------------------------------------- IndexError Traceback (most recent call last) /Users/jstickel/temp/ in () /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/lib/function_base.pyc in diff(a, n, axis) 961 slice1 = [slice(None)]*nd 962 slice2 = [slice(None)]*nd --> 963 slice1[axis] = slice(1, None) 964 slice2[axis] = slice(None, -1) 965 slice1 = tuple(slice1) IndexError: list assignment index out of range I suppose numpy.diff doesn't know how to handle a sparse matrix object. Converting to dense would work, but that would defeat the purpose of using sparse matrices. Any quick solutions? I don't have time now, but if I were to implement a sparse diff, should it go within numpy.diff or be a separate function (maybe scipy.sparse.spdiff)? Thanks, Jonathan From jrennie at gmail.com Tue Sep 7 16:27:26 2010 From: jrennie at gmail.com (Jason Rennie) Date: Tue, 7 Sep 2010 16:27:26 -0400 Subject: [SciPy-User] numpy.diff of sparse matrix In-Reply-To: <4C868930.1060309@vcn.com> References: <4C868930.1060309@vcn.com> Message-ID: My experience is similar to yours---numpy doesn't work with scipy sparse matrices. Not even numpy.dot does the right thing. My guess is that you'd want to write the sparse diff for the sparse library. An example of how to organize such function is the sparse hstack: http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.hstack.html Cheers, Jason On Tue, Sep 7, 2010 at 2:49 PM, Jonathan Stickel wrote: > I'd like to use numpy.diff on a sparse matrix, but I get the following: > > > In [76]: np.diff(np.identity(3),axis=0) > Out[76]: > array([[-1., 1., 0.], > [ 0., -1., 1.]]) > > In [77]: np.diff(sparse.identity(3),axis=0) > --------------------------------------------------------------------------- > IndexError Traceback (most recent call last) > > /Users/jstickel/temp/ in () > > > /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/lib/function_base.pyc > in diff(a, n, axis) > 961 slice1 = [slice(None)]*nd > 962 slice2 = [slice(None)]*nd > --> 963 slice1[axis] = slice(1, None) > 964 slice2[axis] = slice(None, -1) > 965 slice1 = tuple(slice1) > > IndexError: list assignment index out of range > > > I suppose numpy.diff doesn't know how to handle a sparse matrix object. > Converting to dense would work, but that would defeat the purpose of > using sparse matrices. Any quick solutions? I don't have time now, but > if I were to implement a sparse diff, should it go within numpy.diff or > be a separate function (maybe scipy.sparse.spdiff)? > > Thanks, > Jonathan > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.holbrook at gmail.com Tue Sep 7 17:00:12 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Tue, 7 Sep 2010 13:00:12 -0800 Subject: [SciPy-User] Images as markers in matplotlib? Message-ID: Hey y'all, I recently read about Chernoff faces (http://en.wikipedia.org/wiki/Chernoff_face) in one of Edward Tufte's books (great read btw) and would like to mess around with them in matplotlib. My current approach is to generate the faces as images, and then use them as markers on an x-y plot (like this example I found). I just realized, though, that I have no idea how to incorporate images as position markers in matplotlib, or if it's even possible. My search of the mpl docs didn't turn up much. Any ideas? (I know this isn't matplotlib-users, but I was having trouble figuring out where to sign up for that list. If anyone wants would rather I ask them, feel free to point out the sign-up page---I'd appreciate it.) Thanks! --Joshua holbrook From robert.kern at gmail.com Tue Sep 7 17:01:57 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 7 Sep 2010 16:01:57 -0500 Subject: [SciPy-User] Images as markers in matplotlib? In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 16:00, Joshua Holbrook wrote: > (I know this isn't matplotlib-users, but I was having trouble figuring > out where to sign up for that list. If anyone wants would rather I ask > them, feel free to point out the sign-up page---I'd appreciate it.) https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From daniele at grinta.net Wed Sep 8 08:48:39 2010 From: daniele at grinta.net (Daniele Nicolodi) Date: Wed, 08 Sep 2010 14:48:39 +0200 Subject: [SciPy-User] Gaussian Beam propagation Message-ID: <4C878627.6060306@grinta.net> Hello. Sorry for the slightly off topic question. I'm looking for a Gaussian beam propagation software able to propagate a Gaussian beam trough a simple paraxial lens setup. Of course I would prefer a python one, or one with python binding. Does someone know any such package? Writing the code by myself would not be too difficult, and will probably be a nice learning experience, and I would be glad to release it with an open source license. But I do not have the time to write the code right now. Thanks. Cheers, -- Daniele From kossnair at web.de Wed Sep 8 09:22:26 2010 From: kossnair at web.de (kossnair at web.de) Date: Wed, 8 Sep 2010 15:22:26 +0200 (CEST) Subject: [SciPy-User] MannWhitneyU test critical values Message-ID: <1948359233.5157752.1283952146740.JavaMail.fmail@mwmweb072> An HTML attachment was scrubbed... URL: From jjstickel at vcn.com Wed Sep 8 10:52:09 2010 From: jjstickel at vcn.com (Jonathan Stickel) Date: Wed, 08 Sep 2010 08:52:09 -0600 Subject: [SciPy-User] numpy.diff of sparse matrix In-Reply-To: References: <4C868930.1060309@vcn.com> Message-ID: <4C87A319.4070800@vcn.com> I have found a workaround that meets my needs, in case anyone is interested: In [37]: Is = sparse.identity(3) In [38]: Ds = Is[1:] - Is[:-1] In [39]: Ds.todense() Out[39]: matrix([[-1., 1., 0.], [ 0., -1., 1.]]) I haven't yet tested for computational efficiency vs. dense matrices, but I suppose it should be good. Jonathan On 9/7/10 14:27 , Jason Rennie wrote: > My experience is similar to yours---numpy doesn't work with scipy sparse > matrices. Not even numpy.dot does the right thing. My guess is that > you'd want to write the sparse diff for the sparse library. An example > of how to organize such function is the sparse hstack: > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.hstack.html > > Cheers, > > Jason > > On Tue, Sep 7, 2010 at 2:49 PM, Jonathan Stickel > wrote: > > I'd like to use numpy.diff on a sparse matrix, but I get the following: > > > In [76]: np.diff(np.identity(3),axis=0) > Out[76]: > array([[-1., 1., 0.], > [ 0., -1., 1.]]) > > In [77]: np.diff(sparse.identity(3),axis=0) > --------------------------------------------------------------------------- > IndexError Traceback (most recent > call last) > > /Users/jstickel/temp/ in () > > /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/lib/function_base.pyc > in diff(a, n, axis) > 961 slice1 = [slice(None)]*nd > 962 slice2 = [slice(None)]*nd > --> 963 slice1[axis] = slice(1, None) > 964 slice2[axis] = slice(None, -1) > 965 slice1 = tuple(slice1) > > IndexError: list assignment index out of range > > > I suppose numpy.diff doesn't know how to handle a sparse matrix object. > Converting to dense would work, but that would defeat the purpose of > using sparse matrices. Any quick solutions? I don't have time now, but > if I were to implement a sparse diff, should it go within numpy.diff or > be a separate function (maybe scipy.sparse.spdiff)? > > Thanks, > Jonathan > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > -- > Jason Rennie > Research Scientist, ITA Software > 617-714-2645 > http://www.itasoftware.com/ > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From nwerneck at gmail.com Wed Sep 8 13:15:01 2010 From: nwerneck at gmail.com (Nicolau Werneck) Date: Wed, 8 Sep 2010 14:15:01 -0300 Subject: [SciPy-User] =?windows-1252?q?Building_problem_-_scipy/sparse/spa?= =?windows-1252?q?rsetools/csr=5Fwrap=2Ecxx=3A2917=3A_error=3A_expe?= =?windows-1252?q?cted_=60=29=27_before_=91PRIdPTR=92?= Message-ID: Hello, I am trying to build numpy and scipy for my newly built Python 2.7, and I am getting this error message that seems to be a classic problem happening again: compile options: '-I/usr/local/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include/python2.7 -c' g++: scipy/sparse/sparsetools/csr_wrap.cxx scipy/sparse/sparsetools/csr_wrap.cxx: In function ?int require_size(PyArrayObject*, npy_intp*, int)?: scipy/sparse/sparsetools/csr_wrap.cxx:2910: error: expected `)' before ?PRIdPTR? scipy/sparse/sparsetools/csr_wrap.cxx:2910: warning: spurious trailing ?%? in format scipy/sparse/sparsetools/csr_wrap.cxx:2910: warning: too many arguments for format scipy/sparse/sparsetools/csr_wrap.cxx:2917: error: expected `)' before ?PRIdPTR? scipy/sparse/sparsetools/csr_wrap.cxx:2917: warning: spurious trailing ?%? in format scipy/sparse/sparsetools/csr_wrap.cxx:2917: warning: too many arguments for format scipy/sparse/sparsetools/csr_wrap.cxx: In function ?int require_size(PyArrayObject*, npy_intp*, int)?: scipy/sparse/sparsetools/csr_wrap.cxx:2910: error: expected `)' before ?PRIdPTR? scipy/sparse/sparsetools/csr_wrap.cxx:2910: warning: spurious trailing ?%? in format scipy/sparse/sparsetools/csr_wrap.cxx:2910: warning: too many arguments for format scipy/sparse/sparsetools/csr_wrap.cxx:2917: error: expected `)' before ?PRIdPTR? scipy/sparse/sparsetools/csr_wrap.cxx:2917: warning: spurious trailing ?%? in format scipy/sparse/sparsetools/csr_wrap.cxx:2917: warning: too many arguments for format error: Command "g++ -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include/python2.7 -c scipy/sparse/sparsetools/csr_wrap.cxx -o build/temp.linux-x86_64-2.7/scipy/sparse/sparsetools/csr_wrap.o" failed with exit status 1 Numpy seems to have compiled fine. Any ideas of how to fix that? Thanks, ++nicolau -- Nicolau Werneck ? ? ? ?? 25A1 1BCB 714C 29E7 D84C http://www.lti.pcs.usp.br/~nwerneck? ? ? ? ? ? ? ? ?? 8514 152D 103F B001 8740 Linux user #460716 From timmichelsen at gmx-topmail.de Wed Sep 8 14:03:18 2010 From: timmichelsen at gmx-topmail.de (Timmie) Date: Wed, 8 Sep 2010 18:03:18 +0000 (UTC) Subject: [SciPy-User] MemoryError with tsfromtxt Message-ID: Hello, using scikits.timeseries.tsfromtxt I got the following error: File "C:\Python26\lib\site-packages\scikits\timeseries\extras.py", line 504, in tsfromtxt mrec = genfromtxt(fname, **kwargs) File "C:\Python26\lib\site-packages\scikits\timeseries\_preview.py", line 1221, in genfromtxt append_to_rows(tuple(values)) MemoryError Where does this come from? And what could I do to mitigate it? Thanks in advance! Regards, Timmie From pgmdevlist at gmail.com Wed Sep 8 15:04:28 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 8 Sep 2010 21:04:28 +0200 Subject: [SciPy-User] MemoryError with tsfromtxt In-Reply-To: References: Message-ID: On Sep 8, 2010, at 8:03 PM, Timmie wrote: > Hello, > using scikits.timeseries.tsfromtxt I got the following error: > > File "C:\Python26\lib\site-packages\scikits\timeseries\extras.py", line 504, > in tsfromtxt > mrec = genfromtxt(fname, **kwargs) > File "C:\Python26\lib\site-packages\scikits\timeseries\_preview.py", line > 1221, in genfromtxt > append_to_rows(tuple(values)) > MemoryError > > Where does this come from? You must have quite a huge file... Note that it's not a scikits.timeseries pb, just a standard numpy one. > And what could I do to mitigate it? Cut the file in pieces ? From joses.ho at well.ox.ac.uk Wed Sep 8 18:06:33 2010 From: joses.ho at well.ox.ac.uk (Joses Ho) Date: Wed, 8 Sep 2010 22:06:33 +0000 (UTC) Subject: [SciPy-User] help with leastsq and fmin Message-ID: Hi, I am trying to fit to my qPCR data an exponential function where each successive y-value depends on the previous v-value in the following fashion: y[n] = y[n-1] + k ln (1 + (y[n-1]/k) + b where y[n] = current y-value being evaluated and y[n-1] = previous y-value I therefore have three values to fit - y0 (starting value of y), k, and b. I define a residuals function where I compute 20 values of y, given an initial estimate of y0, k and b, and then subtract these from my 20 measured values of y. I have tried to use scipy.optimize.leastsq on the above residuals function, but it doesn't seem to be converging on the correct values of y0,k and b. I have tried scipy.optimize.fmin on the same function, but I get the following error: ...fsim[0] = func(x0) ...ValueError: setting an array element with a sequence. which seems to me that it refuses to accept the x0 array I give to it, consisting of [y0,k,b]. Basically, I am trying to implement the recent method of Boggy and Woolf (PLoS 2010) [doi:10.1371/ journal.pone.0012355] in Python. I am using SciPy 0.8.0b1 and NumPy 1.4.0 on Python 2.6.5 of the Enthought Python Distribution 6.2-2. Any help here would be very much appreciated! Thanks Joses From josef.pktd at gmail.com Wed Sep 8 18:42:26 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 8 Sep 2010 18:42:26 -0400 Subject: [SciPy-User] help with leastsq and fmin In-Reply-To: References: Message-ID: On Wed, Sep 8, 2010 at 6:06 PM, Joses Ho wrote: > Hi, > I am trying to fit to my qPCR data > an exponential function > where each successive y-value depends on the > previous v-value in the following fashion: > y[n] ?= ?y[n-1] + k ln (1 + (y[n-1]/k) + b > where y[n] = current y-value being evaluated > and y[n-1] = previous y-value > > I therefore have three values to fit - y0 (starting value of y), k, and b. you might need good starting values because of the recursive non-linearity. For simpler non-linearities in the recursion, leastsq worked ok for me. > > I define a residuals function where I > compute 20 values of y, > ?given an initial estimate of y0, k and b, and > then subtract these from my 20 > measured values of y. > > I have tried to use scipy.optimize.leastsq on the > above residuals function, but it doesn't seem to be > converging on the correct values of y0,k and b. > I have tried scipy.optimize.fmin on the same function, same function ? see below > but I get the following error: > ...fsim[0] = func(x0) > ...ValueError: setting an array element with a sequence. > which seems to me that it refuses to accept the x0 array > I give to it, consisting of [y0,k,b]. Is your objective function returning a single value? leastsq wants the array of error terms fmin wants the final value of the objective So we cannot use the same function directly in leastsq and fmin. I usually get this kind of exception when I forget this. Josef > > Basically, I am trying to implement the recent method of Boggy and Woolf > (PLoS 2010) [doi:10.1371/ journal.pone.0012355] in Python. > I am using SciPy 0.8.0b1 and NumPy 1.4.0 on > Python 2.6.5 of the Enthought Python Distribution 6.2-2. > > Any help here would be very much appreciated! > > Thanks > Joses > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From kossnair at web.de Thu Sep 9 05:14:30 2010 From: kossnair at web.de (kossnair at web.de) Date: Thu, 9 Sep 2010 11:14:30 +0200 (CEST) Subject: [SciPy-User] MannWhitneyU test critical values Message-ID: <1194593408.7820038.1284023670015.JavaMail.fmail@mwmweb069> An HTML attachment was scrubbed... URL: From lorenzo.isella at gmail.com Thu Sep 9 06:25:37 2010 From: lorenzo.isella at gmail.com (Lorenzo Isella) Date: Thu, 09 Sep 2010 12:25:37 +0200 Subject: [SciPy-User] Making Python Lazy Message-ID: <4C88B621.1000203@gmail.com> Dear All, I hope this is not too off-topic. I am wondering if there is any possibility to make a Python code lazy i.e. to prevent it from calculating quantities which are not used in the code. As an example: you are in a rush to modify your code and at the end it ends up with "dead branches", let's say an array which is calculated but not used in any following calculations, not printed on screen, not stored in a file etc... It would be nice to teach Python to automagically skip its calculation when I run the script (at least in a non-interactive way). I know that such a situation is probably the result of bad programming habits, but it may arise all the same. If I understand correctly, what I am asking for is something different from garbage collection, which would take place, if ever, only after the array has been calculated. Any suggestions (or clarifications if I am on the wrong track) are appreciated. Cheers Lorenzo From joses.ho at well.ox.ac.uk Thu Sep 9 06:40:17 2010 From: joses.ho at well.ox.ac.uk (Joses Ho) Date: Thu, 9 Sep 2010 10:40:17 +0000 (UTC) Subject: [SciPy-User] help with leastsq and fmin References: Message-ID: gmail.com> writes: > > > leastsq wants the array of error terms > fmin wants the final value of the objective > > So we cannot use the same function directly in leastsq and fmin. > > I usually get this kind of exception when I forget this. > > Josef > Hi Josef, Thanks for your advice below. I re-implemented your suggestions, and it seems both leastsq and fmin give the same optimized parameters. I then re-checked the published fit, and my leastsq fit gives slightly better results! So I think my method is good. Thanks, Joses From josef.pktd at gmail.com Thu Sep 9 07:04:09 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Sep 2010 07:04:09 -0400 Subject: [SciPy-User] MannWhitneyU test critical values In-Reply-To: <1194593408.7820038.1284023670015.JavaMail.fmail@mwmweb069> References: <1194593408.7820038.1284023670015.JavaMail.fmail@mwmweb069> Message-ID: On Thu, Sep 9, 2010 at 5:14 AM, wrote: > Hi there, > I just choke on a trivial problem (I think): > I have 13 test data sets and two methods, that return a measure of > performance. (higher value is better) > Now I want to use scipy.stats.mannwhitneyu for significance testing. > However, I did not find a possibility to retrieve a critical value. > If I am right the method returns a critical value, that has to be compared > with a tabulated value. > Can anybody point me towards a way of reading this tabulated value in a > python script? > Thank you very much in advance, > kossnair > All statistical tests (except anderson-darling) return p-values, which you can use to decide whether to reject or not to reject the null hypothesis. For example, a pvalue below 0.05 means, the null hypothesis can be rejected with an alpha of 5 % (confidence level 1-alpha). For the actual critical values, we have to look at the source to see which distribution for the test-statistic is used. I don't remember what mannwhitneyu uses, and I don't have time until later to look it up. Josef > > > Neu: WEB.DE De-Mail - Einfach wie E-Mail, sicher wie ein Brief! > Jetzt De-Mail-Adresse reservieren: https://produkte.web.de/go/demail02 > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.angulo at uam.es Thu Sep 9 07:09:58 2010 From: pablo.angulo at uam.es (Pablo Angulo) Date: Thu, 09 Sep 2010 13:09:58 +0200 Subject: [SciPy-User] Re: Making Python Lazy In-Reply-To: References: Message-ID: <4C88C086.4040005@uam.es> Do you know about generators and generator expressions? http://en.wikipedia.org/wiki/Generator_%28computer_science%29#Python > From: Lorenzo Isella > Subject: [SciPy-User] Making Python Lazy > > Dear All, > I hope this is not too off-topic. > I am wondering if there is any possibility to make a Python code lazy > i.e. to prevent it from calculating quantities which are not used in the > code. > As an example: you are in a rush to modify your code and at the end it > ends up with "dead branches", let's say an array which is calculated but > not used in any following calculations, not printed on screen, not > stored in a file etc... > It would be nice to teach Python to automagically skip its calculation > when I run the script (at least in a non-interactive way). > I know that such a situation is probably the result of bad programming > habits, but it may arise all the same. > If I understand correctly, what I am asking for is something different > from garbage collection, which would take place, if ever, only after the > array has been calculated. > Any suggestions (or clarifications if I am on the wrong track) are > appreciated. > Cheers > > Lorenzo > > > ------------------------------ > From timmichelsen at gmx-topmail.de Thu Sep 9 07:11:52 2010 From: timmichelsen at gmx-topmail.de (Timmie) Date: Thu, 9 Sep 2010 11:11:52 +0000 (UTC) Subject: [SciPy-User] MemoryError with tsfromtxt References: Message-ID: > You must have quite a huge file... Note that it's not a scikits.timeseries pb, just a standard numpy one. The file has 298 MB. 5370772 records (rows); data in minutely frequency. > > And what could I do to mitigate it? > > Cut the file in pieces ? and then concatenate the timeseries? From blattnem at gmail.com Thu Sep 9 07:14:12 2010 From: blattnem at gmail.com (Marcel Blattner) Date: Thu, 9 Sep 2010 13:14:12 +0200 Subject: [SciPy-User] element-wise exponentiation Message-ID: Hi I have a beginners question. I have to exponentiate each element of a sparse lil-matrix. I did: a,b = A.nonzero() for i in range(a.size): A[a[i],b[i]] = A[a[i],b[i]]**0.2 Is there a way to do something like A**0.2 like in numpy? Cheers -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Thu Sep 9 07:19:53 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 9 Sep 2010 13:19:53 +0200 Subject: [SciPy-User] MemoryError with tsfromtxt In-Reply-To: References: Message-ID: <8C562090-D5CE-40EF-918D-249BE6FED835@gmail.com> On Sep 9, 2010, at 1:11 PM, Timmie wrote: >> You must have quite a huge file... Note that it's not a scikits.timeseries > pb, just a standard numpy one. > The file has 298 MB. > > 5370772 records (rows); data in minutely frequency. > >>> And what could I do to mitigate it? >> >> Cut the file in pieces ? > and then concatenate the timeseries? That's the idea. Could you cut it day by day, or week by week, or even month by month to reduce the load ? The issue is that genfromtxt has to keep a lot of information in memory (a list of values, a list of masks) before creating the array, and you're overloading Python's capacity to deal with it... From cmutel at gmail.com Thu Sep 9 07:28:22 2010 From: cmutel at gmail.com (Christopher Mutel) Date: Thu, 9 Sep 2010 13:28:22 +0200 Subject: [SciPy-User] element-wise exponentiation In-Reply-To: References: Message-ID: It is usually easier to mathematical operations on a compressed matrix, e.g. CSR or CSC. Access the arrays that hold actual data is much easier (in my opinion). One simple option would be: A = A.tocsr() A.data = A.data**0.2 # This is the array that hold the actual values in the sparse matrix A = A.tolil() # If desired -Chris On 9 September 2010 13:14, Marcel Blattner wrote: > Hi > I have a beginners question. I have to exponentiate each element of a sparse > lil-matrix. > I did: > a,b = A.nonzero() > for i in range(a.size): > ?? ? ?A[a[i],b[i]] = A[a[i],b[i]]**0.2 > Is there a way to do something like A**0.2 like in numpy? > Cheers > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From gael.varoquaux at normalesup.org Thu Sep 9 07:30:59 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 9 Sep 2010 13:30:59 +0200 Subject: [SciPy-User] Making Python Lazy In-Reply-To: <4C88B621.1000203@gmail.com> References: <4C88B621.1000203@gmail.com> Message-ID: <20100909113059.GC17378@phare.normalesup.org> Not excatly what you want, but joblib, http://packages.python.org/joblib/ addresses partly this issue. I won't say more on the list , os this is code that is still work in progress, and I was meaning to announce it to the scipy community only in a little while. However, we do use it in production on big datasets (0.5 branch on github has issues, use latest release or 0.4.X branch...). Ga?l On Thu, Sep 09, 2010 at 12:25:37PM +0200, Lorenzo Isella wrote: > Dear All, > I hope this is not too off-topic. > I am wondering if there is any possibility to make a Python code lazy > i.e. to prevent it from calculating quantities which are not used in the > code. > As an example: you are in a rush to modify your code and at the end it > ends up with "dead branches", let's say an array which is calculated but > not used in any following calculations, not printed on screen, not > stored in a file etc... > It would be nice to teach Python to automagically skip its calculation > when I run the script (at least in a non-interactive way). > I know that such a situation is probably the result of bad programming > habits, but it may arise all the same. > If I understand correctly, what I am asking for is something different > from garbage collection, which would take place, if ever, only after the > array has been calculated. > Any suggestions (or clarifications if I am on the wrong track) are > appreciated. > Cheers > Lorenzo > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Gael Varoquaux Research Fellow, INRIA Laboratoire de Neuro-Imagerie Assistee par Ordinateur NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France Phone: ++ 33-1-69-08-78-35 Mobile: ++ 33-6-28-25-64-62 http://gael-varoquaux.info From blattnem at gmail.com Thu Sep 9 07:48:55 2010 From: blattnem at gmail.com (Marcel Blattner) Date: Thu, 9 Sep 2010 13:48:55 +0200 Subject: [SciPy-User] element-wise exponentiation In-Reply-To: References: Message-ID: Thankx......everything smooth now.... On Thu, Sep 9, 2010 at 1:28 PM, Christopher Mutel wrote: > It is usually easier to mathematical operations on a compressed > matrix, e.g. CSR or CSC. Access the arrays that hold actual data is > much easier (in my opinion). > > One simple option would be: > A = A.tocsr() > A.data = A.data**0.2 # This is the array that hold the actual values > in the sparse matrix > A = A.tolil() # If desired > > -Chris > > On 9 September 2010 13:14, Marcel Blattner wrote: > > Hi > > I have a beginners question. I have to exponentiate each element of a > sparse > > lil-matrix. > > I did: > > a,b = A.nonzero() > > for i in range(a.size): > > A[a[i],b[i]] = A[a[i],b[i]]**0.2 > > Is there a way to do something like A**0.2 like in numpy? > > Cheers > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_baddeley at yahoo.com.au Thu Sep 9 08:30:44 2010 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Thu, 9 Sep 2010 05:30:44 -0700 (PDT) Subject: [SciPy-User] Making Python Lazy In-Reply-To: <4C88B621.1000203@gmail.com> References: <4C88B621.1000203@gmail.com> Message-ID: <118401.53985.qm@web33006.mail.mud.yahoo.com> Hi Lorenzo, I've got something which I use to perform coordinate transformations which might give you some ideas ... it basically stores a set of mappings and only executes them on demand. I've attached a slightly simplified version cheers, David ----- Original Message ---- From: Lorenzo Isella To: scipy-user at scipy.org Sent: Thu, 9 September, 2010 10:25:37 PM Subject: [SciPy-User] Making Python Lazy Dear All, I hope this is not too off-topic. I am wondering if there is any possibility to make a Python code lazy i.e. to prevent it from calculating quantities which are not used in the code. As an example: you are in a rush to modify your code and at the end it ends up with "dead branches", let's say an array which is calculated but not used in any following calculations, not printed on screen, not stored in a file etc... It would be nice to teach Python to automagically skip its calculation when I run the script (at least in a non-interactive way). I know that such a situation is probably the result of bad programming habits, but it may arise all the same. If I understand correctly, what I am asking for is something different from garbage collection, which would take place, if ever, only after the array has been calculated. Any suggestions (or clarifications if I am on the wrong track) are appreciated. Cheers Lorenzo _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- A non-text attachment was scrubbed... Name: lazyTest.py Type: application/octet-stream Size: 2360 bytes Desc: not available URL: From bsouthey at gmail.com Thu Sep 9 09:29:32 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 09 Sep 2010 08:29:32 -0500 Subject: [SciPy-User] MemoryError with tsfromtxt In-Reply-To: <8C562090-D5CE-40EF-918D-249BE6FED835@gmail.com> References: <8C562090-D5CE-40EF-918D-249BE6FED835@gmail.com> Message-ID: <4C88E13C.1020406@gmail.com> On 09/09/2010 06:19 AM, Pierre GM wrote: > On Sep 9, 2010, at 1:11 PM, Timmie wrote: > >>> You must have quite a huge file... Note that it's not a scikits.timeseries >> pb, just a standard numpy one. >> The file has 298 MB. >> >> 5370772 records (rows); data in minutely frequency. >> >>>> And what could I do to mitigate it? >>> Cut the file in pieces ? >> and then concatenate the timeseries? > That's the idea. Could you cut it day by day, or week by week, or even month by month to reduce the load ? > The issue is that genfromtxt has to keep a lot of information in memory (a list of values, a list of masks) before creating the array, and you're overloading Python's capacity to deal with it... > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user You could buy more memory because 5.4 million rows can add up very quickly with many columns. Note that you also need contiguous memory available. If you know the format of the input, then use something else like loadtxt. If you know the size and format then you can slowly iterate over the file and input the values directly into an empty array or use Chris's code to append to an array. Bruce From eric.moscardi at sophia.inria.fr Thu Sep 9 12:06:20 2010 From: eric.moscardi at sophia.inria.fr (moscardi) Date: Thu, 9 Sep 2010 18:06:20 +0200 Subject: [SciPy-User] Pb with Extending ndimage in C Message-ID: Hi scipy-users, I try to use the geometric_transform function from scipy.ndimage with a mapping function written in C in order to accelerate execution time of my code. For that, I try to use the tutorial " Extending ndimage in C" (http://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html#ndimage-ccallbacks) but I have any difficulties with it. The given example (which implement a simple shift function) doesn't work : ########################################################################### static int _shift_function(int *output_coordinates, double* input_coordinates, int output_rank, int input_rank, void *callback_data) { int ii; /* get the shift from the callback data pointer: */ double shift = *(double*)callback_data; /* calculate the coordinates: */ for(ii = 0; ii < irank; ii++) icoor[ii] = ocoor[ii] - shift; /* return OK status: */ return 1; } ########################################################################### ***************************************************************************** gcc example.c --shared -lpython -o example.so Compile error : example.c: In function ?_shift_function?: example.c:12: error: ?irank? undeclared (first use in this function) example.c:12: error: (Each undeclared identifier is reported only once example.c:12: error: for each function it appears in.) example.c:13: error: ?icoor? undeclared (first use in this function) example.c:13: error: ?ocoor? undeclared (first use in this function) ***************************************************************************** So I replace : icoor ocoor irank by : input_coordinates output_coordinates input_rank So my code is : ########################################################################### static int _shift_function(int *output_coordinates, double* input_coordinates, int output_rank, int input_rank, void *callback_data) { int ii; /* get the shift from the callback data pointer: */ double shift = *(double*)callback_data; /* calculate the coordinates: */ for(ii = 0; ii < input_rank; ii++) input_coordinates[ii] = output_coordinates[ii] - shift; /* return OK status: */ return 1; } ########################################################################### I run this code, but the result is not the same as the example : >>> a = np.arange(12.).reshape((4, 3)) >>> fnc= example.shift_function(0.5) >>> print ndimage.geometric_transform(a, fnc) array([[ 0., 0., 0.], [ 0., 0., 0.], [ 0., 0., 0.], [ 0., 0., 0.]]) instead of : [[ 0. 0. 0. ] [ 0. 1.3625 2.7375] [ 0. 4.8125 6.1875] [ 0. 8.2625 9.6375]] Could someone please suggest what I need to get this working? Thanks, Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnl at cs.wisc.edu Thu Sep 9 12:22:18 2010 From: johnl at cs.wisc.edu (johnl) Date: Thu, 09 Sep 2010 11:22:18 -0500 Subject: [SciPy-User] Weave.inline with multiple processors Message-ID: <1284049338.9171.9.camel@johnl-laptop> Hello. I'm a long-time scipy user, but I'm new to this list. I'm having an issue that I haven't been able to find a solution to. I have several python functions with C code embedded using weave.inline. When the code is run, the functions are called by multiple processes. The first time the code is run, this causes a problem as all the processes try to compile the C code simultaneously leading to a broken object file. Any suggestions for fixing this would be appreciated. Thank you, J. David Lee From Dharhas.Pothina at twdb.state.tx.us Thu Sep 9 13:11:57 2010 From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina) Date: Thu, 09 Sep 2010 12:11:57 -0500 Subject: [SciPy-User] Austin Job Posting Message-ID: <4C88CF0D.63BA.009B.1@twdb.state.tx.us> Hi All, We have a new full time position open in the Surface Water Resources division at the Texas Water Development Board that may be of interest to folks on this mailing list. This person will primarily working under me developing and maintaining software and procedures to manage and analyze various types of surface water data collected across our division. Most of the work will be using python/numpy/scipy. A description of the position and details on how to apply can be found on our website at http://www.twdb.state.tx.us/about/employment_opportunities/Employopport.asp (Scroll Down to Position # 11-01 Environmental Data Analyst). The actual responsibilities of the position and the skills we are looking for are flexible so I encourage potential applicants to call or email me if they have questions about the position. Actual applications need to be made following the procedures outlined on our website. Please circulate this email as you see fit. Thanks - dharhas Dharhas Pothina, Ph.D., P.E. Special Projects Lead Surface Water Resources Division Texas Water Development Board 1700 North Congress Ave. P.O. Box 13231 Austin, TX 78711-3231 Tel: (512) 936-0818 Fax: (512) 936-0816 dharhas.pothina at twdb.state.tx.us www.twdb.state.tx.us From pierre.raybaut at gmail.com Thu Sep 9 14:20:19 2010 From: pierre.raybaut at gmail.com (Pierre Raybaut) Date: Thu, 09 Sep 2010 20:20:19 +0200 Subject: [SciPy-User] ANN: Python(x,y) 2.6.5.3 Message-ID: <4C892563.6000003@gmail.com> Hi all, I am pleased to announce that Python(x,y) 2.6.5.3 has been released. Python(x,y) is a free Python distribution providing a ready-to-use scientific development software for numerical computations, data analysis and data visualization based on Python programming language, Qt graphical user interfaces (and development framework), Eclipse integrated development environment and Spyder interactive development environment. Its purpose is to help scientific programmers used to interpreted languages (such as MATLAB or IDL) or compiled languages (C/C++ or Fortran) to switch to Python. Changes history Version 2.6.5.3 (09/07/2010) Updated * Enthought Tool Suite 3.4.1 * VTK 5.6.0 Version 2.6.5.2 (09/06/2010) Added * guidata 1.2.1 - Automatically generated graphical user interfaces for easy data set edition and display * guiqwt 2.0.3 - Efficient curve/image plotting and other GUI tools for scientific data processing software development * Veusz 1.9 - Scientific plotting application designed to create publication quality output with a friendly interface * pyhdf 0.8.1 - Python interface to HDF4 files (Hierarchical Data Format version 4) Updated * xy 1.2.1 * Spyder 1.1.5 * Matplotlib 1.0.0 * Enthought Tool Suite 3.4.1 * VTK 5.6.0 * pyopencv 2.1.0.wr1.1.0 * NetworkX 1.3 * MDP 2.6 * Parallel Python 1.6.0 * Cython 0.13 * Sphinx 1.0.3 * docutils 0.7 * jinja2 2.5.2 * pylint 0.21.2 * winpdb 1.4.8 * Pydev 1.6.1 * Console 2.0.146.1 * SciTE 2.21 * SWIG 2.0.0 Cheers, Pierre From timmichelsen at gmx-topmail.de Thu Sep 9 18:20:38 2010 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Fri, 10 Sep 2010 00:20:38 +0200 Subject: [SciPy-User] MemoryError with tsfromtxt In-Reply-To: <4C88E13C.1020406@gmail.com> References: <8C562090-D5CE-40EF-918D-249BE6FED835@gmail.com> <4C88E13C.1020406@gmail.com> Message-ID: > If you know the format of the input, then use something else like loadtxt. loadtxt also failed. > If you know the size and format then you can slowly iterate over the > file and input the values directly into an empty array or use Chris's > code to append to an array. Which code from Chris? From timmichelsen at gmx-topmail.de Thu Sep 9 18:20:04 2010 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Fri, 10 Sep 2010 00:20:04 +0200 Subject: [SciPy-User] MemoryError with tsfromtxt In-Reply-To: References: Message-ID: >> Where does this come from? > > You must have quite a huge file... Note that it's not a scikits.timeseries pb, just a standard numpy one. I got it solved using a loop function over the input files that I used to make the large file. I think part of the problem arised because I didn't clearly specify TAB as delimiter at the beginning. From timmichelsen at gmx-topmail.de Thu Sep 9 18:25:06 2010 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Fri, 10 Sep 2010 00:25:06 +0200 Subject: [SciPy-User] AttributeError with scikits.timeseries.tstables Message-ID: Hello, saving a masked array (time series) to a HDF5 file through the pytables interface with scikits.timeseries.tstables I received the following error message: File "C:\Python26\lib\site-packages\scikits\timeseries\lib\tstables.py", line 714, in createTimeSeriesTable chunkshape=chunkshape, byteorder=byteorder) File "C:\Python26\lib\site-packages\scikits\timeseries\lib\tstables.py", line 419, in __init__ _log=_log) File "C:\Python26\lib\site-packages\scikits\timeseries\lib\tstables.py", line 292, in __init__ self.attrs.special_attrs = self._update_special_attrs(maskedarray) File "C:\Python26\lib\site-packages\scikits\timeseries\lib\tstables.py", line 425, in _update_special_attrs special_attrs['recshape'] = tuple(a._varshape) AttributeError: 'MaskedArray' object has no attribute '_varshape' I would love to get help on why the hdf write process fails here. Do I need to fill missing values? Thanks in advance, Timmie From guyer at nist.gov Fri Sep 10 12:07:04 2010 From: guyer at nist.gov (Jonathan Guyer) Date: Fri, 10 Sep 2010 12:07:04 -0400 Subject: [SciPy-User] Making Python Lazy In-Reply-To: <4C88B621.1000203@gmail.com> References: <4C88B621.1000203@gmail.com> Message-ID: FiPy has a lazy evaluate scheme built on top of numpy arrays. It's something I'd like to integrate more tightly with numpy, but have never found the time. In our case, the calculation is (often) done once, in order to determine the shape (and physical units) of the result. Since FiPy is normally performing the same calculations thousands or millions of times, that one calculation is negligible, but it may be a problem for your application. On Sep 9, 2010, at 6:25 AM, Lorenzo Isella wrote: > Dear All, > I hope this is not too off-topic. > I am wondering if there is any possibility to make a Python code lazy > i.e. to prevent it from calculating quantities which are not used in the > code. > As an example: you are in a rush to modify your code and at the end it > ends up with "dead branches", let's say an array which is calculated but > not used in any following calculations, not printed on screen, not > stored in a file etc... > It would be nice to teach Python to automagically skip its calculation > when I run the script (at least in a non-interactive way). > I know that such a situation is probably the result of bad programming > habits, but it may arise all the same. > If I understand correctly, what I am asking for is something different > from garbage collection, which would take place, if ever, only after the > array has been calculated. > Any suggestions (or clarifications if I am on the wrong track) are > appreciated. > Cheers > > Lorenzo > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From Dharhas.Pothina at twdb.state.tx.us Fri Sep 10 14:26:46 2010 From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina) Date: Fri, 10 Sep 2010 13:26:46 -0500 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. Message-ID: <4C8A3215.63BA.009B.1@twdb.state.tx.us> Hi, I have an monotonically increasing array with duplicates in it. e.g. x = np.array([1.0,1.0,1.0,2.0,2.0,3.0,3.0,3.0,4.0,4.0,5.0,5.0,5.0,5.0,6.0]) I need a new array of the same size with linearly interpolated values i.e something like np.array([1.0, 1.33, 1.67, 2.0, 2.5, 3.0, 3.33, 3.67, 4.0, 4.5, 5.0, 5.25, 5.5, 5.75, 6.0]) Is there an efficient way to do this. Right now I'm looping through the array and maintaining position flags of when the value changes and then doing linear interpolation between the start and end flags before resetting the flags moving to the next section. This is pretty slow. I realize there will be a problem on how to deal with duplicates at the end of the array. thanks - dharhas From gokhansever at gmail.com Fri Sep 10 15:48:34 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Fri, 10 Sep 2010 14:48:34 -0500 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: <4C8A3215.63BA.009B.1@twdb.state.tx.us> References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> Message-ID: On Fri, Sep 10, 2010 at 1:26 PM, Dharhas Pothina < Dharhas.Pothina at twdb.state.tx.us> wrote: > Hi, > > I have an monotonically increasing array with duplicates in it. e.g. > > x = np.array([1.0,1.0,1.0,2.0,2.0,3.0,3.0,3.0,4.0,4.0,5.0,5.0,5.0,5.0,6.0]) > > I need a new array of the same size with linearly interpolated values i.e > something like > > np.array([1.0, 1.33, 1.67, 2.0, 2.5, 3.0, 3.33, 3.67, 4.0, 4.5, 5.0, 5.25, > 5.5, 5.75, 6.0]) > > Is there an efficient way to do this. Right now I'm looping through the > array and maintaining position flags of when the value changes and then > doing linear interpolation between the start and end flags before resetting > the flags moving to the next section. This is pretty slow. > > I realize there will be a problem on how to deal with duplicates at the end > of the array. > > thanks > > - dharhas > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Hello again, How did you choose your interpolation method? Are required to go through each switch point? If not you can try the approach below. (Try in ipython -pylab) from scipy.interpolate import interp1d x = np.array([1.0,1.0,1.0,2.0,2.0,3.0,3.0,3.0,4.0,4.0,5.0,5.0,5.0,5.0,6.0]) f = interp1d(range(len(np.unique(x))), np.unique(x)) xnew = np.linspace(0,len(np.unique(x))-1,len(x)); f(xnew) plt.figure(); plt.plot(range(len(x)), x, 'o', range(len(x)), f(xnew), '-') #Comparing to your case plt.figure(); plt.plot(x, "o", y, "-") -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dharhas.Pothina at twdb.state.tx.us Fri Sep 10 16:32:19 2010 From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina) Date: Fri, 10 Sep 2010 15:32:19 -0500 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> Message-ID: <4C8A4F83.63BA.009B.1@twdb.state.tx.us> I may be missing something but I'm not seeing how that would work if the spacing was fairly irregular unlike my example array. I don't mind the interpolation not going through the switching points exactly but the points should not shift to the next interval which it seems would happen if the array was irregularly spaced. > from scipy.interpolate import interp1d > x = np.array([1.0,1.0,1.0,2.0,2.0,3.0,3.0,3.0,4.0,4.0,5.0,5.0,5.0,5.0,6.0]) > f = interp1d(range(len(np.unique(x))), np.unique(x)) > xnew = np.linspace(0,len(np.unique(x))-1,len(x)); f(xnew) > plt.figure(); plt.plot(range(len(x)), x, 'o', range(len(x)), f(xnew), '-') - d From borreguero at gmail.com Fri Sep 10 16:41:43 2010 From: borreguero at gmail.com (Jose Borreguero) Date: Fri, 10 Sep 2010 16:41:43 -0400 Subject: [SciPy-User] adding two pseudoinverse matrixes In-Reply-To: References: Message-ID: Dear Scipy users, I have to solve for Z in the following equation Z^(-1) = A^(-1) - B^(-1), where A and B are covariance matrices with zero determinant. I have never used pseudoinverse matrixes, could anybody please point to me any cautions I have to take when solving this equation for Z? The brute force approach linalg.pinv( linalg.pinv(A) - lingal.pinv(B) ) gives me a matrix with all entries equal to 'infinity'. -Jose Borreguero -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Fri Sep 10 16:53:03 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Fri, 10 Sep 2010 15:53:03 -0500 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: <4C8A4F83.63BA.009B.1@twdb.state.tx.us> References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> <4C8A4F83.63BA.009B.1@twdb.state.tx.us> Message-ID: Could you give one example with a pseudo irregular data? Are the data-points irregular or the x-locations of each-data point? On Fri, Sep 10, 2010 at 3:32 PM, Dharhas Pothina < Dharhas.Pothina at twdb.state.tx.us> wrote: > > I may be missing something but I'm not seeing how that would work if the > spacing was fairly irregular unlike my example array. I don't mind the > interpolation not going through the switching points exactly but the points > should not shift to the next interval which it seems would happen if the > array was irregularly spaced. > > > from scipy.interpolate import interp1d > > x = > np.array([1.0,1.0,1.0,2.0,2.0,3.0,3.0,3.0,4.0,4.0,5.0,5.0,5.0,5.0,6.0]) > > f = interp1d(range(len(np.unique(x))), np.unique(x)) > > xnew = np.linspace(0,len(np.unique(x))-1,len(x)); f(xnew) > > plt.figure(); plt.plot(range(len(x)), x, 'o', range(len(x)), f(xnew), > '-') > > - d > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dtlussier at gmail.com Fri Sep 10 18:31:52 2010 From: dtlussier at gmail.com (Dan Lussier) Date: Fri, 10 Sep 2010 17:31:52 -0500 Subject: [SciPy-User] ask scipy Message-ID: <651C6A9D-8B30-4496-B891-DB94EA2BDEFC@gmail.com> Hi, I am having trouble logging in to Ask Scipy using my GoogleID. On both Safari and Firefox on Mac I cannot get the input field to switch over to take GoodID input. The same appears to be true for Yahoo, while the other options are working well. Just wanted to pass it along. From josh.holbrook at gmail.com Fri Sep 10 18:39:21 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Fri, 10 Sep 2010 14:39:21 -0800 Subject: [SciPy-User] ask scipy In-Reply-To: <651C6A9D-8B30-4496-B891-DB94EA2BDEFC@gmail.com> References: <651C6A9D-8B30-4496-B891-DB94EA2BDEFC@gmail.com> Message-ID: On Fri, Sep 10, 2010 at 2:31 PM, Dan Lussier wrote: > Hi, > > I am having trouble logging in to Ask Scipy using my GoogleID. ?On > both Safari and Firefox on Mac I cannot get the input field to switch > over to take GoodID input. ?The same appears to be true for Yahoo, > while the other options are working well. > > Just wanted to pass it along. > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > I think GoogleID is a known issue, but I hadn't heard of Yahoo being an issue. Huh! I've been using the flickr auth, myself. --Josh From jsseabold at gmail.com Fri Sep 10 18:58:42 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Fri, 10 Sep 2010 18:58:42 -0400 Subject: [SciPy-User] ask scipy In-Reply-To: <651C6A9D-8B30-4496-B891-DB94EA2BDEFC@gmail.com> References: <651C6A9D-8B30-4496-B891-DB94EA2BDEFC@gmail.com> Message-ID: On Fri, Sep 10, 2010 at 6:31 PM, Dan Lussier wrote: > Hi, > > I am having trouble logging in to Ask Scipy using my GoogleID. ?On > both Safari and Firefox on Mac I cannot get the input field to switch > over to take GoodID input. ?The same appears to be true for Yahoo, > while the other options are working well. > > Just wanted to pass it along. Copy and paste https://www.google.com/accounts/o8/id into the login field is the only workaround I have seen suggested. Definitely odd though. Skipper From dtlussier at gmail.com Fri Sep 10 19:21:39 2010 From: dtlussier at gmail.com (Dan Lussier) Date: Fri, 10 Sep 2010 18:21:39 -0500 Subject: [SciPy-User] naming columns in array Message-ID: I was wondering if there was possible to change the dtpye of an array once it exits. Here is my situation. I have either a N x 1 column, or a N x M array where I would like to be able to name the columns after the array exists. I have tried working with record arrays a bit, but no luck. If someone might be able to scratch a quick example I'd be grateful. Thanks! From josef.pktd at gmail.com Fri Sep 10 19:28:44 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 10 Sep 2010 19:28:44 -0400 Subject: [SciPy-User] naming columns in array In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 7:21 PM, Dan Lussier wrote: > I was wondering if there was possible to change the dtpye of an array > once it exits. > > Here is my situation. > > I have either a N x 1 column, or a N x M array where I would like to > be able to name the columns after the array exists. > > I have tried working with record arrays a bit, but no luck. ?If > someone might be able to scratch a quick example I'd be grateful. creating a view with structured dtype >>> x = np.array([[1, 2],[3,4]]) >>> x.view(dtype=[('a', np.int32), ('b', np.int32)]) array([[(1, 2)], [(3, 4)]], dtype=[('a', ' > Thanks! > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From ben.root at ou.edu Fri Sep 10 21:12:23 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 10 Sep 2010 20:12:23 -0500 Subject: [SciPy-User] naming columns in array In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 6:28 PM, wrote: > On Fri, Sep 10, 2010 at 7:21 PM, Dan Lussier wrote: > > I was wondering if there was possible to change the dtpye of an array > > once it exits. > > > > Here is my situation. > > > > I have either a N x 1 column, or a N x M array where I would like to > > be able to name the columns after the array exists. > > > > I have tried working with record arrays a bit, but no luck. If > > someone might be able to scratch a quick example I'd be grateful. > > creating a view with structured dtype > > >>> x = np.array([[1, 2],[3,4]]) > >>> x.view(dtype=[('a', np.int32), ('b', np.int32)]) > array([[(1, 2)], > [(3, 4)]], > dtype=[('a', ' > Maybe a labeled array (a.k.a. - larry)? larry, the manual ? la v0.4.0 documentation Ben Root > Josef > > > > > Thanks! > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timmichelsen at gmx-topmail.de Sat Sep 11 03:30:17 2010 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Sat, 11 Sep 2010 09:30:17 +0200 Subject: [SciPy-User] [SOLVED] Re: AttributeError with scikits.timeseries.tstables In-Reply-To: References: Message-ID: > saving a masked array (time series) to a HDF5 file through the pytables > interface with scikits.timeseries.tstables I received the following > error message: [...] > > I would love to get help on why the hdf write process fails here. I solved it by using tables.File.createMaskedTable instead of tables.File.createTimeSeriesTable. Is there really need for two different functions? Can tables.File.createTimeSeriesTable not chnage the behaviour depending if the array is masked or not? From massimodisasha at gmail.com Sat Sep 11 08:45:38 2010 From: massimodisasha at gmail.com (Massimo Di Stefano) Date: Sat, 11 Sep 2010 14:45:38 +0200 Subject: [SciPy-User] scan array to extract min-max values (with if condition) Message-ID: Hello All, i need to extract data from an array, that are inside a rectangle area defined as : N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 the data are in a csv (comma delimited text file, with 3 columns X,Y,Z) #X,Y,Z 3020081.5500,769999.3100,0.0300 3020086.2000,769991.6500,0.4600 3020099.6600,769996.2700,0.9000 ... ... i read it using " numpy.loadtxt " data : http://www.geofemengineering.it/data/csv.txt 5,3 mb (158735 rows) to extract data that are inside the boundy-box area (N, S, E, W) i'm using a loop inside a function like : import numpy as np def getMinMaxBB(data, N, S, E, W): mydata = data * 0.3048006096012 for i in range(len(mydata)): if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : if i == 0: newdata = np.array((mydata[i,0],mydata[i,1],mydata[i,2]), float) else : newdata = np.vstack((newdata,(mydata[i,0], mydata[i,1], mydata[i,2]))) results = {} results['Max_Z'] = newdata.max(0)[2] results['Min_Z'] = newdata.min(0)[2] results['Num_P'] = len(newdata) return results N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 data = '/Users/sasha/csv.txt' mydata = np.loadtxt(data, comments='#', delimiter=',') out = getMinMaxBB(mydata, N, S, E, W) print out This method works, but maybe is not soo fast, have you any hints on how to improve code with better performance ? thanks!!! Massimo. From thoeger at fys.ku.dk Sat Sep 11 11:40:14 2010 From: thoeger at fys.ku.dk (=?ISO-8859-1?Q?Th=F8ger?= Emil Juul Thorsen) Date: Sat, 11 Sep 2010 17:40:14 +0200 Subject: [SciPy-User] scan array to extract min-max values (with if condition) In-Reply-To: References: Message-ID: <1284219614.2834.3.camel@falconeer> I had a similar problem lately and never really got an elegant solution. I suggested that polygon clipping be implemented in NumPy, since I believe there are quite a lot of cases where it would be handy, and I hereby renew that request. Cheers; Emil On Sat, 2010-09-11 at 14:45 +0200, Massimo Di Stefano wrote: > Hello All, > > i need to extract data from an array, that are inside a > rectangle area defined as : > > N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 > > the data are in a csv (comma delimited text file, with 3 columns X,Y,Z) > > #X,Y,Z > 3020081.5500,769999.3100,0.0300 > 3020086.2000,769991.6500,0.4600 > 3020099.6600,769996.2700,0.9000 > ... > ... > > i read it using " numpy.loadtxt " > > data : > > http://www.geofemengineering.it/data/csv.txt 5,3 mb (158735 rows) > > to extract data that are inside the boundy-box area (N, S, E, W) i'm using a loop > inside a function like : > > import numpy as np > > def getMinMaxBB(data, N, S, E, W): > mydata = data * 0.3048006096012 > for i in range(len(mydata)): > if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : > if i == 0: > newdata = np.array((mydata[i,0],mydata[i,1],mydata[i,2]), float) > else : > newdata = np.vstack((newdata,(mydata[i,0], mydata[i,1], mydata[i,2]))) > results = {} > results['Max_Z'] = newdata.max(0)[2] > results['Min_Z'] = newdata.min(0)[2] > results['Num_P'] = len(newdata) > return results > > > N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 > data = '/Users/sasha/csv.txt' > mydata = np.loadtxt(data, comments='#', delimiter=',') > out = getMinMaxBB(mydata, N, S, E, W) > > print out > > > This method works, but maybe is not soo fast, have you any hints on how to improve code with better performance ? > > thanks!!! > > Massimo. > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From massimodisasha at gmail.com Sat Sep 11 15:53:27 2010 From: massimodisasha at gmail.com (Massimo Di Stefano) Date: Sat, 11 Sep 2010 21:53:27 +0200 Subject: [SciPy-User] [Numpy-discussion] scan array to extract min-max values (with if condition) In-Reply-To: References: Message-ID: <24AD26B0-C1F7-49B7-9EA5-81DCBF058197@gmail.com> Brett, i tried a different way to solve the problem, using : ############# import os fpath = '/Users/sasha/py/' input_fp = open( os.path.join(fpath, 'BE3730072600WC20050817.txt'), 'r' ) input_file = input_fp.readlines() N = 234560.94503118 S = 234482.56929822 E = 921336.53116178 W = 921185.3779625 xL = [] yL = [] zL = [] for index, line in enumerate( input_file ): if index == 0: print 'skipping header line...' else: x, y, z = line.split(',') xL.append( float(x) * 0.3048006096012 ) yL.append( float(y) * 0.3048006096012 ) zL.append( float(z) * 0.3048006096012 ) xLr = [] yLr = [] zLr = [] for coords in zip(xL, yL, zL): if W < coords[0] < E and S < coords[1] < N: xLr.append( coords[0] ) yLr.append( coords[1] ) zLr.append( coords[2] ) elements = len(xLr) minZ = min(zLr) maxZ = max(zLr) ############ using the same input file i posted early, it give me 966 elements instead of 158734 elements gived by your "MASK" example the input file contains 158734 elements, this means the mask code : > mask |= mydata[:,0] < E > mask |= mydata[:,0] > W > mask |= mydata[:,1] < N > mask |= mydata[:,1] > S is not working as aspected have you hints on how to get working the "MASK" code ? as it is now it pick all the points in the "mydata" array. thanks! Massimo. Il giorno 11/set/2010, alle ore 16.19, Brett Olsen ha scritto: > On Sat, Sep 11, 2010 at 7:45 AM, Massimo Di Stefano > wrote: >> Hello All, >> >> i need to extract data from an array, that are inside a >> rectangle area defined as : >> >> N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 >> >> the data are in a csv (comma delimited text file, with 3 columns X,Y,Z) >> >> #X,Y,Z >> 3020081.5500,769999.3100,0.0300 >> 3020086.2000,769991.6500,0.4600 >> 3020099.6600,769996.2700,0.9000 >> ... >> ... >> >> i read it using " numpy.loadtxt " >> >> data : >> >> http://www.geofemengineering.it/data/csv.txt 5,3 mb (158735 rows) >> >> to extract data that are inside the boundy-box area (N, S, E, W) i'm using a loop >> inside a function like : >> >> import numpy as np >> >> def getMinMaxBB(data, N, S, E, W): >> mydata = data * 0.3048006096012 >> for i in range(len(mydata)): >> if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : >> if i == 0: >> newdata = np.array((mydata[i,0],mydata[i,1],mydata[i,2]), float) >> else : >> newdata = np.vstack((newdata,(mydata[i,0], mydata[i,1], mydata[i,2]))) >> results = {} >> results['Max_Z'] = newdata.max(0)[2] >> results['Min_Z'] = newdata.min(0)[2] >> results['Num_P'] = len(newdata) >> return results >> >> >> N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 >> data = '/Users/sasha/csv.txt' >> mydata = np.loadtxt(data, comments='#', delimiter=',') >> out = getMinMaxBB(mydata, N, S, E, W) >> >> print out > > Use boolean arrays to index the parts of your array that you want to look at: > > def newGetMinMax(data, N, S, E, W): > mydata = data * 0.3048006096012 > mask = np.zeros(mydata.shape[0], dtype=bool) > mask |= mydata[:,0] < E > mask |= mydata[:,0] > W > mask |= mydata[:,1] < N > mask |= mydata[:,1] > S > results = {} > results['Max_Z'] = mydata[mask,2].max() > results['Min_Z'] = mydata[mask,2].min() > results['Num_P'] = mask.sum() > return results > > This runs about 5000 times faster on my machine. > > Brett > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimodisasha at gmail.com Sat Sep 11 17:46:49 2010 From: massimodisasha at gmail.com (Massimo Di Stefano) Date: Sat, 11 Sep 2010 23:46:49 +0200 Subject: [SciPy-User] [Numpy-discussion] scan array to extract min-max values (with if condition) In-Reply-To: <0CD4FE79-4EA7-4181-A119-55C3F96BEA0C@gmail.com> References: <24AD26B0-C1F7-49B7-9EA5-81DCBF058197@gmail.com> <0CD4FE79-4EA7-4181-A119-55C3F96BEA0C@gmail.com> Message-ID: Thanks Pierre, i tried it and all works fine and fast. my apologize :-( i used a wrong "if" statment to represent my needs if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : ^^ totally wrong for my needs^^ this "if" instead : if W < mydata[i,0] < E and S < mydata[i,1] < N: should reflect your example : yselect = (data[:,1] <= N) & (data[:,1] >= S) xselect = (data[:,0] <= E) & (data[:,0] >= W) selected_data = data[xselect & yselect] a question, how to code a masked array, as in the Brett's code, to reflect the new (right) if statment ? i'm asking this to try to learn how to use masked array. thanks a lot for your support!!! Massimo. Il giorno 11/set/2010, alle ore 23.00, Pierre GM ha scritto: > > On Sep 11, 2010, at 9:53 PM, Massimo Di Stefano wrote: > >> have you hints on how to get working the "MASK" code ? >> as it is now it pick all the points in the "mydata" array. > > Brett's code for the mask matched the loop of your post. However, taking a second look at it, I don't see why it would work. Mind trying something like > > # Selection on the NS axis > yselect = (mydata[:,1] <= N) & (mydata[:,1] >= S) > # selection on the EW axis > xselect = (mydata[:,1] <= E) & (mydata[:,1] >= W) > # Global selection > selected_data = mydata[xselect & yselect] > > Now you can check in selected data that the coordinates are indeed in the rectangle that you want, and take the min/max of your data as needed. > let me know how it goes > > P. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From m.abdollahi at gmail.com Sun Sep 12 07:33:22 2010 From: m.abdollahi at gmail.com (Mohammad Abdollahi) Date: Sun, 12 Sep 2010 13:33:22 +0200 Subject: [SciPy-User] How to disable warning display ? Message-ID: Dear All, I keep getting a long list of stupid warnings when i'm running my code. I was just wondering if anybody knows how to disable warning printing at the output, as it takes a lot of time. i just dont want them to be printed. any idea ? Kind Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Sep 12 07:41:02 2010 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 12 Sep 2010 11:41:02 +0000 (UTC) Subject: [SciPy-User] How to disable warning display ? References: Message-ID: Sun, 12 Sep 2010 13:33:22 +0200, Mohammad Abdollahi wrote: > I keep getting a long list of stupid warnings when i'm running my code. > I was just wondering if anybody knows how to disable warning printing at > the output, as it takes a lot of time. i just dont want them to be > printed. any idea ? http://docs.scipy.org/doc/numpy/reference/generated/numpy.seterr.html and http://docs.python.org/library/warnings.html e.g. warnings.simplefilter("ignore", DeprecationWarning) From blattnem at gmail.com Mon Sep 13 04:16:33 2010 From: blattnem at gmail.com (Marcel Blattner) Date: Mon, 13 Sep 2010 10:16:33 +0200 Subject: [SciPy-User] how to get indices Message-ID: Hi there I have a small problem. I have for example a csc_matrix A. I want to find indices of values bigger then a treshold. In numpy I simply can do: a,b = A > threshold...is there something similar in Scipy? Thanks Marcel -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Sep 13 04:28:16 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 13 Sep 2010 10:28:16 +0200 Subject: [SciPy-User] how to get indices In-Reply-To: References: Message-ID: <20100913082816.GA10263@phare.normalesup.org> On Mon, Sep 13, 2010 at 10:16:33AM +0200, Marcel Blattner wrote: > I have a small problem. I have for example a csc_matrix A. I want to find > indices of values bigger then a treshold. > In numpy I simply can do: a,b = A > threshold...is there something similar > in Scipy? The difficulty here is that you are dealing with sparse matrices, for which access to data is not as simple. I believe the easiest option is to convert your matrix to coo, and operate directly on the row/col/data arrays: a = a.tocoo() mask = a.data > threshold col = a.col[mask] row = a.row[mask] HTH, Ga?l From blattnem at gmail.com Mon Sep 13 04:52:21 2010 From: blattnem at gmail.com (Marcel Blattner) Date: Mon, 13 Sep 2010 10:52:21 +0200 Subject: [SciPy-User] how to get indices In-Reply-To: <20100913082816.GA10263@phare.normalesup.org> References: <20100913082816.GA10263@phare.normalesup.org> Message-ID: ...this rocks... On Mon, Sep 13, 2010 at 10:28 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Mon, Sep 13, 2010 at 10:16:33AM +0200, Marcel Blattner wrote: > > I have a small problem. I have for example a csc_matrix A. I want to > find > > indices of values bigger then a treshold. > > In numpy I simply can do: a,b = A > threshold...is there something > similar > > in Scipy? > > The difficulty here is that you are dealing with sparse matrices, for > which access to data is not as simple. I believe the easiest option is to > convert your matrix to coo, and operate directly on the row/col/data > arrays: > > a = a.tocoo() > mask = a.data > threshold > col = a.col[mask] > row = a.row[mask] > > HTH, > > Ga?l > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From schut at sarvision.nl Mon Sep 13 04:57:09 2010 From: schut at sarvision.nl (Vincent Schut) Date: Mon, 13 Sep 2010 10:57:09 +0200 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: <4C8A3215.63BA.009B.1@twdb.state.tx.us> References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> Message-ID: <4C8DE765.1000309@sarvision.nl> On 09/10/2010 08:26 PM, Dharhas Pothina wrote: > Hi, > > I have an monotonically increasing array with duplicates in it. e.g. > > x = np.array([1.0,1.0,1.0,2.0,2.0,3.0,3.0,3.0,4.0,4.0,5.0,5.0,5.0,5.0,6.0]) > > I need a new array of the same size with linearly interpolated values i.e something like > > np.array([1.0, 1.33, 1.67, 2.0, 2.5, 3.0, 3.33, 3.67, 4.0, 4.5, 5.0, 5.25, 5.5, 5.75, 6.0]) > > Is there an efficient way to do this. Right now I'm looping through the array and maintaining position flags of when the value changes and then doing linear interpolation between the start and end flags before resetting the flags moving to the next section. This is pretty slow. > > I realize there will be a problem on how to deal with duplicates at the end of the array. > > thanks > > - dharhas Hmm, something like this should do the trick (not tested): xUnique, xUniqueIndices = scipy.unique(x, return_index=True) # get piecewise linear interpolation xInterpolated = numpy.interp(numpy.arange(len(x), xUniqueIndices, xUnique) note: this will not interpolate duplicates at the beginning/end. You can however adjust the behaviour for those, see the docs for numpy.interp. Vincent. From majoris.ursae at yahoo.co.uk Tue Sep 14 00:45:13 2010 From: majoris.ursae at yahoo.co.uk (majoris.ursae at yahoo.co.uk) Date: Tue, 14 Sep 2010 13:45:13 +0900 Subject: [SciPy-User] Stereoscopic analysis Message-ID: Hello everybody, I would be grateful if someone give me some hints on how to extract average height from two overlapping aerial photographs preferably in Python environment. Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From eavventi at yahoo.it Tue Sep 14 09:55:34 2010 From: eavventi at yahoo.it (enrico avventi) Date: Tue, 14 Sep 2010 13:55:34 +0000 (GMT) Subject: [SciPy-User] optimization routines can not handle infinity values Message-ID: <16213.823.qm@web26703.mail.ukl.yahoo.com> hello all, i am trying out some of the optimization routines for a problem of mine that is on the form: min f(x) s.t M(x) is positive semidefinite where f is strictly convex in the feasible region with compact sublevel sets, M is linear and takes value in some subspace of hermitian matrices. the problem is convex but the costraint can not be handled directly by any of the optimization routines in scipy. So i choose to change it to an uncostrained problem with objective function: f1(x) = f(x) for M(x) pos semi def f1(x) = Inf otherwise the problem is that it seems the routines can not handle the infinity values correctly. Some of the routines (fmin_cg comes to mind) wants to check the gradient at points where the objective function is infinite. Clearly in such cases the gradient is not defined - i.e the calculations fail - and the algorithm terminates. Others (like fmin_bfgs) strangely converge to a point where the objective is infinite despite the fact that the initial point was not. Do you have any suggestion to fix this problem? regards, Enrico ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Tue Sep 14 11:51:24 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 14 Sep 2010 17:51:24 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: <16213.823.qm@web26703.mail.ukl.yahoo.com> References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: Hi, There are two categories of contraints optimizations: - you can evaluate the function outside the constraints - you cannot evaluate the function outsde the constraints. If the first one can be handled by more general algorithms providing some tricks, you cannot use them for the second one. Your problem is clearly a second category problem, so you must use appropriate algorithms (which may not be available in scipy directly, you may want to check OpenOpt). It's not a problem of routines, it's a problem of appropriate algorithms. Matthieu 2010/9/14 enrico avventi > hello all, > > i am trying out some of the optimization routines for a problem of mine > that is on the form: > > min f(x) > > s.t M(x) is positive semidefinite > > where f is strictly convex in the feasible region with compact sublevel > sets, M is linear and takes value in some subspace of hermitian matrices. > > the problem is convex but the costraint can not be handled directly by any > of the optimization routines in scipy. So i choose to change it to an > uncostrained problem with objective function: > > f1(x) = f(x) for M(x) pos semi def > f1(x) = Inf otherwise > > the problem is that it seems the routines can not handle the infinity > values correctly. > > Some of the routines (fmin_cg comes to mind) wants to check the gradient at > points where the objective function is infinite. Clearly in such cases the > gradient is not defined - i.e the calculations fail - and the algorithm > terminates. > > Others (like fmin_bfgs) strangely converge to a point where the objective > is infinite despite the fact that the initial point was not. > > Do you have any suggestion to fix this problem? > > regards, > > Enrico > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dharhas.Pothina at twdb.state.tx.us Tue Sep 14 15:03:57 2010 From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina) Date: Tue, 14 Sep 2010 14:03:57 -0500 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: <4C8DE765.1000309@sarvision.nl> References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> <4C8DE765.1000309@sarvision.nl> Message-ID: <4C8F80CC.63BA.009B.1@twdb.state.tx.us> Vincent, > Hmm, something like this should do the trick (not tested): > xUnique, xUniqueIndices = scipy.unique(x, return_index=True) Neither numpy.unique or scipy.unique seem to have a return_index keyword arg. - dharhas From Dharhas.Pothina at twdb.state.tx.us Tue Sep 14 15:15:58 2010 From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina) Date: Tue, 14 Sep 2010 14:15:58 -0500 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: <4C8F80CC.63BA.009B.1@twdb.state.tx.us> References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> <4C8DE765.1000309@sarvision.nl> <4C8F80CC.63BA.009B.1@twdb.state.tx.us> Message-ID: <4C8F839E.63BA.009B.1@twdb.state.tx.us> Looks like I have numpy 1.3 and this is available in 1.4 onwards. Will try upgrading and try again. thanks - dharhas >>> "Dharhas Pothina" 9/14/2010 2:03 PM >>> Vincent, > Hmm, something like this should do the trick (not tested): > xUnique, xUniqueIndices = scipy.unique(x, return_index=True) Neither numpy.unique or scipy.unique seem to have a return_index keyword arg. - dharhas _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From aarchiba at physics.mcgill.ca Tue Sep 14 15:21:03 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Tue, 14 Sep 2010 15:21:03 -0400 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: <4C8A3215.63BA.009B.1@twdb.state.tx.us> References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> Message-ID: On 10 September 2010 14:26, Dharhas Pothina wrote: > Hi, > > I have an monotonically increasing array with duplicates in it. e.g. > > x = np.array([1.0,1.0,1.0,2.0,2.0,3.0,3.0,3.0,4.0,4.0,5.0,5.0,5.0,5.0,6.0]) > > I need a new array of the same size with linearly interpolated values i.e something like > > np.array([1.0, 1.33, 1.67, 2.0, 2.5, 3.0, 3.33, 3.67, 4.0, 4.5, 5.0, 5.25, 5.5, 5.75, 6.0]) > > Is there an efficient way to do this. Right now I'm looping through the array and maintaining position flags of when the value changes and then doing linear interpolation between the start and end flags before resetting the flags moving to the next section. This is pretty slow. > > I realize there will be a problem on how to deal with duplicates at the end of the array. How's this? In [22]: A = np.array([1,1,1,4,5,5,7,7,7,7,11,11])/100. In [23]: A Out[23]: array([ 0.01, 0.01, 0.01, 0.04, 0.05, 0.05, 0.07, 0.07, 0.07, 0.07, 0.11, 0.11]) In [24]: l=np.searchsorted(A,A,"left") In [25]: r=np.searchsorted(A,A,"right") In [26]: i = np.arange(len(A)) In [27]: AA = np.concatenate((A,[A[-1]])) In [28]: (AA[r]*(i-l) + AA[l]*(r-i))/(r-l) Out[28]: array([ 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 , 0.11, 0.11]) (Start with A, for each element find the leftmost and rightmost end of the run they're in, as well as the index of the element; then the output value is a weighted sum of the value at the left end and the value at the right end weighted by the distance from those same endpoints. The concatenate business just deals with repetitions of the last value.) Anne > thanks > > - dharhas > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From Dharhas.Pothina at twdb.state.tx.us Tue Sep 14 15:37:04 2010 From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina) Date: Tue, 14 Sep 2010 14:37:04 -0500 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> Message-ID: <4C8F8890.63BA.009B.1@twdb.state.tx.us> Thanks to all. Both Anne's and Vincent's solutions work. I'm going to try them on larger real datasets before deciding which to use. I've never used the searchsorted function before. Nice to know that it exists. - dharhas >>> Anne Archibald 9/14/2010 2:21 PM >>> On 10 September 2010 14:26, Dharhas Pothina wrote: > Hi, > > I have an monotonically increasing array with duplicates in it. e.g. > > x = np.array([1.0,1.0,1.0,2.0,2.0,3.0,3.0,3.0,4.0,4.0,5.0,5.0,5.0,5.0,6.0]) > > I need a new array of the same size with linearly interpolated values i.e something like > > np.array([1.0, 1.33, 1.67, 2.0, 2.5, 3.0, 3.33, 3.67, 4.0, 4.5, 5.0, 5.25, 5.5, 5.75, 6.0]) > > Is there an efficient way to do this. Right now I'm looping through the array and maintaining position flags of when the value changes and then doing linear interpolation between the start and end flags before resetting the flags moving to the next section. This is pretty slow. > > I realize there will be a problem on how to deal with duplicates at the end of the array. How's this? In [22]: A = np.array([1,1,1,4,5,5,7,7,7,7,11,11])/100. In [23]: A Out[23]: array([ 0.01, 0.01, 0.01, 0.04, 0.05, 0.05, 0.07, 0.07, 0.07, 0.07, 0.11, 0.11]) In [24]: l=np.searchsorted(A,A,"left") In [25]: r=np.searchsorted(A,A,"right") In [26]: i = np.arange(len(A)) In [27]: AA = np.concatenate((A,[A[-1]])) In [28]: (AA[r]*(i-l) + AA[l]*(r-i))/(r-l) Out[28]: array([ 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 , 0.11, 0.11]) (Start with A, for each element find the leftmost and rightmost end of the run they're in, as well as the index of the element; then the output value is a weighted sum of the value at the left end and the value at the right end weighted by the distance from those same endpoints. The concatenate business just deals with repetitions of the last value.) Anne > thanks > > - dharhas > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From jdh2358 at gmail.com Tue Sep 14 17:39:21 2010 From: jdh2358 at gmail.com (John Hunter) Date: Tue, 14 Sep 2010 16:39:21 -0500 Subject: [SciPy-User] ANN: job opening at TradeLink Securities Message-ID: We are looking to hire a programmer/researcher to help research and develop trading ideas, and to develop infrastructure to put these trading strategies into production. We are looking for someone who is bright and curious with a quantitative background and a strong interest in writing good code and building systems that work. Familiarity with basic probability and statistics is a definite plus. We do not require a financial background, but are looking for someone with an enthusiasm to dive into this industry and learn a lot. We do most of our data modeling and production software in python, utilizing mostly open source tools. We have a lot of ideas to test and hopefully put into production, and you'll be working with a fast paced and friendly small team of traders and programmers. Applying: Please submit a resume and cover letter to qsjobs at trdlnk.com with the subject header "TLS Job". In your cover letter, please address how your background, experience and skills will fit into the position described above. We are looking for a full-time, on-site candidate only. About Us: TradeLink Securities, headquartered in Chicago, IL, is a NASDAQ market maker and registered broker dealer with FINRA. The firm specializes in both quantitative and fundamental investing in the US and abroad. TradeLink Securities is a division of TradeLink Holdings, a diversified alternative investment and proprietary trading firm organized in 1979. For more information visit our website at http://tradelinkll.com. From jdh2358 at gmail.com Tue Sep 14 18:05:32 2010 From: jdh2358 at gmail.com (John Hunter) Date: Tue, 14 Sep 2010 17:05:32 -0500 Subject: [SciPy-User] ANN: job opening at TradeLink Securities In-Reply-To: References: Message-ID: Someone just pointed out the URL for the website contains a typo.... Should read http://tradelinkllc.com/. Sorry for the noise.. On Tue, Sep 14, 2010 at 4:39 PM, John Hunter wrote: > We are looking to hire a programmer/researcher to help research and > develop trading ideas, and to develop infrastructure to put these > trading strategies into production. ?We are looking for someone who is > bright and curious with a quantitative background and a strong > interest in writing good code and building systems that work. > Familiarity with basic probability and statistics is a definite plus. > We do not require a financial background, but are looking for someone > with an enthusiasm to dive into this industry and learn a lot. ?We do > most of our data modeling and production software in python, utilizing > mostly open source tools. ?We have a lot of ideas to test and > hopefully put into production, and you'll be working with a fast paced > and friendly small team of traders and programmers. > > Applying: > > ?Please submit a resume and cover letter to qsjobs at trdlnk.com with > ?the subject header "TLS Job". ?In your cover letter, please address > ?how your background, experience and skills will fit into the > ?position described above. ?We are looking for a full-time, on-site > ?candidate only. > > About Us: > > ?TradeLink Securities, headquartered in Chicago, IL, is a NASDAQ > ?market maker and registered broker dealer with FINRA. ?The firm > ?specializes in both quantitative and fundamental investing in the US > ?and abroad. ?TradeLink Securities is a division of TradeLink > ?Holdings, a diversified alternative investment and proprietary > ?trading firm organized in 1979. ?For more information visit our > ?website at http://tradelinkll.com. > From timmichelsen at gmx-topmail.de Tue Sep 14 21:14:07 2010 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Wed, 15 Sep 2010 03:14:07 +0200 Subject: [SciPy-User] Stereoscopic analysis In-Reply-To: References: Message-ID: > I would be grateful if someone give me some hints on how to extract > average height from two overlapping aerial photographs preferably in > Python environment. there are definately frameworks for that. Check with the OSGEO wiki, GRASS wiki, radTools, etc. From balazovic.peter at gmail.com Wed Sep 15 04:31:42 2010 From: balazovic.peter at gmail.com (pepe_emspin) Date: Wed, 15 Sep 2010 01:31:42 -0700 (PDT) Subject: [SciPy-User] cannot find -llapack Message-ID: http://github.com/avventi/Slycot c:/mingw/bin/../lib/gcc/mingw32/4.5.0/../../../../mingw32/bin/ld.exe: cannot find -llapack I am trying to find out how to fix this... From balazovic.peter at gmail.com Wed Sep 15 04:37:01 2010 From: balazovic.peter at gmail.com (pepe_emspin) Date: Wed, 15 Sep 2010 01:37:01 -0700 (PDT) Subject: [SciPy-User] cannot find -llapack Message-ID: Hello I am sorry for previous incomplete email! I am using pythonxy distribution (windows) including numpy/scipy. I am installing SLYCOT wrapper http://github.com/avventi/Slycot Installation process is halted >python setup.py install build -compiler=mingw32 I am getting this message: c:/mingw/bin/../lib/gcc/mingw32/4.5.0/../../../../mingw32/bin/ld.exe: cannot find -llapack Any idea how to fix it? Thanks! Regards, pepe From balazovic.peter at gmail.com Wed Sep 15 04:37:01 2010 From: balazovic.peter at gmail.com (pepe_emspin) Date: Wed, 15 Sep 2010 01:37:01 -0700 (PDT) Subject: [SciPy-User] cannot find -llapack Message-ID: Hello I am sorry for previous incomplete email! I am using pythonxy distribution (windows) including numpy/scipy. I am installing SLYCOT wrapper http://github.com/avventi/Slycot Installation process is halted >python setup.py install build -compiler=mingw32 I am getting this message: c:/mingw/bin/../lib/gcc/mingw32/4.5.0/../../../../mingw32/bin/ld.exe: cannot find -llapack Any idea how to fix it? Thanks! Regards, pepe From tmp50 at ukr.net Wed Sep 15 05:04:57 2010 From: tmp50 at ukr.net (Dmitrey) Date: Wed, 15 Sep 2010 12:04:57 +0300 Subject: [SciPy-User] ANN: OpenOpt 0.31, FuncDesigner 0.21, DerApproximator 0.21 Message-ID: Hi all, I'm glad to inform you about new releases: OpenOpt 0.31, FuncDesigner 0.21, DerApproximator 0.21 For details see http://forum.openopt.org/viewtopic.php?id=299 or visit our homepage http://openopt.org Regards, Dmitrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From eavventi at yahoo.it Wed Sep 15 05:15:20 2010 From: eavventi at yahoo.it (Enrico Avventi) Date: Wed, 15 Sep 2010 11:15:20 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: Hi Matthieu, thanx for the reply. as far as i know it shouldn't be a problem at all in theory. in fact convergence theorems for newton methods and, in general, descent method do not require the function to be defined everywhere. you need just a function defined in an open convex set that has compact sublevel sets - e.g f(x) = x - log(x). this is exactly my situation. it is just a matter of slightly changing the line search method to reject steps that would lead to an infinite value. i will look at how it is implemented in scipy and see if it can be fixed easily. /Enrico On Tue, Sep 14, 2010 at 5:51 PM, Matthieu Brucher < matthieu.brucher at gmail.com> wrote: > Hi, > > There are two categories of contraints optimizations: > - you can evaluate the function outside the constraints > - you cannot evaluate the function outsde the constraints. > > If the first one can be handled by more general algorithms providing some > tricks, you cannot use them for the second one. Your problem is clearly a > second category problem, so you must use appropriate algorithms (which may > not be available in scipy directly, you may want to check OpenOpt). > > It's not a problem of routines, it's a problem of appropriate algorithms. > > Matthieu > > 2010/9/14 enrico avventi > >> hello all, >> >> i am trying out some of the optimization routines for a problem of mine >> that is on the form: >> >> min f(x) >> >> s.t M(x) is positive semidefinite >> >> where f is strictly convex in the feasible region with compact sublevel >> sets, M is linear and takes value in some subspace of hermitian matrices. >> >> the problem is convex but the costraint can not be handled directly by any >> of the optimization routines in scipy. So i choose to change it to an >> uncostrained problem with objective function: >> >> f1(x) = f(x) for M(x) pos semi def >> f1(x) = Inf otherwise >> >> the problem is that it seems the routines can not handle the infinity >> values correctly. >> >> Some of the routines (fmin_cg comes to mind) wants to check the gradient >> at points where the objective function is infinite. Clearly in such cases >> the gradient is not defined - i.e the calculations fail - and the algorithm >> terminates. >> >> Others (like fmin_bfgs) strangely converge to a point where the objective >> is infinite despite the fact that the initial point was not. >> >> Do you have any suggestion to fix this problem? >> >> regards, >> >> Enrico >> >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Sep 15 05:19:52 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 15 Sep 2010 11:19:52 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: Hi, The line search used in scipy is based on Wolfe-Powell rules and the search for appropriate values is done by interpolation. This is why it cannot be used for your kind of problems. All Wolfe-Powel line searches I've found in the litterature are based on such interpolations. It could be changed to fit your purpose, but it would be slower for 99.99% of the other optimizations. So you may add some additional constraints here. Matthieu 2010/9/15 Enrico Avventi > Hi Matthieu, > > thanx for the reply. as far as i know it shouldn't be a problem at all in > theory. in fact convergence theorems for newton methods and, in general, > descent method do not require the function to be defined everywhere. you > need just a function defined in an open convex set that has compact sublevel > sets - e.g f(x) = x - log(x). this is exactly my situation. > it is just a matter of slightly changing the line search method to reject > steps that would lead to an infinite value. > i will look at how it is implemented in scipy and see if it can be fixed > easily. > > /Enrico > > > On Tue, Sep 14, 2010 at 5:51 PM, Matthieu Brucher < > matthieu.brucher at gmail.com> wrote: > >> Hi, >> >> There are two categories of contraints optimizations: >> - you can evaluate the function outside the constraints >> - you cannot evaluate the function outsde the constraints. >> >> If the first one can be handled by more general algorithms providing some >> tricks, you cannot use them for the second one. Your problem is clearly a >> second category problem, so you must use appropriate algorithms (which may >> not be available in scipy directly, you may want to check OpenOpt). >> >> It's not a problem of routines, it's a problem of appropriate algorithms. >> >> Matthieu >> >> 2010/9/14 enrico avventi >> >>> hello all, >>> >>> i am trying out some of the optimization routines for a problem of mine >>> that is on the form: >>> >>> min f(x) >>> >>> s.t M(x) is positive semidefinite >>> >>> where f is strictly convex in the feasible region with compact sublevel >>> sets, M is linear and takes value in some subspace of hermitian matrices. >>> >>> the problem is convex but the costraint can not be handled directly by >>> any of the optimization routines in scipy. So i choose to change it to an >>> uncostrained problem with objective function: >>> >>> f1(x) = f(x) for M(x) pos semi def >>> f1(x) = Inf otherwise >>> >>> the problem is that it seems the routines can not handle the infinity >>> values correctly. >>> >>> Some of the routines (fmin_cg comes to mind) wants to check the gradient >>> at points where the objective function is infinite. Clearly in such cases >>> the gradient is not defined - i.e the calculations fail - and the algorithm >>> terminates. >>> >>> Others (like fmin_bfgs) strangely converge to a point where the objective >>> is infinite despite the fact that the initial point was not. >>> >>> Do you have any suggestion to fix this problem? >>> >>> regards, >>> >>> Enrico >>> >>> >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> >> -- >> Information System Engineer, Ph.D. >> Blog: http://matt.eifelle.com >> LinkedIn: http://www.linkedin.com/in/matthieubrucher >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From schut at sarvision.nl Wed Sep 15 05:20:20 2010 From: schut at sarvision.nl (Vincent Schut) Date: Wed, 15 Sep 2010 11:20:20 +0200 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: <4C8F80CC.63BA.009B.1@twdb.state.tx.us> References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> <4C8DE765.1000309@sarvision.nl> <4C8F80CC.63BA.009B.1@twdb.state.tx.us> Message-ID: Ah, mine has, but I'm using the development version (from the svn repository) of both. See http://docs.scipy.org/doc/numpy/reference/generated/numpy.unique.html. Hmm, according to the docs version 1.5 of numpy should have it already (http://docs.scipy.org/doc/numpy-1.5.x/reference/generated/numpy.unique.html), so you might just try to upgrade your numpy. Vincent. On 09/14/2010 09:03 PM, Dharhas Pothina wrote: > Vincent, > >> Hmm, something like this should do the trick (not tested): >> xUnique, xUniqueIndices = scipy.unique(x, return_index=True) > > Neither numpy.unique or scipy.unique seem to have a return_index keyword arg. > > - dharhas From pav at iki.fi Wed Sep 15 06:13:15 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Sep 2010 10:13:15 +0000 (UTC) Subject: [SciPy-User] optimization routines can not handle infinity values References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: Wed, 15 Sep 2010 11:15:20 +0200, Enrico Avventi wrote: > thanx for the reply. as far as i know it shouldn't be a problem at all > in theory. in fact convergence theorems for newton methods and, in > general, descent method do not require the function to be defined > everywhere. you need just a function defined in an open convex set that > has compact sublevel sets - e.g f(x) = x - log(x). this is exactly my > situation. it is just a matter of slightly changing the line search > method to reject steps that would lead to an infinite value. i will look > at how it is implemented in scipy and see if it can be fixed easily. The line search routines can be easily changed -- they are separate routines: http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/linesearch.py If you implement an alternative inf-aware line search, I believe we can at least modify the optimization methods to dynamically swap to an inf- aware approach (and add an option to turn it on manually), if the interpolation-based methods detect infinities. -- Pauli Virtanen From sebastian.walter at gmail.com Wed Sep 15 06:18:05 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Wed, 15 Sep 2010 12:18:05 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: I can't quite follow the reasoning that another line search would solve the problem. If the current iterate is at the boundary and the new search direction points into the infeasible set, then a line search is unlikely to help. Example: min_x dot([2,1],x) s.t. x >= 0 the minimizer is x_* = [0,0]. If the current iterate x_k is at [3,0], then the new search direction woud be s_k = -[2,1] which points directly into the infeasible set which would yield infty. I'm not sure what the best approach for your problem is. Since SDP is still an active topic of research one could conjecture that possibly it's not as easy as you think. If you find a good solution I'd be happy to hear about it. regards, Sebastian On Wed, Sep 15, 2010 at 11:19 AM, Matthieu Brucher < matthieu.brucher at gmail.com> wrote: > Hi, > > The line search used in scipy is based on Wolfe-Powell rules and the search > for appropriate values is done by interpolation. This is why it cannot be > used for your kind of problems. All Wolfe-Powel line searches I've found in > the litterature are based on such interpolations. > It could be changed to fit your purpose, but it would be slower for 99.99% > of the other optimizations. So you may add some additional constraints here. > > Matthieu > > 2010/9/15 Enrico Avventi > > Hi Matthieu, >> >> thanx for the reply. as far as i know it shouldn't be a problem at all in >> theory. in fact convergence theorems for newton methods and, in general, >> descent method do not require the function to be defined everywhere. you >> need just a function defined in an open convex set that has compact sublevel >> sets - e.g f(x) = x - log(x). this is exactly my situation. >> it is just a matter of slightly changing the line search method to reject >> steps that would lead to an infinite value. >> i will look at how it is implemented in scipy and see if it can be fixed >> easily. >> >> /Enrico >> >> >> On Tue, Sep 14, 2010 at 5:51 PM, Matthieu Brucher < >> matthieu.brucher at gmail.com> wrote: >> >>> Hi, >>> >>> There are two categories of contraints optimizations: >>> - you can evaluate the function outside the constraints >>> - you cannot evaluate the function outsde the constraints. >>> >>> If the first one can be handled by more general algorithms providing some >>> tricks, you cannot use them for the second one. Your problem is clearly a >>> second category problem, so you must use appropriate algorithms (which may >>> not be available in scipy directly, you may want to check OpenOpt). >>> >>> It's not a problem of routines, it's a problem of appropriate algorithms. >>> >>> Matthieu >>> >>> 2010/9/14 enrico avventi >>> >>>> hello all, >>>> >>>> i am trying out some of the optimization routines for a problem of mine >>>> that is on the form: >>>> >>>> min f(x) >>>> >>>> s.t M(x) is positive semidefinite >>>> >>>> where f is strictly convex in the feasible region with compact sublevel >>>> sets, M is linear and takes value in some subspace of hermitian matrices. >>>> >>>> the problem is convex but the costraint can not be handled directly by >>>> any of the optimization routines in scipy. So i choose to change it to an >>>> uncostrained problem with objective function: >>>> >>>> f1(x) = f(x) for M(x) pos semi def >>>> f1(x) = Inf otherwise >>>> >>>> the problem is that it seems the routines can not handle the infinity >>>> values correctly. >>>> >>>> Some of the routines (fmin_cg comes to mind) wants to check the gradient >>>> at points where the objective function is infinite. Clearly in such cases >>>> the gradient is not defined - i.e the calculations fail - and the algorithm >>>> terminates. >>>> >>>> Others (like fmin_bfgs) strangely converge to a point where the >>>> objective is infinite despite the fact that the initial point was not. >>>> >>>> Do you have any suggestion to fix this problem? >>>> >>>> regards, >>>> >>>> Enrico >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> >>> -- >>> Information System Engineer, Ph.D. >>> Blog: http://matt.eifelle.com >>> LinkedIn: http://www.linkedin.com/in/matthieubrucher >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eavventi at yahoo.it Wed Sep 15 06:52:46 2010 From: eavventi at yahoo.it (Enrico Avventi) Date: Wed, 15 Sep 2010 12:52:46 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: my function tends to infinity as you approach the boundary, as it is the case for all convex functions that can be defined only inside an open convex set. On Wed, Sep 15, 2010 at 12:18 PM, Sebastian Walter < sebastian.walter at gmail.com> wrote: > I can't quite follow the reasoning that another line search would solve the > problem. > If the current iterate is at the boundary and the new search direction > points into the infeasible set, then a line search is unlikely to help. > > Example: > min_x dot([2,1],x) > s.t. x >= 0 > > the minimizer is x_* = [0,0]. If the current iterate x_k is at [3,0], then > the new search direction woud be > s_k = -[2,1] which points directly into the infeasible set which would > yield infty. > > I'm not sure what the best approach for your problem is. > Since SDP is still an active topic of research one could conjecture that > possibly it's not as easy as you think. > If you find a good solution I'd be happy to hear about it. > > regards, > Sebastian > > > > > On Wed, Sep 15, 2010 at 11:19 AM, Matthieu Brucher < > matthieu.brucher at gmail.com> wrote: > >> Hi, >> >> The line search used in scipy is based on Wolfe-Powell rules and the >> search for appropriate values is done by interpolation. This is why it >> cannot be used for your kind of problems. All Wolfe-Powel line searches I've >> found in the litterature are based on such interpolations. >> It could be changed to fit your purpose, but it would be slower for 99.99% >> of the other optimizations. So you may add some additional constraints here. >> >> Matthieu >> >> 2010/9/15 Enrico Avventi >> >> Hi Matthieu, >>> >>> thanx for the reply. as far as i know it shouldn't be a problem at all in >>> theory. in fact convergence theorems for newton methods and, in general, >>> descent method do not require the function to be defined everywhere. you >>> need just a function defined in an open convex set that has compact sublevel >>> sets - e.g f(x) = x - log(x). this is exactly my situation. >>> it is just a matter of slightly changing the line search method to reject >>> steps that would lead to an infinite value. >>> i will look at how it is implemented in scipy and see if it can be fixed >>> easily. >>> >>> /Enrico >>> >>> >>> On Tue, Sep 14, 2010 at 5:51 PM, Matthieu Brucher < >>> matthieu.brucher at gmail.com> wrote: >>> >>>> Hi, >>>> >>>> There are two categories of contraints optimizations: >>>> - you can evaluate the function outside the constraints >>>> - you cannot evaluate the function outsde the constraints. >>>> >>>> If the first one can be handled by more general algorithms providing >>>> some tricks, you cannot use them for the second one. Your problem is clearly >>>> a second category problem, so you must use appropriate algorithms (which may >>>> not be available in scipy directly, you may want to check OpenOpt). >>>> >>>> It's not a problem of routines, it's a problem of appropriate >>>> algorithms. >>>> >>>> Matthieu >>>> >>>> 2010/9/14 enrico avventi >>>> >>>>> hello all, >>>>> >>>>> i am trying out some of the optimization routines for a problem of mine >>>>> that is on the form: >>>>> >>>>> min f(x) >>>>> >>>>> s.t M(x) is positive semidefinite >>>>> >>>>> where f is strictly convex in the feasible region with compact sublevel >>>>> sets, M is linear and takes value in some subspace of hermitian matrices. >>>>> >>>>> the problem is convex but the costraint can not be handled directly by >>>>> any of the optimization routines in scipy. So i choose to change it to an >>>>> uncostrained problem with objective function: >>>>> >>>>> f1(x) = f(x) for M(x) pos semi def >>>>> f1(x) = Inf otherwise >>>>> >>>>> the problem is that it seems the routines can not handle the infinity >>>>> values correctly. >>>>> >>>>> Some of the routines (fmin_cg comes to mind) wants to check the >>>>> gradient at points where the objective function is infinite. Clearly in such >>>>> cases the gradient is not defined - i.e the calculations fail - and the >>>>> algorithm terminates. >>>>> >>>>> Others (like fmin_bfgs) strangely converge to a point where the >>>>> objective is infinite despite the fact that the initial point was not. >>>>> >>>>> Do you have any suggestion to fix this problem? >>>>> >>>>> regards, >>>>> >>>>> Enrico >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> SciPy-User mailing list >>>>> SciPy-User at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>>> >>>>> >>>> >>>> >>>> -- >>>> Information System Engineer, Ph.D. >>>> Blog: http://matt.eifelle.com >>>> LinkedIn: http://www.linkedin.com/in/matthieubrucher >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> >> -- >> Information System Engineer, Ph.D. >> Blog: http://matt.eifelle.com >> LinkedIn: http://www.linkedin.com/in/matthieubrucher >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Sep 15 09:51:01 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 15 Sep 2010 21:51:01 +0800 Subject: [SciPy-User] =?windows-1252?q?Building_problem_-_scipy/sparse/spa?= =?windows-1252?q?rsetools/csr=5Fwrap=2Ecxx=3A2917=3A_error=3A_expe?= =?windows-1252?q?cted_=60=29=27_before_=91PRIdPTR=92?= In-Reply-To: References: Message-ID: On Thu, Sep 9, 2010 at 1:15 AM, Nicolau Werneck wrote: > Hello, I am trying to build numpy and scipy for my newly built > Python 2.7, and I am getting this error message that seems to be a > classic problem happening again: > > compile options: > '-I/usr/local/lib/python2.7/site-packages/numpy/core/include > -I/usr/local/include/python2.7 -c' g++: > scipy/sparse/sparsetools/csr_wrap.cxx > scipy/sparse/sparsetools/csr_wrap.cxx: In function ?int > require_size(PyArrayObject*, npy_intp*, int)?: > scipy/sparse/sparsetools/csr_wrap.cxx:2910: error: expected `)' before > ?PRIdPTR? > scipy/sparse/sparsetools/csr_wrap.cxx:2910: warning: spurious trailing > ?%? in format > scipy/sparse/sparsetools/csr_wrap.cxx:2910: warning: too many > arguments for format > scipy/sparse/sparsetools/csr_wrap.cxx:2917: error: expected `)' before > ?PRIdPTR? > scipy/sparse/sparsetools/csr_wrap.cxx:2917: warning: spurious trailing > ?%? in format > scipy/sparse/sparsetools/csr_wrap.cxx:2917: warning: too many > arguments for format > scipy/sparse/sparsetools/csr_wrap.cxx: In function ?int > require_size(PyArrayObject*, npy_intp*, int)?: > scipy/sparse/sparsetools/csr_wrap.cxx:2910: error: expected `)' before > ?PRIdPTR? > scipy/sparse/sparsetools/csr_wrap.cxx:2910: warning: spurious trailing > ?%? in format > scipy/sparse/sparsetools/csr_wrap.cxx:2910: warning: too many > arguments for format > scipy/sparse/sparsetools/csr_wrap.cxx:2917: error: expected `)' before > ?PRIdPTR? > scipy/sparse/sparsetools/csr_wrap.cxx:2917: warning: spurious trailing > ?%? in format > scipy/sparse/sparsetools/csr_wrap.cxx:2917: warning: too many > arguments for format > error: Command "g++ -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g > -fwrapv -O3 -Wall -fPIC > -I/usr/local/lib/python2.7/site-packages/numpy/core/include > -I/usr/local/include/python2.7 -c > scipy/sparse/sparsetools/csr_wrap.cxx -o > build/temp.linux-x86_64-2.7/scipy/sparse/sparsetools/csr_wrap.o" > failed with exit status 1 > > > Numpy seems to have compiled fine. Any ideas of how to fix that? > This should be fixed in trunk (r6645), the fix is not in 0.8.0. If that doesn't work for you, it would be helpful to know your OS, compiler versions and exact numpy/scipy you're trying to build. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian.walter at gmail.com Wed Sep 15 11:28:56 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Wed, 15 Sep 2010 17:28:56 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: you said you formulated your problem as follows: f1(x) = f(x) for M(x) pos semi def f1(x) = Inf otherwise I don't see any indication that anything goes to infinity when the sequence of iterates approaches a boundary. It seems to me that what you are describing is the behavior of a barrier function which is used in interior point methods, i.e. of the form min_x f(x) - a log( g(M(x)) where g(M(x)) is a sufficiently smooth function in x and g(M(x)) = 0 if M(x) is not positive definite and a>0. Is this what you are doing? Sebastian On Wed, Sep 15, 2010 at 12:52 PM, Enrico Avventi wrote: > my function tends to infinity as you approach the boundary, as it is the > case for all convex functions that can be defined only inside an open convex > set. > > > On Wed, Sep 15, 2010 at 12:18 PM, Sebastian Walter < > sebastian.walter at gmail.com> wrote: > >> I can't quite follow the reasoning that another line search would solve >> the problem. >> If the current iterate is at the boundary and the new search direction >> points into the infeasible set, then a line search is unlikely to help. >> >> Example: >> min_x dot([2,1],x) >> s.t. x >= 0 >> >> the minimizer is x_* = [0,0]. If the current iterate x_k is at [3,0], then >> the new search direction woud be >> s_k = -[2,1] which points directly into the infeasible set which would >> yield infty. >> >> I'm not sure what the best approach for your problem is. >> Since SDP is still an active topic of research one could conjecture that >> possibly it's not as easy as you think. >> If you find a good solution I'd be happy to hear about it. >> >> regards, >> Sebastian >> >> >> >> >> On Wed, Sep 15, 2010 at 11:19 AM, Matthieu Brucher < >> matthieu.brucher at gmail.com> wrote: >> >>> Hi, >>> >>> The line search used in scipy is based on Wolfe-Powell rules and the >>> search for appropriate values is done by interpolation. This is why it >>> cannot be used for your kind of problems. All Wolfe-Powel line searches I've >>> found in the litterature are based on such interpolations. >>> It could be changed to fit your purpose, but it would be slower for >>> 99.99% of the other optimizations. So you may add some additional >>> constraints here. >>> >>> Matthieu >>> >>> 2010/9/15 Enrico Avventi >>> >>> Hi Matthieu, >>>> >>>> thanx for the reply. as far as i know it shouldn't be a problem at all >>>> in theory. in fact convergence theorems for newton methods and, in general, >>>> descent method do not require the function to be defined everywhere. you >>>> need just a function defined in an open convex set that has compact sublevel >>>> sets - e.g f(x) = x - log(x). this is exactly my situation. >>>> it is just a matter of slightly changing the line search method to >>>> reject steps that would lead to an infinite value. >>>> i will look at how it is implemented in scipy and see if it can be fixed >>>> easily. >>>> >>>> /Enrico >>>> >>>> >>>> On Tue, Sep 14, 2010 at 5:51 PM, Matthieu Brucher < >>>> matthieu.brucher at gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> There are two categories of contraints optimizations: >>>>> - you can evaluate the function outside the constraints >>>>> - you cannot evaluate the function outsde the constraints. >>>>> >>>>> If the first one can be handled by more general algorithms providing >>>>> some tricks, you cannot use them for the second one. Your problem is clearly >>>>> a second category problem, so you must use appropriate algorithms (which may >>>>> not be available in scipy directly, you may want to check OpenOpt). >>>>> >>>>> It's not a problem of routines, it's a problem of appropriate >>>>> algorithms. >>>>> >>>>> Matthieu >>>>> >>>>> 2010/9/14 enrico avventi >>>>> >>>>>> hello all, >>>>>> >>>>>> i am trying out some of the optimization routines for a problem of >>>>>> mine that is on the form: >>>>>> >>>>>> min f(x) >>>>>> >>>>>> s.t M(x) is positive semidefinite >>>>>> >>>>>> where f is strictly convex in the feasible region with compact >>>>>> sublevel sets, M is linear and takes value in some subspace of hermitian >>>>>> matrices. >>>>>> >>>>>> the problem is convex but the costraint can not be handled directly by >>>>>> any of the optimization routines in scipy. So i choose to change it to an >>>>>> uncostrained problem with objective function: >>>>>> >>>>>> f1(x) = f(x) for M(x) pos semi def >>>>>> f1(x) = Inf otherwise >>>>>> >>>>>> the problem is that it seems the routines can not handle the infinity >>>>>> values correctly. >>>>>> >>>>>> Some of the routines (fmin_cg comes to mind) wants to check the >>>>>> gradient at points where the objective function is infinite. Clearly in such >>>>>> cases the gradient is not defined - i.e the calculations fail - and the >>>>>> algorithm terminates. >>>>>> >>>>>> Others (like fmin_bfgs) strangely converge to a point where the >>>>>> objective is infinite despite the fact that the initial point was not. >>>>>> >>>>>> Do you have any suggestion to fix this problem? >>>>>> >>>>>> regards, >>>>>> >>>>>> Enrico >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> SciPy-User mailing list >>>>>> SciPy-User at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Information System Engineer, Ph.D. >>>>> Blog: http://matt.eifelle.com >>>>> LinkedIn: http://www.linkedin.com/in/matthieubrucher >>>>> >>>>> _______________________________________________ >>>>> SciPy-User mailing list >>>>> SciPy-User at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> >>> -- >>> Information System Engineer, Ph.D. >>> Blog: http://matt.eifelle.com >>> LinkedIn: http://www.linkedin.com/in/matthieubrucher >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrennie at gmail.com Wed Sep 15 12:48:49 2010 From: jrennie at gmail.com (Jason Rennie) Date: Wed, 15 Sep 2010 12:48:49 -0400 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: <16213.823.qm@web26703.mail.ukl.yahoo.com> References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: On Tue, Sep 14, 2010 at 9:55 AM, enrico avventi wrote: > Some of the routines (fmin_cg comes to mind) wants to check the gradient at > points where the objective function is infinite. Clearly in such cases the > gradient is not defined - i.e the calculations fail - and the algorithm > terminates. IIUC, CG requires that the function is smooth, so you can't use CG for your problem. I.e. there's nothing wrong with fmin_cg. You really need a semidefinite programming solver, such as yalmip or sedumi. My experience from ~5 years ago is that SDP solvers only work on relatively small problems (1000s of variables). http://en.wikipedia.org/wiki/Semidefinite_programming Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eavventi at yahoo.it Wed Sep 15 15:44:25 2010 From: eavventi at yahoo.it (Enrico Avventi) Date: Wed, 15 Sep 2010 21:44:25 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: you are right i didn't mention it. but it is indeed the case f(x) is finite only for M(x) pos def. On Wed, Sep 15, 2010 at 5:28 PM, Sebastian Walter < sebastian.walter at gmail.com> wrote: > you said you formulated your problem as follows: > > > f1(x) = f(x) for M(x) pos semi def > f1(x) = Inf otherwise > > I don't see any indication that anything goes to infinity when the sequence > of iterates approaches a boundary. > It seems to me that what you are describing is the behavior of a barrier > function which is used in interior point methods, i.e. of the form > > min_x f(x) - a log( g(M(x)) > > where g(M(x)) is a sufficiently smooth function in x and g(M(x)) = 0 if > M(x) is not positive definite > and a>0. Is this what you are doing? > > > Sebastian > > > > > On Wed, Sep 15, 2010 at 12:52 PM, Enrico Avventi wrote: > >> my function tends to infinity as you approach the boundary, as it is the >> case for all convex functions that can be defined only inside an open convex >> set. >> >> >> On Wed, Sep 15, 2010 at 12:18 PM, Sebastian Walter < >> sebastian.walter at gmail.com> wrote: >> >>> I can't quite follow the reasoning that another line search would solve >>> the problem. >>> If the current iterate is at the boundary and the new search direction >>> points into the infeasible set, then a line search is unlikely to help. >>> >>> Example: >>> min_x dot([2,1],x) >>> s.t. x >= 0 >>> >>> the minimizer is x_* = [0,0]. If the current iterate x_k is at [3,0], >>> then the new search direction woud be >>> s_k = -[2,1] which points directly into the infeasible set which would >>> yield infty. >>> >>> I'm not sure what the best approach for your problem is. >>> Since SDP is still an active topic of research one could conjecture that >>> possibly it's not as easy as you think. >>> If you find a good solution I'd be happy to hear about it. >>> >>> regards, >>> Sebastian >>> >>> >>> >>> >>> On Wed, Sep 15, 2010 at 11:19 AM, Matthieu Brucher < >>> matthieu.brucher at gmail.com> wrote: >>> >>>> Hi, >>>> >>>> The line search used in scipy is based on Wolfe-Powell rules and the >>>> search for appropriate values is done by interpolation. This is why it >>>> cannot be used for your kind of problems. All Wolfe-Powel line searches I've >>>> found in the litterature are based on such interpolations. >>>> It could be changed to fit your purpose, but it would be slower for >>>> 99.99% of the other optimizations. So you may add some additional >>>> constraints here. >>>> >>>> Matthieu >>>> >>>> 2010/9/15 Enrico Avventi >>>> >>>> Hi Matthieu, >>>>> >>>>> thanx for the reply. as far as i know it shouldn't be a problem at all >>>>> in theory. in fact convergence theorems for newton methods and, in general, >>>>> descent method do not require the function to be defined everywhere. you >>>>> need just a function defined in an open convex set that has compact sublevel >>>>> sets - e.g f(x) = x - log(x). this is exactly my situation. >>>>> it is just a matter of slightly changing the line search method to >>>>> reject steps that would lead to an infinite value. >>>>> i will look at how it is implemented in scipy and see if it can be >>>>> fixed easily. >>>>> >>>>> /Enrico >>>>> >>>>> >>>>> On Tue, Sep 14, 2010 at 5:51 PM, Matthieu Brucher < >>>>> matthieu.brucher at gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> There are two categories of contraints optimizations: >>>>>> - you can evaluate the function outside the constraints >>>>>> - you cannot evaluate the function outsde the constraints. >>>>>> >>>>>> If the first one can be handled by more general algorithms providing >>>>>> some tricks, you cannot use them for the second one. Your problem is clearly >>>>>> a second category problem, so you must use appropriate algorithms (which may >>>>>> not be available in scipy directly, you may want to check OpenOpt). >>>>>> >>>>>> It's not a problem of routines, it's a problem of appropriate >>>>>> algorithms. >>>>>> >>>>>> Matthieu >>>>>> >>>>>> 2010/9/14 enrico avventi >>>>>> >>>>>>> hello all, >>>>>>> >>>>>>> i am trying out some of the optimization routines for a problem of >>>>>>> mine that is on the form: >>>>>>> >>>>>>> min f(x) >>>>>>> >>>>>>> s.t M(x) is positive semidefinite >>>>>>> >>>>>>> where f is strictly convex in the feasible region with compact >>>>>>> sublevel sets, M is linear and takes value in some subspace of hermitian >>>>>>> matrices. >>>>>>> >>>>>>> the problem is convex but the costraint can not be handled directly >>>>>>> by any of the optimization routines in scipy. So i choose to change it to an >>>>>>> uncostrained problem with objective function: >>>>>>> >>>>>>> f1(x) = f(x) for M(x) pos semi def >>>>>>> f1(x) = Inf otherwise >>>>>>> >>>>>>> the problem is that it seems the routines can not handle the infinity >>>>>>> values correctly. >>>>>>> >>>>>>> Some of the routines (fmin_cg comes to mind) wants to check the >>>>>>> gradient at points where the objective function is infinite. Clearly in such >>>>>>> cases the gradient is not defined - i.e the calculations fail - and the >>>>>>> algorithm terminates. >>>>>>> >>>>>>> Others (like fmin_bfgs) strangely converge to a point where the >>>>>>> objective is infinite despite the fact that the initial point was not. >>>>>>> >>>>>>> Do you have any suggestion to fix this problem? >>>>>>> >>>>>>> regards, >>>>>>> >>>>>>> Enrico >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> SciPy-User mailing list >>>>>>> SciPy-User at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Information System Engineer, Ph.D. >>>>>> Blog: http://matt.eifelle.com >>>>>> LinkedIn: http://www.linkedin.com/in/matthieubrucher >>>>>> >>>>>> _______________________________________________ >>>>>> SciPy-User mailing list >>>>>> SciPy-User at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> SciPy-User mailing list >>>>> SciPy-User at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>>> >>>>> >>>> >>>> >>>> -- >>>> Information System Engineer, Ph.D. >>>> Blog: http://matt.eifelle.com >>>> LinkedIn: http://www.linkedin.com/in/matthieubrucher >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eavventi at yahoo.it Wed Sep 15 16:05:58 2010 From: eavventi at yahoo.it (Enrico Avventi) Date: Wed, 15 Sep 2010 22:05:58 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: i'm aware of SDP solvers but they handle only linear objective functions AFAIK. and the costraints are not the problem. it is just that the function is not defined everywhere. i will experiment by changing the line search methods as i think they are the only part of the methods that needs to be aware of the domain. thanx for the help, i will post my eventual findings. On Wed, Sep 15, 2010 at 6:48 PM, Jason Rennie wrote: > On Tue, Sep 14, 2010 at 9:55 AM, enrico avventi wrote: > >> Some of the routines (fmin_cg comes to mind) wants to check the gradient >> at points where the objective function is infinite. Clearly in such cases >> the gradient is not defined - i.e the calculations fail - and the algorithm >> terminates. > > > IIUC, CG requires that the function is smooth, so you can't use CG for your > problem. I.e. there's nothing wrong with fmin_cg. You really need a > semidefinite programming solver, such as yalmip or sedumi. My experience > from ~5 years ago is that SDP solvers only work on relatively small problems > (1000s of variables). > > http://en.wikipedia.org/wiki/Semidefinite_programming > > Jason > > -- > Jason Rennie > Research Scientist, ITA Software > 617-714-2645 > http://www.itasoftware.com/ > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian.walter at gmail.com Wed Sep 15 16:16:51 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Wed, 15 Sep 2010 22:16:51 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: well, good luck then. I'm still curious what the objective and constraint functions of your original problem are. Would it be possible to post it here? On Wed, Sep 15, 2010 at 10:05 PM, Enrico Avventi wrote: > i'm aware of SDP solvers but they handle only linear objective functions > AFAIK. > and the costraints are not the problem. it is just that the function is not > defined everywhere. > i will experiment by changing the line search methods as i think they are > the only > part of the methods that needs to be aware of the domain. > > thanx for the help, i will post my eventual findings. > > On Wed, Sep 15, 2010 at 6:48 PM, Jason Rennie wrote: > >> On Tue, Sep 14, 2010 at 9:55 AM, enrico avventi wrote: >> >>> Some of the routines (fmin_cg comes to mind) wants to check the gradient >>> at points where the objective function is infinite. Clearly in such cases >>> the gradient is not defined - i.e the calculations fail - and the algorithm >>> terminates. >> >> >> IIUC, CG requires that the function is smooth, so you can't use CG for >> your problem. I.e. there's nothing wrong with fmin_cg. You really need a >> semidefinite programming solver, such as yalmip or sedumi. My experience >> from ~5 years ago is that SDP solvers only work on relatively small problems >> (1000s of variables). >> >> http://en.wikipedia.org/wiki/Semidefinite_programming >> >> Jason >> >> -- >> Jason Rennie >> Research Scientist, ITA Software >> 617-714-2645 >> http://www.itasoftware.com/ >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eavventi at yahoo.it Thu Sep 16 03:57:45 2010 From: eavventi at yahoo.it (Enrico Avventi) Date: Thu, 16 Sep 2010 09:57:45 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: sure, no problem. the objective function is f(\Lambda) = trace(\Sigma \Lambda) - \int_\Pi \log [G(z) \Lambda G(z^-1)'] z^-1 dz where \Sigma and \Lambda are hermitian matrices, G(z) is complex matrix valued and analytic inside the unit disc and the integration is along the unit circle. the function is only defined when G(z) \Lambda G(z^-1)' is positive definite in the unit circle and tends to infinity when approaching a value of \Lambda that makes it losing rank. in some special cases you can then substitute w.l.o.g \lambda with some linear M(x) where x is a real vector in order to obtain a problem of the form that i was talking about. On Wed, Sep 15, 2010 at 10:16 PM, Sebastian Walter < sebastian.walter at gmail.com> wrote: > well, good luck then. > > I'm still curious what the objective and constraint functions of your > original problem are. > Would it be possible to post it here? > > > On Wed, Sep 15, 2010 at 10:05 PM, Enrico Avventi wrote: > >> i'm aware of SDP solvers but they handle only linear objective functions >> AFAIK. >> and the costraints are not the problem. it is just that the function is >> not defined everywhere. >> i will experiment by changing the line search methods as i think they are >> the only >> part of the methods that needs to be aware of the domain. >> >> thanx for the help, i will post my eventual findings. >> >> On Wed, Sep 15, 2010 at 6:48 PM, Jason Rennie wrote: >> >>> On Tue, Sep 14, 2010 at 9:55 AM, enrico avventi wrote: >>> >>>> Some of the routines (fmin_cg comes to mind) wants to check the >>>> gradient at points where the objective function is infinite. Clearly in such >>>> cases the gradient is not defined - i.e the calculations fail - and the >>>> algorithm terminates. >>> >>> >>> IIUC, CG requires that the function is smooth, so you can't use CG for >>> your problem. I.e. there's nothing wrong with fmin_cg. You really need a >>> semidefinite programming solver, such as yalmip or sedumi. My experience >>> from ~5 years ago is that SDP solvers only work on relatively small problems >>> (1000s of variables). >>> >>> http://en.wikipedia.org/wiki/Semidefinite_programming >>> >>> Jason >>> >>> -- >>> Jason Rennie >>> Research Scientist, ITA Software >>> 617-714-2645 >>> http://www.itasoftware.com/ >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adamdryan at gmail.com Thu Sep 16 13:20:14 2010 From: adamdryan at gmail.com (Adam Ryan) Date: Thu, 16 Sep 2010 10:20:14 -0700 Subject: [SciPy-User] weighted griddata Message-ID: Hi. First, kudos to the scipy community; great package. I was wondering if there is a way to use scipy.interpolate.griddata with weights. Specifically, I have a list of lines. Each line represents the position of a wave on a beach at a timestamp, and is comprised of a list of mostly connected points. I can use griddata and all the points of all the lines to create an interpolation of the position of the wave vs time. The problem is that the lines are the result of image processing and the points vary in confidence level. Currently I'm just tossing out points that don't pass muster, but I'd like a more robust solution, something like a weighted griddata, or some other method. Any advice would be great. Thanks, Adam From pav at iki.fi Thu Sep 16 16:27:46 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 16 Sep 2010 20:27:46 +0000 (UTC) Subject: [SciPy-User] weighted griddata References: Message-ID: Thu, 16 Sep 2010 10:20:14 -0700, Adam Ryan wrote: [clip] > I was wondering if there is a way to use scipy.interpolate.griddata with > weights. You are probably looking for routines for data smoothing, and not for interpolation. griddata only computes interpolants, functions f(x) that go through all the data points f(x_i) = y_i If you want some data points to have more weight than others, then probably you don't want to this condition to hold (otherwise you get sharp peaks around points with low weights). Data smoothing is a different problem than interpolation, and the algorithms in griddata cannot do it, and they are not easily modified to do it either. > Specifically, I have a list of lines. Each line represents the position > of a wave on a beach at a timestamp, and is comprised of a list of > mostly connected points. I can use griddata and all the points of all > the lines to create an interpolation of the position of the wave vs > time. The problem is that the lines are the result of image processing > and the points vary in confidence level. So you have data (x[i], y[i], t[i]) and you'd like to fit a 2-D surface to it? I guess you used griddata to find the graph of the function y = f(x, t)? > Currently I'm just tossing out > points that don't pass muster, but I'd like a more robust solution, > something like a weighted griddata, or some other method. Any advice > would be great. Since your data is 2-D, you can in principle use the spline routines in scipy.interpolate for data smoothing. For example, ip = interpolate.SmoothBivariateSpline(x=x, y=t, z=y, w=weights) xi, ti = mgrid[0:1:70j,0:1:80j] yi = ip.ev(xi.ravel(), ti.ravel()).reshape(xi.shape) instead of yi = griddata((x, t), y, (xi, ti)) You may need to fiddle with the smoothing parameter `s=...` for SmoothBivariateSpline. Ok, a word of warning: personally, I've found that these 2-D spline routines often produce garbage, especially when `s` is small, and you can waste a lot of time fiddling with them. These routines come from the ancient FITPACK library (http://www.netlib.org/dierckx/), and I suppose it's not fully refined in this respect... Or, you can maybe fit a 1-D spline (UnivariateSpline) at each `t` separately, and use the smoothed results in griddata. The 1D spline routines are robust. Another option that you can try is to cook up some inverse distance weighting scheme -- you can use scipy.spatial.cKDTree/KDTree to do the fast N nearest neighbour lookups. Scipy doesn't have an implementation for these algorithms at the moment, so you'd have to do it from scratch. Maybe other people have more ideas? -- Pauli Virtanen From pav at iki.fi Thu Sep 16 16:42:51 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 16 Sep 2010 20:42:51 +0000 (UTC) Subject: [SciPy-User] weighted griddata References: Message-ID: Thu, 16 Sep 2010 20:27:46 +0000, Pauli Virtanen wrote: [clip] > Data smoothing is a different problem than interpolation, and the > algorithms in griddata cannot do it, and they are not easily modified to > do it either. No, I tell a lie, the Clough-Tocher 2D spline could easily be used for weighed data smoothing. The only change needed would be to adjust the gradient estimation routine so that instead of minimizing ||surface curvature|| it would allow for changes also in the data points, and would minimize ||weighed deviation from data points|| + ||surface curvature|| instead. Interesting idea. I'll need to look up some literature before thinking more about it -- this particular wheel must already be invented... -- Pauli Virtanen From charlesr.harris at gmail.com Thu Sep 16 18:53:54 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 16 Sep 2010 16:53:54 -0600 Subject: [SciPy-User] weighted griddata In-Reply-To: References: Message-ID: On Thu, Sep 16, 2010 at 2:27 PM, Pauli Virtanen wrote: > Thu, 16 Sep 2010 10:20:14 -0700, Adam Ryan wrote: > [clip] > > I was wondering if there is a way to use scipy.interpolate.griddata with > > weights. > > You are probably looking for routines for data smoothing, and not for > interpolation. griddata only computes interpolants, functions f(x) that > go through all the data points > > f(x_i) = y_i > > If you want some data points to have more weight than others, then > probably you don't want to this condition to hold (otherwise you get > sharp peaks around points with low weights). > > Data smoothing is a different problem than interpolation, and the > algorithms in griddata cannot do it, and they are not easily modified to > do it either. > > > Specifically, I have a list of lines. Each line represents the position > > of a wave on a beach at a timestamp, and is comprised of a list of > > mostly connected points. I can use griddata and all the points of all > > the lines to create an interpolation of the position of the wave vs > > time. The problem is that the lines are the result of image processing > > and the points vary in confidence level. > > So you have data > > (x[i], y[i], t[i]) > > and you'd like to fit a 2-D surface to it? I guess you used griddata to > find the graph of the function y = f(x, t)? > > > Currently I'm just tossing out > > points that don't pass muster, but I'd like a more robust solution, > > something like a weighted griddata, or some other method. Any advice > > would be great. > > Since your data is 2-D, you can in principle use the spline routines in > scipy.interpolate for data smoothing. For example, > > ip = interpolate.SmoothBivariateSpline(x=x, y=t, z=y, > w=weights) > xi, ti = mgrid[0:1:70j,0:1:80j] > yi = ip.ev(xi.ravel(), ti.ravel()).reshape(xi.shape) > > instead of > > yi = griddata((x, t), y, (xi, ti)) > > You may need to fiddle with the smoothing parameter `s=...` for > SmoothBivariateSpline. > > Ok, a word of warning: personally, I've found that these 2-D spline > routines often produce garbage, especially when `s` is small, and you can > waste a lot of time fiddling with them. These routines come from the > ancient FITPACK library (http://www.netlib.org/dierckx/), and I suppose > it's not fully refined in this respect... > > Or, you can maybe fit a 1-D spline (UnivariateSpline) at each `t` > separately, and use the smoothed results in griddata. The 1D spline > routines are robust. > > I've found the automatic 1D spline fits to also produce garbage. However, they work OK with user supplied knots. > Another option that you can try is to cook up some inverse distance > weighting scheme -- you can use scipy.spatial.cKDTree/KDTree to do the > fast N nearest neighbour lookups. Scipy doesn't have an implementation > for these algorithms at the moment, so you'd have to do it from scratch. > > Maybe other people have more ideas? > > I think the rbf routines might be appropriate. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aarchiba at physics.mcgill.ca Thu Sep 16 19:14:26 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Thu, 16 Sep 2010 19:14:26 -0400 Subject: [SciPy-User] weighted griddata In-Reply-To: References: Message-ID: On 16 September 2010 16:42, Pauli Virtanen wrote: > Thu, 16 Sep 2010 20:27:46 +0000, Pauli Virtanen wrote: > [clip] >> Data smoothing is a different problem than interpolation, and the >> algorithms in griddata cannot do it, and they are not easily modified to >> do it either. > > No, I tell a lie, the Clough-Tocher 2D spline could easily be used for > weighed data smoothing. The only change needed would be to adjust the > gradient estimation routine so that instead of minimizing > > ? ? ? ?||surface curvature|| > > it would allow for changes also in the data points, and would minimize > > ? ? ? ?||weighed deviation from data points|| + ||surface curvature|| > > instead. > > Interesting idea. I'll need to look up some literature before thinking > more about it -- this particular wheel must already be invented... I think so - in fact it's what the FITPACK splines do. At least the 1D splines. They have the additional wrinkle that in addition to the above optimization they try to find the "right" number of knots. For the OP's original purpose, isn't that what "kriging" is for? I've never used it, but as I understand it it's a form of interpolation where the value at a point is the average of the nearest data values, weighted by distance. Anne From charlesr.harris at gmail.com Thu Sep 16 21:27:45 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 16 Sep 2010 19:27:45 -0600 Subject: [SciPy-User] weighted griddata In-Reply-To: References: Message-ID: On Thu, Sep 16, 2010 at 2:42 PM, Pauli Virtanen wrote: > Thu, 16 Sep 2010 20:27:46 +0000, Pauli Virtanen wrote: > [clip] > > Data smoothing is a different problem than interpolation, and the > > algorithms in griddata cannot do it, and they are not easily modified to > > do it either. > > No, I tell a lie, the Clough-Tocher 2D spline could easily be used for > weighed data smoothing. The only change needed would be to adjust the > gradient estimation routine so that instead of minimizing > > ||surface curvature|| > > it would allow for changes also in the data points, and would minimize > > ||weighed deviation from data points|| + ||surface curvature|| > > If you use the sqrt of the squares it is a Sobolev space due to the fact that the first derivative can be omitted. but there are lots of variations on this theme. Another is least squares with, say, lambda times the square of the second difference added to the normal squared deviation. That tends to keep the wiggles down but is very sensitive to lambda. There might even be an official name for that method, it goes back a long ways. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From adamdryan at gmail.com Fri Sep 17 00:41:28 2010 From: adamdryan at gmail.com (Adam Ryan) Date: Thu, 16 Sep 2010 21:41:28 -0700 Subject: [SciPy-User] weighted griddata In-Reply-To: References: Message-ID: Thanks to all for the input. I'm a Python programmer trying to understand the math I need. My comments in line... > So you have data > > (x[i], y[i], t[i]) > > and you'd like to fit a 2-D surface to it? I guess you used griddata to > find the graph of the function y = f(x, t)? Right, but griddata alone just fills in the grid, it doesn't return a function, and as you rightly point out, only fills in and doesn't smooth the original data. > You may need to fiddle with the smoothing parameter `s=...` for > SmoothBivariateSpline. I was using interpolate.SmoothBivariateSpline previously and had good results when it worked, but I struggled with the smoothing parameter for so long I had to put it down. I could not figure out how to consistently calculate/estimate/predict a smoothing factor (from the data) that was not too low (and never return) or too high (and give blocky results). Is there a way to calculate an optimal smoothing factor from a data set? > Or, you can maybe fit a 1-D spline (UnivariateSpline) at each `t` > separately, and use the smoothed results in griddata. The 1D spline > routines are robust. Tried that too, but that didn't really do the trick, since the actual position of a point is more a function of the neighbors in time then along shore, due to obstacles, etc. > No, I tell a lie, the Clough-Tocher 2D spline could easily be used for > weighed data smoothing. The only change needed would be to adjust the > gradient estimation routine ... Yea, that's what I was thinking, ha ha, I just don't know how. Pointers? > I think the rbf routines might be appropriate. I've used the rbf routines to ortho-rectify our barrel-distorted images (where the data comes from), but here the positional data is not based around a center point. Or maybe I don't understand the rbf stuff fully. How would I use weights with the rbf stuff? > If you use the sqrt of the squares it is a Sobolev space due to the fact > that the first derivative can be omitted. but there are lots of variations > on this theme. Another is least squares with, say, lambda times the square > of the second difference added to the normal squared deviation. That tends > to keep the wiggles down but is very sensitive to lambda. There might even > be an official name for that method, it goes back a long ways. For clarification, all the points of a line have the same t value, but each point has a weight of 0 to 1. Would LSQBivariateSpline be any better than SmoothBivariateSpline? Again, the smoothing factor drove me nuts. - Adam From charlesr.harris at gmail.com Fri Sep 17 01:10:10 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 16 Sep 2010 23:10:10 -0600 Subject: [SciPy-User] weighted griddata In-Reply-To: References: Message-ID: On Thu, Sep 16, 2010 at 10:41 PM, Adam Ryan wrote: > Thanks to all for the input. I'm a Python programmer trying to > understand the math I need. My comments in line... > > > So you have data > > > > (x[i], y[i], t[i]) > > > > and you'd like to fit a 2-D surface to it? I guess you used griddata to > > find the graph of the function y = f(x, t)? > > Right, but griddata alone just fills in the grid, it doesn't return a > function, and as you rightly point out, only fills in and doesn't > smooth the original data. > > > You may need to fiddle with the smoothing parameter `s=...` for > > SmoothBivariateSpline. > > I was using interpolate.SmoothBivariateSpline previously and had good > results when it worked, but I struggled with the smoothing parameter > for so long I had to put it down. I could not figure out how to > consistently calculate/estimate/predict a smoothing factor (from the > data) that was not too low (and never return) or too high (and give > blocky results). > > Is there a way to calculate an optimal smoothing factor from a data set? > > > Or, you can maybe fit a 1-D spline (UnivariateSpline) at each `t` > > separately, and use the smoothed results in griddata. The 1D spline > > routines are robust. > > Tried that too, but that didn't really do the trick, since the actual > position of a point is more a function of the neighbors in time then > along shore, due to obstacles, etc. > > > No, I tell a lie, the Clough-Tocher 2D spline could easily be used for > > weighed data smoothing. The only change needed would be to adjust the > > gradient estimation routine ... > > Yea, that's what I was thinking, ha ha, I just don't know how. Pointers? > > > I think the rbf routines might be appropriate. > > I've used the rbf routines to ortho-rectify our barrel-distorted > images (where the data comes from), How well did it work? I've been thinking of trying rbf for some inkblot-distorted images -- nothing so simple as barrel-distortion here. > but here the positional data is > not based around a center point. Or maybe I don't understand the rbf > stuff fully. How would I use weights with the rbf stuff? > > > If you use the sqrt of the squares it is a Sobolev space due to the fact > > that the first derivative can be omitted. but there are lots of > variations > > on this theme. Another is least squares with, say, lambda times the > square > > of the second difference added to the normal squared deviation. That > tends > > to keep the wiggles down but is very sensitive to lambda. There might > even > > be an official name for that method, it goes back a long ways. > > For clarification, all the points of a line have the same t value, but > each point has a weight of 0 to 1. Would LSQBivariateSpline be any > better than SmoothBivariateSpline? Again, the smoothing factor drove > me nuts. > > Probably. You need to pick the knot points, which sort like picking the smoothing factor but more robust. It's a least squares fit with curves/surfaces that are less "stiff" than polynomials. I don't understand the points of a line bit, you can probably fit the surface directly. Anne's suggestion of looking into kriging is also worth following up. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Sep 17 05:19:06 2010 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 17 Sep 2010 09:19:06 +0000 (UTC) Subject: [SciPy-User] weighted griddata References: Message-ID: Thu, 16 Sep 2010 21:41:28 -0700, Adam Ryan wrote: [clip] > > You may need to fiddle with the smoothing parameter `s=...` for > > SmoothBivariateSpline. > > I was using interpolate.SmoothBivariateSpline previously and had good > results when it worked, but I struggled with the smoothing parameter > for so long I had to put it down. I could not figure out how to > consistently calculate/estimate/predict a smoothing factor (from the > data) that was not too low (and never return) or too high (and give > blocky results). > > Is there a way to calculate an optimal smoothing factor from a data set? If there is, it's probably explained in P. Dierckx's book http://books.google.fi/books?id=-RIQ3SR0sZMC about FITPACK. [clip] >> No, I tell a lie, the Clough-Tocher 2D spline could easily be used for >> weighed data smoothing. The only change needed would be to adjust the >> gradient estimation routine ... > > Yea, that's what I was thinking, ha ha, I just don't know how. > Pointers? You'd have to change the approach in the _estimate_gradients_2d_global function to also minimize some norm over the data points: http://projects.scipy.org/scipy/browser/trunk/scipy/interpolate/ interpnd.pyx#L270 The optimization problem solved iteratively at each vertex should be changed to also optimize the function value, and a quadratic penalty term should be added to the error functional. (Quadratic, since we want the optimization problem to remain linear.) The main problem here is choosing the weights of the error term for each vertex in a way that makes sense. I suppose some good recipe for this can be found in the literature. Actually, I think I'll take a look at this during the weekend, it sounds interesting (some of us have weird tastes in fun...). I won't promise anything, though :) >> I think the rbf routines might be appropriate. > > I've used the rbf routines to ortho-rectify our barrel-distorted images > (where the data comes from), but here the positional data is not based > around a center point. Or maybe I don't understand the rbf stuff fully. > How would I use weights with the rbf stuff? You'd probably need to add the weights to the distance function, and somehow balance it against the smooth parameter. The distance function you pass to Rbf() has the data points always along the last axis, so weights can in principle be added. Anyway, I guess some inverse distance weighting scheme (i.e. the same thing as simple kriging) would be more efficient than RBF. The RBF in Scipy always looks at all points, whereas IDW's typically look only at N nearest neighbours, which is probably more efficient and more stable. [clip] > For clarification, all the points of a line have the same t value, but > each point has a weight of 0 to 1. Would LSQBivariateSpline be any > better than SmoothBivariateSpline? Again, the smoothing factor drove me > nuts. LSQBivariateSpline might be better. I somehow suppose part of the problems with FITPACK come from the knot selection, at least Chuck's comment points to that direction... Now, I really wonder: is the spline fitting problem inherently difficult, or is there just some simple bug in FITPACK making it work like this? -- Pauli Virtanen From david.douard at logilab.fr Fri Sep 17 06:40:06 2010 From: david.douard at logilab.fr (David Douard) Date: Fri, 17 Sep 2010 12:40:06 +0200 Subject: [SciPy-User] ANN: new version of pylint (in fact, astng) Message-ID: <201009171240.12318.david.douard@logilab.fr> Hi all, a few days ago, we (at Logilab) took time (well, Sylvain did) to fix a long outstanding bug in astng (which is the heart of pylint): http://www.logilab.org/ticket/23008 The result is that, at last, pylint can be used with numpy. Thank you Sylvain! So go and get the latest version of astng and enjoy pylint on your numpy-based projects! http://www.logilab.org/project/logilab-astng Regards, David -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From henrylindsaysmith at gmail.com Fri Sep 17 09:44:45 2010 From: henrylindsaysmith at gmail.com (henry lindsay smith) Date: Fri, 17 Sep 2010 14:44:45 +0100 Subject: [SciPy-User] performant envelope follower Message-ID: I have the following code backported from C envelope follower and converted into python code (using numpy arrays) for i in range(len(self.m_signal)): envIn=np.abs(self.m_signal[i]**2)#square for energy if(self.envelope < envIn): self.envelope *= self.m_ga self.envelope +=(1-self.m_ga)*envIn else: self.envelope *= self.m_gr self.envelope +=(1-self.m_gr)*envIn self.sigEnv[i]=self.envelope it runs but slowly as I am iterating through the whole signal which is pretty slow in python (but super fast in C) Can anyone think of a more efficient way to run this possibly as as a filtering operation? I need the same result as I'm going to port back into C at somepoint. thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjstickel at vcn.com Fri Sep 17 09:47:02 2010 From: jjstickel at vcn.com (Jonathan Stickel) Date: Fri, 17 Sep 2010 07:47:02 -0600 Subject: [SciPy-User] weighted griddata In-Reply-To: References: Message-ID: <4C937156.5080103@vcn.com> On 9/16/10 22:41 , scipy-user-request at scipy.org wrote: >> You may need to fiddle with the smoothing parameter `s=...` for >> > SmoothBivariateSpline. > I was using interpolate.SmoothBivariateSpline previously and had good > results when it worked, but I struggled with the smoothing parameter > for so long I had to put it down. I could not figure out how to > consistently calculate/estimate/predict a smoothing factor (from the > data) that was not too low (and never return) or too high (and give > blocky results). > > Is there a way to calculate an optimal smoothing factor from a data set? > I know your problem is 2-D, but for 1-D I have written the scikit "datasmooth" that implements smoothing by regularization and includes generalized cross validation for determining the optimal regularization parameter (aka smoothing factor): http://pypi.python.org/pypi/scikits.datasmooth/0.5 I know the method can be extended to 2-D, but I don't have a need or the time to do it myself. Jonathan From ben.root at ou.edu Fri Sep 17 11:11:47 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 17 Sep 2010 10:11:47 -0500 Subject: [SciPy-User] performant envelope follower In-Reply-To: References: Message-ID: On Fri, Sep 17, 2010 at 8:44 AM, henry lindsay smith < henrylindsaysmith at gmail.com> wrote: > I have the following code backported from C envelope follower and converted > into python code (using numpy arrays) > > for i in range(len(self.m_signal)): > > envIn=np.abs(self.m_signal[i]**2)#square for energy > > if(self.envelope < envIn): > > self.envelope *= self.m_ga > > self.envelope +=(1-self.m_ga)*envIn > > else: > > self.envelope *= self.m_gr > > self.envelope +=(1-self.m_gr)*envIn > > self.sigEnv[i]=self.envelope > > > it runs but slowly as I am iterating through the whole signal which is > pretty slow in python (but super fast in C) > > Can anyone think of a more efficient way to run this possibly as as a > filtering operation? I need the same result as I'm going to port back into C > at somepoint. > > thanks > > First, I would just go ahead and calculate envln outside the loop, that will save you some time. After that, it looks like you have a conditional accumulation operation going on. If it was just a conditional, I would suggest np.where(), or if it was just an accumulation there is np.cumprod() and np.cumsum(). Even then, one could just use 1's and 0's to do the correct operation within cumprod() or cumsum(). However, you are doing both a product and a summation in the accumulator, which is a little harder to figure out. I am not sure where to go for that. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Sep 17 15:00:35 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 17 Sep 2010 12:00:35 -0700 Subject: [SciPy-User] weighted griddata In-Reply-To: References: Message-ID: <4C93BAD3.7080201@noaa.gov> Adam Ryan wrote: > For clarification, all the points of a line have the same t value, but > each point has a weight of 0 to 1. does that mean the ones with weight 0 you want to ignore, and ones with weight 1 you want to fit exactly? i.e. you know which ones are more robust/accurate? If so then it doesn't sound to me like any of the smoothing routines being talked about are going to do the right thing, at least not out of the box -- I think they all assume that all points are equally valid, and weight according to how far away points are, or, more generally, how well they fit a smooth function. Of the top of me head, I imagine you may be able to do some sort of least squares type fit to a known function (maybe a polynomial or piecewise-polynomial), but with the error term weighted according to to your known weights. That could look a lot like a spline fit, but you'd have to insert your weighting in there somehow. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From pav at iki.fi Fri Sep 17 15:09:43 2010 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 17 Sep 2010 19:09:43 +0000 (UTC) Subject: [SciPy-User] weighted griddata References: <4C93BAD3.7080201@noaa.gov> Message-ID: Fri, 17 Sep 2010 12:00:35 -0700, Christopher Barker wrote: [clip] > does that mean the ones with weight 0 you want to ignore, and ones with > weight 1 you want to fit exactly? i.e. you know which ones are more > robust/accurate? > > If so then it doesn't sound to me like any of the smoothing routines > being talked about are going to do the right thing, at least not out of > the box -- I think they all assume that all points are equally valid, > and weight according to how far away points are, or, more generally, how > well they fit a smooth function. The spline routines in Scipy have weight parameters, which should do exactly that out of the box. Rbf/inverse distance/Kriging type methods can use weights, but Scipy doesn't contain out-of-the-box solutions. For the triangulation-based methods in griddata(), support for weighed smoothing can in principle be added. -- Pauli Virtanen From ben.root at ou.edu Fri Sep 17 16:03:23 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 17 Sep 2010 15:03:23 -0500 Subject: [SciPy-User] weighted griddata In-Reply-To: <4C93BAD3.7080201@noaa.gov> References: <4C93BAD3.7080201@noaa.gov> Message-ID: On Fri, Sep 17, 2010 at 2:00 PM, Christopher Barker wrote: > Adam Ryan wrote: > > For clarification, all the points of a line have the same t value, but > > each point has a weight of 0 to 1. > > > does that mean the ones with weight 0 you want to ignore, and ones with > weight 1 you want to fit exactly? i.e. you know which ones are more > robust/accurate? > > If so then it doesn't sound to me like any of the smoothing routines > being talked about are going to do the right thing, at least not out of > the box -- I think they all assume that all points are equally valid, > and weight according to how far away points are, or, more generally, how > well they fit a smooth function. > > Of the top of me head, I imagine you may be able to do some sort of > least squares type fit to a known function (maybe a polynomial or > piecewise-polynomial), but with the error term weighted according to to > your known weights. That could look a lot like a spline fit, but you'd > have to insert your weighting in there somehow. > > -Chris > > > > Come to think of it, wouldn't something like 3DVar or 4DVar (data assimilation) be more what he is looking for? I know in meteorology, we use various data assimilation techniques to not only "snap" the observation data to a regular grid, but to also give proper weight to data that is more reliable than others. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Sep 19 21:36:23 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 20 Sep 2010 01:36:23 +0000 (UTC) Subject: [SciPy-User] weighted griddata References: Message-ID: Fri, 17 Sep 2010 09:19:06 +0000, Pauli Virtanen wrote: [clip] > Actually, I think I'll take a look at this during the weekend, it sounds > interesting (some of us have weird tastes in fun...). I won't promise > anything, though :) Turns out that the minimum norm network idea works: http://pav.iki.fi/blog/2010-09-19/nd-smoothing.html If the OP is interested in playing around with this: git clone git://github.com/pv/scipy-work.git git checkout feature/interpnd-smooth -- Pauli Virtanen From pav at iki.fi Mon Sep 20 05:26:44 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 20 Sep 2010 09:26:44 +0000 (UTC) Subject: [SciPy-User] weighted griddata References: Message-ID: Mon, 20 Sep 2010 01:36:23 +0000, Pauli Virtanen wrote: [clip] > git clone git://github.com/pv/scipy-work.git > git checkout feature/interpnd-smooth Should be git checkout origin/feature/interpnd-smooth or just download it from http://github.com/pv/scipy-work/tree/feature/interpnd-smooth From eric.moscardi at sophia.inria.fr Mon Sep 20 07:35:48 2010 From: eric.moscardi at sophia.inria.fr (moscardi) Date: Mon, 20 Sep 2010 13:35:48 +0200 Subject: [SciPy-User] Error in scipy.ndimage documentation In-Reply-To: References: Message-ID: <7578D643-CD3C-4498-ADE1-96BD98A31623@sophia.inria.fr> Nobody has tried to extend ndimage with C...? On Sep 10, 2010, at 7:18 PM, Matthieu Brucher wrote: > Hi, > > The following function is referenced on the scipy.ndimage page > (http://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html#ndimage-ccallbacks) > : > > static int > _shift_function(int *output_coordinates, double* input_coordinates, > int output_rank, int input_rank, void *callback_data) > { > int ii; > /* get the shift from the callback data pointer: */ > double shift = *(double*)callback_data; > /* calculate the coordinates: */ > for(ii = 0; ii < irank; ii++) > icoor[ii] = ocoor[ii] - shift; > /* return OK status: */ > return 1; > } > > It obviously cannot be compiled (irank -> input_rank, icoor -> > input_coordinates, ocoor -> output_coordinates) > > Matthieu > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev Eric MOSCARDI INRIA - Virtual Plants CIRAD, Avenue Agropolis 34398 Montpellier Cedex 5, France 04 67 61 58 00 (ask number 60 09) email : eric.moscardi at sophia.inria.fr From eavventi at yahoo.it Mon Sep 20 10:20:24 2010 From: eavventi at yahoo.it (Enrico Avventi) Date: Mon, 20 Sep 2010 16:20:24 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: so, updating the situation a bit... i wrote a new scalar search method that allows functions that tends to infinity. it works quite well on toy examples like phi(x) = - d*x - log(1-x) or phi(x) = -d*x + (1-x)^-k i changed the line searh method and fmin_bfgs to detect when a function is not defined everywhere and call the appropriate line search method. i also added test cases (consisting of the obove functions with d=100 and random k between 1 and 5) for checking if the new scalar search returns valid points - i.e satisfying wolfe conditions. all tests old and new seems topass. the last thing i want to check before submitting the patch is if this works in my problem as it is a real-world and high dimensional. so the perfect test case. unfortunately i couldn't try it out. the problem i stumbled with is that i rely on some fortran code wrapped with f2py. i recompiled it after installing the development version of numpy and scipy but nonetheless this is what i get: In [1]: run graph_test.py [[ 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0.] [ 0. 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1. 0.] [ 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1.] [ 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0.] [ 0. 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1. 0.] [ 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1.] [ 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0.] [ 0. 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1. 0.] [ 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1.]] check: 4.68754204281 34.9603190301 --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) RuntimeError: module compiled against ABI version 1000009 but this version of numpy is 2000000 --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /home/avventi/my_code/python_code/xspectral/graph_test.py in () 33 print "check:", min(np.linalg.eig(L+dL)[0].real), max(np.linalg.eig(L+dL)[0].real) 34 ---> 35 Sigma = compute_moment(fL,A,B,F,G,H,D) 36 print "R_0=" 37 print Sigma[0:m,0:m] /home/avventi/my_code/python_code/xspectral/armalib.pyc in compute_moment(Lambda, A, B, F, G, H, D) 359 from numpy import zeros, zeros_like, dot, concatenate, shape, bmat, kron, eye 360 from numpy.linalg import inv --> 361 from slycot import sb03md 362 n1 = A.shape[0] 363 n2 = F.shape[0] /usr/lib/python2.6/site-packages/slycot/__init__.pyc in () 2 3 # Analysis routines (5/40 wrapped) ----> 4 from analysis import ab01nd,ab05md,ab05nd,ab07nd,ab08nd 5 6 # Data analysis routines (0/7 wrapped) /usr/lib/python2.6/site-packages/slycot/analysis.py in () 20 # MA 02110-1301, USA. 21 ---> 22 from slycot import _wrapper 23 24 def ab01nd(n,m,A,B,jobz='N',tol=0,ldwork=None): ImportError: numpy.core.multiarray failed to import WARNING: Failure executing file: what did i miss? On Thu, Sep 16, 2010 at 9:57 AM, Enrico Avventi wrote: > sure, no problem. the objective function is > > f(\Lambda) = trace(\Sigma \Lambda) - \int_\Pi \log [G(z) \Lambda > G(z^-1)'] z^-1 dz > > where \Sigma and \Lambda are hermitian matrices, G(z) is complex matrix > valued and analytic inside the unit disc and the integration is along the > unit circle. the function is only defined when G(z) \Lambda G(z^-1)' is > positive definite in the unit circle and tends to infinity when approaching > a value of \Lambda that makes it losing rank. > in some special cases you can then substitute w.l.o.g \lambda with some > linear M(x) where x is a real vector in order to obtain a problem of the > form that i was talking about. > > On Wed, Sep 15, 2010 at 10:16 PM, Sebastian Walter < > sebastian.walter at gmail.com> wrote: > >> well, good luck then. >> >> I'm still curious what the objective and constraint functions of your >> original problem are. >> Would it be possible to post it here? >> >> >> On Wed, Sep 15, 2010 at 10:05 PM, Enrico Avventi wrote: >> >>> i'm aware of SDP solvers but they handle only linear objective functions >>> AFAIK. >>> and the costraints are not the problem. it is just that the function is >>> not defined everywhere. >>> i will experiment by changing the line search methods as i think they are >>> the only >>> part of the methods that needs to be aware of the domain. >>> >>> thanx for the help, i will post my eventual findings. >>> >>> On Wed, Sep 15, 2010 at 6:48 PM, Jason Rennie wrote: >>> >>>> On Tue, Sep 14, 2010 at 9:55 AM, enrico avventi wrote: >>>> >>>>> Some of the routines (fmin_cg comes to mind) wants to check the >>>>> gradient at points where the objective function is infinite. Clearly in such >>>>> cases the gradient is not defined - i.e the calculations fail - and the >>>>> algorithm terminates. >>>> >>>> >>>> IIUC, CG requires that the function is smooth, so you can't use CG for >>>> your problem. I.e. there's nothing wrong with fmin_cg. You really need a >>>> semidefinite programming solver, such as yalmip or sedumi. My experience >>>> from ~5 years ago is that SDP solvers only work on relatively small problems >>>> (1000s of variables). >>>> >>>> http://en.wikipedia.org/wiki/Semidefinite_programming >>>> >>>> Jason >>>> >>>> -- >>>> Jason Rennie >>>> Research Scientist, ITA Software >>>> 617-714-2645 >>>> http://www.itasoftware.com/ >>>> >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Sep 20 10:40:01 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 20 Sep 2010 14:40:01 +0000 (UTC) Subject: [SciPy-User] optimization routines can not handle infinity values References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: Mon, 20 Sep 2010 16:20:24 +0200, Enrico Avventi wrote: [clip] > the problem i stumbled with is that i rely on some fortran code wrapped > with f2py. > i recompiled it after installing the development version of numpy and > scipy but nonetheless > this is what i get: [clip] > RuntimeError: module compiled against ABI version 1000009 but this > version of numpy is 2000000 [clip] > what did i miss? The development version of Numpy (which will become Numpy 2.0) is binary incompatible with extension modules compiled with older versions. In your case, slycot. You'd need to recompile all extension modules that use the Numpy API to fix this. (Also, check which version of numpy you get by 'import numpy; print numpy.__file__') I'd suggest to stick with Numpy 1.5.0 or the 1.5.x branch for developing stuff in Scipy. -- Pauli Virtanen From eavventi at yahoo.it Mon Sep 20 10:46:40 2010 From: eavventi at yahoo.it (Enrico Avventi) Date: Mon, 20 Sep 2010 16:46:40 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: nevermind. i just compiled in the wrong order. it now works although the convergence rate is quite slow in some cases. i will submit the patch at scipy-dev mailing list. On Mon, Sep 20, 2010 at 4:20 PM, Enrico Avventi wrote: > so, updating the situation a bit... > > i wrote a new scalar search method that allows functions that tends to > infinity. > it works quite well on toy examples like > > phi(x) = - d*x - log(1-x) > > or > > phi(x) = -d*x + (1-x)^-k > > i changed the line searh method and fmin_bfgs to detect when a function is > not defined > everywhere and call the appropriate line search method. > > i also added test cases (consisting of the obove functions with d=100 and > random k > between 1 and 5) for checking if the new scalar search returns valid > points - i.e satisfying > wolfe conditions. all tests old and new seems topass. > > the last thing i want to check before submitting the patch is if this works > in my problem > as it is a real-world and high dimensional. so the perfect test case. > unfortunately i couldn't > try it out. > > the problem i stumbled with is that i rely on some fortran code wrapped > with f2py. > i recompiled it after installing the development version of numpy and scipy > but nonetheless > this is what i get: > > In [1]: run graph_test.py > [[ 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0.] > [ 0. 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1.] > [ 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1. 0.] > [ 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1.] > [ 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0.] > [ 0. 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1.] > [ 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1. 0.] > [ 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1.] > [ 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0.] > [ 0. 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1.] > [ 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1. 0.] > [ 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1.]] > check: 4.68754204281 34.9603190301 > --------------------------------------------------------------------------- > RuntimeError Traceback (most recent call last) > > > RuntimeError: module compiled against ABI version 1000009 but this version > of numpy is 2000000 > --------------------------------------------------------------------------- > ImportError Traceback (most recent call last) > > /home/avventi/my_code/python_code/xspectral/graph_test.py in () > 33 print "check:", min(np.linalg.eig(L+dL)[0].real), > max(np.linalg.eig(L+dL)[0].real) > 34 > ---> 35 Sigma = compute_moment(fL,A,B,F,G,H,D) > 36 print "R_0=" > 37 print Sigma[0:m,0:m] > > /home/avventi/my_code/python_code/xspectral/armalib.pyc in > compute_moment(Lambda, A, B, F, G, H, D) > 359 from numpy import zeros, zeros_like, dot, concatenate, shape, > bmat, kron, > eye > > 360 from numpy.linalg import > inv > > --> 361 from slycot import > sb03md > > 362 n1 = > A.shape[0] > > 363 n2 = > F.shape[0] > > > > /usr/lib/python2.6/site-packages/slycot/__init__.pyc in > () > > > 2 > > 3 # Analysis routines (5/40 > wrapped) > > > > ----> 4 from analysis import > ab01nd,ab05md,ab05nd,ab07nd,ab08nd > > > 5 > > 6 # Data analysis routines (0/7 > wrapped) > > > > > > /usr/lib/python2.6/site-packages/slycot/analysis.py in > () > > 20 # MA 02110-1301, > USA. > > > > > 21 > > ---> 22 from slycot import > _wrapper > > > 23 > > 24 def ab01nd(n,m,A,B,jobz='N',tol=0,ldwork=None): > > ImportError: numpy.core.multiarray failed to import > WARNING: Failure executing file: > > what did i miss? > > > On Thu, Sep 16, 2010 at 9:57 AM, Enrico Avventi wrote: > >> sure, no problem. the objective function is >> >> f(\Lambda) = trace(\Sigma \Lambda) - \int_\Pi \log [G(z) \Lambda >> G(z^-1)'] z^-1 dz >> >> where \Sigma and \Lambda are hermitian matrices, G(z) is complex matrix >> valued and analytic inside the unit disc and the integration is along the >> unit circle. the function is only defined when G(z) \Lambda G(z^-1)' is >> positive definite in the unit circle and tends to infinity when approaching >> a value of \Lambda that makes it losing rank. >> in some special cases you can then substitute w.l.o.g \lambda with some >> linear M(x) where x is a real vector in order to obtain a problem of the >> form that i was talking about. >> >> On Wed, Sep 15, 2010 at 10:16 PM, Sebastian Walter < >> sebastian.walter at gmail.com> wrote: >> >>> well, good luck then. >>> >>> I'm still curious what the objective and constraint functions of your >>> original problem are. >>> Would it be possible to post it here? >>> >>> >>> On Wed, Sep 15, 2010 at 10:05 PM, Enrico Avventi wrote: >>> >>>> i'm aware of SDP solvers but they handle only linear objective functions >>>> AFAIK. >>>> and the costraints are not the problem. it is just that the function is >>>> not defined everywhere. >>>> i will experiment by changing the line search methods as i think they >>>> are the only >>>> part of the methods that needs to be aware of the domain. >>>> >>>> thanx for the help, i will post my eventual findings. >>>> >>>> On Wed, Sep 15, 2010 at 6:48 PM, Jason Rennie wrote: >>>> >>>>> On Tue, Sep 14, 2010 at 9:55 AM, enrico avventi wrote: >>>>> >>>>>> Some of the routines (fmin_cg comes to mind) wants to check the >>>>>> gradient at points where the objective function is infinite. Clearly in such >>>>>> cases the gradient is not defined - i.e the calculations fail - and the >>>>>> algorithm terminates. >>>>> >>>>> >>>>> IIUC, CG requires that the function is smooth, so you can't use CG for >>>>> your problem. I.e. there's nothing wrong with fmin_cg. You really need a >>>>> semidefinite programming solver, such as yalmip or sedumi. My experience >>>>> from ~5 years ago is that SDP solvers only work on relatively small problems >>>>> (1000s of variables). >>>>> >>>>> http://en.wikipedia.org/wiki/Semidefinite_programming >>>>> >>>>> Jason >>>>> >>>>> -- >>>>> Jason Rennie >>>>> Research Scientist, ITA Software >>>>> 617-714-2645 >>>>> http://www.itasoftware.com/ >>>>> >>>>> >>>>> _______________________________________________ >>>>> SciPy-User mailing list >>>>> SciPy-User at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Mon Sep 20 18:10:49 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Mon, 20 Sep 2010 17:10:49 -0500 Subject: [SciPy-User] [Matplotlib-users] Sigmoid Curve Fitting In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 3:36 PM, Chris Spencer wrote: > Hi, > > Does Matplotlib/Numpy/Scipy contain the ability to fit a sigmoid curve > to a set of data points? > > Regards, > Chris > I am very curious to know how this is done. I have seen in many Cloud Condensation Nuclei (CCN) counter calibration experiments [e.g. Rose et. al. (2008) - p1156, fig2 http://www.atmos-chem-phys.net/8/1153/2008/acp-8-1153-2008.html] people use sigmoidal fits. Never seen a code is shared so far or how it is obtained. from scipy.optimize import leastsq could be a good point to start. To me this fit is very likely a least square minimization problem. You have your tanh(x) or 1/(1+e^-x) as a sigmoidal function, and you should find the points that will make the error minimal for a given discrete points/measurements. from scipy.optimize import curve_fit might be another way to approach. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Mon Sep 20 18:35:00 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Mon, 20 Sep 2010 17:35:00 -0500 Subject: [SciPy-User] [Matplotlib-users] Sigmoid Curve Fitting In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 3:36 PM, Chris Spencer wrote: > Hi, > > Does Matplotlib/Numpy/Scipy contain the ability to fit a sigmoid curve > to a set of data points? > You could also experiment at zunzun.com. Choose 2D sigmoidal from the 2D function scroll list. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From adamdryan at gmail.com Mon Sep 20 19:20:55 2010 From: adamdryan at gmail.com (Adam Ryan) Date: Mon, 20 Sep 2010 16:20:55 -0700 Subject: [SciPy-User] weighted griddata In-Reply-To: References: Message-ID: To Pauli Virtanen, I'm not sure how to thank you. I just started reading this and it will take me a while to digest but it looks fantastic. Cheers, Adam On Mon, Sep 20, 2010 at 2:26 AM, Pauli Virtanen wrote: > Mon, 20 Sep 2010 01:36:23 +0000, Pauli Virtanen wrote: > [clip] >> ? ? git clone git://github.com/pv/scipy-work.git >> ? ? git checkout feature/interpnd-smooth > > Should be > > ? ?git checkout origin/feature/interpnd-smooth > > or just download it from > > ? ?http://github.com/pv/scipy-work/tree/feature/interpnd-smooth > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From chrisspen at gmail.com Mon Sep 20 21:38:32 2010 From: chrisspen at gmail.com (Chris Spencer) Date: Mon, 20 Sep 2010 21:38:32 -0400 Subject: [SciPy-User] Sigmoid Curve Fitting Message-ID: Hi, Does Scipy contain the ability to fit a sigmoid curve to a set of data points? I found some Numpy code (http://pingswept.org/2009/01/24/least-squares-polynomial-fitting-in-python/) for fitting curves using the least squares method, but it only seems to fit parabolas to my sigmoid data. Regards, Chris From warren.weckesser at enthought.com Mon Sep 20 23:54:44 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 20 Sep 2010 22:54:44 -0500 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: References: Message-ID: <4C982C84.9010001@enthought.com> On 9/20/10 8:38 PM, Chris Spencer wrote: > Hi, > > Does Scipy contain the ability to fit a sigmoid curve to a set of data points? > > I found some Numpy code > (http://pingswept.org/2009/01/24/least-squares-polynomial-fitting-in-python/) > for fitting curves using the least squares method, but it only seems > to fit parabolas to my sigmoid data. > You can use curve_fit (scipy.optimize.curve_fit). Which family of sigmoid functions do you want to use? See http://en.wikipedia.org/wiki/Sigmoid_function for a few possibilities. If, for example, you want to fit the following family to your data: f(x) = 1/(1 + exp(-k*(x-x0))) (which has two parameters, k and x0), you can do something like this: ----- import numpy as np import pylab from scipy.optimize import curve_fit def sigmoid(x, x0, k): y = 1 / (1 + np.exp(-k*(x-x0))) return y xdata = np.array([0.0, 1.0, 3.0, 4.3, 7.0, 8.0, 8.5, 10.0, 12.0]) ydata = np.array([0.01, 0.02, 0.04, 0.11, 0.43, 0.7, 0.89, 0.95, 0.99]) popt, pcov = curve_fit(sigmoid, xdata, ydata) print popt x = np.linspace(-1, 15, 50) y = sigmoid(x, *popt) pylab.plot(xdata, ydata, 'o', label='data') pylab.plot(x,y, label='fit') pylab.ylim(0, 1.05) pylab.legend(loc='best') pylab.show() ----- This script generates the attached plot. Warren -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 23138 bytes Desc: not available URL: From millman at berkeley.edu Tue Sep 21 03:14:38 2010 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 21 Sep 2010 00:14:38 -0700 Subject: [SciPy-User] [ANN] SciPy India 2009 Call for Presentations Message-ID: ========================== SciPy 2010 Call for Papers ========================== The second `SciPy India Conference `_ will be held from December 13th to 18th, 2010 at `IIIT-Hyderabad `_. At this conference, novel applications and breakthroughs made in the pursuit of science using Python are presented. Attended by leading figures from both academia and industry, it is an excellent opportunity to experience the cutting edge of scientific software development. The conference is followed by two days of tutorials and a code sprint, during which community experts provide training on several scientific Python packages. We invite you to take part by submitting a talk abstract on the conference website at: http://scipy.in Talk/Paper Submission ========================== We solicit talks and accompanying papers (either formal academic or magazine-style articles) that discuss topics regarding scientific computing using Python, including applications, teaching, development and research. Papers are included in the peer-reviewed conference proceedings, published online. Please note that submissions primarily aimed at the promotion of a commercial product or service will not be considered. Important Dates ========================== Monday, Oct. 11: Abstracts Due Saturday, Oct. 30: Schedule announced Tuesday, Nov. 30: Proceedings paper submission due Monday-Tuesday, Dec. 13-14: Conference Wednesday-Friday, Dec. 15-17: Tutorials/Sprints Saturday, Dec. 18: Sprints Organizers ========================== * Jarrod Millman, Neuroscience Institute, UC Berkeley, USA (Conference Co-Chair) * Prabhu Ramachandran, Department of Aerospace Engineering, IIT Bombay, India (Conference Co-Chair) * FOSSEE Team From chrisspen at gmail.com Tue Sep 21 09:13:07 2010 From: chrisspen at gmail.com (Chris Spencer) Date: Tue, 21 Sep 2010 09:13:07 -0400 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: <4C982C84.9010001@enthought.com> References: <4C982C84.9010001@enthought.com> Message-ID: Thank you. Since that feature only appears to be in the bleeding-edge version, it took me a while to track-down and compile all the libraries on Ubuntu 10.04, but it works perfectly. Regards, Chris On Mon, Sep 20, 2010 at 11:54 PM, Warren Weckesser wrote: > ?On 9/20/10 8:38 PM, Chris Spencer wrote: >> >> Hi, >> >> Does Scipy contain the ability to fit a sigmoid curve to a set of data >> points? >> >> I found some Numpy code >> >> (http://pingswept.org/2009/01/24/least-squares-polynomial-fitting-in-python/) >> for fitting curves using the least squares method, but it only seems >> to fit parabolas to my sigmoid data. >> > > You can use curve_fit (scipy.optimize.curve_fit). > > Which family of sigmoid functions do you want to use? > See http://en.wikipedia.org/wiki/Sigmoid_function for a few possibilities. > > If, for example, you want to fit the following family to your data: > > f(x) = 1/(1 + exp(-k*(x-x0))) > > (which has two parameters, k and x0), you can do something like this: > > ----- > import numpy as np > import pylab > from scipy.optimize import curve_fit > > def sigmoid(x, x0, k): > ? ?y = 1 / (1 + np.exp(-k*(x-x0))) > ? ?return y > > xdata = np.array([0.0, ? 1.0, ?3.0, 4.3, 7.0, ? 8.0, ? 8.5, 10.0, 12.0]) > ydata = np.array([0.01, 0.02, 0.04, 0.11, 0.43, ?0.7, 0.89, 0.95, 0.99]) > > popt, pcov = curve_fit(sigmoid, xdata, ydata) > print popt > > x = np.linspace(-1, 15, 50) > y = sigmoid(x, *popt) > > > pylab.plot(xdata, ydata, 'o', label='data') > pylab.plot(x,y, label='fit') > pylab.ylim(0, 1.05) > pylab.legend(loc='best') > pylab.show() > ----- > > This script generates the attached plot. > > > Warren > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From chrisspen at gmail.com Tue Sep 21 11:24:09 2010 From: chrisspen at gmail.com (Chris Spencer) Date: Tue, 21 Sep 2010 11:24:09 -0400 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: <4C982C84.9010001@enthought.com> References: <4C982C84.9010001@enthought.com> Message-ID: Is it possible to get it to determine the asymptotes as well? It seems to assume the curve will be bounded between y=0 and 1, whereas my data can have arbitrary limits. I tried changing sigmoid() to: def sigmoid(x, x0, k, a): y = a * 1 / (1 + np.exp(-k*(x-x0))) return y but that only results in a curve of f(x)=0.5. Regards, Chris On Mon, Sep 20, 2010 at 11:54 PM, Warren Weckesser wrote: > ?On 9/20/10 8:38 PM, Chris Spencer wrote: >> >> Hi, >> >> Does Scipy contain the ability to fit a sigmoid curve to a set of data >> points? >> >> I found some Numpy code >> >> (http://pingswept.org/2009/01/24/least-squares-polynomial-fitting-in-python/) >> for fitting curves using the least squares method, but it only seems >> to fit parabolas to my sigmoid data. >> > > You can use curve_fit (scipy.optimize.curve_fit). > > Which family of sigmoid functions do you want to use? > See http://en.wikipedia.org/wiki/Sigmoid_function for a few possibilities. > > If, for example, you want to fit the following family to your data: > > f(x) = 1/(1 + exp(-k*(x-x0))) > > (which has two parameters, k and x0), you can do something like this: > > ----- > import numpy as np > import pylab > from scipy.optimize import curve_fit > > def sigmoid(x, x0, k): > ? ?y = 1 / (1 + np.exp(-k*(x-x0))) > ? ?return y > > xdata = np.array([0.0, ? 1.0, ?3.0, 4.3, 7.0, ? 8.0, ? 8.5, 10.0, 12.0]) > ydata = np.array([0.01, 0.02, 0.04, 0.11, 0.43, ?0.7, 0.89, 0.95, 0.99]) > > popt, pcov = curve_fit(sigmoid, xdata, ydata) > print popt > > x = np.linspace(-1, 15, 50) > y = sigmoid(x, *popt) > > > pylab.plot(xdata, ydata, 'o', label='data') > pylab.plot(x,y, label='fit') > pylab.ylim(0, 1.05) > pylab.legend(loc='best') > pylab.show() > ----- > > This script generates the attached plot. > > > Warren > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From warren.weckesser at enthought.com Tue Sep 21 12:16:40 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 21 Sep 2010 11:16:40 -0500 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: References: <4C982C84.9010001@enthought.com> Message-ID: <4C98DA68.9090408@enthought.com> On 9/21/10 10:24 AM, Chris Spencer wrote: > Is it possible to get it to determine the asymptotes as well? It seems > to assume the curve will be bounded between y=0 and 1, whereas my data > can have arbitrary limits. I tried changing sigmoid() to: > > def sigmoid(x, x0, k, a): > y = a * 1 / (1 + np.exp(-k*(x-x0))) > return y > > but that only results in a curve of f(x)=0.5. > The following is a variation that includes more parameters in the family of sigmoid functions. But bear in mind, I chose this family of functions just as a demonstration of curve_fit. I don't know if it makes sense to use this family for your data. The appropriate family to use depends on the nature of the data. ----- import numpy as np import pylab from scipy.optimize import curve_fit def sigmoid(x, x0, k, a, c): y = a / (1 + np.exp(-k*(x-x0))) + c return y xdata = np.array([0.0, 1.0, 3.0, 4.3, 7.0, 8.0, 8.5, 10.0, 12.0, 14.0]) ydata = np.array([0.11, 0.12, 0.14, 0.21, 0.83, 1.45, 1.78, 1.9, 1.98, 2.02]) popt, pcov = curve_fit(sigmoid, xdata, ydata) print "Fit:" print "x0 =", popt[0] print "k =", popt[1] print "a =", popt[2] print "c =", popt[3] print "Asymptotes are", popt[3], "and", popt[3] + popt[2] x = np.linspace(-1, 15, 50) y = sigmoid(x, *popt) pylab.plot(xdata, ydata, 'o', label='data') pylab.plot(x,y, label='fit') pylab.ylim(0, 2.05) pylab.legend(loc='upper left') pylab.grid(True) pylab.show() ----- Warren > Regards, > Chris > > On Mon, Sep 20, 2010 at 11:54 PM, Warren Weckesser > wrote: >> On 9/20/10 8:38 PM, Chris Spencer wrote: >>> Hi, >>> >>> Does Scipy contain the ability to fit a sigmoid curve to a set of data >>> points? >>> >>> I found some Numpy code >>> >>> (http://pingswept.org/2009/01/24/least-squares-polynomial-fitting-in-python/) >>> for fitting curves using the least squares method, but it only seems >>> to fit parabolas to my sigmoid data. >>> >> You can use curve_fit (scipy.optimize.curve_fit). >> >> Which family of sigmoid functions do you want to use? >> See http://en.wikipedia.org/wiki/Sigmoid_function for a few possibilities. >> >> If, for example, you want to fit the following family to your data: >> >> f(x) = 1/(1 + exp(-k*(x-x0))) >> >> (which has two parameters, k and x0), you can do something like this: >> >> ----- >> import numpy as np >> import pylab >> from scipy.optimize import curve_fit >> >> def sigmoid(x, x0, k): >> y = 1 / (1 + np.exp(-k*(x-x0))) >> return y >> >> xdata = np.array([0.0, 1.0, 3.0, 4.3, 7.0, 8.0, 8.5, 10.0, 12.0]) >> ydata = np.array([0.01, 0.02, 0.04, 0.11, 0.43, 0.7, 0.89, 0.95, 0.99]) >> >> popt, pcov = curve_fit(sigmoid, xdata, ydata) >> print popt >> >> x = np.linspace(-1, 15, 50) >> y = sigmoid(x, *popt) >> >> >> pylab.plot(xdata, ydata, 'o', label='data') >> pylab.plot(x,y, label='fit') >> pylab.ylim(0, 1.05) >> pylab.legend(loc='best') >> pylab.show() >> ----- >> >> This script generates the attached plot. >> >> >> Warren >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From chrisspen at gmail.com Tue Sep 21 13:02:47 2010 From: chrisspen at gmail.com (Chris Spencer) Date: Tue, 21 Sep 2010 13:02:47 -0400 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: <4C98DA68.9090408@enthought.com> References: <4C982C84.9010001@enthought.com> <4C98DA68.9090408@enthought.com> Message-ID: On Tue, Sep 21, 2010 at 12:16 PM, Warren Weckesser wrote: > The following is a variation that includes more parameters in the family > of sigmoid functions. ?But bear in mind, I chose this family of > functions just as a demonstration of curve_fit. ?I don't know if it > makes sense to use this family for your data. ?The appropriate family to > use depends on the nature of the data. I see what you mean. That modification only fits a low-to-high sigmoid, but that's close enough for me to adapt by reversing my data set. Thank you for the excellent example. Regards, Chris From adessler at tamu.edu Tue Sep 21 13:28:27 2010 From: adessler at tamu.edu (Andrew Dessler) Date: Tue, 21 Sep 2010 17:28:27 +0000 (UTC) Subject: [SciPy-User] Help required regding Eigenvectors References: Message-ID: Sudheer Phani james.hut.fi> writes: > > Hello All, > > I have been encountering some strange problems with the calculation of > eigen vectors using python. The problem is with the sign of the eigen > vectors calculated. On Very few occasions the eigen vectors calculated > using python have an opposite sign than the actual eigenvectors, though > the magnitude is correct. > > Can any one of you plz help me in this regard and suggest me some solution > if you have encoutered similar problems . > > Any help in this regd is appreciated > > Thanks in Advace > > regds > Sudheer > I had a similar experience. However, I solved it when I realized that the eig routine in scipy.linalg does not necessarily return the biggest eigenvalue/eigenvector first. It often does, but not always. Once I realized I had to find the biggest one on my own, the problem was solved. From chrisspen at gmail.com Tue Sep 21 14:04:45 2010 From: chrisspen at gmail.com (Chris Spencer) Date: Tue, 21 Sep 2010 14:04:45 -0400 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: References: <4C982C84.9010001@enthought.com> <4C98DA68.9090408@enthought.com> Message-ID: I found this modification allows for the inversion of the estimated sigmoid curve: def sigmoid(x, x0, k, a, c, d): y = 1 / (1 + np.exp(-k*(x-x0))) y = (1 - y)*(1 - d) + y*d y = a * y + c return y Regards, Chris On Tue, Sep 21, 2010 at 1:02 PM, Chris Spencer wrote: > On Tue, Sep 21, 2010 at 12:16 PM, Warren Weckesser > wrote: >> The following is a variation that includes more parameters in the family >> of sigmoid functions. ?But bear in mind, I chose this family of >> functions just as a demonstration of curve_fit. ?I don't know if it >> makes sense to use this family for your data. ?The appropriate family to >> use depends on the nature of the data. > > I see what you mean. That modification only fits a low-to-high > sigmoid, but that's close enough for me to adapt by reversing my data > set. Thank you for the excellent example. > > Regards, > Chris > From warren.weckesser at enthought.com Tue Sep 21 14:21:17 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 21 Sep 2010 13:21:17 -0500 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: References: <4C982C84.9010001@enthought.com> <4C98DA68.9090408@enthought.com> Message-ID: <4C98F79D.20707@enthought.com> On 9/21/10 1:04 PM, Chris Spencer wrote: > I found this modification allows for the inversion of the estimated > sigmoid curve: > > def sigmoid(x, x0, k, a, c, d): > y = 1 / (1 + np.exp(-k*(x-x0))) > y = (1 - y)*(1 - d) + y*d > y = a * y + c > return y > A negative value of k "reverses" the sigmoid shape, so you shouldn't have to define a new function. If you prefer to have k be positive, you could use def sigmoid(x, x0, k, a, c): y = a / (1 + np.exp(k*(x-x0))) + c return y (I changed "-k" to "k".) Warren > Regards, > Chris > > On Tue, Sep 21, 2010 at 1:02 PM, Chris Spencer wrote: >> On Tue, Sep 21, 2010 at 12:16 PM, Warren Weckesser >> wrote: >>> The following is a variation that includes more parameters in the family >>> of sigmoid functions. But bear in mind, I chose this family of >>> functions just as a demonstration of curve_fit. I don't know if it >>> makes sense to use this family for your data. The appropriate family to >>> use depends on the nature of the data. >> I see what you mean. That modification only fits a low-to-high >> sigmoid, but that's close enough for me to adapt by reversing my data >> set. Thank you for the excellent example. >> >> Regards, >> Chris >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From chrisspen at gmail.com Tue Sep 21 14:30:52 2010 From: chrisspen at gmail.com (Chris Spencer) Date: Tue, 21 Sep 2010 14:30:52 -0400 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: <4C98F79D.20707@enthought.com> References: <4C982C84.9010001@enthought.com> <4C98DA68.9090408@enthought.com> <4C98F79D.20707@enthought.com> Message-ID: Right, I noticed that negating k should theoretically have the same effect. However, when I reversed your sample data (i.e. ydata = ydata[::-1]), I was surprised to find that curve_fit gives me f(x)=0.92 instead of a proper sigmoid curve. Adding my parameter seems to work around this problem. Regards, Chris On Tue, Sep 21, 2010 at 2:21 PM, Warren Weckesser wrote: > > On 9/21/10 1:04 PM, Chris Spencer wrote: >> I found this modification allows for the inversion of the estimated >> sigmoid curve: >> >> def sigmoid(x, x0, k, a, c, d): >> ? ? ?y = 1 / (1 + np.exp(-k*(x-x0))) >> ? ? ?y = (1 - y)*(1 - d) + y*d >> ? ? ?y = a * y + c >> ? ? ?return y >> > > > A negative value of k "reverses" the sigmoid shape, so you shouldn't > have to define a new function. ?If you prefer to have k be positive, you > could use > > def sigmoid(x, x0, k, a, c): > ? ? y = a / (1 + np.exp(k*(x-x0))) + c > ? ? return y > > (I changed "-k" to "k".) > > > Warren > >> Regards, >> Chris >> >> On Tue, Sep 21, 2010 at 1:02 PM, Chris Spencer ?wrote: >>> On Tue, Sep 21, 2010 at 12:16 PM, Warren Weckesser >>> ?wrote: >>>> The following is a variation that includes more parameters in the family >>>> of sigmoid functions. ?But bear in mind, I chose this family of >>>> functions just as a demonstration of curve_fit. ?I don't know if it >>>> makes sense to use this family for your data. ?The appropriate family to >>>> use depends on the nature of the data. >>> I see what you mean. That modification only fits a low-to-high >>> sigmoid, but that's close enough for me to adapt by reversing my data >>> set. Thank you for the excellent example. >>> >>> Regards, >>> Chris >>> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From warren.weckesser at enthought.com Tue Sep 21 14:36:57 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 21 Sep 2010 13:36:57 -0500 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: References: <4C982C84.9010001@enthought.com> <4C98DA68.9090408@enthought.com> <4C98F79D.20707@enthought.com> Message-ID: <4C98FB49.6050106@enthought.com> On 9/21/10 1:30 PM, Chris Spencer wrote: > Right, I noticed that negating k should theoretically have the same > effect. However, when I reversed your sample data (i.e. ydata = > ydata[::-1]), I was surprised to find that curve_fit gives me > f(x)=0.92 instead of a proper sigmoid curve. Ah, in that case, it can help to give curve_fit a better initial guess for the parameters. For example, if I use popt, pcov = curve_fit(sigmoid, xdata, ydata, p0=(1.0, -1.0, 1.0, 0.0)) with the original function, it works as expected. Warren > Adding my parameter seems > to work around this problem. > > Regards, > Chris > > On Tue, Sep 21, 2010 at 2:21 PM, Warren Weckesser > wrote: >> On 9/21/10 1:04 PM, Chris Spencer wrote: >>> I found this modification allows for the inversion of the estimated >>> sigmoid curve: >>> >>> def sigmoid(x, x0, k, a, c, d): >>> y = 1 / (1 + np.exp(-k*(x-x0))) >>> y = (1 - y)*(1 - d) + y*d >>> y = a * y + c >>> return y >>> >> >> A negative value of k "reverses" the sigmoid shape, so you shouldn't >> have to define a new function. If you prefer to have k be positive, you >> could use >> >> def sigmoid(x, x0, k, a, c): >> y = a / (1 + np.exp(k*(x-x0))) + c >> return y >> >> (I changed "-k" to "k".) >> >> >> Warren >> >>> Regards, >>> Chris >>> >>> On Tue, Sep 21, 2010 at 1:02 PM, Chris Spencer wrote: >>>> On Tue, Sep 21, 2010 at 12:16 PM, Warren Weckesser >>>> wrote: >>>>> The following is a variation that includes more parameters in the family >>>>> of sigmoid functions. But bear in mind, I chose this family of >>>>> functions just as a demonstration of curve_fit. I don't know if it >>>>> makes sense to use this family for your data. The appropriate family to >>>>> use depends on the nature of the data. >>>> I see what you mean. That modification only fits a low-to-high >>>> sigmoid, but that's close enough for me to adapt by reversing my data >>>> set. Thank you for the excellent example. >>>> >>>> Regards, >>>> Chris >>>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From sturla at molden.no Tue Sep 21 15:29:28 2010 From: sturla at molden.no (Sturla Molden) Date: Tue, 21 Sep 2010 21:29:28 +0200 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: References: Message-ID: > Hi, > > Does Scipy contain the ability to fit a sigmoid curve to a set of data > points? > > I found some Numpy code > (http://pingswept.org/2009/01/24/least-squares-polynomial-fitting-in-python/) > for fitting curves using the least squares method, but it only seems > to fit parabolas to my sigmoid data. scipy.optimize.leastsq can fit any non-linear regression curve using Levenberg-Marquardt. You will have to supply a function that computes the residuals, and optionally a function that returns their Jacobian unless you want it estimated. Sturla From gokhansever at gmail.com Tue Sep 21 16:31:29 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Tue, 21 Sep 2010 15:31:29 -0500 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: References: Message-ID: On Tue, Sep 21, 2010 at 2:29 PM, Sturla Molden wrote: > scipy.optimize.leastsq can fit any non-linear regression curve using > Levenberg-Marquardt. You will have to supply a function that computes the > residuals, and optionally a function that returns their Jacobian unless > you want it estimated. curve_fit is just a specific version of leastsq, right? From their documentation I see that they both employ the same nonlinear regression algorithm. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy at jeremysanders.net Tue Sep 21 17:49:03 2010 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Tue, 21 Sep 2010 22:49:03 +0100 Subject: [SciPy-User] numpy.nansum bug? Message-ID: I've hit this problem with numpy (1.3.0 on x86-64). In [12]: a = numpy.array([numpy.nan]) In [13]: numpy.nansum(a) Out[13]: nan I assume that this is a bug? It's pretty nasty if you assume that nansum always returns a non nan. This is also pretty weird: In [14]: numpy.nansum([]) Out[14]: nan Jeremy From charlesr.harris at gmail.com Tue Sep 21 20:04:27 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 21 Sep 2010 18:04:27 -0600 Subject: [SciPy-User] Help required regding Eigenvectors In-Reply-To: References: Message-ID: On Tue, Sep 21, 2010 at 11:28 AM, Andrew Dessler wrote: > Sudheer Phani james.hut.fi> writes: > > > > > Hello All, > > > > I have been encountering some strange problems with the calculation of > > eigen vectors using python. The problem is with the sign of the eigen > > vectors calculated. On Very few occasions the eigen vectors calculated > > using python have an opposite sign than the actual eigenvectors, though > > the magnitude is correct. > > > > Can any one of you plz help me in this regard and suggest me some > solution > > if you have encoutered similar problems . > > > > Any help in this regd is appreciated > > > > Thanks in Advace > > > > regds > > Sudheer > > > > > I had a similar experience. However, I solved it when I realized that the > eig > routine in scipy.linalg does not necessarily return the biggest > eigenvalue/eigenvector first. It often does, but not always. Once I > realized I > had to find the biggest one on my own, the problem was solved. > > Note that eigen vectors don't *have* a sign and this is the case even if they are normalized. It is always possible to multiply them by +/- 1, or even by a complex number of magnitude 1 if you are working with complex arrays. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Tue Sep 21 21:18:00 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 21 Sep 2010 20:18:00 -0500 Subject: [SciPy-User] Sigmoid Curve Fitting In-Reply-To: References: Message-ID: On Tue, Sep 21, 2010 at 2:29 PM, Sturla Molden wrote: >> Hi, >> >> Does Scipy contain the ability to fit a sigmoid curve to a set of data >> points? >> >> I found some Numpy code >> (http://pingswept.org/2009/01/24/least-squares-polynomial-fitting-in-python/) >> for fitting curves using the least squares method, but it only seems >> to fit parabolas to my sigmoid data. > > scipy.optimize.leastsq can fit any non-linear regression curve using > Levenberg-Marquardt. You will have to supply a function that computes the > residuals, and optionally a function that returns their Jacobian unless > you want it estimated. > > Sturla > Ignoring that you data is the opposite, there are a lot of 'growth curves' such as Richard's (growth) curve or generalized logistic function. http://en.wikipedia.org/wiki/Generalised_logistic_function Also, you can fit a linear model if your model can be linearized. The downside is that the errors are multiplicative rather than additive. For nonlinear models, it usually helps to standardize your data. Bruce From ben.root at ou.edu Tue Sep 21 21:41:05 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 21 Sep 2010 20:41:05 -0500 Subject: [SciPy-User] numpy.nansum bug? In-Reply-To: References: Message-ID: On Tue, Sep 21, 2010 at 4:49 PM, Jeremy Sanders wrote: > I've hit this problem with numpy (1.3.0 on x86-64). > > In [12]: a = numpy.array([numpy.nan]) > > In [13]: numpy.nansum(a) > Out[13]: nan > > I assume that this is a bug? It's pretty nasty if you assume that nansum > always returns a non nan. > > This is also pretty weird: > In [14]: numpy.nansum([]) > Out[14]: nan > > Jeremy > > > I am not certain if this is a bug or not, but I would like to point out the documentation: "Return the sum of array elements over a given axis treating Not a Numbers (NaNs) as zero." So, either there is a bug, or the documentation needs to be clearer. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Sep 21 21:45:29 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 21 Sep 2010 21:45:29 -0400 Subject: [SciPy-User] numpy.nansum bug? In-Reply-To: References: Message-ID: On Tue, Sep 21, 2010 at 9:41 PM, Benjamin Root wrote: > On Tue, Sep 21, 2010 at 4:49 PM, Jeremy Sanders > wrote: >> >> I've hit this problem with numpy (1.3.0 on x86-64). >> >> In [12]: a = numpy.array([numpy.nan]) >> >> In [13]: numpy.nansum(a) >> Out[13]: nan >> >> I assume that this is a bug? It's pretty nasty if you assume that nansum >> always returns a non nan. >> >> This is also pretty weird: >> In [14]: numpy.nansum([]) >> Out[14]: nan >> >> Jeremy >> >> > > I am not certain if this is a bug or not, but I would like to point out the > documentation: > > "Return the sum of array elements over a given axis treating Not a Numbers > (NaNs) as zero." > > So, either there is a bug, or the documentation needs to be clearer. There was a long thread earlier this year where most but not all agreed that this is a bug. (my opinion is on record) Josef > Ben Root > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From matthew.brett at gmail.com Tue Sep 21 21:46:17 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 21 Sep 2010 18:46:17 -0700 Subject: [SciPy-User] numpy.nansum bug? In-Reply-To: References: Message-ID: Hi, On Tue, Sep 21, 2010 at 6:41 PM, Benjamin Root wrote: > On Tue, Sep 21, 2010 at 4:49 PM, Jeremy Sanders > wrote: >> >> I've hit this problem with numpy (1.3.0 on x86-64). >> >> In [12]: a = numpy.array([numpy.nan]) >> >> In [13]: numpy.nansum(a) >> Out[13]: nan >> >> I assume that this is a bug? It's pretty nasty if you assume that nansum >> always returns a non nan. >> >> This is also pretty weird: >> In [14]: numpy.nansum([]) >> Out[14]: nan >> >> Jeremy >> >> > > I am not certain if this is a bug or not, but I would like to point out the > documentation: > > "Return the sum of array elements over a given axis treating Not a Numbers > (NaNs) as zero." > > So, either there is a bug, or the documentation needs to be clearer. Right. And given that: In [2]: np.sum([]) Out[2]: 0.0 I'd vote for both to be bugs, that is: np.nansum([np.nan]) should return 0 and np.nansum([]) should return 0 See you, Matthew From lev at columbia.edu Tue Sep 21 22:14:25 2010 From: lev at columbia.edu (Lev Givon) Date: Tue, 21 Sep 2010 22:14:25 -0400 Subject: [SciPy-User] scikits.cuda - initial public release Message-ID: <20100922021425.GB20128@avicenna.ee.columbia.edu> I recently released a scikit containing wrappers for functions in CUBLAS, CUFFT, and the free-of-charge (i.e., non-premium) version of the CULA library [1], as well as some higher-level numpy-like functions that make use of them. The wrappers are ctypes-based [2] and do not attempt to be exhaustive (primarily because of the research needs of myself and my colleagues). The current release of the scikit is available here: http://pypi.python.org/pypi/scikits.cuda/ Feedback welcome. L.G. [1] http://www.culatools.com [2] and make use of some the ideas in PARRET: http://www.mathcs.emory.edu/~yfan/PARRET/doc/index.html From dswoods at gmail.com Tue Sep 21 20:27:03 2010 From: dswoods at gmail.com (Darryl Woods) Date: Wed, 22 Sep 2010 00:27:03 +0000 (UTC) Subject: [SciPy-User] =?utf-8?q?scikits=2Etimeseries=3A_How_to_define_freq?= =?utf-8?q?uency_of=0915minutes?= References: <4BF13906.9030707@internet.lu> Message-ID: Matt Knox gmail.com> writes: > > > > On 05/05/2010 12:03, Martin Felder wrote: > > > Hi *, > > > > > > just for the record, I'm having the exact same problem as Georges. I > > > read through your discussion from three weeks ago, but I also don't feel > > > up to modifying the C code myself (being a Fortran kind of guy...). > > > > > > I understand implementing custom user-defined frequencies is probably a > > > lot of effort, but maybe it's less troublesome to just add some > > > frequencies often used (=by Georges and me, and hopefully others?) to > > > the currently implemented ones? I'd be extremely happy to have 12h, 6h, > > > 3h, 15min and 10min intervals in addition to the existing ones. > > > > > > If you could point me to the part of the code that would have to be > > > modified for that, maybe I can find someone more apt in C who can > > > implement it. > > > > > > Thanks, > > > Martin > > > Hi All, I know this was posted some time ago now, but just wondering if anyone has created any patches or found a way to use a 15-minute frequency with timeseries. I'm currently using pandas because it supports it, but some of the functions of timeseries would be very useful in my case for dealing with messy input data. Thanks, Darryl From wesmckinn at gmail.com Tue Sep 21 23:43:10 2010 From: wesmckinn at gmail.com (Wes McKinney) Date: Tue, 21 Sep 2010 23:43:10 -0400 Subject: [SciPy-User] scikits.timeseries: How to define frequency of 15minutes In-Reply-To: References: <4BF13906.9030707@internet.lu> Message-ID: On Tue, Sep 21, 2010 at 8:27 PM, Darryl Woods wrote: > Matt Knox gmail.com> writes: > >> > >> > On 05/05/2010 12:03, Martin Felder wrote: >> > > Hi *, >> > > >> > > just for the record, I'm having the exact same problem as Georges. I >> > > read through your discussion from three weeks ago, but I also don't feel >> > > up to modifying the C code myself (being a Fortran kind of guy...). >> > > >> > > I understand implementing custom user-defined frequencies is probably a >> > > lot of effort, but maybe it's less troublesome to just add some >> > > frequencies often used (=by Georges and me, and hopefully others?) to >> > > the currently implemented ones? I'd be extremely happy to have 12h, 6h, >> > > 3h, 15min and 10min intervals in addition to the existing ones. >> > > >> > > If you could point me to the part of the code that would have to be >> > > modified for that, maybe I can find someone more apt in C who can >> > > implement it. >> > > >> > > Thanks, >> > > Martin >> > > > > > Hi All, > > I know this was posted some time ago now, but just wondering if anyone has > created any patches or found a way to use a 15-minute frequency with timeseries. > ?I'm currently using pandas because it supports it, but some of the functions of > timeseries would be very useful in my case for dealing with messy input data. > > Thanks, > Darryl > > > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Hi Darryl, I'm curious which functions in scikits.timeseries you're interested in using-- I'd be willing to implement some new features in pandas if they would be useful. I've stagnated a bit on development in the last few months due to lots of other things taking up my time, but I've picked up development again and hope to get out some new releases in the next couple of months. Cheers, Wes From pgmdevlist at gmail.com Wed Sep 22 02:37:43 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 22 Sep 2010 08:37:43 +0200 Subject: [SciPy-User] scikits.timeseries: How to define frequency of 15minutes In-Reply-To: References: <4BF13906.9030707@internet.lu> Message-ID: <3CC2BB33-A226-48E5-80AE-2AE08E77551A@gmail.com> On Sep 22, 2010, at 2:27 AM, Darryl Woods wrote: > > I know this was posted some time ago now, but just wondering if anyone has > created any patches or found a way to use a 15-minute frequency with timeseries. > I'm currently using pandas because it supports it, but some of the functions of > timeseries would be very useful in my case for dealing with messy input data. AAMOF, yes. You can find it on http://github.com/pierregm/scikits.timeseries-sandbox. Now, you can add a `timestep` to a function like time_series, which lets you define a regular interval between two consecutive events. Because the functionality hasn't been thoroughly tested yet, it's not the in the official branch yet (hence the scikits.timeseries-sandbox) and is not well documented. Last time I checked, it worked well, though. Your feedback will therefore be invaluable. Let me know how it goes. P From jeremy at jeremysanders.net Wed Sep 22 04:36:32 2010 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Wed, 22 Sep 2010 09:36:32 +0100 Subject: [SciPy-User] numpy.nansum bug? References: Message-ID: josef.pktd at gmail.com wrote: > There was a long thread earlier this year where most but not all > agreed that this is a bug. Found the bug here: http://projects.scipy.org/numpy/ticket/1123 Jeremy From stefan at sun.ac.za Wed Sep 22 07:39:11 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 22 Sep 2010 13:39:11 +0200 Subject: [SciPy-User] scikits.cuda - initial public release In-Reply-To: <20100922021425.GB20128@avicenna.ee.columbia.edu> References: <20100922021425.GB20128@avicenna.ee.columbia.edu> Message-ID: Hi Lev On Wed, Sep 22, 2010 at 4:14 AM, Lev Givon wrote: > The current release of the scikit is available here: > http://pypi.python.org/pypi/scikits.cuda/ Your scikit also got picked up by the scikits web app: http://scikits.appspot.com/cuda It's great to see a fairly light-weight GPU library out there. Regards St?fan From flip-per at pimmer.info Wed Sep 22 08:07:57 2010 From: flip-per at pimmer.info (Flipper) Date: Wed, 22 Sep 2010 14:07:57 +0200 Subject: [SciPy-User] Integer dtype comparisons Message-ID: <4C99F19D.30905@pimmer.info> Hello, is there a nice way to check whether an object is an Integer? Including all python and numpy Integers? I tried out subclass, but uint is not a subclass of Int: >>> a = numpy.zeros(1,numpy.uint32) >>> aType = type(a[0]) >>> issubclass(aType,types.IntType) False I tried out enumerating, but numpy seems to slightly change datatypes during some calculations, which complicates comparisons: >>> a = numpy.zeros(1,numpy.uint8) >>> aValue = a[0] >>> b = numpy.zeros(1,numpy.int32) >>> bValue = b[0] >>> result = aValue + bValue >>> type(result) >>> numpy.int32 >>> type(result) == numpy.int32 False thanks, Michael From josef.pktd at gmail.com Wed Sep 22 10:18:57 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 10:18:57 -0400 Subject: [SciPy-User] format for chunked file save and read ? Message-ID: What is the best file format for storing temporary data, for chunked saving and loading, that only uses numpy and scipy? I would like a file format that could be shared cross-platform and across python/numpy versions if needed. usecase: Stata is (optionally) saving all Bootstrap samples to a file so that the same samples will be available if a follow-up analysis is desired/required. We could also just save the seed and redo the same samples which might however not be fast for some models Josef From robert.kern at gmail.com Wed Sep 22 10:36:13 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Sep 2010 09:36:13 -0500 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 09:18, wrote: > What is the best file format for storing temporary data, for chunked > saving and loading, that only uses numpy and scipy? An mmapped .npy file would probably serve the purpose well enough. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From robert.kern at gmail.com Wed Sep 22 10:37:21 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Sep 2010 09:37:21 -0500 Subject: [SciPy-User] Integer dtype comparisons In-Reply-To: <4C99F19D.30905@pimmer.info> References: <4C99F19D.30905@pimmer.info> Message-ID: On Wed, Sep 22, 2010 at 07:07, Flipper wrote: > Hello, > > is there a nice way to check whether an object is an Integer? Including > all python and numpy Integers? > > I tried out subclass, but uint is not a subclass of Int: > >>>> a = numpy.zeros(1,numpy.uint32) >>>> aType = type(a[0]) >>>> issubclass(aType,types.IntType) > False isinstance(a[0], (int, numpy.integer)) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Wed Sep 22 11:03:36 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 11:03:36 -0400 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 10:36 AM, Robert Kern wrote: > On Wed, Sep 22, 2010 at 09:18, ? wrote: >> What is the best file format for storing temporary data, for chunked >> saving and loading, that only uses numpy and scipy? > > An mmapped .npy file would probably serve the purpose well enough. Thanks, I will try. The old stats.models had one example with memmapped temporary storage, where I never managed to figure out how to find and close an open file handle (on Windows) that was not cleaned up on delete or when the instance went out of scope. Josef > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From robert.kern at gmail.com Wed Sep 22 11:12:26 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Sep 2010 10:12:26 -0500 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 10:03, wrote: > On Wed, Sep 22, 2010 at 10:36 AM, Robert Kern wrote: >> On Wed, Sep 22, 2010 at 09:18, ? wrote: >>> What is the best file format for storing temporary data, for chunked >>> saving and loading, that only uses numpy and scipy? >> >> An mmapped .npy file would probably serve the purpose well enough. > > Thanks, I will try. > > The old stats.models had one example with memmapped temporary storage, > where I never managed to figure out how to find and close an open file > handle (on Windows) that was not cleaned up on delete or when the > instance went out of scope. Use numpy.lib.format.open_memmap(). When the mmapped array goes out of scope, the associated file object will also get finalized. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Wed Sep 22 11:41:09 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 11:41:09 -0400 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 11:12 AM, Robert Kern wrote: > On Wed, Sep 22, 2010 at 10:03, ? wrote: >> On Wed, Sep 22, 2010 at 10:36 AM, Robert Kern wrote: >>> On Wed, Sep 22, 2010 at 09:18, ? wrote: >>>> What is the best file format for storing temporary data, for chunked >>>> saving and loading, that only uses numpy and scipy? >>> >>> An mmapped .npy file would probably serve the purpose well enough. >> >> Thanks, I will try. >> >> The old stats.models had one example with memmapped temporary storage, >> where I never managed to figure out how to find and close an open file >> handle (on Windows) that was not cleaned up on delete or when the >> instance went out of scope. > > Use numpy.lib.format.open_memmap(). When the mmapped array goes out of > scope, the associated file object will also get finalized. I will have to experiment more systematically. Now I have several memmapped files in my working directory that windows blocks as accessed, and I'm not able to close them until I shut down the python shell (which I don't want to do since I have other work in it.) Thanks, Josef > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From robert.kern at gmail.com Wed Sep 22 11:52:18 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Sep 2010 10:52:18 -0500 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 10:41, wrote: > On Wed, Sep 22, 2010 at 11:12 AM, Robert Kern wrote: >> On Wed, Sep 22, 2010 at 10:03, ? wrote: >>> On Wed, Sep 22, 2010 at 10:36 AM, Robert Kern wrote: >>>> On Wed, Sep 22, 2010 at 09:18, ? wrote: >>>>> What is the best file format for storing temporary data, for chunked >>>>> saving and loading, that only uses numpy and scipy? >>>> >>>> An mmapped .npy file would probably serve the purpose well enough. >>> >>> Thanks, I will try. >>> >>> The old stats.models had one example with memmapped temporary storage, >>> where I never managed to figure out how to find and close an open file >>> handle (on Windows) that was not cleaned up on delete or when the >>> instance went out of scope. >> >> Use numpy.lib.format.open_memmap(). When the mmapped array goes out of >> scope, the associated file object will also get finalized. > > I will have to experiment more systematically. Now I have several > memmapped files in my working directory that windows blocks as > accessed, and I'm not able to close them until I shut down the python > shell (which I don't want to do since I have other work in it.) Are you using IPython? IPython can hold onto references to many of the objects that you use. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Wed Sep 22 12:02:16 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 12:02:16 -0400 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 11:52 AM, Robert Kern wrote: > On Wed, Sep 22, 2010 at 10:41, ? wrote: >> On Wed, Sep 22, 2010 at 11:12 AM, Robert Kern wrote: >>> On Wed, Sep 22, 2010 at 10:03, ? wrote: >>>> On Wed, Sep 22, 2010 at 10:36 AM, Robert Kern wrote: >>>>> On Wed, Sep 22, 2010 at 09:18, ? wrote: >>>>>> What is the best file format for storing temporary data, for chunked >>>>>> saving and loading, that only uses numpy and scipy? >>>>> >>>>> An mmapped .npy file would probably serve the purpose well enough. >>>> >>>> Thanks, I will try. >>>> >>>> The old stats.models had one example with memmapped temporary storage, >>>> where I never managed to figure out how to find and close an open file >>>> handle (on Windows) that was not cleaned up on delete or when the >>>> instance went out of scope. >>> >>> Use numpy.lib.format.open_memmap(). When the mmapped array goes out of >>> scope, the associated file object will also get finalized. >> >> I will have to experiment more systematically. Now I have several >> memmapped files in my working directory that windows blocks as >> accessed, and I'm not able to close them until I shut down the python >> shell (which I don't want to do since I have other work in it.) > > Are you using IPython? IPython can hold onto references to many of the > objects that you use. I'm back to POI, plain old IDLE Josef > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From njs at pobox.com Wed Sep 22 12:29:38 2010 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 22 Sep 2010 09:29:38 -0700 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 7:18 AM, wrote: > What is the best file format for storing temporary data, for chunked > saving and loading, that only uses numpy and scipy? > I would like a file format that could be shared cross-platform and > across python/numpy versions if needed. Why not just use pickle? Mmap isn't giving you any advantages here that I can see, and pickles are much easier to handle when you want to write things out incrementally. > usecase: Stata is (optionally) saving all Bootstrap samples to a file > so that the same samples will be available if a follow-up analysis is > desired/required. > > We could also just save the seed and redo the same samples which might > however not be fast for some models You should save the seed in any case! For probably most bootstrap purposes, it would work fine to just save the samples themselves in the bootstrap object, or have them as an extra return value. The 'boot' package for R does this. Most bootstrap results don't involve huge amounts of memory. On a more general note, I think APIs that take a filename and store some of their (logical) return values there are somewhat "smelly"[1]. Managing temporary files programmatically is a huge pain, esp. when I'll just need to read the results back out again or whatever. If you're worried about memory use, maybe let the user pass in a callback that will be called with each sample in turn, instead of hard-coding this temporary file thing? [1] http://c2.com/xp/CodeSmell.html Cheers, -- Nathaniel From robert.kern at gmail.com Wed Sep 22 12:40:32 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Sep 2010 11:40:32 -0500 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 11:29, Nathaniel Smith wrote: > On Wed, Sep 22, 2010 at 7:18 AM, ? wrote: >> What is the best file format for storing temporary data, for chunked >> saving and loading, that only uses numpy and scipy? >> I would like a file format that could be shared cross-platform and >> across python/numpy versions if needed. > > Why not just use pickle? Mmap isn't giving you any advantages here > that I can see, and pickles are much easier to handle when you want to > write things out incrementally. Large arrays are not written or read incrementally in a pickle. We have some tricks in order to not duplicate memory, but they don't always work. Using np.save() and np.load() would probably also work fine. For most data, they will use fromfile() and .tofile() so they will not duplicate memory during the process. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Wed Sep 22 13:04:24 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 13:04:24 -0400 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 12:29 PM, Nathaniel Smith wrote: > On Wed, Sep 22, 2010 at 7:18 AM, ? wrote: >> What is the best file format for storing temporary data, for chunked >> saving and loading, that only uses numpy and scipy? >> I would like a file format that could be shared cross-platform and >> across python/numpy versions if needed. > > Why not just use pickle? Mmap isn't giving you any advantages here > that I can see, and pickles are much easier to handle when you want to > write things out incrementally. I don't like pickles much for anything that needs to be stored for more than 5 minutes, because several times I wasn't able to read them anymore after some version or code changes. > >> usecase: Stata is (optionally) saving all Bootstrap samples to a file >> so that the same samples will be available if a follow-up analysis is >> desired/required. >> >> We could also just save the seed and redo the same samples which might >> however not be fast for some models > > You should save the seed in any case! > > For probably most bootstrap purposes, it would work fine to just save > the samples themselves in the bootstrap object, or have them as an > extra return value. The 'boot' package for R does this. Most bootstrap > results don't involve huge amounts of memory. The bootstrap module is still in planning stage. For all current examples of bootstrap, I always throw away the original sample and only keep the summary statistics. But I thought the Stata idea of saving the samples to a file is a good idea, and I won't be needing it in most cases. But there are bootstrap (I think) and simulation based estimators where the same simulated samples have to be used several times. Right now I'm mainly planning ahead, while going through STATA, SAS and SPSS manuals. > > On a more general note, I think APIs that take a filename and store > some of their (logical) return values there are somewhat "smelly"[1]. > Managing temporary files programmatically is a huge pain, esp. when > I'll just need to read the results back out again or whatever. If > you're worried about memory use, maybe let the user pass in a callback > that will be called with each sample in turn, instead of hard-coding > this temporary file thing? Using a callback is a good idea to allow for different storage backends, (I ran into a similar problem for storing datasets downloaded from the internet.), but doesn't remove the decision to pick a default storage format, that I can also use myself. Josef > > [1] http://c2.com/xp/CodeSmell.html > > Cheers, > -- Nathaniel > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From jrennie at gmail.com Wed Sep 22 13:30:49 2010 From: jrennie at gmail.com (Jason Rennie) Date: Wed, 22 Sep 2010 13:30:49 -0400 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 1:04 PM, wrote: > I don't like pickles much for anything that needs to be stored for > more than 5 minutes, because several times I wasn't able to read them > anymore after some version or code changes. The only not-particularly-well-documented issue I've run into with pickle (after using it in a production system where we've gone through a number of "upgrades") is that classes defined at the point of "dump" must still be defined at the point of "load". Also, whether the class is old-style or new-style (object) must not change. We have an upgrade infrastructure with version numbers and upgrade functions to lessen the chance that we do something silly. At least, one must be careful in changing code of classes which are pickled. Cheers, Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 22 13:35:45 2010 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 22 Sep 2010 10:35:45 -0700 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 9:40 AM, Robert Kern wrote: > On Wed, Sep 22, 2010 at 11:29, Nathaniel Smith wrote: >> On Wed, Sep 22, 2010 at 7:18 AM, ? wrote: >>> What is the best file format for storing temporary data, for chunked >>> saving and loading, that only uses numpy and scipy? >>> I would like a file format that could be shared cross-platform and >>> across python/numpy versions if needed. >> >> Why not just use pickle? Mmap isn't giving you any advantages here >> that I can see, and pickles are much easier to handle when you want to >> write things out incrementally. > > Large arrays are not written or read incrementally in a pickle. We > have some tricks in order to not duplicate memory, but they don't > always work. Oh, I see, we're talking past each other. I don't think Josef's problem is how to save a large array where you need to avoid copies; I think the problem is to compute and save one array, then compute and save another array, then compute and save another array, etc. Pickles can handle that sort of incrementality just fine :-). For bootstrapping, if we can construct the whole array in memory even once, then there's no need to save them out to a file at all -- the bootstrap routine can just return that array and let the user decide what they want to do with it! On Wed, Sep 22, 2010 at 10:04 AM, wrote: > I don't like pickles much for anything that needs to be stored for > more than 5 minutes, because several times I wasn't able to read them > anymore after some version or code changes. Sure, if you pickle some object, and then change the in-memory definition of the object, the pickle system won't magically know how to translate the old version of the object into the new version -- I assume that's the issue you ran into? (I often use pickles for quick ad-hoc storage, but restrict myself to built-in types like tuples and dicts for just this reason.) But here you're just talking about ndarray's, where pickle compatibility *is* guaranteed (right?). -- Nathaniel From wesmckinn at gmail.com Wed Sep 22 13:38:22 2010 From: wesmckinn at gmail.com (Wes McKinney) Date: Wed, 22 Sep 2010 13:38:22 -0400 Subject: [SciPy-User] scikits.cuda - initial public release In-Reply-To: References: <20100922021425.GB20128@avicenna.ee.columbia.edu> Message-ID: 2010/9/22 St?fan van der Walt : > Hi Lev > > On Wed, Sep 22, 2010 at 4:14 AM, Lev Givon wrote: >> The current release of the scikit is available here: >> http://pypi.python.org/pypi/scikits.cuda/ > > Your scikit also got picked up by the scikits web app: > > http://scikits.appspot.com/cuda > > It's great to see a fairly light-weight GPU library out there. > > Regards > St?fan > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Very cool-- is the project source repository hosted somewhere public? If not would you be opposed to creating one? - Wes From josef.pktd at gmail.com Wed Sep 22 14:09:05 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 14:09:05 -0400 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 1:35 PM, Nathaniel Smith wrote: > On Wed, Sep 22, 2010 at 9:40 AM, Robert Kern wrote: >> On Wed, Sep 22, 2010 at 11:29, Nathaniel Smith wrote: >>> On Wed, Sep 22, 2010 at 7:18 AM, ? wrote: >>>> What is the best file format for storing temporary data, for chunked >>>> saving and loading, that only uses numpy and scipy? >>>> I would like a file format that could be shared cross-platform and >>>> across python/numpy versions if needed. >>> >>> Why not just use pickle? Mmap isn't giving you any advantages here >>> that I can see, and pickles are much easier to handle when you want to >>> write things out incrementally. >> >> Large arrays are not written or read incrementally in a pickle. We >> have some tricks in order to not duplicate memory, but they don't >> always work. > > Oh, I see, we're talking past each other. I don't think Josef's > problem is how to save a large array where you need to avoid copies; I > think the problem is to compute and save one array, then compute and > save another array, then compute and save another array, etc. Pickles > can handle that sort of incrementality just fine :-). No, Stata appends to the file (as it is described, I don't know what they are doing internally.) If you save to a new file, then several files would need to be pieced together, or the previous data needs to be loaded and saved again. For example, when they use an optimal stopping rule (estimating the error of a given number of bootstrap samples), they have to do it in at least two step, initial number of samples, then update error estimate, then sample more given the new estimate. > > For bootstrapping, if we can construct the whole array in memory even > once, then there's no need to save them out to a file at all -- the > bootstrap routine can just return that array and let the user decide > what they want to do with it! > > On Wed, Sep 22, 2010 at 10:04 AM, ? wrote: >> I don't like pickles much for anything that needs to be stored for >> more than 5 minutes, because several times I wasn't able to read them >> anymore after some version or code changes. > > Sure, if you pickle some object, and then change the in-memory > definition of the object, the pickle system won't magically know how > to translate the old version of the object into the new version -- I > assume that's the issue you ran into? (I often use pickles for quick > ad-hoc storage, but restrict myself to built-in types like tuples and > dicts for just this reason.) But here you're just talking about > ndarray's, where pickle compatibility *is* guaranteed (right?). Yes that would be possible, I got my aversion to pickle after pickling a Pandas dataframe. Are ndarrays guaranteed to be pickle compatible with numpy 2.0? Are all python types guaranteed to be pickle compatible between python 2.5 and 3.x ? Just a preference given my tastes: I like open standards where I don't have to worry about any compatibility questions. My favorite data file format is a csv file. (I lost hdf5 support with numpy 1.4.0) Josef > > -- Nathaniel > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From bsouthey at gmail.com Wed Sep 22 14:29:49 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 22 Sep 2010 13:29:49 -0500 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: <4C9A4B1D.9030909@gmail.com> On 09/22/2010 01:09 PM, josef.pktd at gmail.com wrote: > On Wed, Sep 22, 2010 at 1:35 PM, Nathaniel Smith wrote: >> On Wed, Sep 22, 2010 at 9:40 AM, Robert Kern wrote: >>> On Wed, Sep 22, 2010 at 11:29, Nathaniel Smith wrote: >>>> On Wed, Sep 22, 2010 at 7:18 AM, wrote: >>>>> What is the best file format for storing temporary data, for chunked >>>>> saving and loading, that only uses numpy and scipy? >>>>> I would like a file format that could be shared cross-platform and >>>>> across python/numpy versions if needed. >>>> Why not just use pickle? Mmap isn't giving you any advantages here >>>> that I can see, and pickles are much easier to handle when you want to >>>> write things out incrementally. >>> Large arrays are not written or read incrementally in a pickle. We >>> have some tricks in order to not duplicate memory, but they don't >>> always work. >> Oh, I see, we're talking past each other. I don't think Josef's >> problem is how to save a large array where you need to avoid copies; I >> think the problem is to compute and save one array, then compute and >> save another array, then compute and save another array, etc. Pickles >> can handle that sort of incrementality just fine :-). > No, Stata appends to the file (as it is described, I don't know what > they are doing internally.) > If you save to a new file, then several files would need to be pieced > together, or the previous data needs to be loaded and saved again. > > For example, when they use an optimal stopping rule (estimating the > error of a given number of bootstrap samples), they have to do it in > at least two step, initial number of samples, then update error > estimate, then sample more given the new estimate. With Python2.5+ you do have the option of sqlite. Sure it has various issues but it is builtin. Bruce From jrennie at gmail.com Wed Sep 22 14:40:26 2010 From: jrennie at gmail.com (Jason Rennie) Date: Wed, 22 Sep 2010 14:40:26 -0400 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 2:09 PM, wrote: > Are all python types guaranteed to be pickle compatible between python > 2.5 and 3.x ? > The pickle format is python-version neutral. It sounds like the biggest issue is with name changes: http://peadrop.com/blog/2009/04/05/porting-your-code-to-python-3/ (search for "pickle") Some gymnastics will be required if you are storing objects from modules which were deleted and/or renamed. Sounds like there was an issue with going forward, but it was resolved: http://bugs.python.org/issue6137 Also, since everything is new-style in 3.x, make sure that all classes you pickle are new-style. Allowing old-style classes to be loaded in 3.x was rejected: http://bugs.python.org/issue2572 One major issue was the elimination of str, but it sounds like this is being dealt with in a reasonable manner per this thread: http://www.mail-archive.com/python-3000 at python.org/msg10949.html Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 22 15:05:28 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Sep 2010 14:05:28 -0500 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 13:09, wrote: > Just a preference given my tastes: I like open standards where I don't > have to worry about any compatibility questions. My favorite data file > format is a csv file. (I lost hdf5 support with numpy 1.4.0) What do you mean you "lost hdf5 support with numpy 1.4.0"? In any case, just use the NPY format using np.save() and np.load() and be done with it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Wed Sep 22 16:25:03 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 16:25:03 -0400 Subject: [SciPy-User] format for chunked file save and read ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 3:05 PM, Robert Kern wrote: > On Wed, Sep 22, 2010 at 13:09, ? wrote: > >> Just a preference given my tastes: I like open standards where I don't >> have to worry about any compatibility questions. My favorite data file >> format is a csv file. (I lost hdf5 support with numpy 1.4.0) > > What do you mean you "lost hdf5 support with numpy 1.4.0"? Just that there never was a h5py or pytabels available that was binary compatible. (but that's http://www.goenglish.com/waterunderthebridge.asp) > > In any case, just use the NPY format using np.save() and np.load() and > be done with it. I'm back to `work` Josef > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From senajohns at gmail.com Wed Sep 22 23:43:20 2010 From: senajohns at gmail.com (Sena Johns) Date: Thu, 23 Sep 2010 13:43:20 +1000 Subject: [SciPy-User] Cannot build SciPy : File contains no section headers. Message-ID: Hi, I'm trying to install Numpy and SciPy using this following method : http://www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974 But I get this error : python setup.py config --compiler=intel --fcompiler=intel build_clib --compiler=intel --fcompiler=intel build_ext --compiler=intel --fcompiler=intel install Warning: No configuration returned, assuming unavailable.Traceback (most recent call last): File "setup.py", line 160, in setup_package() File "setup.py", line 152, in setup_package configuration=configuration ) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/core.py", line 152, in setup config = configuration() File "setup.py", line 118, in configuration config.add_subpackage('scipy') File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/setup.py", line 8, in configuration config.add_subpackage('integrate') File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/integrate/setup.py", line 10, in configuration blas_opt = get_info('blas_opt',notfound_action=2) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/system_info.py", line 310, in get_info return cl().get_info(notfound_action) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/system_info.py", line 409, in __init__ self.parse_config_files() File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/system_info.py", line 416, in parse_config_files self.cp.read(self.files) File "/opt/cluster/software/python-2.5.4//lib/python2.5/ConfigParser.py", line 267, in read self._read(fp, filename) File "/opt/cluster/software/python-2.5.4//lib/python2.5/ConfigParser.py", line 462, in _read raise MissingSectionHeaderError(fpname, lineno, line) ConfigParser.MissingSectionHeaderError: File contains no section headers. file: /var/tmp/NUMPY/scipy-0.8.0/site.cfg, line: 57 'library_dirs = /usr/lib/:/usr/lib64/\n' terrawulf:/var/tmp/NUMPY/scipy-0.8.0 # vi site.cfg terrawulf:/var/tmp/NUMPY/scipy-0.8.0 # python setup.py config --compiler=intel --fcompiler=intel build_clib --compiler=intel --fcompiler=intel build_ext --compiler=intel --fcompiler=intel install Warning: No configuration returned, assuming unavailable.Traceback (most recent call last): File "setup.py", line 160, in setup_package() File "setup.py", line 152, in setup_package configuration=configuration ) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/core.py", line 152, in setup config = configuration() File "setup.py", line 118, in configuration config.add_subpackage('scipy') File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/setup.py", line 8, in configuration config.add_subpackage('integrate') File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/integrate/setup.py", line 10, in configuration blas_opt = get_info('blas_opt',notfound_action=2) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/system_info.py", line 310, in get_info return cl().get_info(notfound_action) File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/system_info.py", line 409, in __init__ self.parse_config_files() File "/opt/cluster/software/python-2.5.4/lib/python2.5/site-packages/numpy/distutils/system_info.py", line 416, in parse_config_files self.cp.read(self.files) File "/opt/cluster/software/python-2.5.4//lib/python2.5/ConfigParser.py", line 267, in read self._read(fp, filename) File "/opt/cluster/software/python-2.5.4//lib/python2.5/ConfigParser.py", line 462, in _read raise MissingSectionHeaderError(fpname, lineno, line) ConfigParser.MissingSectionHeaderError: File contains no section headers. file: /var/tmp/NUMPY/scipy-0.8.0/site.cfg, line: 60 ' library_dirs = /opt/intel/mkl/10.0.1.014/lib/64/\n' Do you have any suggestions ? I'm on Suse 10.1. Thanks & regards, Sena -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Thu Sep 23 21:00:24 2010 From: david at silveregg.co.jp (David) Date: Fri, 24 Sep 2010 10:00:24 +0900 Subject: [SciPy-User] Cannot build SciPy : File contains no section headers. In-Reply-To: References: Message-ID: <4C9BF828.1070001@silveregg.co.jp> On 09/23/2010 12:43 PM, Sena Johns wrote: > Hi, > > I'm trying to install Numpy and SciPy using this following method > :http://www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974 Most likely, you have an invalid site.cfg file. Could you paste it ? cheers, David From dav at alum.mit.edu Thu Sep 23 21:48:50 2010 From: dav at alum.mit.edu (Dav Clark) Date: Thu, 23 Sep 2010 18:48:50 -0700 Subject: [SciPy-User] Old meaning of numpy mgrid command Message-ID: <5DF38E2A-DE14-4792-9D54-95EA7A2E590F@alum.mit.edu> Hi all, I have some old code that implemented the equivalent of the make3dgabor matlab command. However, the usage of mgrid seems to have changed. The old way, with some variables subbed in for brevity (hence strange arithmetic with constants): ix, iy, it = np.mgrid[0:1.00001:(1/(10-1.0)), 0:1.00001:(1/(10-1.0)), 0.5] Those first two indices can be converted to the more straightforward complex-number approach, yielding: ix, iy, it = np.mgrid[0:1:10*1j, 0:1:10*1j, 0.5] But the float there gives an error, and it's unclear to me what was intended. Does anyone that knows the old behavior of mgrid know what the 0.5 in the third index position would have done? Thanks! Dav From oliphant at enthought.com Fri Sep 24 00:45:11 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Fri, 24 Sep 2010 00:45:11 -0400 Subject: [SciPy-User] Old meaning of numpy mgrid command In-Reply-To: <5DF38E2A-DE14-4792-9D54-95EA7A2E590F@alum.mit.edu> References: <5DF38E2A-DE14-4792-9D54-95EA7A2E590F@alum.mit.edu> Message-ID: On Sep 23, 2010, at 9:48 PM, Dav Clark wrote: > Hi all, > > I have some old code that implemented the equivalent of the make3dgabor matlab command. However, the usage of mgrid seems to have changed. The old way, with some variables subbed in for brevity (hence strange arithmetic with constants): > > ix, iy, it = np.mgrid[0:1.00001:(1/(10-1.0)), 0:1.00001:(1/(10-1.0)), 0.5] > > Those first two indices can be converted to the more straightforward complex-number approach, yielding: > > ix, iy, it = np.mgrid[0:1:10*1j, 0:1:10*1j, 0.5] > > But the float there gives an error, and it's unclear to me what was intended. Does anyone that knows the old behavior of mgrid know what the 0.5 in the third index position would have done? How old is this code? I don't recall any old behavior of mgrid. What version of NumPy did the previous code used to work with? Note, you can also just write 10j for your endpoints: mgrid[0:1:10j, 0:1:10j] > > Thanks! > Dav > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com From pav at iki.fi Fri Sep 24 05:17:17 2010 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 24 Sep 2010 09:17:17 +0000 (UTC) Subject: [SciPy-User] Old meaning of numpy mgrid command References: <5DF38E2A-DE14-4792-9D54-95EA7A2E590F@alum.mit.edu> Message-ID: Fri, 24 Sep 2010 00:45:11 -0400, Travis Oliphant wrote: [clip] > How old is this code? I don't recall any old behavior of mgrid. > What version of NumPy did the previous code used to work with? The __getitem__ in mgrid has stayed essentially the same since 2002. Seems very likely it has never worked in a different way. -- Pauli Virtanen From icpup at verizon.net Fri Sep 24 09:56:27 2010 From: icpup at verizon.net (Jing) Date: Fri, 24 Sep 2010 09:56:27 -0400 Subject: [SciPy-User] scipy odr testing failed Message-ID: <4C9CAE0B.5030004@verizon.net> I have Lucid 10.04 ubuntu x86_64 and python 2.6.5. After install numpy1.3.0 and scipy 0.7.0 from repository, I run a test on numpy and scipy. The numpy test was OK, but the scipy test was failed on ODR testing. The screen print-out was shown below: >>> scipy.test() Running unit tests for scipy NumPy version 1.3.0 NumPy is installed in /usr/lib/python2.6/dist-packages/numpy SciPy version 0.7.0 SciPy is installed in /usr/lib/python2.6/dist-packages/scipy Python version 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] nose version 0.11.4 ... .........warning: specified build_dir '_bad_path_' does not exist or is not writable. Trying default locations ...warning: specified build_dir '..' does not exist or is not writable. Trying default locations ..warning: specified build_dir '_bad_path_' does not exist or is not writable. Trying default locations ...warning: specified build_dir '..' does not exist or is not writable. Trying default locations .................................................. .................................................. ........................ ================================================== ==================== ERROR: test_implicit (test_odr.TestODR) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/dist-packages/scipy/odr/tests/test_odr.py", line 88, in test_implicit out = implicit_odr.run() File "/usr/lib/python2.6/dist-packages/scipy/odr/odrpack.py", line 1055, in run self.output = Output(apply(odr, args, kwds)) TypeError: y must be a sequence or integer (if model is implicit) ---------------------------------------------------------------------- Ran 2587 tests in 27.774s FAILED (KNOWNFAIL=3, SKIP=13, errors=1) >>> Is scipy broken in this build or do I miss something? Thanks for the help. From hmgaudecker at gmail.com Fri Sep 24 09:56:45 2010 From: hmgaudecker at gmail.com (Hans-Martin v. Gaudecker) Date: Fri, 24 Sep 2010 15:56:45 +0200 Subject: [SciPy-User] Installation of SciPy 0.8.0 for Python 2.7 on MacOS 10.6 Message-ID: <0A4E1644-DAE0-42FC-BA7B-394EBAD6CB29@gmail.com> Hi, I am trying to compile SciPy for use with python.org Python 2.7 on Snow Leopard, but I can't seem to get the architectures to work together correctly (examples all use numscons, the install didn't work with the standard procedure, either). Same behaviour on Core2 Duo MacBook Pro and a Mac Pro with Quad-Core Xeon. Setting the flags to 64-bit as described on the SciPy website leads to the build to break (F77 BLAS not found). The build runs through if I leave them out, but then many tests fail with [...] no suitable image found. Did find [...] mach-o, but wrong architecture. I am lost -- any help would be greatly appreciated. More details below. Best wishes, Hans-Martin gfortran is 4.2.3 for SL from r.resarch.att.com: $ gfortran -v Using built-in specs. Target: i686-apple-darwin8 Configured with: /Builds/unix/gcc/gcc-4.2/configure --prefix=/usr/local --mandir=/share/man --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --build=i686-apple-darwin8 --host=i686-apple-darwin8 --target=i686-apple-darwin8 --enable-languages=fortran Thread model: posix gcc version 4.2.3 Error for: sudo LDFLAGS="-arch x86_64" FFLAGS="-arch x86_64" python setupscons.py scons --jobs=4 install Mkdir("build/scons/scipy/integrate") Checking if gfortran needs dummy main - Failed ! Exception: Could not find F77 BLAS, needed for integrate package: File "~/Downloads/scipy-0.8.0/scipy/integrate/SConstruct", line 2: GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript') File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numscons/core/numpyenv.py", line 135: build_dir = '$build_dir', src_dir = '$src_dir') File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py", line 553: return apply(_SConscript, [self.fs,] + files, subst_kw) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py", line 262: exec _file_ in call_stack[-1].globals File "~/Downloads/scipy-0.8.0/build/scons/scipy/integrate/SConscript", line 15: raise Exception("Could not find F77 BLAS, needed for integrate package") error: Error while executing scons command. See above for more information. If you think it is a problem in numscons, you can also try executing the scons command with --log-level option for more detailed output of what numscons is doing, for example --log-level=0; the lowest the level is, the more detailed the output it. Typical test output after "sudo python setupscons.py scons --jobs=4 install" seemed to work fine: ====================================================================== ERROR: Failure: ImportError (dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): no suitable image found. Did find: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so: mach-o, but wrong architecture) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/loader.py", line 382, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 9, in import vq, hierarchy File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 199, in import _hierarchy_wrap ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): no suitable image found. Did find: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so: mach-o, but wrong architecture From warren.weckesser at enthought.com Fri Sep 24 12:33:40 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Fri, 24 Sep 2010 11:33:40 -0500 Subject: [SciPy-User] scipy odr testing failed In-Reply-To: <4C9CAE0B.5030004@verizon.net> References: <4C9CAE0B.5030004@verizon.net> Message-ID: <4C9CD2E4.4000909@enthought.com> On 9/24/10 8:56 AM, Jing wrote: > I have Lucid 10.04 ubuntu x86_64 and python 2.6.5. After install > numpy1.3.0 and scipy 0.7.0 from repository, I run a test on numpy and > scipy. The numpy test was OK, but the scipy test was failed on ODR > testing. The screen print-out was shown below: > >>> scipy.test() > Running unit tests for scipy > NumPy version 1.3.0 > NumPy is installed in /usr/lib/python2.6/dist-packages/numpy > SciPy version 0.7.0 > SciPy is installed in /usr/lib/python2.6/dist-packages/scipy > Python version 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] > nose version 0.11.4 > ... > .........warning: specified build_dir '_bad_path_' does not exist or is > not writable. Trying default locations > ...warning: specified build_dir '..' does not exist or is not writable. > Trying default locations > ..warning: specified build_dir '_bad_path_' does not exist or is not > writable. Trying default locations > ...warning: specified build_dir '..' does not exist or is not writable. > Trying default locations > .................................................. > .................................................. ........................ > ================================================== ==================== > ERROR: test_implicit (test_odr.TestODR) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib/python2.6/dist-packages/scipy/odr/tests/test_odr.py", > line 88, in test_implicit > out = implicit_odr.run() > File "/usr/lib/python2.6/dist-packages/scipy/odr/odrpack.py", line 1055, > in run > self.output = Output(apply(odr, args, kwds)) > TypeError: y must be a sequence or integer (if model is implicit) > > ---------------------------------------------------------------------- > Ran 2587 tests in 27.774s > > FAILED (KNOWNFAIL=3, SKIP=13, errors=1) > > >>> > > Is scipy broken in this build or do I miss something? Thanks for the help. > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user This problem was fixed after 0.7.0 was released: http://projects.scipy.org/scipy/ticket/829 Warren From ralf.gommers at googlemail.com Fri Sep 24 20:13:19 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 25 Sep 2010 08:13:19 +0800 Subject: [SciPy-User] Installation of SciPy 0.8.0 for Python 2.7 on MacOS 10.6 In-Reply-To: <0A4E1644-DAE0-42FC-BA7B-394EBAD6CB29@gmail.com> References: <0A4E1644-DAE0-42FC-BA7B-394EBAD6CB29@gmail.com> Message-ID: On Fri, Sep 24, 2010 at 9:56 PM, Hans-Martin v. Gaudecker < hmgaudecker at gmail.com> wrote: > Hi, > > I am trying to compile SciPy for use with python.org Python 2.7 on Snow > Leopard, but I can't seem to get the architectures to work together > correctly (examples all use numscons, the install didn't work with the > standard procedure, either). Same behaviour on Core2 Duo MacBook Pro and a > Mac Pro with Quad-Core Xeon. > > Setting the flags to 64-bit as described on the SciPy website leads to the > build to break (F77 BLAS not found). The build runs through if I leave them > out, but then many tests fail with [...] no suitable image found. Did find > [...] mach-o, but wrong architecture. > Which Python are you using? Setting "-arch x86_64" should only be done if you use a home-built 64-bit capable Python or Apple Python, not for the Python from python.org. Are you using numpy 1.5.0? If so, try deleting 'ppc64' from distutils/fcompiler/gnu.py, that may fix the architecture errors. Also, don't use "sudo" to install. Cheers, Ralf > I am lost -- any help would be greatly appreciated. More details below. > > Best wishes, > Hans-Martin > > > gfortran is 4.2.3 for SL from r.resarch.att.com: > > $ gfortran -v > Using built-in specs. > Target: i686-apple-darwin8 > Configured with: /Builds/unix/gcc/gcc-4.2/configure --prefix=/usr/local > --mandir=/share/man --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ > --build=i686-apple-darwin8 --host=i686-apple-darwin8 > --target=i686-apple-darwin8 --enable-languages=fortran > Thread model: posix > gcc version 4.2.3 > > > Error for: sudo LDFLAGS="-arch x86_64" FFLAGS="-arch x86_64" python > setupscons.py scons --jobs=4 install > > Mkdir("build/scons/scipy/integrate") > Checking if gfortran needs dummy main - Failed ! > Exception: Could not find F77 BLAS, needed for integrate package: > File "~/Downloads/scipy-0.8.0/scipy/integrate/SConstruct", line 2: > GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript') > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numscons/core/numpyenv.py", > line 135: > build_dir = '$build_dir', src_dir = '$src_dir') > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py", > line 553: > return apply(_SConscript, [self.fs,] + files, subst_kw) > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py", > line 262: > exec _file_ in call_stack[-1].globals > File "~/Downloads/scipy-0.8.0/build/scons/scipy/integrate/SConscript", > line 15: > raise Exception("Could not find F77 BLAS, needed for integrate package") > error: Error while executing scons command. See above for more information. > If you think it is a problem in numscons, you can also try executing the > scons > command with --log-level option for more detailed output of what numscons > is > doing, for example --log-level=0; the lowest the level is, the more > detailed > the output it. > > > Typical test output after "sudo python setupscons.py scons --jobs=4 > install" seemed to work fine: > > ====================================================================== > ERROR: Failure: ImportError > (dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, > 2): no suitable image found. Did find: > > /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so: > mach-o, but wrong architecture) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/loader.py", > line 382, in loadTestsFromName > addr.filename, addr.module) > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/importer.py", > line 39, in importFromPath > return self.importFromDir(dir_path, fqname) > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/importer.py", > line 86, in importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", > line 9, in > import vq, hierarchy > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", > line 199, in > import _hierarchy_wrap > ImportError: > dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, > 2): no suitable image found. Did find: > > /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so: > mach-o, but wrong architecture > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joses.ho at magd.ox.ac.uk Wed Sep 8 17:54:37 2010 From: joses.ho at magd.ox.ac.uk (Joses Ho) Date: Wed, 08 Sep 2010 21:54:37 -0000 Subject: [SciPy-User] leastsq not converging to optimal solution, fmin won't accept parameters Message-ID: Hi, I am trying to fit to my qPCR data an exponential function where each successive y-value depends on the previous v-value in the following fashion: y[n] = y[n-1] + k ln (1 + (y[n-1]/k) + b where y[n] = current y-value being evaluated and y[n-1] = previous y-value I therefore have three values to fit - y0 (starting value of y), k, and b. I define a residuals function where I compute 20 values of y, given an initial estimate of y0, k and b, and then subtract these from my 20 measured values of y. I have tried to use scipy.optimize.leastsq on the above residuals function, but it doesn't seem to be converging on the correct values of y0,k and b. I have tried scipy.optimize.fmin on the same function, but I get the following error: ...fsim[0] = func(x0) ...ValueError: setting an array element with a sequence. which seems to me that it refuses to accept the x0 array I give to it, consisting of [y0,k,b]. Basically, I am trying to implement the recent method of Boggy and Woolf (PLoS 2010) [doi:10.1371/ journal.pone.0012355] in Python. I am using SciPy 0.8.0b1 and NumPy 1.4.0 on Python 2.6.5 of the Enthought Python Distribution 6.2-2. Any help here would be very much appreciated! Thanks Joses --- This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify the sender immediately. Do not copy or use it for any purpose, or disclose its contents to any other person. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yannis1962 at gmail.com Fri Sep 10 08:46:18 2010 From: yannis1962 at gmail.com (Yannis Haralambous) Date: Fri, 10 Sep 2010 12:46:18 -0000 Subject: [SciPy-User] [SciPy-user] 64 bit on Mac? In-Reply-To: References: <60cc3bb5-ab28-42e6-874c-ef49dd2bf015@d2g2000pra.googlegroups.com> <6595CCDD-785D-448E-AE21-1D184BEF6330@cs.toronto.edu> <3f919b92-d7a6-4a82-9cb7-755172bd4af9@v23g2000pro.googlegroups.com> <3d375d730905221143i55d2869ch27d4a3eee8f17085@mail.gmail.com> <97867327-59d4-4d5a-b53b-24827ff619d6@c7g2000prc.googlegroups.com> <3d375d730905221359s1ec1943chb80d6507678b405f@mail.gmail.com> <622e6568-afa3-4dd1-8414-4f5640b07b45@p21g2000prn.googlegroups.com> <3f4d203f-a8cf-4870-823d-d1fc96d7cb07@f38g2000pra.googlegroups.com> <3d375d730906220939x4fe3c5a5wa06c7ff7f8836f2d@mail.gmail.com> <393a3a2b-df8c-4177-b474-58865cf5e581@w31g2000prd.googlegroups.com> <3d375d730906221333o12a7fe7as1752fc0d6720f48b@mail.gmail.com> Message-ID: <29676452.post@talk.nabble.com> I have the same problem but I don't have a friend with a successfully-installed 64 bit python like you. Is there a possibility you put those files on some server? thanks in advance Yannis keflavich wrote: > > me too: > file /usr/X11/lib/libfreetype.6.dylib > /usr/X11/lib/libfreetype.6.dylib: Mach-O universal binary with 4 > architectures > /usr/X11/lib/libfreetype.6.dylib (for architecture ppc7400): Mach-O > dynamically linked shared library ppc > /usr/X11/lib/libfreetype.6.dylib (for architecture ppc64): Mach-O > 64-bit dynamically linked shared library ppc64 > /usr/X11/lib/libfreetype.6.dylib (for architecture i386): Mach-O > dynamically linked shared library i386 > /usr/X11/lib/libfreetype.6.dylib (for architecture x86_64): Mach-O > 64-bit dynamically linked shared library x86_64 > > I ended up copying over my friend's successfully-installed 64 bit > python and matplotlib imports fine. There's no obvious difference in > the procedures we used. So, my problem has been worked around, but I > didn't find a solution. > > Adam > > On Jun 22, 2:33?pm, Robert Kern wrote: >> On Mon, Jun 22, 2009 at 15:15, Adam wrote: >> > Ah, good point: >> > otool -L ?/Library/Frameworks/Python.framework/Versions/2.6/lib/ >> > python2.6/site-packages/matplotlib/ft2font.so >> > /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- >> > packages/matplotlib/ft2font.so: >> > ? ? ? ?/usr/X11/lib/libfreetype.6.dylib (compatibility version >> > 10.0.0, current version 10.20.0) >> >> Hmm, this is 64-bit on my machine: >> >> $ file /usr/X11/lib/libfreetype.6.dylib >> /usr/X11/lib/libfreetype.6.dylib: Mach-O universal binary with 4 >> architectures >> /usr/X11/lib/libfreetype.6.dylib (for architecture ppc7400): ? ?Mach-O >> dynamically linked shared library ppc >> /usr/X11/lib/libfreetype.6.dylib (for architecture ppc64): ? ? ?Mach-O >> 64-bit dynamically linked shared library ppc64 >> /usr/X11/lib/libfreetype.6.dylib (for architecture i386): ? ? ? Mach-O >> dynamically linked shared library i386 >> /usr/X11/lib/libfreetype.6.dylib (for architecture x86_64): ? ? Mach-O >> 64-bit dynamically linked shared library x86_64 >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma that is made terrible by our own mad attempt to interpret it as >> though it had an underlying truth." >> ? -- Umberto Eco >> _______________________________________________ >> SciPy-user mailing list >> SciPy-u... at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-user > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- View this message in context: http://old.nabble.com/64-bit-on-Mac--tp23101275p29676452.html Sent from the Scipy-User mailing list archive at Nabble.com. From kossnair at web.de Sat Sep 11 04:32:36 2010 From: kossnair at web.de (kossnair at web.de) Date: Sat, 11 Sep 2010 10:32:36 +0200 (CEST) Subject: [SciPy-User] MannWhitneyU test critical values In-Reply-To: References: <1194593408.7820038.1284023670015.JavaMail.fmail@mwmweb069>, Message-ID: <1315441757.6302620.1284193956287.JavaMail.fmail@mwmweb070> An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user ___________________________________________________________ WEB.DE DSL SOMMER-SPECIAL: Surf & Phone Flat 16.000 f?r nur 19,99 €/mtl.!* http://produkte.web.de/go/DSL-Doppel-Flatrate/2 From ahilan.kanagasundaram at student.qut.edu.au Sun Sep 12 21:09:24 2010 From: ahilan.kanagasundaram at student.qut.edu.au (AHILAN KANAGASUNDARAM) Date: Mon, 13 Sep 2010 11:09:24 +1000 Subject: [SciPy-User] need assistance Message-ID: <1CF9E7945C0D304FA45AA41277D319CD02B4EF5280@QUTEXMBX02.qut.edu.au> Hi, I'm working on python and scipy. I need to create block diagonal matrix, When I tried to use below function, it says that there is not block module >>> a=numpy.ones([2,2]) >>> b=numpy.random.rand(3,3) >>> import block >>> c=block.block_diag(a,b) >>> print c How can I fix this problem. Is there any alternative way to create block diagonal matrix. I look forward to hearing from you. Thanks K.Ahilan -------------- next part -------------- An HTML attachment was scrubbed... URL: From schut at sarvision.nl Mon Sep 13 04:57:09 2010 From: schut at sarvision.nl (Vincent Schut) Date: Mon, 13 Sep 2010 10:57:09 +0200 Subject: [SciPy-User] Replace duplicates in a monotonically increasing array with linearly interpolated values. In-Reply-To: <4C8A3215.63BA.009B.1@twdb.state.tx.us> References: <4C8A3215.63BA.009B.1@twdb.state.tx.us> Message-ID: <4C8DE765.1000309@sarvision.nl> On 09/10/2010 08:26 PM, Dharhas Pothina wrote: > Hi, > > I have an monotonically increasing array with duplicates in it. e.g. > > x = np.array([1.0,1.0,1.0,2.0,2.0,3.0,3.0,3.0,4.0,4.0,5.0,5.0,5.0,5.0,6.0]) > > I need a new array of the same size with linearly interpolated values i.e something like > > np.array([1.0, 1.33, 1.67, 2.0, 2.5, 3.0, 3.33, 3.67, 4.0, 4.5, 5.0, 5.25, 5.5, 5.75, 6.0]) > > Is there an efficient way to do this. Right now I'm looping through the array and maintaining position flags of when the value changes and then doing linear interpolation between the start and end flags before resetting the flags moving to the next section. This is pretty slow. > > I realize there will be a problem on how to deal with duplicates at the end of the array. > > thanks > > - dharhas Hmm, something like this should do the trick (not tested): xUnique, xUniqueIndices = scipy.unique(x, return_index=True) # get piecewise linear interpolation xInterpolated = numpy.interp(numpy.arange(len(x), xUniqueIndices, xUnique) note: this will not interpolate duplicates at the beginning/end. You can however adjust the behaviour for those, see the docs for numpy.interp. Vincent. From tonightslastsong at gmail.com Tue Sep 14 14:10:19 2010 From: tonightslastsong at gmail.com (Tim Valenta) Date: Tue, 14 Sep 2010 11:10:19 -0700 Subject: [SciPy-User] scipy.linalg.lstsq Message-ID: Hello all, Longtime Python user here. I'm a computer science guy, went through mostly calculus and discrete math courses-- never much in pure statistics. Which leads me into my dilemma: I'm doing least-square linear regression on a scatter plot of points, where several inputs lead to one output. That is, it's not just one system of x,y coordinates, but instead several parallel systems something more like (x1,y) (x2, y) (x3, y), upwards to (xn, y). I'm attempting to get my company off of a severely broken calculation engine, but throwing my data at linalg.lstsq or stats.linregress isn't just a magic one-step fix to my desired solution. Basically, the company ends up using data from Microsoft Office 2003 Excel, which can do hardly-configurable regression calculation on a matrix of data, each row formatted like y, x1, x2, x3 (to follow my example given above). It spits out all kinds of variables and data, and the company draws on the calculated 'coefficients' to make a pie graph, the coefficients taken as percentages. The Excel coefficient numbers come out as something in the range of (-1,1). Their goal is to take the 2 or 3 largest of the coefficients and look at the corresponding xn value (that is, x1, x2, or x3) to decide which of the x values is most influential in the resulting y. So first: linalg.lstsq gives me back four variables, none of which, but `rank`, do I completely understand. The first returned value, `x`, claims to be a vector, but I'm a little lost on that. If it's a vector, I assume it's projecting from the origin point? But that seems too easy, and likely incorrect, but I've got no `intercept` value to observe. Is this capable of giving me the result I'm trying to chase? Second: stats.linregress gives me much more familiar data, but seems to only compute one (x,y) plot at a time, leading me to conclude that I should do 3 separate computations: linregress with all (x1,y) data points, then again with (x2,y) points, and then again with (x3,y) points. Using this method, I get slopes, intercepts, r-values, etc. Is it then up to me to minimize the r^2 value? Simply my lack of exposure to stats leaves me unsure of this step. Third: Is the procedure my company uses totally flawed? That is, if I want to determine which of x1, x2, and x3 is most influential on y, across all samples, is there a more direct calculation that yields the correct conclusion? Thanks in advance-- I'm eager to wrap my head around this! I just need some direction as I continue to play with scipy. Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From eschow at uci.edu Wed Sep 15 20:52:05 2010 From: eschow at uci.edu (Eric Schow) Date: Wed, 15 Sep 2010 17:52:05 -0700 Subject: [SciPy-User] trouble installing scipy on Mac10.6.4 with Python2.7 Message-ID: <3B13DA19-A94B-4B78-A4FB-C7BB12B18229@uci.edu> Hi all, I am new to Python and NumPy/SciPy, and I'm having trouble getting scipy installed on my machine. I am working on an Intel Mac running OS 10.6.4, and I've recently updated the python installation to 2.7. I have installed NumPy 1.5.0, and I am trying to install scipy 0.8.0, which I downloaded as a tarball from sourceforge. I have gcc 4.2.1 installed (through Xcode), although the build process seems to be calling gcc4.0. I have gfortran, which I just re-installed from http://r.research.att.com/tools/, as suggested in the installation notes that came with scipy. The problem seems to be with gfortran, although I admit I'm a little out of my league as I try to troubleshoot this one. The log file is attached. Finally, I initiated the build with this command: LDFLAGS="-arch x86_64" FFLAGS="-arch x86_64" python setup.py build Any help on this issue will be appreciated. BTW, I would prefer to go with Python 2.7 (as opposed to 2.6, for which there is a .dmg available) because of its purported increased compatibility with Python 3.x. Thanks a mole, Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: scipyBuild.log Type: application/octet-stream Size: 215697 bytes Desc: not available URL: -------------- next part -------------- From enrico.avventi at gmail.com Thu Sep 16 03:59:00 2010 From: enrico.avventi at gmail.com (Enrico Avventi) Date: Thu, 16 Sep 2010 09:59:00 +0200 Subject: [SciPy-User] optimization routines can not handle infinity values In-Reply-To: References: <16213.823.qm@web26703.mail.ukl.yahoo.com> Message-ID: forgot the determinant... f(\Lambda) = trace(\Sigma \Lambda) - \int_\Pi \log \det [G(z) \Lambda G(z^-1)'] z^-1 dz On Thu, Sep 16, 2010 at 9:57 AM, Enrico Avventi wrote: > sure, no problem. the objective function is > > f(\Lambda) = trace(\Sigma \Lambda) - \int_\Pi \log [G(z) \Lambda > G(z^-1)'] z^-1 dz > > where \Sigma and \Lambda are hermitian matrices, G(z) is complex matrix > valued and analytic inside the unit disc and the integration is along the > unit circle. the function is only defined when G(z) \Lambda G(z^-1)' is > positive definite in the unit circle and tends to infinity when approaching > a value of \Lambda that makes it losing rank. > in some special cases you can then substitute w.l.o.g \lambda with some > linear M(x) where x is a real vector in order to obtain a problem of the > form that i was talking about. > > On Wed, Sep 15, 2010 at 10:16 PM, Sebastian Walter < > sebastian.walter at gmail.com> wrote: > >> well, good luck then. >> >> I'm still curious what the objective and constraint functions of your >> original problem are. >> Would it be possible to post it here? >> >> >> On Wed, Sep 15, 2010 at 10:05 PM, Enrico Avventi wrote: >> >>> i'm aware of SDP solvers but they handle only linear objective functions >>> AFAIK. >>> and the costraints are not the problem. it is just that the function is >>> not defined everywhere. >>> i will experiment by changing the line search methods as i think they are >>> the only >>> part of the methods that needs to be aware of the domain. >>> >>> thanx for the help, i will post my eventual findings. >>> >>> On Wed, Sep 15, 2010 at 6:48 PM, Jason Rennie wrote: >>> >>>> On Tue, Sep 14, 2010 at 9:55 AM, enrico avventi wrote: >>>> >>>>> Some of the routines (fmin_cg comes to mind) wants to check the >>>>> gradient at points where the objective function is infinite. Clearly in such >>>>> cases the gradient is not defined - i.e the calculations fail - and the >>>>> algorithm terminates. >>>> >>>> >>>> IIUC, CG requires that the function is smooth, so you can't use CG for >>>> your problem. I.e. there's nothing wrong with fmin_cg. You really need a >>>> semidefinite programming solver, such as yalmip or sedumi. My experience >>>> from ~5 years ago is that SDP solvers only work on relatively small problems >>>> (1000s of variables). >>>> >>>> http://en.wikipedia.org/wiki/Semidefinite_programming >>>> >>>> Jason >>>> >>>> -- >>>> Jason Rennie >>>> Research Scientist, ITA Software >>>> 617-714-2645 >>>> http://www.itasoftware.com/ >>>> >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdavis.webb at gmail.com Tue Sep 21 00:49:33 2010 From: sdavis.webb at gmail.com (Stephen Webb) Date: Tue, 21 Sep 2010 00:49:33 -0400 Subject: [SciPy-User] scipy install issues Message-ID: I recently had to rebuild my entire Python library from scratch after doing some very foolish things. After installing SciPy on my Mac (OS X 10.6) using the Fink scipy-26 version, I had no errors after the install (I had to use mv /usr/library /usr/library.moved to get around a gcc issue while installing) I still cannot import scipy through python26. I can import numpy without a problem. For reference: uname -a : Darwin Macintosh-4.local 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i386 Any help would be greatly appreciated. Best, Stephen Webb sdavis.webb at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at pimmer.info Wed Sep 22 08:04:09 2010 From: michael at pimmer.info (Michael Pimmer) Date: Wed, 22 Sep 2010 14:04:09 +0200 Subject: [SciPy-User] Integer dtype comparisons Message-ID: <4C99F0B9.9040409@pimmer.info> Hello, is there a nice way to check whether an object is an Integer? Including all python and numpy Integers? I tried out subclass, but uint is not a subclass of Int: >>> a = numpy.zeros(1,numpy.uint32) >>> aType = type(a[0]) >>> issubclass(aType,types.IntType) False I tried out enumerating, but numpy seems to slightly change datatypes during some calculations, which complicates comparisons: >>> a = numpy.zeros(1,numpy.uint8) >>> aValue = a[0] >>> b = numpy.zeros(1,numpy.int32) >>> bValue = b[0] >>> result = aValue + bValue >>> type(result) >>> numpy.int32 >>> type(result) == numpy.int32 False thanks, Michael From mail.till at gmx.de Sat Sep 25 13:15:09 2010 From: mail.till at gmx.de (Till Stensitzki) Date: Sat, 25 Sep 2010 19:15:09 +0200 Subject: [SciPy-User] leastsq and cython Message-ID: <4C9E2E1D.6000905@gmx.de> Hello, i have a problem with scipy.optimize.leastsq and cython. To make my fitting procedure faster and cythonized parts of it and got a factor 2 speedup. But if i try to use the cython version with leastsq i get: AttributeError: 'builtin_function_or_method' object has no attribute 'func_name' any suggestions? I am using scipy 0.8 and cython 0.13 on Windows with Python 2.6. From josef.pktd at gmail.com Sat Sep 25 14:35:32 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 25 Sep 2010 14:35:32 -0400 Subject: [SciPy-User] need assistance In-Reply-To: <1CF9E7945C0D304FA45AA41277D319CD02B4EF5280@QUTEXMBX02.qut.edu.au> References: <1CF9E7945C0D304FA45AA41277D319CD02B4EF5280@QUTEXMBX02.qut.edu.au> Message-ID: On Sun, Sep 12, 2010 at 9:09 PM, AHILAN KANAGASUNDARAM wrote: > Hi, > > I?m working on python and scipy. ?I need to create block diagonal matrix, > When I tried to use below function, ?it says that there is not block module > > > >>>> a=numpy.ones([2,2]) > >>>> b=numpy.random.rand(3,3) > >>>> import block > >>>> c=block.block_diag(a,b) > >>>> print c In case you still need it, I don't know why the message shows up today, block_diag is in a recent scipy >>> from scipy.linalg import block_diag >>> block_diag(np.random.randint(1,4,size=(3,3)), np.random.randint(1,2,size=(2,4))) array([[2, 2, 1, 0, 0, 0, 0], [2, 1, 1, 0, 0, 0, 0], [1, 2, 1, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 1, 1, 1, 1]]) Josef > > > > > How can I fix this problem. Is there any alternative way to create block > diagonal matrix. > > > > I look forward to hearing from you. > > > > > > Thanks > > > > > > K.Ahilan > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From josef.pktd at gmail.com Sat Sep 25 15:02:44 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 25 Sep 2010 15:02:44 -0400 Subject: [SciPy-User] scipy.linalg.lstsq In-Reply-To: References: Message-ID: On Tue, Sep 14, 2010 at 2:10 PM, Tim Valenta wrote: > Hello all, > Longtime Python user here. ?I'm a computer science guy, went through mostly > calculus and discrete math courses-- never much in pure statistics. ?Which > leads me into my dilemma: > I'm doing least-square linear regression on a scatter plot of points, where > several inputs lead to one output. ?That is, it's not just one system of x,y > coordinates, but instead several parallel systems something more like (x1,y) > (x2, y) (x3, y), upwards to (xn, y). ?I'm attempting to get my company off > of a severely broken calculation engine, but throwing my data at > linalg.lstsq or stats.linregress isn't just a magic one-step fix to my > desired solution. 11 days delay until it showed up on the list for me. Here are some comments in case they are still relevant. > Basically, the company ends up using data from Microsoft Office 2003 Excel, > which can do hardly-configurable regression calculation on a matrix of data, > each row formatted like y, x1, x2, x3 (to follow my example given above). >From your above description it's not really clear, whether you have different observations or variables. the excel matrix looks more like you have rows of observations and several explanatory variables in the columns > ?It spits out all kinds of variables and data, and the company draws on the > calculated 'coefficients' to make a pie graph, the coefficients taken as > percentages. ?The Excel coefficient numbers come out as something in the > range of (-1,1). ?Their goal is to take the 2 or 3 largest of the > coefficients and look at the corresponding xn value (that is, x1, x2, or x3) > to decide which of the x values is most influential in the resulting y. If the excel coefficients in the range (-1,1), that would be "unusual" for regression coefficients. Do the x1, x2, x3, ... all use the same scale? Otherwise it wouldn't make sense to compare the regression coefficients directly. > So first: > linalg.lstsq gives me back four variables, none of which, but `rank`, do I > completely understand. ?The first returned value, `x`, claims to be a > vector, but I'm a little lost on that. ?If it's a vector, I assume it's > projecting from the origin point? ?But that seems too easy, and likely > incorrect, but I've got no `intercept` value to observe. ?Is this capable of > giving me the result I'm trying to chase? > Second: > stats.linregress gives me much more familiar data, but seems to only compute > one (x,y) plot at a time, leading me to conclude that I should do 3 separate > computations: linregress with all (x1,y) data points, then again with (x2,y) > points, and then again with (x3,y) points. ?Using this method, I get slopes, > intercepts, r-values, etc. ?Is it then up to me to minimize the r^2 value? > ?Simply my lack of exposure to stats leaves me unsure of this step. with linalg.leastsq you would have to do some work to get all desired results. linregress can only have one regressor. I would recommend to use either the OLS class from the scipy cookbook, or, better, use statsmodels. If you really have a lot of explanatory variables, then scikits.learn has the best algorithms to select a reduced number of them. > Third: > Is the procedure my company uses totally flawed? ?That is, if I want to > determine which of x1, x2, and x3 is most influential on y, across all > samples, is there a more direct calculation that yields the correct > conclusion? I don't understand enough of what the selection is actually supposed to do, to make a comment on this. Josef > Thanks in advance-- I'm eager to wrap my head around this! ?I just need some > direction as I continue to play with scipy. > Tim > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From charlesr.harris at gmail.com Sat Sep 25 15:27:38 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 25 Sep 2010 13:27:38 -0600 Subject: [SciPy-User] scipy.linalg.lstsq In-Reply-To: References: Message-ID: On Tue, Sep 14, 2010 at 12:10 PM, Tim Valenta wrote: > Hello all, > > Longtime Python user here. I'm a computer science guy, went through mostly > calculus and discrete math courses-- never much in pure statistics. Which > leads me into my dilemma: > > I'm doing least-square linear regression on a scatter plot of points, where > several inputs lead to one output. That is, it's not just one system of x,y > coordinates, but instead several parallel systems something more like (x1,y) > (x2, y) (x3, y), upwards to (xn, y). I'm attempting to get my company off > of a severely broken calculation engine, but throwing my data at > linalg.lstsq or stats.linregress isn't just a magic one-step fix to my > desired solution. > > Basically, the company ends up using data from Microsoft Office 2003 Excel, > which can do hardly-configurable regression calculation on a matrix of data, > each row formatted like y, x1, x2, x3 (to follow my example given above). > It spits out all kinds of variables and data, and the company draws on the > calculated 'coefficients' to make a pie graph, the coefficients taken as > percentages. The Excel coefficient numbers come out as something in the > range of (-1,1). Their goal is to take the 2 or 3 largest of the > coefficients and look at the corresponding xn value (that is, x1, x2, or x3) > to decide which of the x values is most influential in the resulting y. > > So first: > > linalg.lstsq gives me back four variables, none of which, but `rank`, do I > completely understand. The first returned value, `x`, claims to be a > vector, but I'm a little lost on that. If it's a vector, I assume it's > projecting from the origin point? But that seems too easy, and likely > incorrect, but I've got no `intercept` value to observe. Is this capable of > giving me the result I'm trying to chase? > > Second: > > stats.linregress gives me much more familiar data, but seems to only > compute one (x,y) plot at a time, leading me to conclude that I should do 3 > separate computations: linregress with all (x1,y) data points, then again > with (x2,y) points, and then again with (x3,y) points. Using this method, I > get slopes, intercepts, r-values, etc. Is it then up to me to minimize the > r^2 value? Simply my lack of exposure to stats leaves me unsure of this > step. > > Third: > > Is the procedure my company uses totally flawed? That is, if I want to > determine which of x1, x2, and x3 is most influential on y, across all > samples, is there a more direct calculation that yields the correct > conclusion? > > Thanks in advance-- I'm eager to wrap my head around this! I just need > some direction as I continue to play with scipy. > > I'm not clear on what the excel spreadsheet is doing, but I suspect something like Ax ~= y, where A is a matrix formed from the column values and the x are the coefficients. The least squares solution to this is x, which is indeed a vector, and the columns with the largest corresponding elements in x would contribute the most to the solution. However, there are other possibilities in this situation. It would help if you could supply a small example along with the result. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_baddeley at yahoo.com.au Sat Sep 25 17:03:38 2010 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Sat, 25 Sep 2010 14:03:38 -0700 (PDT) Subject: [SciPy-User] leastsq and cython In-Reply-To: <4C9E2E1D.6000905@gmx.de> References: <4C9E2E1D.6000905@gmx.de> Message-ID: <407519.98277.qm@web113415.mail.gq1.yahoo.com> I've, run into a similar problem, it's due to the fact that native methods (ie ones loaded from c libraries) are not quite the same as pure python function objects & don't possess all the nice attributes. It's been a while, but I seem to remember that this error crops up when leastsq is trying to print some form of status message with the function name in it. I've circumvented the problem by making a thin python wrapper function which just calls my c method. This works, but is not ideal, as you introduce additional function call overhead. If anyone out there knows of a better solution I'd also be interested. cheers, David ----- Original Message ---- From: Till Stensitzki To: scipy-user at scipy.org Sent: Sun, 26 September, 2010 6:15:09 AM Subject: [SciPy-User] leastsq and cython Hello, i have a problem with scipy.optimize.leastsq and cython. To make my fitting procedure faster and cythonized parts of it and got a factor 2 speedup. But if i try to use the cython version with leastsq i get: AttributeError: 'builtin_function_or_method' object has no attribute 'func_name' any suggestions? I am using scipy 0.8 and cython 0.13 on Windows with Python 2.6. _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From charlesr.harris at gmail.com Sat Sep 25 17:27:57 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 25 Sep 2010 15:27:57 -0600 Subject: [SciPy-User] leastsq and cython In-Reply-To: <407519.98277.qm@web113415.mail.gq1.yahoo.com> References: <4C9E2E1D.6000905@gmx.de> <407519.98277.qm@web113415.mail.gq1.yahoo.com> Message-ID: On Sat, Sep 25, 2010 at 3:03 PM, David Baddeley wrote: > I've, run into a similar problem, it's due to the fact that native methods > (ie > ones loaded from c libraries) are not quite the same as pure python > function > objects & don't possess all the nice attributes. It's been a while, but I > seem > to remember that this error crops up when leastsq is trying to print some > form > of status message with the function name in it. I've circumvented the > problem by > making a thin python wrapper function which just calls my c method. This > works, > but is not ideal, as you introduce additional function call overhead. If > anyone > out there knows of a better solution I'd also be interested. > > Maybe it shouldn't try to print the function name? The context of the message might be enough. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Sat Sep 25 17:45:08 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sat, 25 Sep 2010 16:45:08 -0500 Subject: [SciPy-User] leastsq and cython In-Reply-To: References: <4C9E2E1D.6000905@gmx.de> <407519.98277.qm@web113415.mail.gq1.yahoo.com> Message-ID: <4C9E6D64.1070602@enthought.com> On 9/25/10 4:27 PM, Charles R Harris wrote: > > > On Sat, Sep 25, 2010 at 3:03 PM, David Baddeley > > wrote: > > I've, run into a similar problem, it's due to the fact that > native methods (ie > ones loaded from c libraries) are not quite the same as pure > python function > objects & don't possess all the nice attributes. It's been a > while, but I seem > to remember that this error crops up when leastsq is trying to > print some form > of status message with the function name in it. I've circumvented > the problem by > making a thin python wrapper function which just calls my c > method. This works, > but is not ideal, as you introduce additional function call > overhead. If anyone > out there knows of a better solution I'd also be interested. > > > Maybe it shouldn't try to print the function name? The context of the > message might be enough. > The function 'check_func' in minpack.py uses the attribute 'func_name' in an error message. I'm tweaking 'check_func' now so that it first checks for the 'func_name' attribute before trying to use it. Any suggestions for incorporating a cython or ctypes function into a test, so I can include tests of the change? Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Sep 25 19:30:53 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 25 Sep 2010 17:30:53 -0600 Subject: [SciPy-User] leastsq not converging to optimal solution, fmin won't accept parameters In-Reply-To: References: Message-ID: On Wed, Sep 8, 2010 at 3:54 PM, Joses Ho wrote: > Hi, > > I am trying to fit to my qPCR data an exponential function where each > successive y-value depends on the previous v-value in the following fashion: > y[n] = y[n-1] + k ln (1 + (y[n-1]/k) + b > where y[n] = current y-value being evaluated > and y[n-1] = previous y-value > > I therefore have three values to fit - y0 (starting value of y), k, and b. > > I define a residuals function where I compute 20 values of y, given an > initial estimate of y0, k and b, and then subtract these from my 20 measured > values of y. > > I have tried to use scipy.optimize.leastsq on the above residuals function, > but it doesn't seem to be converging on the correct values of y0,k and b. > I have tried scipy.optimize.fmin on the same function, but I get the > following error: > ...fsim[0] = func(x0) > ...ValueError: setting an array element with a sequence. > which seems to me that it refuses to accept the x0 array I give to it, > consisting of [y0,k,b]. > > Basically, I am trying to implement the recent method of Boggy and Woolf > (PLoS 2010) [doi:10.1371/ journal.pone.0012355] in Python. > I am using SciPy 0.8.0b1 and NumPy 1.4.0 on Python 2.6.5 of the Enthought > Python Distribution 6.2-2. > > Any help here would be very much appreciated! > > Can you supply 20 data values so that we can play with the problem? Is it real data or synthesized test data? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pebarrett at gmail.com Sat Sep 25 19:53:05 2010 From: pebarrett at gmail.com (Paul Barrett) Date: Sat, 25 Sep 2010 19:53:05 -0400 Subject: [SciPy-User] Integer dtype comparisons In-Reply-To: <4C99F0B9.9040409@pimmer.info> References: <4C99F0B9.9040409@pimmer.info> Message-ID: Have you tried issubdtype ? >>> numpy.issubdtype(numpy.int32, numpy.int) True On Wed, Sep 22, 2010 at 8:04 AM, Michael Pimmer wrote: > Hello, > > is there a nice way to check whether an object is an Integer? Including > all python and numpy Integers? > > I tried out subclass, but uint is not a subclass of Int: > > ?>>> a = numpy.zeros(1,numpy.uint32) > ?>>> aType = type(a[0]) > ?>>> issubclass(aType,types.IntType) > False > > > I tried out enumerating, but numpy seems to slightly change datatypes > during some calculations, which complicates comparisons: > > ?>>> a = numpy.zeros(1,numpy.uint8) > ?>>> aValue = a[0] > ?>>> b = numpy.zeros(1,numpy.int32) > ?>>> bValue = b[0] > ?>>> result = aValue + bValue > ?>>> type(result) > > ?>>> numpy.int32 > > ?>>> type(result) == numpy.int32 > False > > > thanks, > Michael > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From lorenzo.isella at gmail.com Sat Sep 25 20:49:40 2010 From: lorenzo.isella at gmail.com (Lorenzo Isella) Date: Sun, 26 Sep 2010 02:49:40 +0200 Subject: [SciPy-User] Finding Multiple Zeros of a Function within Interval Message-ID: <4C9E98A4.90501@gmail.com> Dear All, I am looking for an algorithm able to find the (possibly more than one) zeros of a nonlinear function f(x,y,z) for x in [a,b] (maybe bad notation: let us say that y and z act as parameters since the only variable I can really tune is x) which does not require f(a)f(b)<0. I had a look at the optimization module in scipy, but it seems to be that most of the functions would return the first zero they can find and then stop. Any suggestion is really appreciated. Cheers Lorenzo From charlesr.harris at gmail.com Sat Sep 25 21:19:13 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 25 Sep 2010 19:19:13 -0600 Subject: [SciPy-User] Finding Multiple Zeros of a Function within Interval In-Reply-To: <4C9E98A4.90501@gmail.com> References: <4C9E98A4.90501@gmail.com> Message-ID: On Sat, Sep 25, 2010 at 6:49 PM, Lorenzo Isella wrote: > Dear All, > I am looking for an algorithm able to find the (possibly more than one) > zeros of a nonlinear function f(x,y,z) for x in [a,b] (maybe bad > notation: let us say that y and z act as parameters since the only > variable I can really tune is x) which does not require f(a)f(b)<0. > I had a look at the optimization module in scipy, but it seems to be > that most of the functions would return the first zero they can find and > then stop. > Any suggestion is really appreciated. > Cheers > > What does the function look like? One quick way if y and z are given is to fit the function with a Chebyshev series and use the zeros of that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Sat Sep 25 23:04:57 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sat, 25 Sep 2010 22:04:57 -0500 Subject: [SciPy-User] leastsq and cython In-Reply-To: <4C9E6D64.1070602@enthought.com> References: <4C9E2E1D.6000905@gmx.de> <407519.98277.qm@web113415.mail.gq1.yahoo.com> <4C9E6D64.1070602@enthought.com> Message-ID: <4C9EB859.7010100@enthought.com> On 9/25/10 4:45 PM, Warren Weckesser wrote: > On 9/25/10 4:27 PM, Charles R Harris wrote: >> >> >> On Sat, Sep 25, 2010 at 3:03 PM, David Baddeley >> > wrote: >> >> I've, run into a similar problem, it's due to the fact that >> native methods (ie >> ones loaded from c libraries) are not quite the same as pure >> python function >> objects & don't possess all the nice attributes. It's been a >> while, but I seem >> to remember that this error crops up when leastsq is trying to >> print some form >> of status message with the function name in it. I've circumvented >> the problem by >> making a thin python wrapper function which just calls my c >> method. This works, >> but is not ideal, as you introduce additional function call >> overhead. If anyone >> out there knows of a better solution I'd also be interested. >> >> >> Maybe it shouldn't try to print the function name? The context of the >> message might be enough. >> > > > The function 'check_func' in minpack.py uses the attribute 'func_name' > in an error message. I'm tweaking 'check_func' now so that it first > checks for the 'func_name' attribute before trying to use it. > > Any suggestions for incorporating a cython or ctypes function into a > test, so I can include tests of the change? > I fixed the code so that it only uses the 'func_name' attribute if it exists. A callable class also does not have a 'func_name' attribute, so I used that to create tests without having to actually have a cython or ctypes function. Till, the problem only arises if the function that leastsq is calling returns the wrong shape, so if you update to the trunk of scipy, you will still get an exception, but now it should be a bit more appropriate to the problem. For the record, I created a ticket here: http://projects.scipy.org/scipy/ticket/1287 Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Sun Sep 26 05:20:48 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 26 Sep 2010 11:20:48 +0200 Subject: [SciPy-User] leastsq and cython In-Reply-To: <4C9E6D64.1070602@enthought.com> References: <4C9E2E1D.6000905@gmx.de> <407519.98277.qm@web113415.mail.gq1.yahoo.com> <4C9E6D64.1070602@enthought.com> Message-ID: <4C9F1070.4020105@student.matnat.uio.no> Warren Weckesser wrote: > On 9/25/10 4:27 PM, Charles R Harris wrote: >> >> >> On Sat, Sep 25, 2010 at 3:03 PM, David Baddeley >> > wrote: >> >> I've, run into a similar problem, it's due to the fact that >> native methods (ie >> ones loaded from c libraries) are not quite the same as pure >> python function >> objects & don't possess all the nice attributes. It's been a >> while, but I seem >> to remember that this error crops up when leastsq is trying to >> print some form >> of status message with the function name in it. I've circumvented >> the problem by >> making a thin python wrapper function which just calls my c >> method. This works, >> but is not ideal, as you introduce additional function call >> overhead. If anyone >> out there knows of a better solution I'd also be interested. >> >> >> Maybe it shouldn't try to print the function name? The context of the >> message might be enough. >> > > > The function 'check_func' in minpack.py uses the attribute 'func_name' > in an error message. I'm tweaking 'check_func' now so that it first > checks for the 'func_name' attribute before trying to use it. Both Cython and Python functions do have the __name__ attribute though, can't you just use that? Dag Sverre From hmgaudecker at gmail.com Sun Sep 26 08:00:03 2010 From: hmgaudecker at gmail.com (Hans-Martin v. Gaudecker) Date: Sun, 26 Sep 2010 14:00:03 +0200 Subject: [SciPy-User] Installation of SciPy 0.8.0 for Python 2.7 on MacOS 10.6 In-Reply-To: References: Message-ID: On 25 Sep 2010, at 08:13, Ralf Gommers wrote: > Which Python are you using? Setting "-arch x86_64" should only be done if > you use a home-built 64-bit capable Python or Apple Python, not for the > Python from python.org. Okay, I guess I thought that since python.org now provides a 64-bit version alongside the 32-bit one, this was the way to go when it failed without the flags... > Are you using numpy 1.5.0? If so, try deleting 'ppc64' from > distutils/fcompiler/gnu.py, that may fix the architecture errors. Also, > don't use "sudo" to install. Many thanks, I would not have been able to find that out myself... For the record, changing line 257 of that file from: for arch in ["ppc", "i686", "x86_64", "ppc64"]: to: for arch in ["i686", "x86_64"]: did the job for me, together with the standard "python setup.py install". All tests pass: >>> scipy.test() Ran 4405 tests in 54.433s OK (KNOWNFAIL=19, SKIP=39) If anybody is interested: I could not get it to work using numscons. On the MacBook Pro, the "Exception: Could not find F77 BLAS, needed for integrate package:" error showed up again. On the Mac Pro, it compiled, but the architecture error creeped back up. Cheerio, HM From ralf.gommers at googlemail.com Sun Sep 26 11:02:21 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 26 Sep 2010 23:02:21 +0800 Subject: [SciPy-User] scipy install issues In-Reply-To: References: Message-ID: On Tue, Sep 21, 2010 at 12:49 PM, Stephen Webb wrote: > I recently had to rebuild my entire Python library from scratch after doing > some very foolish things. After installing SciPy on my Mac (OS X 10.6) using > the Fink scipy-26 version, I had no errors after the install (I had to use > mv /usr/library /usr/library.moved to get around a gcc issue while > installing) I still cannot import scipy through python26. I can import numpy > without a problem. > > For reference: > > uname -a : > Darwin Macintosh-4.local 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr 23 > 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i386 > > Any help would be greatly appreciated. > Hard to answer this without more info. What error do you get when trying to import scipy? Is the install location on your PYTHONPATH? Why are you using Fink by the way? This is not recommended, just building everything the standard way on top of a Python from python.org will make your life a lot easier probably. Cheers, Ralf > Best, > > Stephen Webb > sdavis.webb at gmail.com > > > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lev at columbia.edu Sun Sep 26 12:15:51 2010 From: lev at columbia.edu (Lev Givon) Date: Sun, 26 Sep 2010 12:15:51 -0400 Subject: [SciPy-User] scikits.cuda - initial public release In-Reply-To: References: <20100922021425.GB20128@avicenna.ee.columbia.edu> Message-ID: <20100926161551.GH20128@avicenna.ee.columbia.edu> Received from Wes McKinney on Wed, Sep 22, 2010 at 01:38:22PM EDT: > 2010/9/22 St?fan van der Walt : > > Hi Lev > > > > On Wed, Sep 22, 2010 at 4:14 AM, Lev Givon wrote: > >> The current release of the scikit is available here: > >> http://pypi.python.org/pypi/scikits.cuda/ > > > > Your scikit also got picked up by the scikits web app: > > > > http://scikits.appspot.com/cuda > > > > It's great to see a fairly light-weight GPU library out there. > > > > Regards > > St?fan > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > Very cool-- is the project source repository hosted somewhere public? > If not would you be opposed to creating one? > > - Wes I'm planning to post it on BitBucket in the near future. L.G. From mail.till at gmx.de Sat Sep 25 18:45:20 2010 From: mail.till at gmx.de (Till Stensitzki) Date: Sat, 25 Sep 2010 22:45:20 +0000 (UTC) Subject: [SciPy-User] leastsq and cython References: <4C9E2E1D.6000905@gmx.de> <407519.98277.qm@web113415.mail.gq1.yahoo.com> <4C9E6D64.1070602@enthought.com> Message-ID: Warren Weckesser enthought.com> writes: > > > On 9/25/10 4:27 PM, Charles R Harris wrote: > > On Sat, Sep 25, 2010 at 3:03 PM, David Baddeley yahoo.com.au> > wrote: > I've, ?run into a similar problem, it's due to the fact that > native methods (ie > ones loaded from c libraries) are not quite the same as pure > python function > objects & don't possess all the nice attributes. It's been > a while, but I seem > to remember that this error crops up when leastsq is trying to > print some form > of status message with the function name in it. I've > circumvented the problem by > making a thin python wrapper function which just calls my c > method. This works, > but is not ideal, as you introduce additional function call > overhead. If anyone > out there knows of a better solution I'd also be interested. > > > > Maybe it shouldn't try to print the function name? The context > of the message might be enough. > > > > > The function 'check_func' in minpack.py uses the attribute > 'func_name' in an error message.? I'm tweaking 'check_func' now so > that it first checks for the 'func_name' attribute before trying to > use it. > Any suggestions for incorporating a cython or ctypes function into a > test, so I can include tests of the change? > Warren > > > _______________________________________________ > SciPy-User mailing list > SciPy-User scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Hmm, i tried to change my check_func too, but it seems the error stems also from c/fortran code. While cython function don't have the func_name attribute, the have the __name__ attribute, maybe it help. While we are at it, i have some questions about leastsq: 1. it seems to use always the double precision versions of lmdif, is there a way to change it? I am using cuda for the calculation of my function values, so no double precision for me. Another reason for using the single precision are memory concerns, as i am fitting around 70 000 data points with 600 parameters, so the size of jacobian is quite big. 2. does it use the same blas/lapack liberays as numpy? 3. does it copy the Jacobian every time, or is there a way to write directly to the right memory? thanks for the fast answers and help, Till From wardefar at iro.umontreal.ca Sun Sep 26 14:01:52 2010 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Sun, 26 Sep 2010 14:01:52 -0400 Subject: [SciPy-User] [SciPy-user] 64 bit on Mac? In-Reply-To: <29676452.post@talk.nabble.com> References: <60cc3bb5-ab28-42e6-874c-ef49dd2bf015@d2g2000pra.googlegroups.com> <6595CCDD-785D-448E-AE21-1D184BEF6330@cs.toronto.edu> <3f919b92-d7a6-4a82-9cb7-755172bd4af9@v23g2000pro.googlegroups.com> <3d375d730905221143i55d2869ch27d4a3eee8f17085@mail.gmail.com> <97867327-59d4-4d5a-b53b-24827ff619d6@c7g2000prc.googlegroups.com> <3d375d730905221359s1ec1943chb80d6507678b405f@mail.gmail.com> <622e6568-afa3-4dd1-8414-4f5640b07b45@p21g2000prn.googlegroups.com> <3f4d203f-a8cf-4870-823d-d1fc96d7cb07@f38g2000pra.googlegroups.com> <3d375d730906220939x4fe3c5a5wa06c7ff7f8836f2d@mail.gmail.com> <393a3a2b-df8c-4177-b474-58865cf5e581@w31g2000prd.googlegroups.com> <3d375d730906221333o12a7fe7as1752fc0d6720f48b@mail.gmail.com> <29676452.post@talk.nabble.com> Message-ID: <1FFEB9BE-A751-44B6-BB06-34BE6E9B843F@iro.umontreal.ca> This isn't the right mailing list for matplotlib discussions, but I've never had trouble using the 'make.osx' Makefile provided in the tarball. Maybe try that? David On 2010-09-10, at 8:46 AM, Yannis Haralambous wrote: > > I have the same problem but I don't have a friend with a > successfully-installed 64 bit > python like you. Is there a possibility you put those files on some server? > thanks in advance > Yannis > > > keflavich wrote: >> >> me too: >> file /usr/X11/lib/libfreetype.6.dylib >> /usr/X11/lib/libfreetype.6.dylib: Mach-O universal binary with 4 >> architectures >> /usr/X11/lib/libfreetype.6.dylib (for architecture ppc7400): Mach-O >> dynamically linked shared library ppc >> /usr/X11/lib/libfreetype.6.dylib (for architecture ppc64): Mach-O >> 64-bit dynamically linked shared library ppc64 >> /usr/X11/lib/libfreetype.6.dylib (for architecture i386): Mach-O >> dynamically linked shared library i386 >> /usr/X11/lib/libfreetype.6.dylib (for architecture x86_64): Mach-O >> 64-bit dynamically linked shared library x86_64 >> >> I ended up copying over my friend's successfully-installed 64 bit >> python and matplotlib imports fine. There's no obvious difference in >> the procedures we used. So, my problem has been worked around, but I >> didn't find a solution. >> >> Adam >> >> On Jun 22, 2:33 pm, Robert Kern wrote: >>> On Mon, Jun 22, 2009 at 15:15, Adam wrote: >>>> Ah, good point: >>>> otool -L /Library/Frameworks/Python.framework/Versions/2.6/lib/ >>>> python2.6/site-packages/matplotlib/ft2font.so >>>> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- >>>> packages/matplotlib/ft2font.so: >>>> /usr/X11/lib/libfreetype.6.dylib (compatibility version >>>> 10.0.0, current version 10.20.0) >>> >>> Hmm, this is 64-bit on my machine: >>> >>> $ file /usr/X11/lib/libfreetype.6.dylib >>> /usr/X11/lib/libfreetype.6.dylib: Mach-O universal binary with 4 >>> architectures >>> /usr/X11/lib/libfreetype.6.dylib (for architecture ppc7400): Mach-O >>> dynamically linked shared library ppc >>> /usr/X11/lib/libfreetype.6.dylib (for architecture ppc64): Mach-O >>> 64-bit dynamically linked shared library ppc64 >>> /usr/X11/lib/libfreetype.6.dylib (for architecture i386): Mach-O >>> dynamically linked shared library i386 >>> /usr/X11/lib/libfreetype.6.dylib (for architecture x86_64): Mach-O >>> 64-bit dynamically linked shared library x86_64 >>> >>> -- >>> Robert Kern >>> >>> "I have come to believe that the whole world is an enigma, a harmless >>> enigma that is made terrible by our own mad attempt to interpret it as >>> though it had an underlying truth." >>> -- Umberto Eco >>> _______________________________________________ >>> SciPy-user mailing list >>> SciPy-u... at scipy.orghttp://mail.scipy.org/mailman/listinfo/scipy-user >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > -- > View this message in context: http://old.nabble.com/64-bit-on-Mac--tp23101275p29676452.html > Sent from the Scipy-User mailing list archive at Nabble.com. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From wesmckinn at gmail.com Sun Sep 26 14:59:17 2010 From: wesmckinn at gmail.com (Wes McKinney) Date: Sun, 26 Sep 2010 14:59:17 -0400 Subject: [SciPy-User] scikits.cuda - initial public release In-Reply-To: <20100926161551.GH20128@avicenna.ee.columbia.edu> References: <20100922021425.GB20128@avicenna.ee.columbia.edu> <20100926161551.GH20128@avicenna.ee.columbia.edu> Message-ID: On Sun, Sep 26, 2010 at 12:15 PM, Lev Givon wrote: > Received from Wes McKinney on Wed, Sep 22, 2010 at 01:38:22PM EDT: >> 2010/9/22 St?fan van der Walt : >> > Hi Lev >> > >> > On Wed, Sep 22, 2010 at 4:14 AM, Lev Givon wrote: >> >> The current release of the scikit is available here: >> >> http://pypi.python.org/pypi/scikits.cuda/ >> > >> > Your scikit also got picked up by the scikits web app: >> > >> > http://scikits.appspot.com/cuda >> > >> > It's great to see a fairly light-weight GPU library out there. >> > >> > Regards >> > St?fan >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> > >> >> Very cool-- is the project source repository hosted somewhere public? >> If not would you be opposed to creating one? >> >> - Wes > > I'm planning to post it on BitBucket in the near future. > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?L.G. > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Excellent. If you have no particular preference I would be +1 for GitHub mainly because NumPy and friends are headed that way (I'm migrating my projects there too to make my life easier) Thanks, Wes From gokhansever at gmail.com Sun Sep 26 15:02:41 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Sun, 26 Sep 2010 14:02:41 -0500 Subject: [SciPy-User] stats.chisquare issues Message-ID: Hello, Consider these examples: I[35]: np.histogram(ydata, bins=6) O[35]: (array([4, 1, 3, 0, 0, 1]), array([ 2.8 , 146.33333333, 289.86666667, 433.4 , 576.93333333, 720.46666667, 864. ])) I[36]: np.histogram(ypred, bins=6) O[36]: (array([4, 2, 2, 0, 0, 1]), array([ 22.08895 , 166.34439167, 310.59983333, 454.855275 , 599.11071667, 743.36615833, 887.6216 ])) I[45]: stats.chisquare([4, 1, 3, 0, 0, 1], [4, 2, 2, 0, 0, 1])--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /home/gsever/Desktop/ in () /usr/lib/python2.6/site-packages/scipy/stats/stats.pyc in chisquare(f_obs, f_exp, ddof) 2516 if f_exp is None: 2517 f_exp = array([np.sum(f_obs,axis=0)/float(k)] * len(f_obs),float) -> 2518 f_exp = f_exp.astype(float) 2519 chisq = np.add.reduce((f_obs-f_exp)**2 / f_exp) 2520 return chisq, chisqprob(chisq, k-1-ddof) AttributeError: 'list' object has no attribute 'astype' Here, I expect any scipy function including chisquare should be able to handle lists??? ############################################ This one throws: I[46]: stats.chisquare(np.array([4, 1, 3, 0, 0, 1]), np.array([4, 2, 2, 0, 0, 1])) O[46]: (nan, nan) again I should be aware since the division has 0 in it. after masking: I[47]: a1 = np.ma.masked_equal([4,1,3,0,0,1], 0) I[48]: a2 = np.ma.masked_equal([4,2,2,0,0,1], 0) Further, I[49]: stats.chisquare(a1, a2) O[49]: (1.0, 0.96256577324729631) I[50]: stats.mstats.chisquare(a1, a2) O[50]: (1.0, 0.80125195690120077) p-values differ, is this expected? Thanks. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Sep 26 16:17:16 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 26 Sep 2010 16:17:16 -0400 Subject: [SciPy-User] stats.chisquare issues In-Reply-To: References: Message-ID: On Sun, Sep 26, 2010 at 3:02 PM, G?khan Sever wrote: > Hello, > Consider these examples: > I[35]: np.histogram(ydata, bins=6) > O[35]: > (array([4, 1, 3, 0, 0, 1]), > ?array([ ? 2.8 ? ? ? , ?146.33333333, ?289.86666667, ?433.4 ? ? ? , > ?? ? ? ?576.93333333, ?720.46666667, ?864. ? ? ? ?])) > I[36]: np.histogram(ypred, bins=6) > O[36]: > (array([4, 2, 2, 0, 0, 1]), > ?array([ ?22.08895 ? , ?166.34439167, ?310.59983333, ?454.855275 ?, > ?? ? ? ?599.11071667, ?743.36615833, ?887.6216 ? ?])) > I[45]: stats.chisquare([4, 1, 3, 0, 0, 1], [4, 2, 2, 0, 0, > 1])--------------------------------------------------------------------------- > AttributeError ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call last) > /home/gsever/Desktop/ in () > /usr/lib/python2.6/site-packages/scipy/stats/stats.pyc in chisquare(f_obs, > f_exp, ddof) > ?? 2516 ? ? if f_exp is None: > ?? 2517 ? ? ? ? f_exp = array([np.sum(f_obs,axis=0)/float(k)] * > len(f_obs),float) > -> 2518 ? ? f_exp = f_exp.astype(float) > ?? 2519 ? ? chisq = np.add.reduce((f_obs-f_exp)**2 / f_exp) > ?? 2520 ? ? return chisq, chisqprob(chisq, k-1-ddof) > AttributeError: 'list' object has no attribute 'astype' > Here, I expect any scipy function including chisquare should be able to > handle lists??? > ############################################ > This one throws: > I[46]: stats.chisquare(np.array([4, 1, 3, 0, 0, 1]), np.array([4, 2, 2, 0, > 0, 1])) > O[46]: (nan, nan) > again I should be aware since the division has 0 in it. > after masking: > I[47]: a1 = np.ma.masked_equal([4,1,3,0,0,1], 0) > I[48]: a2 = np.ma.masked_equal([4,2,2,0,0,1], 0) > Further, > I[49]: stats.chisquare(a1, a2) > O[49]: (1.0, 0.96256577324729631) > I[50]: stats.mstats.chisquare(a1, a2) > O[50]: (1.0, 0.80125195690120077) masking doesn't remove the values, so when you have a masked array, then you should use compressed or similar dropping the zero bins >>> stats.chisquare(np.array([4, 1, 3, 1.]),np.array([4, 2, 2, 1.])) (1.0, 0.80125195690120077) Not accepting list is a bug Returning nans in the case when you expect zero in a bin might be by design. But we need to check this. >>> stats.chisquare(np.array([4, 1, 3, 1.]),np.array([4, 2, 0, 1.])) (inf, nan) >>> stats.chisquare(np.array([4, 0, 3, 1.]),np.array([4, 2, 2, 1.])) (2.5, 0.4752910833430205) For chisquare test the standard recommendation is that you need at least 5 expected observations in each bin, otherwise you should combine bins. Josef > p-values differ, is this expected? > Thanks. > > -- > G?khan > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From ralf.gommers at googlemail.com Sun Sep 26 19:39:45 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 27 Sep 2010 07:39:45 +0800 Subject: [SciPy-User] Installation of SciPy 0.8.0 for Python 2.7 on MacOS 10.6 In-Reply-To: References: Message-ID: On Sun, Sep 26, 2010 at 8:00 PM, Hans-Martin v. Gaudecker < hmgaudecker at gmail.com> wrote: > On 25 Sep 2010, at 08:13, Ralf Gommers > wrote: > > > Which Python are you using? Setting "-arch x86_64" should only be done if > > you use a home-built 64-bit capable Python or Apple Python, not for the > > Python from python.org. > > Okay, I guess I thought that since python.org now provides a 64-bit > version alongside the 32-bit one, this was the way to go when it failed > without the flags... > Sorry, this was my autopilot response - x86_64 should work with Python from python.org (but not ppc64). > > > Are you using numpy 1.5.0? If so, try deleting 'ppc64' from > > distutils/fcompiler/gnu.py, that may fix the architecture errors. Also, > > don't use "sudo" to install. > > Many thanks, I would not have been able to find that out myself... For the > record, changing line 257 of that file from: > > for arch in ["ppc", "i686", "x86_64", "ppc64"]: > > to: > > for arch in ["i686", "x86_64"]: > > did the job for me, together with the standard "python setup.py install". Hmm, I'm afraid I introduced this problem myself when trying to fix http://projects.scipy.org/numpy/ticket/1399. Will try to fix this properly asap. > All tests pass: > > >>> scipy.test() > > Ran 4405 tests in 54.433s > > OK (KNOWNFAIL=19, SKIP=39) > > > > If anybody is interested: I could not get it to work using numscons. On the > MacBook Pro, the "Exception: Could not find F77 BLAS, needed for integrate > package:" error showed up again. On the Mac Pro, it compiled, but the > architecture error creeped back up. > Numscons with 2.7 on OS X has not been properly tested yet, this should be fixed before the next release as well. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Sep 26 19:45:34 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 27 Sep 2010 07:45:34 +0800 Subject: [SciPy-User] trouble installing scipy on Mac10.6.4 with Python2.7 In-Reply-To: <3B13DA19-A94B-4B78-A4FB-C7BB12B18229@uci.edu> References: <3B13DA19-A94B-4B78-A4FB-C7BB12B18229@uci.edu> Message-ID: On Thu, Sep 16, 2010 at 8:52 AM, Eric Schow wrote: > Hi all, > > I am new to Python and NumPy/SciPy, and I'm having trouble getting scipy > installed on my machine. > > I am working on an Intel Mac running OS 10.6.4, and I've recently updated > the python installation to 2.7. I have installed NumPy 1.5.0, and I am > trying to install scipy 0.8.0, which I downloaded as a tarball from > sourceforge. 0.8.0 was not fully working with 2.7, some fixes went into trunk after the release. please use a recent svn checkout. I have gcc 4.2.1 installed (through Xcode), although the build process seems > to be calling gcc4.0. I have gfortran, which I just re-installed from > http://r.research.att.com/tools/, as suggested in the installation notes > that came with scipy. The problem seems to be with gfortran, although I > admit I'm a little out of my league as I try to troubleshoot this one. The > log file is attached. Finally, I initiated the build with this command: > > LDFLAGS="-arch x86_64" FFLAGS="-arch x86_64" python setup.py build > This doesn't extend but overrides the build flags, that's how distutils works. With numscons this should work as advertised (although it looks like numscons has some issue with 2.7 ATM). I've just added a warning that describes this better at http://www.scipy.org/Installing_SciPy/Mac_OS_X, because people keep running into this issue. > Any help on this issue will be appreciated. BTW, I would prefer to go with > Python 2.7 (as opposed to 2.6, for which there is a .dmg available) because > of its purported increased compatibility with Python 3.x. > I'd go with 2.6 for now, that's properly tested and binaries are available. Any 2.7 issues should be worked out soon, but at the moment YMMV. Sorry for the very slow reply. Cheers, Ralf > Thanks a mole, > > Eric > > > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lev at columbia.edu Sun Sep 26 23:03:48 2010 From: lev at columbia.edu (Lev Givon) Date: Sun, 26 Sep 2010 23:03:48 -0400 Subject: [SciPy-User] scikits.cuda - initial public release In-Reply-To: References: <20100922021425.GB20128@avicenna.ee.columbia.edu> <20100926161551.GH20128@avicenna.ee.columbia.edu> Message-ID: <20100927030348.GM20128@avicenna.ee.columbia.edu> Received from Wes McKinney on Sun, Sep 26, 2010 at 02:59:17PM EDT: > On Sun, Sep 26, 2010 at 12:15 PM, Lev Givon wrote: > > Received from Wes McKinney on Wed, Sep 22, 2010 at 01:38:22PM EDT: > >> 2010/9/22 St?fan van der Walt : > >> > Hi Lev > >> > > >> > On Wed, Sep 22, 2010 at 4:14 AM, Lev Givon wrote: > >> >> The current release of the scikit is available here: > >> >> http://pypi.python.org/pypi/scikits.cuda/ > >> > > >> > Your scikit also got picked up by the scikits web app: > >> > > >> > http://scikits.appspot.com/cuda > >> > > >> > It's great to see a fairly light-weight GPU library out there. > >> > > >> > Regards > >> > St?fan > >> > _______________________________________________ > >> > SciPy-User mailing list > >> > SciPy-User at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/scipy-user > >> > > >> > >> Very cool-- is the project source repository hosted somewhere public? > >> If not would you be opposed to creating one? > >> > >> - Wes > > > > I'm planning to post it on BitBucket in the near future. > > > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?L.G. > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > Excellent. If you have no particular preference I would be +1 for > GitHub mainly because NumPy and friends are headed that way (I'm > migrating my projects there too to make my life easier) > > Thanks, > Wes Sure; I was planning on using BitBucket for the simple reason that I and my colleagues have been using Mercurial for our revision control needs for some time, but hg-git [1] seems to have become sufficiently stable to enable me to put my code on GitHub: http://github.com/lebedov/scikits.cuda L.G. [1] http://hg-git.github.com From justinbois at gmail.com Mon Sep 27 02:02:37 2010 From: justinbois at gmail.com (Justin Bois) Date: Sun, 26 Sep 2010 23:02:37 -0700 Subject: [SciPy-User] Speeding up odeint for big problems Message-ID: Hello all, I'm using odeint to solve a system of PDEs using the method of lines. This works well for problems in one dimension where there are about ~10^2 coupled ODEs. In 2D, I have ~10^4, and speed is a major issue. Does anyone have any general tips on ways to optimally use odeint to solve large problems using method of lines? Some useful info/questions about my problem: --The Jacobian is dense, even if I use finite differencing for the spatial derivatives. --Using something like PyDSTool might be difficult to implement because I need many of SciPy's handy functions like FFTs, etc. --I'm worried I might be wasting time by passing a large object as an argument to the RHS function. This is out of convenience because it contains useful information about the problem. Does odeint do type conversions on all attributes/methods of the input argument, or only on what is needed? Would paring down the input save time? Any input is greatly appreciated. I'd really like to avoid having to code this up in C. Best, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From Gregor.Thalhammer at i-med.ac.at Mon Sep 27 09:50:15 2010 From: Gregor.Thalhammer at i-med.ac.at (Gregor Thalhammer) Date: Mon, 27 Sep 2010 15:50:15 +0200 Subject: [SciPy-User] leastsq and cython In-Reply-To: References: <4C9E2E1D.6000905@gmx.de> <407519.98277.qm@web113415.mail.gq1.yahoo.com> <4C9E6D64.1070602@enthought.com> Message-ID: <4CA0A117.9030006@i-med.ac.at> Am 26.09.2010 00:45, schrieb Till Stensitzki: > Warren Weckesser enthought.com> writes: > >> >> On 9/25/10 4:27 PM, Charles R Harris wrote: >> >> On Sat, Sep 25, 2010 at 3:03 PM, David Baddeley > yahoo.com.au> >> wrote: >> I've, run into a similar problem, it's due to the fact that >> native methods (ie >> ones loaded from c libraries) are not quite the same as pure >> python function >> objects& don't possess all the nice attributes. It's been >> a while, but I seem >> to remember that this error crops up when leastsq is trying to >> print some form >> of status message with the function name in it. I've >> circumvented the problem by >> making a thin python wrapper function which just calls my c >> method. This works, >> but is not ideal, as you introduce additional function call >> overhead. If anyone >> out there knows of a better solution I'd also be interested. >> >> >> >> Maybe it shouldn't try to print the function name? The context >> of the message might be enough. >> >> >> >> >> The function 'check_func' in minpack.py uses the attribute >> 'func_name' in an error message. I'm tweaking 'check_func' now so >> that it first checks for the 'func_name' attribute before trying to >> use it. >> Any suggestions for incorporating a cython or ctypes function into a >> test, so I can include tests of the change? >> Warren >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > Hmm, i tried to change my check_func too, but it seems the error stems also from > c/fortran code. While cython function don't have the func_name attribute, the > have the __name__ attribute, maybe it help. While we are at it, i have some > questions about leastsq: > > 1. it seems to use always the double precision versions of lmdif, is there a way > to change it? I am using cuda for the calculation of my function values, so no > double precision for me. Another reason for using the single precision are > memory concerns, as i am fitting around 70 000 data points with 600 parameters, > so the size of jacobian is quite big. I had a similar problem with analyzing images - millions of data points. I ended up in implementing the Levenberg-Marquardt-algorithm (on which leastsq is based) in pure Python, so I could use single precision arithmetic. I can send you the code. > 2. does it use the same blas/lapack liberays as numpy? lmdif uses its own implementation of the QR factorization, which is very memory efficient, but slower than state-of-the-art (multithreaded) lapack implementations. In my case I could achieve a large speed improvement by directly solving the normal equations instead of using the QR factorization (I know, this is less accurate and typically not recommended in literature). > 3. does it copy the Jacobian every time, or is there a way to write directly to > the right memory? Don't know how much of copying takes place, but copying is not the limiting the performance. Gregor > thanks for the fast answers and help, > Till > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Dr. Gregor Thalhammer Sektion f?r Biomedizinische Physik Medizinische Universit?t Innsbruck M?llerstr. 44 A-6020 Innsbruck +43 512 9003 70875 From mpan at cesga.es Mon Sep 27 08:54:18 2010 From: mpan at cesga.es (Miguel Pan Fidalgo) Date: Mon, 27 Sep 2010 14:54:18 +0200 Subject: [SciPy-User] problems configuring scipy (0.8.0) from numpy (1.5.0) in a itanium cluster Message-ID: <201009271454.18949.mpan@cesga.es> Technical information ++++++++++++++++++++++ $ python -c 'import sys;print sys.version' 2.4.2 (#1, Apr 13 2007, 16:14:04) [GCC 4.1.0 (SUSE Linux)] $ module load numpy/1.5.0 $ module list Currently Loaded Modulefiles: 1) icc/11.1.056 3) mkl/10.2.2 5) nose/0.11.2 7) numpy/1.5.0 2) ifort/11.1.056 4) fftw/3.2.2 6) SuiteSparse/3.4.0 $ python -c 'import numpy;print numpy.__version__' 1.5.0 $ uname -a Linux cn002 2.6.16.53-0.8_SFS2.3_0-default #1 SMP Mon Nov 5 14:15:47 CET 2007 ia64 ia64 ia64 GNU/Linux numpy-1.5.0 +++++++++++++++++ I have installed numpy-1.5.0 in our Itanium cluster. This is my site.cfg file: $ cat /opt/cesga/numpy-1.5.0/src_numpy-1.5.0/site.cfg [DEFAULT] library_dirs = /opt/cesga/fftw-3.2.2/lib:/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64:/opt/cesga/SuiteSparse-3.4.0/lib include_dirs = /opt/cesga/fftw-3.2.2/include:/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include:/opt/cesga/SuiteSparse-3.4.0/include # UMFPACK # ------- [amd] amd_libs = amd [umfpack] umfpack_libs = umfpack # FFT libraries # ------------- [fftw] libraries = fftw3 # MKL # ---- [mkl] library_dirs = /opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 lapack_libs = mkl_lapack mkl_libs = mkl_solver_lp64, mkl_intel_lp64, mkl_intel_thread, mkl_i2p, mkl_core These are the hacks: $ diff system_info.py.original system_info.py 818c818 < _lib_mkl = ['mkl','vml','guide'] --- > _lib_mkl = ['mkl','mkl_lapack','guide'] $ diff intelccompiler.py.original intelccompiler.py 28a29 > cc_exe=cc_exe + ' -liomp5 -lpthread -fPIC' scipy-0.8.0 +++++++++++++++ When I try to install scipy-0.8.0, the configuration process fails: $ tar zxvf scipy-0.8.0.tar.gz $ mv scipy-0.8.0 src $ cp -va run.sh src/ $ cd src/ $ cat run.sh #!/bin/bash module load numpy/1.5.0 python setup.py config_fc --fcompiler=intele #install --prefix=/sfs/home/cesga/mpan/scipy-0.8.0 $ ./run.sh Warning: No configuration returned, assuming unavailable.blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] FOUND: libraries = ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] lapack_opt_info: lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] FOUND: libraries = ['mkl_lapack', 'mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] FOUND: libraries = ['mkl_lapack', 'mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] umfpack_info: libraries umfpack not found in /opt/cesga/fftw-3.2.2/lib libraries umfpack not found in /opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 amd_info: libraries amd not found in /opt/cesga/fftw-3.2.2/lib libraries amd not found in /opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 FOUND: libraries = ['amd'] library_dirs = ['/opt/cesga/SuiteSparse-3.4.0/lib'] swig_opts = ['-I/opt/cesga/SuiteSparse-3.4.0/include'] define_macros = [('SCIPY_AMD_H', None)] include_dirs = ['/opt/cesga/SuiteSparse-3.4.0/include'] FOUND: libraries = ['umfpack', 'amd'] library_dirs = ['/opt/cesga/SuiteSparse-3.4.0/lib'] swig_opts = ['-I/opt/cesga/SuiteSparse-3.4.0/include', '- I/opt/cesga/SuiteSparse-3.4.0/include'] define_macros = [('SCIPY_UMFPACK_H', None), ('SCIPY_AMD_H', None)] include_dirs = ['/opt/cesga/SuiteSparse-3.4.0/include'] running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options the warning (first line) implies that BLAS primitives have to be recompiled. At the end the installation ends properly... this is not the point, but scipy doesn't use MKL BLAS primitives. BUT according to numpy... blas_mkl and blas_opt should be the same (like lapack_mkl and lapack_opt): $ module load numpy/1.5.0 $ python Python 2.4.2 (#1, Apr 13 2007, 16:14:04) [GCC 4.1.0 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy.distutils.system_info import get_info >>> get_info('blas_mkl') {'libraries': ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'], 'library_dirs': ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'], 'define_macros': [('SCIPY_MKL_H', None)], 'include_dirs': ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include']} >>> get_info('blas_opt') {'libraries': ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'], 'library_dirs': ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'], 'define_macros': [('SCIPY_MKL_H', None)], 'include_dirs': ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include']} >>> There is no need to recompile BLAS!!!! any idea or suggestion ? WHAT I'm doing wrong ? -- Atentamente, Miguel Pan Fidalgo (Applications Technician) mail: mpan at cesga.es web: http://www.cesga.es Avda. de Vigo s/n 15705, Santiago de Compostela Telf.: +34 981 569810 - Fax: 981 594616 ------------------------------------------------------------------------- From rob.clewley at gmail.com Mon Sep 27 12:28:22 2010 From: rob.clewley at gmail.com (Rob Clewley) Date: Mon, 27 Sep 2010 12:28:22 -0400 Subject: [SciPy-User] Speeding up odeint for big problems In-Reply-To: References: Message-ID: > I'm using odeint to solve a system of PDEs using the method of lines. This > works well for problems in one dimension where there are about ~10^2 coupled > ODEs. In 2D, I have ~10^4, and speed is a major issue. Does anyone have > any general tips on ways to optimally use odeint to solve large problems > using method of lines? Some useful info/questions about my problem: Justin, If you could post some examples of the code that you'd like in the right hand side of your ODEs then it will be easier to give you advice. -Rob From bsouthey at gmail.com Mon Sep 27 15:41:36 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 27 Sep 2010 14:41:36 -0500 Subject: [SciPy-User] stats.chisquare issues In-Reply-To: References: Message-ID: <4CA0F370.2080406@gmail.com> On 09/26/2010 03:17 PM, josef.pktd at gmail.com wrote: > On Sun, Sep 26, 2010 at 3:02 PM, G?khan Sever wrote: >> Hello, >> Consider these examples: >> I[35]: np.histogram(ydata, bins=6) >> O[35]: >> (array([4, 1, 3, 0, 0, 1]), >> array([ 2.8 , 146.33333333, 289.86666667, 433.4 , >> 576.93333333, 720.46666667, 864. ])) >> I[36]: np.histogram(ypred, bins=6) >> O[36]: >> (array([4, 2, 2, 0, 0, 1]), >> array([ 22.08895 , 166.34439167, 310.59983333, 454.855275 , >> 599.11071667, 743.36615833, 887.6216 ])) >> I[45]: stats.chisquare([4, 1, 3, 0, 0, 1], [4, 2, 2, 0, 0, >> 1])--------------------------------------------------------------------------- >> AttributeError Traceback (most recent call last) >> /home/gsever/Desktop/ in() >> /usr/lib/python2.6/site-packages/scipy/stats/stats.pyc in chisquare(f_obs, >> f_exp, ddof) >> 2516 if f_exp is None: >> 2517 f_exp = array([np.sum(f_obs,axis=0)/float(k)] * >> len(f_obs),float) >> -> 2518 f_exp = f_exp.astype(float) >> 2519 chisq = np.add.reduce((f_obs-f_exp)**2 / f_exp) >> 2520 return chisq, chisqprob(chisq, k-1-ddof) >> AttributeError: 'list' object has no attribute 'astype' >> Here, I expect any scipy function including chisquare should be able to >> handle lists??? >> ############################################ >> This one throws: >> I[46]: stats.chisquare(np.array([4, 1, 3, 0, 0, 1]), np.array([4, 2, 2, 0, >> 0, 1])) >> O[46]: (nan, nan) >> again I should be aware since the division has 0 in it. >> after masking: >> I[47]: a1 = np.ma.masked_equal([4,1,3,0,0,1], 0) >> I[48]: a2 = np.ma.masked_equal([4,2,2,0,0,1], 0) >> Further, >> I[49]: stats.chisquare(a1, a2) >> O[49]: (1.0, 0.96256577324729631) >> I[50]: stats.mstats.chisquare(a1, a2) >> O[50]: (1.0, 0.80125195690120077) > masking doesn't remove the values, so when you have a masked array, > then you should use compressed or similar > > dropping the zero bins You should use the masked version of chisquare() in mstats for masked array inputs. However, hiding zeros is not correct unless both observed and expected equal zero. >>>> stats.chisquare(np.array([4, 1, 3, 1.]),np.array([4, 2, 2, 1.])) > (1.0, 0.80125195690120077) > > Not accepting list is a bug It is not a bug because the docstring says arrays not array-like. > Returning nans in the case when you expect zero in a bin might be by > design. But we need to check this. > >>>> stats.chisquare(np.array([4, 1, 3, 1.]),np.array([4, 2, 0, 1.])) > (inf, nan) This is correct since the expected value for a cell is zero (results in division by zero). You can not use the chi-square test in this situation. You might be able to get the fisher exact test (see ticket 956 http://projects.scipy.org/scipy/ticket/956) to work here. If you are doing something like density estimation then you probably need to select your bins (especially in the tails) more carefully to avoid this from happening. Bruce From sebastian.walter at gmail.com Mon Sep 27 17:58:47 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Mon, 27 Sep 2010 23:58:47 +0200 Subject: [SciPy-User] Speeding up odeint for big problems In-Reply-To: References: Message-ID: Well, my first guess would be that the Jacobian computation (+ its factorizations) is the culprit since it is a 10000x10000 array for 10000 coupled ODEs. Leaves the questions how many times the Jacobian is computed? At each step of the integration? Then it would be no surprise that the runtime is really high. What kind of spatial discretization are you using to get a dense Jacobian? Sebastian On Mon, Sep 27, 2010 at 8:02 AM, Justin Bois wrote: > Hello all, > > I'm using odeint to solve a system of PDEs using the method of lines.? This > works well for problems in one dimension where there are about ~10^2 coupled > ODEs.? In 2D, I have ~10^4, and speed is a major issue.? Does anyone have > any general tips on ways to optimally use odeint to solve large problems > using method of lines?? Some useful info/questions about my problem: > > --The Jacobian is dense, even if I use finite differencing for the spatial > derivatives. > --Using something like PyDSTool might be difficult to implement because I > need many of SciPy's handy functions like FFTs, etc. > --I'm worried I might be wasting time by passing a large object as an > argument to the RHS function.? This is out of convenience because it > contains useful information about the problem.? Does odeint do type > conversions on all attributes/methods of the input argument, or only on what > is needed?? Would paring down the input save time? > > Any input is greatly appreciated.? I'd really like to avoid having to code > this up in C. > > Best, > Justin > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From yliu73 at wisc.edu Mon Sep 27 19:02:33 2010 From: yliu73 at wisc.edu (22Nick22 Yanpei Liu) Date: Mon, 27 Sep 2010 18:02:33 -0500 Subject: [SciPy-User] Can't install Scipy on my Mac 10.5.2 Message-ID: <729099b72c58d.4ca0dc39@wiscmail.wisc.edu> Hi Guys: During the installation I keep getting this message: "You cannot install scipy 0.8.0 on this volume. scipy requires Python 2.6 from www.python.org to install." This has been confusing me for months since I installed Python 2.7 directly from www.python.org. It should have replaced the Apple's stock Python 2.5... When I type python it returns: Python 2.7 (r27:82508, Jul 3 2010, 20:17:05) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Any suggestion is appreciated. Thanks Nick From gokhansever at gmail.com Mon Sep 27 19:39:04 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Mon, 27 Sep 2010 18:39:04 -0500 Subject: [SciPy-User] stats.chisquare issues In-Reply-To: <4CA0F370.2080406@gmail.com> References: <4CA0F370.2080406@gmail.com> Message-ID: On Mon, Sep 27, 2010 at 2:41 PM, Bruce Southey wrote: > >>>> stats.chisquare(np.array([4, 1, 3, 1.]),np.array([4, 2, 2, 1.])) > > (1.0, 0.80125195690120077) > > > > Not accepting list is a bug > > It is not a bug because the docstring says arrays not array-like. > This is interesting. I am too much used to then getting array-like sequences converted to np.arrays under the hood when I forget explicitly using np.arrays for numpy, scipy and matplotlib libraries. > If you are doing something like density estimation then you probably > need to select your bins (especially in the tails) more carefully to > avoid this from happening. I was testing chi-square test to see how well my model representing my actual observed data. Right know I don't need to use the function for further calculation as I have different methods to estimate the quality of my fit. Thanks for your suggestions. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 27 19:56:29 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 27 Sep 2010 18:56:29 -0500 Subject: [SciPy-User] Can't install Scipy on my Mac 10.5.2 In-Reply-To: <729099b72c58d.4ca0dc39@wiscmail.wisc.edu> References: <729099b72c58d.4ca0dc39@wiscmail.wisc.edu> Message-ID: On Mon, Sep 27, 2010 at 18:02, 22Nick22 Yanpei Liu wrote: > Hi Guys: > > During the installation I keep getting this message: > > "You cannot install scipy 0.8.0 on this volume. scipy requires Python 2.6 from www.python.org to install." > > This has been confusing me for months since I installed Python 2.7 directly from www.python.org. It should have replaced the Apple's stock Python 2.5... When I type python it returns: > > Python 2.7 (r27:82508, Jul ?3 2010, 20:17:05) > [GCC 4.0.1 (Apple Inc. build 5493)] on darwin > > Any suggestion is appreciated. That binary package for scipy was built for Python 2.6. It will not work for Python 2.7. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From bsouthey at gmail.com Mon Sep 27 21:42:27 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 27 Sep 2010 20:42:27 -0500 Subject: [SciPy-User] stats.chisquare issues In-Reply-To: References: <4CA0F370.2080406@gmail.com> Message-ID: On Mon, Sep 27, 2010 at 6:39 PM, G?khan Sever wrote: > On Mon, Sep 27, 2010 at 2:41 PM, Bruce Southey wrote: >> >> >>>> stats.chisquare(np.array([4, 1, 3, 1.]),np.array([4, 2, 2, 1.])) >> > (1.0, 0.80125195690120077) >> > >> > Not accepting list is a bug >> >> It is not a bug because the docstring says arrays not array-like. > > This is interesting. I am too much used to then getting array-like?sequences > converted to np.arrays under the hood when I forget explicitly using > np.arrays for numpy, scipy and matplotlib libraries. Actually I think that this is rather typical of the scipy.stats functions. Some functions use _chk_asarray that can convert array-like input into arrays. But enhancement patches to handle array-like inputs or other array types like masked arrays are very welcome! > >> >> If you are doing something like density estimation then you probably >> need to select your bins (especially in the tails) more carefully to >> avoid this from happening. > > I was testing chi-square test to see how well my model representing my > actual observed data. Right know I don't need to use the function for > further calculation as I have different methods to estimate the quality of > my fit. > Thanks for your suggestions. > > -- > G?khan > Good as I was just guessing. Bruce From senajohns at gmail.com Tue Sep 28 00:10:54 2010 From: senajohns at gmail.com (Sena Johns) Date: Tue, 28 Sep 2010 14:10:54 +1000 Subject: [SciPy-User] Cannot build SciPy : File contains no section headers. Message-ID: > > Message: 1 > Date: Fri, 24 Sep 2010 10:00:24 +0900 > From: David > Subject: Re: [SciPy-User] Cannot build SciPy : File contains no > section headers. > To: SciPy Users List > Message-ID: <4C9BF828.1070001 at silveregg.co.jp> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 09/23/2010 12:43 PM, Sena Johns wrote: > > Hi, > > > > I'm trying to install Numpy and SciPy using this following method > > : > http://www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974 > > Most likely, you have an invalid site.cfg file. Could you paste it ? > > cheers, > > David > > Thanks David, I've just checked it and found out I misspelled an entry. Cheers, Sena -------------- next part -------------- An HTML attachment was scrubbed... URL: From georges.schutz at internet.lu Tue Sep 28 03:40:27 2010 From: georges.schutz at internet.lu (Georges Schutz) Date: Tue, 28 Sep 2010 09:40:27 +0200 Subject: [SciPy-User] scikits.timeseries: How to define frequency of 15minutes In-Reply-To: <3CC2BB33-A226-48E5-80AE-2AE08E77551A@gmail.com> References: <4BF13906.9030707@internet.lu> <3CC2BB33-A226-48E5-80AE-2AE08E77551A@gmail.com> Message-ID: On 22/09/2010 08:37, Pierre GM wrote: > > On Sep 22, 2010, at 2:27 AM, Darryl Woods wrote: >> >> I know this was posted some time ago now, but just wondering if anyone has >> created any patches or found a way to use a 15-minute frequency with timeseries. >> I'm currently using pandas because it supports it, but some of the functions of >> timeseries would be very useful in my case for dealing with messy input data. > > AAMOF, yes. > You can find it on http://github.com/pierregm/scikits.timeseries-sandbox. > Now, you can add a `timestep` to a function like time_series, which lets you define a regular interval between two consecutive events. > Because the functionality hasn't been thoroughly tested yet, it's not the in the official branch yet (hence the scikits.timeseries-sandbox) and is not well documented. Last time I checked, it worked well, though. Your feedback will therefore be invaluable. > Let me know how it goes. > P Thank you for this development. I would like to analyse the sandbox branch but how can I do this without impacting my stable scikits.timeseries installation. I can clone the git repository into an local directory but it seems to me that I need to install the the package using python setup.py install And this will probably impact my stable scikits.timeseries no? As I use Win-XP as OS it is recommended to use the pre-build installer (README.txt). What do you suggest me to do for testing and analysing your sandbox branch? Georges Schutz From pgmdevlist at gmail.com Tue Sep 28 04:38:28 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 28 Sep 2010 10:38:28 +0200 Subject: [SciPy-User] scikits.timeseries: How to define frequency of 15minutes In-Reply-To: References: <4BF13906.9030707@internet.lu> <3CC2BB33-A226-48E5-80AE-2AE08E77551A@gmail.com> Message-ID: <7A83B122-BB03-4E2F-AAD8-9F2F44666FB4@gmail.com> On Sep 28, 2010, at 9:40 AM, Georges Schutz wrote: > > Thank you for this development. I would like to analyse the sandbox > branch but how can I do this without impacting my stable > scikits.timeseries installation. > I can clone the git repository into an local directory but it seems to > me that I need to install the the package using > python setup.py install > And this will probably impact my stable scikits.timeseries no? Yes. > As I use Win-XP as OS it is recommended to use the pre-build installer > (README.txt). What do you suggest me to do for testing and analysing > your sandbox branch? Use a virtual environment. http://pypi.python.org/pypi/virtualenv I advise you to use the following wrappers to simplify handling multiple virtualenvs http://www.doughellmann.com/projects/virtualenvwrapper/ It's quite straightforward, but don't hesitate to contact me if you have any question. From massimodisasha at gmail.com Tue Sep 28 07:07:19 2010 From: massimodisasha at gmail.com (Massimo Di Stefano) Date: Tue, 28 Sep 2010 13:07:19 +0200 Subject: [SciPy-User] port a PDE system from matab to scipy Message-ID: <6F9BAF99-1161-4AAB-8EBF-DF2B4D39E7B6@gmail.com> Hello All, i'm tring to port a matlab code to run under scipy. the code solve a pde system using ode45, the matlab code is like : tmax = 100 ; x_d = 60 ; x_s = 0 ; n = 10 ; C_0 = 10 ; V_0 = 5 ; a = 0.9 ; p = 0.15 ; b = 0.65 ; r = 0.1 ; x=linspace(x_s,x_d,n+2); h=(x_d-x_s)/(n+1); C_x_s = 0 ; C_x_d = 0 ; V_x_s = 0 ; V_x_d = 0 ; CViniz=zeros(2*n,1); CViniz(1)=C_0; CViniz(n+1)=V_0; [t,CV] = ode45('AvvDiffPredRatevar',[0 tmax],CViniz); % where AvvDiffPredRatevar is : function CVprimo = AvvDiffPredRatevar(t,CV) global n h x C_x_s C_x_d V_x_s V_x_d a b p r CVprimo(1:n) = -0.2/h*(CV(1:n)-[C_x_s; CV(1:(n-1))])+0.2/(h^2)*.... ([C_x_s; CV(1:n-1)]-2*CV(1:n)+[CV(2:n); C_x_d])+a*CV(1:n)-b*CV(1:n).*CV((n+1):end); CVprimo(n+1:2*n) = -0.2/h*(CV(n+1:end)-[V_x_s; CV(n+1:end-1)])+0.2/(h^2)*.... ([V_x_s; CV(n+1:end-1)]-2*CV(n+1:end)+[CV(n+2:end); V_x_d])+p*CV(n+1:end)-r*CV(1:n).*CV(n+1:end); CVprimo=CVprimo'; in scipy i start to write : tmax = 100 ; x_d = 60 ; x_s = 0 ; n = 10 ; C_0 = 10 ; V_0 = 5 ; a = 0.9 ; p = 0.15 ; b = 0.65 ; r = 0.1 ; x=linspace(x_s,x_d,n+2); h=(x_d-x_s)/(n+1); C_x_s = 0 ; C_x_d = 0 ; V_x_s = 0 ; V_x_d = 0 ; CViniz=zeros(2*n,1); CViniz(1)=C_0; CViniz(n+1)=V_0; scip now i have to define the "ratefunction" AvvDiffPredRatevar(t,CV) have you any suggestion on how to translate the matlab ratefunciotn code to python ? i suppose i have to write a new python function : def AvvDiffPredRatevar(t,CV) and then i have to pass it to scipy : scipy.integrate.ode(f).set_integrator('dopri5') where "f" is "AvvDiffPredRatevar(t,CV)" make sense for you ? or i'm wrong ? thanks for any hints!!! Best regards, Massimo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nico.schloemer at gmail.com Tue Sep 28 09:34:21 2010 From: nico.schloemer at gmail.com (=?ISO-8859-1?Q?Nico_Schl=F6mer?=) Date: Tue, 28 Sep 2010 15:34:21 +0200 Subject: [SciPy-User] scipy.sparse.linalg.cg statistics? Message-ID: Hi all, I was just running some test problems with scipy.sparse.linalg.cg and thing seem work work out fine. To compare preconditioners, I'd like to get more detailed information about the convergence history, but I haven't been able to figure out how to, e.g., get the number of performed iterations or the (relative) residual for each of those steps. Compare this with http://www.mathworks.com/help/techdoc/ref/pcg.html. Any hints? Cheers, Nico From pav at iki.fi Tue Sep 28 11:19:50 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 28 Sep 2010 15:19:50 +0000 (UTC) Subject: [SciPy-User] scipy.sparse.linalg.cg statistics? References: Message-ID: Tue, 28 Sep 2010 15:34:21 +0200, Nico Schl?mer wrote: > Hi all, > > I was just running some test problems with scipy.sparse.linalg.cg and > thing seem work work out fine. > To compare preconditioners, I'd like to get more detailed information > about the convergence history, but I haven't been able to figure out how > to, e.g., get the number of performed iterations or the (relative) > residual for each of those steps. Compare this with > http://www.mathworks.com/help/techdoc/ref/pcg.html. > > Any hints? Use the 'callback' argument. >>> help(scipy.sparse.linalg.cg) ... callback : function User-supplied function to call after each iteration. It is called as callback(xk), where xk is the current solution vector. ... From justinbois at gmail.com Tue Sep 28 13:15:16 2010 From: justinbois at gmail.com (Justin Bois) Date: Tue, 28 Sep 2010 19:15:16 +0200 Subject: [SciPy-User] Speeding up odeint for big problems In-Reply-To: References: Message-ID: Thanks Ron and Sebastian for your help. The code to calculate the RHS of the ODEs is really large so I can't really post it. I am solving a reaction-diffusion-advection system in 2D. The velocity (which contributes to the advection) is a complicated function of the concentrations of the species in the reaction-diffusion system. At each time step, I have to solve a separate, time-independent ODE for the velocity, which is then incorporated into the RHS. Because this ODE happens to be linear, I can solve it using a spectral method (for my boundary conditions of interest, regardless of my differencing scheme in the reaction-diffusion-advection system). The presence of this velocity, manifest in the advection term, makes the Jacobian dense, even if I use low-order finite difference for my differencing. Since I necessarily have a dense Jacobian, I typically use pseudospectral methods for computing the derivatives to get high accuracy with fewer grid points, though I have also implemented finite difference methods. I have pasted below the first few lines of output of a profile I did using the magic function %prun with ipython -pylab. 36447356 function calls in 1212.986 CPU seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1 903.674 903.674 1212.976 1212.976 {scipy.integrate._odepack.odeint} 678300 157.372 0.000 239.023 0.000 basic.py:358(_raw_fftnd) 1356600 24.225 0.000 35.504 0.000 basic.py:109(_fix_shape) So, aside from odeint itself, most of the computational effort is spent computing derivatives and solving for the velocity, all of which is just using 2D FFTs. Curiously, odeint spends a lot of time doing something that does not require evaluation of the RHS of the ODEs. As I watch the integration proceed, it takes several time steps very rapidly, and then pauses at a particular time point, presumably to adjust the step size. My guess is that Sebastian is right: this is slow because the Jacobian is large and dense. Do you think just doing an explicit RK4 time stepping or something might help? Or are there tricks we can play to get odeint to roll along more quickly? I am passing an object with all sorts of attributes as an argument to the RHS function, though only some are used. I am assuming the RHS function is all executed in Python and then the return values are all that undergo type conversion, right? I'm guessing it's not so much type conversion that's the big slowdown, or do you think I'm mistaken? odeinthas been a nice little workhorse in my work so far! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dtlussier at gmail.com Wed Sep 29 13:05:15 2010 From: dtlussier at gmail.com (Dan Lussier) Date: Wed, 29 Sep 2010 12:05:15 -0500 Subject: [SciPy-User] extracting row from 2D array Message-ID: I want to extract a row from an 2D array given a criteria that is based on each entry in that row. By way of an example: xyz = np.array([[ 159.278 , 14.7733 , 11.6785 ], [ 158.251 , 1.11666, 4.76546], [ 21.1418 , 84.8988 , 56.5169 ], [ 154.149 , 4.9442 , 131.672 ], [ 31.578 , 11.1078 , 8.47435]]) magnitude_sq = xyz*xyz.sum(axis=1) I want to pull out the rows that satisfy a condition per row like magnitude_sq < 16. I've found and tried np.extract, np.take and np.compress. While np.extract fails to do what I want given that it effectively wants to treat the 2D array was a 1D vector, I moved onto np.compress and can get what I want via: np.compress( magnitude_sq < 16 , xyz, axis=0) What I'm trying to understand is the relationship between these different functions (i.e. which one(s) are more fundamental) as the documentation doesn't really identify how they might relate to each other. I have also posted the question to http://ask.scipy.org/en/topic/50-extracting-entries-from-2d-numpy-array to give that system at try. From josef.pktd at gmail.com Wed Sep 29 13:09:48 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 29 Sep 2010 13:09:48 -0400 Subject: [SciPy-User] extracting row from 2D array In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 1:05 PM, Dan Lussier wrote: > I want to extract a row from an 2D array given a criteria that is > based on each entry in that row. > > By way of an example: > > xyz = np.array([[ 159.278 ?, ? 14.7733 , ? 11.6785 ], > ? ? ? ? ? ? ? [ 158.251 ?, ? ?1.11666, ? ?4.76546], > ? ? ? ? ? ? ? [ ?21.1418 , ? 84.8988 , ? 56.5169 ], > ? ? ? ? ? ? ? [ 154.149 ?, ? ?4.9442 , ?131.672 ?], > ? ? ? ? ? ? ? [ ?31.578 ?, ? 11.1078 , ? ?8.47435]]) > > magnitude_sq = xyz*xyz.sum(axis=1) > > I want to pull out the rows that satisfy a condition per row like > magnitude_sq < 16. I've found and tried np.extract, np.take and > np.compress. While np.extract fails to do what I want given that it > effectively wants to treat the 2D array was a 1D vector, I moved onto > np.compress and can get what I want via: > > np.compress( magnitude_sq < 16 , xyz, axis=0) with indexing xyz[magnitude_sq < 16, :] Josef > > What I'm trying to understand is the relationship between these > different functions (i.e. which one(s) are more fundamental) as the > documentation doesn't really identify how they might relate to each > other. > > I have also posted the question to > http://ask.scipy.org/en/topic/50-extracting-entries-from-2d-numpy-array > to give that system at try. > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From oliphant at enthought.com Wed Sep 29 14:59:59 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 29 Sep 2010 13:59:59 -0500 Subject: [SciPy-User] Integer dtype comparisons In-Reply-To: <4C99F0B9.9040409@pimmer.info> References: <4C99F0B9.9040409@pimmer.info> Message-ID: On Sep 22, 2010, at 7:04 AM, Michael Pimmer wrote: > Hello, > > is there a nice way to check whether an object is an Integer? Including > all python and numpy Integers? All numpy integers inherit from a base-class numpy.integer. Thus, isinstance(obj, (numpy.integer, int, long)) should work. But, you might want to think about "duck-typing" (i.e why do you need to do this check) and use a try, except approach with the __index__() method of the object instead. The __index__() object returns a real Python int from any object that supports it. Example ======= try: obj_as_integer = obj.__index__() # code to use obj_as_integer except (TypeError, AttributeError): # some other code.... -Travis > > I tried out subclass, but uint is not a subclass of Int: > >>>> a = numpy.zeros(1,numpy.uint32) >>>> aType = type(a[0]) >>>> issubclass(aType,types.IntType) > False > > > I tried out enumerating, but numpy seems to slightly change datatypes > during some calculations, which complicates comparisons: > >>>> a = numpy.zeros(1,numpy.uint8) >>>> aValue = a[0] >>>> b = numpy.zeros(1,numpy.int32) >>>> bValue = b[0] >>>> result = aValue + bValue >>>> type(result) > >>>> numpy.int32 > >>>> type(result) == numpy.int32 > False > > > thanks, > Michael > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com From Michael.Graham at cdicorp.com Thu Sep 30 11:31:58 2010 From: Michael.Graham at cdicorp.com (Graham, Michael) Date: Thu, 30 Sep 2010 11:31:58 -0400 Subject: [SciPy-User] Integer dtype comparisons In-Reply-To: References: <4C99F0B9.9040409@pimmer.info> Message-ID: <61A807FD6A92CC45A6064B98D79D343B260988AF56@CDIMSXMBCL01.CDI.CDICorp.net> On Wednesday, September 29, 2010 at 3:00PM, Travis Oliphant wrote: > try: > obj_as_integer = obj.__index__() > # code to use obj_as_integer > except (TypeError, AttributeError): > # some other code.... > > -Travis Surely you mean try: obj_as_integer = obj.__index__() except (TypeError, AttributeError): # some other code.... code to use obj_as_integer -or- try: obj_as_integer = obj.__index__() except (TypeError, AttributeError): # some other code.... else: code to use obj_as_integer Mike Mike Graham Software Engineer CDI Marine - Band Lavis Division 6960 Aviation Blvd, Suite A Glen Burnie, MD 21061 ______________________________________________________________________ CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain information which is confidential to, and/or privileged in favor of, CDI Corporation or its affiliated companies (CDI) or CDI's customers. Any review, use, reproduction, disclosure or distribution by the recipient is prohibited without prior written approval from an authorized CDI representative. This notice must appear in any such authorized reproduction, disclosure or distribution. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and any attachments. Thank you. From dav at alum.mit.edu Thu Sep 30 21:46:32 2010 From: dav at alum.mit.edu (Dav Clark) Date: Thu, 30 Sep 2010 18:46:32 -0700 Subject: [SciPy-User] Old meaning of numpy mgrid command In-Reply-To: References: <5DF38E2A-DE14-4792-9D54-95EA7A2E590F@alum.mit.edu> Message-ID: On Sep 24, 2010, at 2:17 AM, Pauli Virtanen wrote: > Fri, 24 Sep 2010 00:45:11 -0400, Travis Oliphant wrote: > [clip] >> How old is this code? I don't recall any old behavior of mgrid. >> What version of NumPy did the previous code used to work with? > > The __getitem__ in mgrid has stayed essentially the same since 2002. > Seems very likely it has never worked in a different way. After some digging, I am inclined to agree. Fortunately, I know enough to have (I hope) figured it out from the underlying math. Thanks! DC From bfrenay at gmail.com Wed Sep 29 07:20:01 2010 From: bfrenay at gmail.com (bfrenay) Date: Wed, 29 Sep 2010 04:20:01 -0700 (PDT) Subject: [SciPy-User] [SciPy-user] Using Sphinx with numpy/scipy template Message-ID: <29837454.post@talk.nabble.com> Hello, I just migrated all my code documentation to the numpy/scipy format. However, I have some problems with sphinx. I have (i) installed sphinx and numpydoc using easyinstall, (ii) inserted the following line in conf.py extensions= ['sphinx.ext.autodoc', 'sphinx.ext.autosummary', 'numpydoc'] and (iii) created a simple, dummy index.rst file reproduced here bellow .. project documentation master file .. automodule:: project.datasets :members: However, I only get errors (reproduced after the message). Does anyone have an idea about what is going on, please ? Thank you, Beno?t $ make html sphinx-build -b html -d _build/doctrees . _build/html Running Sphinx v1.0.4 Exception occurred: File "/usr/local/lib/python2.6/dist-packages/numpydoc-0.3.1-py2.6.egg/numpydoc/numpydoc.py", line 136, in get_directive raise RuntimeError("No directive named '%s' found" % name) RuntimeError: No directive named 'function' found The full traceback has been saved in /tmp/sphinx-err-pCQ8Df.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. Either send bugs to the mailing list at , or report them in the tracker at . Thanks! make: *** [html] Erreur 1 -- View this message in context: http://old.nabble.com/Using-Sphinx-with-numpy-scipy-template-tp29837454p29837454.html Sent from the Scipy-User mailing list archive at Nabble.com.