From pearu@egoist.ioc.ee Fri Nov 13 13:46:39 1998 From: pearu@egoist.ioc.ee (Pearu Peterson) Date: Fri, 13 Nov 1998 15:46:39 +0200 (EET) Subject: [Matrix-SIG] umath bug => bug in Numeric module Message-ID: Hi! umath.arcsin(3) gives 'ValueError: math domain error' I would expect a complex number. The same holds also for umath.arccos. It seems that the author of umath has forgot to implement these cases correctly. The corresponding functions in math module work correctly. This is also Numeric bug since it uses umath (I have LLNLPython4). With best regards, Pearu From hinsen@cnrs-orleans.fr Fri Nov 13 15:11:51 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Fri, 13 Nov 1998 16:11:51 +0100 Subject: [Matrix-SIG] umath bug => bug in Numeric module In-Reply-To: (message from Pearu Peterson on Fri, 13 Nov 1998 15:46:39 +0200 (EET)) References: Message-ID: <199811131511.QAA13618@dirac.cnrs-orleans.fr> > umath.arcsin(3) gives 'ValueError: math domain error' > I would expect a complex number. Try: >>> umath.arcsin(3.+0j) (1.57079632679-1.76274717404j) The philosophy of all umath functions is to return complex values only for complex arguments. The reason is that in many circumstances only real numbers should be used and arguments leading to complex results should be considered an error. You will also notice that "sqrt(-1)" is an error, you need to write "sqrt(-1+0j)" to get the result 1j. > The corresponding functions in math module work correctly. The module math does not handle complex numbers at all, and math.asin(3) raises a math domain error. I suppose you mean cmath, which is a specialized module for complex math. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From frank@ned.dem.csiro.au Sat Nov 14 03:18:46 1998 From: frank@ned.dem.csiro.au (Frank Horowitz) Date: Sat, 14 Nov 1998 11:18:46 +0800 Subject: [Matrix-SIG] inverse fft2d (complex case) Message-ID: <364CF696.FC7B81C1@ned.dem.csiro.au> G'Day folks, I've been using NumPy (and Python) for a bit over a week now. To all who've contributed, I'd like to say "Great Job"! It's truly a pleasure to use... Even though it may be too trivial to warrant comment, I've implemented something that I felt the acute lack of in the LLNL NumPy distribution: the inverse of a 2d FFT (at least for the complex valued case; I hope to follow up with the equivalent real-valued inverse, once I grok the way that the wavenumbers are packed upon output from the forward 2dfft). Without further ado, here is the code (all 2 lines of it :-) def inverse_fft2d(a, s=None, axes=(-2,-1)): return _raw_fft2d(a,s,axes,inverse_fft) I suppose it's debatable whether such a trivial extension belongs in the distribution. Indeed it's absence suggests some deeper issues (that I'm missing) have already been debated, and somebody decided that the implementation of an inverse properly belongs with the individual users. If so, pardon me for the intrusion :-) Obviously, I've also implemented ways of determining the appropriate wavenumbers, and will be happy to post them if anyone is interested. Cheers, Frank Horowitz From Paul F. Dubois" Frank, I have added your new function to fft.py. It will appear in the next LLNL release. Thanks for sending it in. If you have time, please improve the test routine at the bottom of fft.py so that it indicates whether or not the answers are correct, as I don't remember enough about FFTs to do this without some research. I added a test to see if your inverse inversed fft2d, which it did, but that is all I knew how to do. I'm sure any other functions you add to fft.py will be appreciated. Thanks, Paul -----Original Message----- From: Frank Horowitz To: matrix-sig@python.org Date: Friday, November 13, 1998 7:23 PM Subject: [Matrix-SIG] inverse fft2d (complex case) >G'Day folks, > > I've been using NumPy (and Python) for a bit over a week now. To all >who've contributed, I'd like to say "Great Job"! It's truly a pleasure >to use... > > Even though it may be too trivial to warrant comment, I've implemented >something that I felt the acute lack of in the LLNL NumPy distribution: >the inverse of a 2d FFT (at least for the complex valued case; I hope >to follow up with the equivalent real-valued inverse, once I grok the >way that the wavenumbers are packed upon output from the forward >2dfft). Without further ado, here is the code (all 2 lines of it :-) > >def inverse_fft2d(a, s=None, axes=(-2,-1)): > return _raw_fft2d(a,s,axes,inverse_fft) > > I suppose it's debatable whether such a trivial extension belongs in >the distribution. Indeed it's absence suggests some deeper issues (that >I'm missing) have already been debated, and somebody decided that the >implementation of an inverse properly belongs with the individual >users. If so, pardon me for the intrusion :-) > > Obviously, I've also implemented ways of determining the appropriate >wavenumbers, and will be happy to post them if anyone is interested. > > Cheers, > Frank Horowitz > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig > > From pearu@egoist.ioc.ee Mon Nov 16 07:17:31 1998 From: pearu@egoist.ioc.ee (Pearu Peterson) Date: Mon, 16 Nov 1998 09:17:31 +0200 (EET) Subject: [Matrix-SIG] NumericFix.py Message-ID: Hi! The problem is that in order to get sqrt(-1) one has to write sqrt(-1+0j) which, I think, is quite weird. The same holds for arcsin(a),arccos(a) if |a|>1. So, I propose the following fixes in the Numeric module that is realized in the file 'NumericFix.py' (see below). With best regards, Pearu Pearu Peterson , MSc, Researcer Department of Mechanics and Applied Mathematics http://koer.ioc.ee/~pearu/ Institute of Cybernetics at Tallinn Technical University Phone: (+3722) 527907 Akadeemia rd. 21, 12618 Tallinn ESTONIA Fax: (+372) 6397039 --------------- NumericFix.py ----------------- #!/usr/bin/env python """ In order to use the following statements (that in the Numeric module give ValueError): Numeric.sqrt(-1) Numeric.arcsin(3) Numeric.arccos(3) use `from NumericFix import Numeric` instead of `import Numeric`. Use `from NumericFix import *` instead of `from Numeric import *`. In addition, the functions Numeric.ceil, Numeric.floor, and Numeric.fabs will work with complex arguments as well. For example, Numeric.ceil(2.1-5.3j) gives (3-5j). PS: These fixes should really be done in Numeric module. November 16, 1998 Pearu Peterson, """ __version__ = "01" import Numeric,umath from Numeric import * def sqrt(a): try: return umath.sqrt(a) except ValueError: return umath.sqrt(a+0j) def arcsin(a): try: return umath.arcsin(a) except ValueError: return umath.arcsin(a+0j) def arccos(a): try: return umath.arccos(a) except ValueError: return umath.arccos(a+0j) def ceil(a): try: return umath.ceil(a) except AttributeError: return umath.ceil(a.real)+1j*umath.ceil(a.imag) def floor(a): try: return umath.floor(a) except AttributeError: return umath.floor(a.real)+1j*umath.floor(a.imag) def fabs(a): try: return umath.fabs(a) except AttributeError: return umath.fabs(a.real)+1j*umath.fabs(a.imag) Numeric.__dict__['sqrt']=sqrt Numeric.__dict__['arcsin']=arcsin Numeric.__dict__['arccos']=arccos Numeric.__dict__['ceil']=ceil Numeric.__dict__['floor']=floor Numeric.__dict__['fabs']=fabs def test(): for a in [-1,-1+0j,3.4,Numeric.array([2,3]),Numeric.array([-5,3j])]: print 'Numeric.sqrt(%s)=%s'%(`a`,`Numeric.sqrt(a)`) print 'Numeric.arcsin(%s)=%s'%(`a`,`Numeric.arcsin(a)`) print 'Numeric.arccos(%s)=%s'%(`a`,`Numeric.arccos(a)`) for a in [-1.5,-1+4.3j,Numeric.array([-5.2,3.7j])]: print 'Numeric.ceil(%s)=%s'%(`a`,`Numeric.ceil(a)`) print 'Numeric.floor(%s)=%s'%(`a`,`Numeric.floor(a)`) print 'Numeric.fabs(%s)=%s'%(`a`,`Numeric.fabs(a)`) if __name__ == "__main__": test() From hinsen@cnrs-orleans.fr Mon Nov 16 16:12:58 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Mon, 16 Nov 1998 17:12:58 +0100 Subject: [Matrix-SIG] NumericFix.py In-Reply-To: (message from Pearu Peterson on Mon, 16 Nov 1998 09:17:31 +0200 (EET)) References: Message-ID: <199811161612.RAA22396@dirac.cnrs-orleans.fr> > The problem is that in order to get sqrt(-1) one has > to write sqrt(-1+0j) which, I think, is quite weird. Well, it may be surprising, but it's the outcome of a design decision, not a bug. There are good arguments for making the functions behave like they do, once you consider that most practical applications do not require complex numbers. > So, I propose the following fixes in the Numeric module > that is realized in the file 'NumericFix.py' (see below). I wouldn't call it a fix, but that's debatable. More importantly, some of your definitions don't make sense to me: > def ceil(a): > try: return umath.ceil(a) > except AttributeError: return umath.ceil(a.real)+1j*umath.ceil(a.imag) > def floor(a): > try: return umath.floor(a) > except AttributeError: return umath.floor(a.real)+1j*umath.floor(a.imag) Both ceil() and floor() make sense only for numbers with an order relation, i.e. not for complex numbers. > def fabs(a): > try: return umath.fabs(a) > except AttributeError: return umath.fabs(a.real)+1j*umath.fabs(a.imag) The proper generalization of absolute values to complex numbers is umath.sqrt(a*conjugate()). > Numeric.__dict__['sqrt']=sqrt >... It's not a good idea to have one module modify another one. This can create a lot of confusion, because the functions you get depend on when you import them. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From amullhau@ix.netcom.com Mon Nov 16 19:22:42 1998 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Mon, 16 Nov 1998 14:22:42 -0500 Subject: [Matrix-SIG] NumericFix.py Message-ID: <008301be1196$7d3bf300$ce0709c0@amullhau.ix.netcom.com> -----Original Message----- From: Konrad Hinsen To: pearu@egoist.ioc.ee Cc: matrix-sig@python.org Date: Monday, November 16, 1998 11:19 AM Subject: Re: [Matrix-SIG] NumericFix.py >> The problem is that in order to get sqrt(-1) one has >> to write sqrt(-1+0j) which, I think, is quite weird. > >Well, it may be surprising, but it's the outcome of a design >decision, not a bug. There are good arguments for making the >functions behave like they do, once you consider that most >practical applications do not require complex numbers. The arguments for automatically widening to complex in an interpreted language are better. The two sensible choices are either a domain error (which is what I get with python) or automatic widening, which is better. It's not the end of the world that python uses the domain error, though. You can get around this with try/except. It all boils down to whether you believe the programmer _expects_ the result to widen or not. The consistency of this with the rest of the language has a lot to do with this expectation. If integer division and floating point division share the same syntax (in python they do) then since 3/5 is expected to produce 0 and not 0.6 then the dice have already been rolled. It's definitely advantageous in scientific computing to make the other choice, but this is one that would require pervasive modification of python, and so we're about ten years too late to do anything about it now. The choice that has been made is not catastrophic, although in some cases annoying, but shouldn't be reconsidered at this point. Later, Andrew Mullhaupt From pearu@egoist.ioc.ee Tue Nov 17 07:33:51 1998 From: pearu@egoist.ioc.ee (Pearu Peterson) Date: Tue, 17 Nov 1998 09:33:51 +0200 (EET) Subject: [Matrix-SIG] NumericFix.py In-Reply-To: <008301be1196$7d3bf300$ce0709c0@amullhau.ix.netcom.com> Message-ID: Thank you, Andrew Mullhaupt. Yours answer explained me well why my ideas raised so much oposition from the python people in matrix-sig. So, I will rest to call my proposals as fixes to python as the choise is made so many years ago by programmers and I will respect this because they created this great language. In conclusion, all that I want is that my computer code will contain minimal amount of character and information that are not directly related to my mathematical problem in hand. I believe that this is realizable in python and I will certainly continue to shape the python possibilities to my needs. With best regards, Pearu Pearu Peterson , MSc, Researcer Department of Mechanics and Applied Mathematics http://koer.ioc.ee/~pearu/ Institute of Cybernetics at Tallinn Technical University Phone: (+3722) 527907 Akadeemia rd. 21, 12618 Tallinn ESTONIA Fax: (+372) 6397039 On Mon, 16 Nov 1998, Andrew P. Mullhaupt wrote: > > -----Original Message----- > From: Konrad Hinsen > To: pearu@egoist.ioc.ee > Cc: matrix-sig@python.org > Date: Monday, November 16, 1998 11:19 AM > Subject: Re: [Matrix-SIG] NumericFix.py > > > >> The problem is that in order to get sqrt(-1) one has > >> to write sqrt(-1+0j) which, I think, is quite weird. > > > >Well, it may be surprising, but it's the outcome of a design > >decision, not a bug. There are good arguments for making the > >functions behave like they do, once you consider that most > >practical applications do not require complex numbers. > > > The arguments for automatically widening to complex in an interpreted > language are better. > > The two sensible choices are either a domain error (which is what I get with > python) or automatic widening, which is better. > > It's not the end of the world that python uses the domain error, though. You > can get around this with try/except. > > It all boils down to whether you believe the programmer _expects_ the result > to widen or not. > > The consistency of this with the rest of the language has a lot to do with > this expectation. If integer division and floating point division share the > same syntax (in python they do) then since 3/5 is expected to produce 0 and > not 0.6 then the dice have already been rolled. > > It's definitely advantageous in scientific computing to make the other > choice, but this is one that would require pervasive modification of python, > and so we're about ten years too late to do anything about it now. > > The choice that has been made is not catastrophic, although in some cases > annoying, but shouldn't be reconsidered at this point. > > Later, > Andrew Mullhaupt > > > From tim_one@email.msn.com Tue Nov 17 22:31:04 1998 From: tim_one@email.msn.com (Tim Peters) Date: Tue, 17 Nov 1998 17:31:04 -0500 Subject: [Matrix-SIG] NumericFix.py In-Reply-To: <008301be1196$7d3bf300$ce0709c0@amullhau.ix.netcom.com> Message-ID: <000301be1279$f63eb4c0$399e2299@tim> [pearu@egoist.ioc.ee] > The problem is that in order to get sqrt(-1) one has > to write sqrt(-1+0j) which, I think, is quite weird. [Konrad Hinsen] > Well, it may be surprising, but it's the outcome of a design > decision, not a bug. There are good arguments for making the > functions behave like they do, once you consider that most > practical applications do not require complex numbers. [Andrew P. Mullhaupt] > The arguments for automatically widening to complex in an > interpreted language are better. I don't know that it has much to do with interpreting (vs compiling?), but it has a lot to do with the principle of least surprise across *all* a language's users. sqrt in particular is often used in mundane business math (your basic $5 calculator has a sqrt key for a reason ), and in those apps sqrt(negative) is always "an error". The presumption is that Python users sophisticated enough to know what a complex number is are also sophisticated enough to do something to get one. > The two sensible choices are either a domain error (which is what > I get with python) or automatic widening, which is better. Better for some users, definitely. > ... > It all boils down to whether you believe the programmer _expects_ > the result to widen or not. Yes, and it's thought that most Python users do not. > The consistency of this with the rest of the language has a lot to do > with this expectation. If integer division and floating point division > share the same syntax (in python they do) then since 3/5 is expected to > produce 0 and not 0.6 then the dice have already been rolled. There was quite a firestorm over int/int on c.l.py a few months ago. This is a case Guido would change *if he could*, because truncation surprises many new users of all stripes. So much so that the Alice (3D graphics for beginners) project actually changed the semantics of int/int in their embedded version of Python. Alas, there's far too much code out there that relies on truncation now, and even the suggestion that it be changed in the hypothetical "Python 2" provoked curiously outraged howling. Nevertheless, if Python 2 ever does come to pass, I bet half of an apple will finally be better than nothing . although-feeding-it-sqrt(negative-apple)-will-still-barf-ly y'rs - tim From Warren B. Focke" I am having trouble assigning to extended slices of object arrays using strides > 1 on Linux/Intel: -pfiesteria:~> python -Python 1.5.1 (#1, Aug 8 1998, 13:33:40) [GCC egcs-2.90.29 980515 (egc on linux2 -Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam ->>> import Numeric ->>> Numeric.__version__ -'1.5' ->>> a=Numeric.zeros((4,),'O') ->>> a -array([0 , 0 , 0 , 0 ],'O') ->>> a[::2]=2 -Segmentation fault and: ->>> a=Numeric.zeros((2,2),"O") ->>> a -array([[0 , 0 ], - [0 , 0 ]],'O') ->>> a[:,1]=1 -Segmentation fault but: ->>> a[1:3]=1 ->>> a -array([0 , 1 , 1 , 0 ],'O') This happens with python 1.5(.1)? with NumPy 1.[01245] on various combinations of (486|pentium|PII) running kernel 2.0.3[0246] and libc [56], python and NumPy compiled with gcc2.7.2.x or egcs, with or without optimization. It does not happen on a sparc10 running 1.5.1, 1.0, and 4.1.4 or a 21064 running 1.5, 1.0, and 4.0. Warren Focke From zcm@llnl.gov Wed Nov 18 15:15:23 1998 From: zcm@llnl.gov (Zane Motteler) Date: Wed, 18 Nov 1998 07:15:23 -0800 Subject: [Matrix-SIG] bug with extended slices vs. object arrays? In-Reply-To: Message-ID: Warren: You wrote >I am having trouble assigning to extended slices of object arrays using >strides > 1 on Linux/Intel: ...... We'll have a look at it. Unfortunately I do not have access to a Linux box, but we'll do what we can. Zane -------------------- Zane C. Motteler, Ph. D. Professor Emeritus of Computer Science and Engineering California Polytechnic State University, San Luis Obispo zmottel@calpoly.edu Currently: Computer Scientist Lawrence Livermore National Laboratory P O Box 808, L-038 (Street address 7000 East Avenue, L-038) Livermore, CA 94551-9900 zcm@llnl.gov 925/423-2143, FAX 925/423-9969 From warren@pfiesteria.gsfc.nasa.gov Wed Nov 18 22:02:45 1998 From: warren@pfiesteria.gsfc.nasa.gov (Warren B. Focke) Date: Wed, 18 Nov 1998 17:02:45 -0500 (EST) Subject: [Matrix-SIG] broadcast rules Message-ID: Is broadcasting expected to work differently for arithmetic operations and assignment? -pfiesteria:~> python -Python 1.5.1 (#1, Aug 8 1998, 13:33:40) [GCC egcs-2.90.29 980515 (egc on linux2 -Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam ->>> import Numeric ->>> a=Numeric.ones((4,2)) ->>> b=Numeric.ones((4,1)) ->>> a[...]+b -array([[2, 2], - [2, 2], - [2, 2], - [2, 2]]) ->>> a[...]=b -Traceback (innermost last): - File "", line 1, in ? -ValueError: matrices are not aligned for copy Happens with 1.5/1.0 on linux/pentium, alpha/osf, and sparc10/4.1.4; and 1.5.1/1.5 on linux/pII. Warren Focke From Oliphant.Travis@mayo.edu Wed Nov 18 23:34:04 1998 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Wed, 18 Nov 1998 17:34:04 -0600 (CST) Subject: [Matrix-SIG] Special functions with Numerical Python In-Reply-To: <199811181700.MAA06901@python.org> Message-ID: Hi all, Does anybody know of a package that allows evaluation of special functions, like bessel, airy, or error functions on Numerical Python arrays. It wouldn't be too hard to wrap up one of the many libraries on the net, and it would help in my move from MATLAB to NumPy. Before I did something like that I wanted to know if anyone has already done it. Thanks, Travis ---------------------------------------------------- Travis Oliphant 200 First St SW Graduate Student Rochester MN 55901 Ultrasound Research Lab (507) 286-5923 Mayo Graduate School Oliphant.Travis@mayo.edu From mhagger@blizzard.harvard.edu Thu Nov 19 00:00:13 1998 From: mhagger@blizzard.harvard.edu (Michael Haggerty) Date: Wed, 18 Nov 1998 19:00:13 -0500 Subject: [Matrix-SIG] ANNOUNCE: New Gnuplot.py (plotting from python using gnuplot) Message-ID: <9811190000.AA28852@tsunami.harvard.edu> Hi, I would like to announce the release of a major new version of Gnuplot.py. Gnuplot.py is an interface between python and the gnuplot plotting program. Gnuplot.py can be obtained from The module has extensive documentation in the form of comments and doc-strings within the code, and a simple demonstration can be run by typing `python Gnuplot.py'. Of course you must have the gnuplot program installed to use Gnuplot.py; see to obtain gnuplot. This version was inspired and partly derived from an earlier Gnuplot.py written by Konrad Hinsen, and should be drop-in compatible with the older version. However, the new version adds an object-oriented interface, which is much more flexible than the old functional interface. Konrad tells me that he plans to declare the old Gnuplot.py obsolete and recommend migration to this one. Example of creating a simple plot using the new interface: import Gnuplot g = Gnuplot.Gnuplot() # Start up a gnuplot process g.title('Simple plot') # Can add a title g.xlabel('time') # Can set the x axis label g.plot(array, title='Data') # `array' is an Nx2 array of numbers; # `Data' goes in the plot's key Some features and limitations are described below. Yours, Michael -- Michael Haggerty mhagger@blizzard.harvard.edu [Taken from Gnuplot.py file comments:] # Features: # + A gnuplot session is an instance of class `Gnuplot', so multiple # sessions can be open at once: # g1 = Gnuplot.Gnuplot(); g2 = Gnuplot.Gnuplot() # + The implicitly-generated gnuplot commands can be stored to a file # instead of executed immediately: # g = Gnuplot.Gnuplot("commands.gnuplot") # The file can then be run later with gnuplot's `load' command. # Note, however, that this option does not cause the life of any # temporary data files to be extended. # + Can pass arbitrary commands to the gnuplot command interpreter: # g("set pointsize 2") # + A Gnuplot object knows how to plot three types of `PlotItem': # `Data', `File', and `Func'tion. See those classes for # information. # + Any PlotItem can have optional `title' and/or `with' suboptions. # + Builtin PlotItem types: # * Data(array1) -- data from a Python list or NumPy array # (permits additional option `cols') # * File("filename") -- data from an existing data file (permits # additional option `using') # * Func("exp(4.0 * sin(x))") -- functions (passed as a string # for gnuplot to evaluate) # + PlotItems are implemented as objects that can be assigned to # variables (including their options) and plotted # repeatedly---this also saves much of the overhead of plotting # the same data multiple times. # + Communication of data between python and gnuplot is via # temporary files, which are deleted automatically when their # associated PlotItem is deleted. (Communication of commands is # via a pipe.) The PlotItems currently in use by a Gnuplot object # are stored in an internal list so that they won't be deleted # prematurely. # + Can use `replot' method to add datasets to an existing plot. # + Can make persistent gnuplot windows by using the constructor # option `persist=1'. (`persist' is no longer the default.) Such # windows stay around even after the gnuplot program is exited. # Note that only newer version of gnuplot support this option. # + Plotting to a postscript file is via new `hardcopy' method, # which outputs the currently-displayed plot to either a # postscript printer or to a postscript file. # + There is a `plot' command which is roughly compatible with the # command from the old Gnuplot.py. # # Restrictions: # - Relies on the Numeric Python extension. This can be obtained # from LLNL (See ftp://ftp-icf.llnl.gov/pub/python/README.html). # If you're interested in gnuplot, you would probably also want # NumPy anyway. # - Probably depends on a unix-type environment. Anyone who wants # to remedy this situation should get in contact with me. # - Only a small fraction of gnuplot functionality is implemented as # explicit Gnuplot method functions. However, you can give # arbitrary commands to gnuplot manually; for example: # g = Gnuplot.Gnuplot() # g('set data style linespoints') # g('set pointsize 5') # etc. I might add a more organized way of setting arbitrary # options. # - Only 2-d plots are supported so far. # - There is no provision for missing data points in array data # (which gnuplot would allow by specifying `?' as a data point). # I can't think of a clean way to implement this; maybe one could # use NaN for machines that support IEEE floating point. # - There is no supported way to change the plotting options of # PlotItems after they have been created. # - The object-oriented interface doesn't automatically plot # datasets column-by-column using 1:2, 1:3, 1:4, etc as did the # old version of Gnuplot.py. Instead, make a temporary data file # then plot that file multiple times with different `using=' # options: # a = Gnuplot.TemparrayFile(array_nx3) # g.plot(Gnuplot.File(a, using=(1,2)), Gnuplot.File(a, using=(1,3))) # - Does not support parallel axis plots, as did the old Gnuplot.py. # # Bugs: # - No attempt is made to check for errors reported by gnuplot (but # they will appear on stderr). # - All of these classes perform their resource deallocation when # __del__ is called. If you delete things explicitly, there will # be no problem. If you don't, an attempt is made to delete # remaining objects when the interpreter is exited, but this is # not completely reliable, so sometimes temporary files will be # left around. If anybody knows how to fix this problem, please # let me know. From mhagger@blizzard.harvard.edu Thu Nov 19 06:02:06 1998 From: mhagger@blizzard.harvard.edu (mhagger@blizzard.harvard.edu) Date: Thu, 19 Nov 1998 01:02:06 -0500 (EST) Subject: [Matrix-SIG] (no subject) Message-ID: <199811190602.BAA15661@python.org> From mhagger@blizzard.harvard.edu Thu Nov 19 00:00:13 1998 From: mhagger@blizzard.harvard.edu (Michael Haggerty) Date: Wed, 18 Nov 1998 19:00:13 -0500 Subject: [Plot-SIG] ANNOUNCE: New Gnuplot.py (plotting from python using gnuplot) Message-ID: <9811190000.AA28852@tsunami.harvard.edu> Hi, I would like to announce the release of a major new version of Gnuplot.py. Gnuplot.py is an interface between python and the gnuplot plotting program. Gnuplot.py can be obtained from The module has extensive documentation in the form of comments and doc-strings within the code, and a simple demonstration can be run by typing `python Gnuplot.py'. Of course you must have the gnuplot program installed to use Gnuplot.py; see to obtain gnuplot. This version was inspired and partly derived from an earlier Gnuplot.py written by Konrad Hinsen, and should be drop-in compatible with the older version. However, the new version adds an object-oriented interface, which is much more flexible than the old functional interface. Konrad tells me that he plans to declare the old Gnuplot.py obsolete and recommend migration to this one. Example of creating a simple plot using the new interface: import Gnuplot g = Gnuplot.Gnuplot() # Start up a gnuplot process g.title('Simple plot') # Can add a title g.xlabel('time') # Can set the x axis label g.plot(array, title='Data') # `array' is an Nx2 array of numbers; # `Data' goes in the plot's key Some features and limitations are described below. Yours, Michael -- Michael Haggerty mhagger@blizzard.harvard.edu [Taken from Gnuplot.py file comments:] # Features: # + A gnuplot session is an instance of class `Gnuplot', so multiple # sessions can be open at once: # g1 = Gnuplot.Gnuplot(); g2 = Gnuplot.Gnuplot() # + The implicitly-generated gnuplot commands can be stored to a file # instead of executed immediately: # g = Gnuplot.Gnuplot("commands.gnuplot") # The file can then be run later with gnuplot's `load' command. # Note, however, that this option does not cause the life of any # temporary data files to be extended. # + Can pass arbitrary commands to the gnuplot command interpreter: # g("set pointsize 2") # + A Gnuplot object knows how to plot three types of `PlotItem': # `Data', `File', and `Func'tion. See those classes for # information. # + Any PlotItem can have optional `title' and/or `with' suboptions. # + Builtin PlotItem types: # * Data(array1) -- data from a Python list or NumPy array # (permits additional option `cols') # * File("filename") -- data from an existing data file (permits # additional option `using') # * Func("exp(4.0 * sin(x))") -- functions (passed as a string # for gnuplot to evaluate) # + PlotItems are implemented as objects that can be assigned to # variables (including their options) and plotted # repeatedly---this also saves much of the overhead of plotting # the same data multiple times. # + Communication of data between python and gnuplot is via # temporary files, which are deleted automatically when their # associated PlotItem is deleted. (Communication of commands is # via a pipe.) The PlotItems currently in use by a Gnuplot object # are stored in an internal list so that they won't be deleted # prematurely. # + Can use `replot' method to add datasets to an existing plot. # + Can make persistent gnuplot windows by using the constructor # option `persist=1'. (`persist' is no longer the default.) Such # windows stay around even after the gnuplot program is exited. # Note that only newer version of gnuplot support this option. # + Plotting to a postscript file is via new `hardcopy' method, # which outputs the currently-displayed plot to either a # postscript printer or to a postscript file. # + There is a `plot' command which is roughly compatible with the # command from the old Gnuplot.py. # # Restrictions: # - Relies on the Numeric Python extension. This can be obtained # from LLNL (See ftp://ftp-icf.llnl.gov/pub/python/README.html). # If you're interested in gnuplot, you would probably also want # NumPy anyway. # - Probably depends on a unix-type environment. Anyone who wants # to remedy this situation should get in contact with me. # - Only a small fraction of gnuplot functionality is implemented as # explicit Gnuplot method functions. However, you can give # arbitrary commands to gnuplot manually; for example: # g = Gnuplot.Gnuplot() # g('set data style linespoints') # g('set pointsize 5') # etc. I might add a more organized way of setting arbitrary # options. # - Only 2-d plots are supported so far. # - There is no provision for missing data points in array data # (which gnuplot would allow by specifying `?' as a data point). # I can't think of a clean way to implement this; maybe one could # use NaN for machines that support IEEE floating point. # - There is no supported way to change the plotting options of # PlotItems after they have been created. # - The object-oriented interface doesn't automatically plot # datasets column-by-column using 1:2, 1:3, 1:4, etc as did the # old version of Gnuplot.py. Instead, make a temporary data file # then plot that file multiple times with different `using=' # options: # a = Gnuplot.TemparrayFile(array_nx3) # g.plot(Gnuplot.File(a, using=(1,2)), Gnuplot.File(a, using=(1,3))) # - Does not support parallel axis plots, as did the old Gnuplot.py. # # Bugs: # - No attempt is made to check for errors reported by gnuplot (but # they will appear on stderr). # - All of these classes perform their resource deallocation when # __del__ is called. If you delete things explicitly, there will # be no problem. If you don't, an attempt is made to delete # remaining objects when the interpreter is exited, but this is # not completely reliable, so sometimes temporary files will be # left around. If anybody knows how to fix this problem, please # let me know. _______________________________________________ Plot-SIG maillist - Plot-SIG@python.org http://www.python.org/mailman/listinfo/plot-sig From jhauser@ifm.uni-kiel.de Thu Nov 19 08:26:04 1998 From: jhauser@ifm.uni-kiel.de (Janko Hauser) Date: Thu, 19 Nov 1998 09:26:04 +0100 (CET) Subject: [Matrix-SIG] broadcast rules In-Reply-To: References: Message-ID: <13907.54506.382464.944237@lisboa.ifm.uni-kiel.de> Broadcasting doesn't really exist for assignment. Only scalars can be broadcasted. So this also doesn't work: >>> a[:]=array([2]) Traceback (innermost last): File "", line 1, in ? ValueError: matrices are not aligned for copy But this does: >>> a[:]=3 >>> a array([[3, 3], [3, 3], [3, 3], [3, 3]]) So the answer to your question is ``Yes'' __Janko From jhauser@ifm.uni-kiel.de Thu Nov 19 08:30:45 1998 From: jhauser@ifm.uni-kiel.de (Janko Hauser) Date: Thu, 19 Nov 1998 09:30:45 +0100 (CET) Subject: [Matrix-SIG] Special functions with Numerical Python In-Reply-To: References: <199811181700.MAA06901@python.org> Message-ID: <13907.54830.243344.851790@lisboa.ifm.uni-kiel.de> I have looked into this, but haven't done anything. A good candidate for a library is the cephes package from netlib, because it has many special functions for different numerical types. But to be really useful one would build something like ufunc. This would be fast and independent of dimensions. __Janko From johann@physics.berkeley.edu Thu Nov 19 18:54:08 1998 From: johann@physics.berkeley.edu (Johann Hibschman) Date: Thu, 19 Nov 1998 10:54:08 -0800 (PST) Subject: [Matrix-SIG] Special functions with Numerical Python In-Reply-To: (message from Travis Oliphant on Wed, 18 Nov 1998 17:34:04 -0600 (CST)) References: Message-ID: <199811191854.KAA21076@leporello.berkeley.EdU> Travis Oliphant writes: > Does anybody know of a package that allows evaluation of special > functions, like bessel, airy, or error functions on Numerical Python > arrays. I don't know of any. If you want to see examples of C modules which use python arrays, I have a basic linkage module to NR in C bessik (I and K) routines on my web page, http://astro.berkeley.edu/~johann/python/numpy.html It's not complete, but it sketches the idea. --Johann From Warren B. Focke" Message-ID: [me: Are brodcast rules for arithmetic and assignment expected to be different?] On Thu, 19 Nov 1998, Janko Hauser wrote: > So the answer to your question is ``Yes'' Ok. > Broadcasting doesn't really exist for assignment. Only scalars can be > broadcasted. But it's not quite this restrictive. Further poking reveals that, while length-1 dimensions are not broadcast on assingment (this differs from how it is handled for arithmetic), missing dimensions on the left are (this is the same way it is handled in arithmetic). >>> big=ones((5,3,2)) >>> junk=big+ones((2,)) >>> junk=big+ones((3,2)) >>> junk=big+ones((5,3)) Traceback (innermost last): File "", line 1, in ? ValueError: frames are not aligned >>> big[...]=ones((2,)) >>> big[...]=ones((3,2)) >>> big[...]=ones((5,3)) Traceback (innermost last): File "", line 1, in ? ValueError: matrices are not aligned for copy Warren Focke From HYoon@exchange.ml.com Sat Nov 21 23:03:46 1998 From: HYoon@exchange.ml.com (Yoon, Hoon (CICG - NY Program Trading)) Date: Sat, 21 Nov 1998 18:03:46 -0500 Subject: [Matrix-SIG] Special functions with Numerical Python Message-ID: Since we are on the subject. Has anyone did any work on Loess, GAM, GLM, or any statistical method in Numeric? Most of the code for those resides at http://netlib.bell-labs.com/netlib/a/, but I had hard time finding Libraries that goes with it. I know this is not NumPy Q, but if anyone knows... dloess packages: cc -o gas.x gas.o loessc.o loess.o predict.o misc.o loessf.o -llinpack -lcor -lm -lF77 ld: fatal: library -llinpack: not found ld: fatal: library -lcor: not found ld: fatal: File processing errors. No output written to gas.x *** Error code 1 make: Fatal error: Command failed for target `gas.x' (BTW: I tried contacting all the authors, but all messages just bounced back. Also this is where most of Splus' higher level regression models came from. I suspect R is using same thing (free version of S based on Scheme).) Thanks, ************************************************************** S. Hoon Yoon (Quant) Merrill Lynch Equity Trading yelled@yahoo.com hoon@bigfoot.com(w) "Miracle is always only few standard deviations away, but so is catastrophe." * Expressed opinions are often my own, but NOT my employer's. "I feel like a fugitive from the law of averages." Maudin ************************************************************** > -----Original Message----- > From: Johann Hibschman > Sent: Thursday, November 19, 1998 1:54 PM > To: Oliphant.Travis@mayo.edu > Cc: matrix-sig@python.org > Subject: Re: [Matrix-SIG] Special functions with Numerical Python > > Travis Oliphant writes: > > > Does anybody know of a package that allows evaluation of special > > functions, like bessel, airy, or error functions on Numerical Python > > arrays. > > I don't know of any. If you want to see examples of C modules which > use python arrays, I have a basic linkage module to NR in C bessik (I > and K) routines on my web page, > > http://astro.berkeley.edu/~johann/python/numpy.html > > It's not complete, but it sketches the idea. > > --Johann > > _______________________________________________ > Matrix-SIG maillist - Matrix-SIG@python.org > http://www.python.org/mailman/listinfo/matrix-sig From Oliphant.Travis@mayo.edu Mon Nov 9 10:56:12 1998 From: Oliphant.Travis@mayo.edu (Travis E. Oliphant) Date: Mon, 09 Nov 1998 04:56:12 -0600 Subject: [Matrix-SIG] Special functions in NumPy References: <199811211700.MAA14541@python.org> Message-ID: <3646CA4C.7EBEEC2@mayo.edu> To those interested in special functions under NumPy: Thanks for the suggestions I've received. I've spent a little while playing with the umathmodule.c code and managed to get a working (but minimal) cephes module made --- ufunc style. I must say I am very impressed with the ufunc interface. It was really a snap to get a function working with NumPy arrays and types. Great job to thos involved in its design and coding. I don't expect it to take too long to have most of the cephes (double) library wrapped up into a beta module. If anyone has anymore suggestions regarding this project please let me know. Thanks, Travis -- Travis Oliphant 200 First St SW Graduate Student Rochester MN 55901 Ultrasound Research Lab (507) 286-5923 Mayo Graduate School Oliphant.Travis@mayo.edu From vanandel@ucar.edu Mon Nov 23 17:30:53 1998 From: vanandel@ucar.edu (Joe Van Andel) Date: Mon, 23 Nov 1998 10:30:53 -0700 Subject: [Matrix-SIG] advocacy - happy users of NumPy? Message-ID: <199811231730.KAA02989@stout.atd.ucar.edu> Having been freshly inspired by attending the Python Conference, I want to try to sell NumPy to my management as an alternative to using MATLAB, PV-WAVE, and IDL. Would any of you be willing to send me your success stories about how you/your organization has used NumPy (and associated packages) to build data analysis software? It would be particularly helpful if you could supply a URL to any results generated by your tools, and/or explanations of your NumPy based tools. Please indicate if you'd be willing to have your comments added to a Numeric Python Satisfied User's Page. Of course, if you had problems using NumPy, I'd sure like to know why, as well . . . I'd like to show that this approach worked well for other scientific researchers, and that it isn't totally "off-the-wall". Thanks much for your help! Joe VanAndel Internet: vanandel@ucar.edu National Center for http://www.atd.ucar.edu/~vanandel/home.html Atmospheric Research From David.Buscher@durham.ac.uk Mon Nov 23 19:06:32 1998 From: David.Buscher@durham.ac.uk (David Buscher) Date: Mon, 23 Nov 1998 19:06:32 +0000 (BST) Subject: [Matrix-SIG] FITS image file interface for NumPy Message-ID: Please find enclosed a Python module to read and write FITS image files to/from NumPy arrays. FITS is a format widely used in the astronomical community for storage and interchange of data, especially multidimensional image data. The code is so short, it does not seem worth the trouble of putting it on an ftp archive. Others have probably written the same thing themselves, but I couldn't find one anywhere, and I wanted to save others needing to reinvent this wheel again. The module defines only a Read() and a Write() function. The comments in the module should be sufficient for people to work out how to use it, but please let me know if it is too obscure. I was going to add proper docstrings and a test function, but waiting for the spare time to do that has already taken 3 months. enjoy, David p.s. This code has been regularly used on various types of data, but the usual caveats apply... ----------------------------------------+------------------------------------- David Buscher | Phone +44 191 374 7462 Dept of Physics, University of Durham, | Fax +44 191 374 3709 South Road, Durham DH1 3LE, UK | Email david.buscher@durham.ac.uk ----------------------------------------+------------------------------------- #!/usr/bin/env python # $Id: FITS.py,v 1.1 1998/08/24 14:49:57 dfb Exp dfb $ # # Functions to read and write FITS image files # For more information on the FITS format, see http://www.cv.nrao.edu/fits. # # This module reads and writes only the image formats, not the # more complex binary and ascii tables. # # It will handle 8, 16 and 32 bit integer and 32 bit float formats, but # it does not handle the recently-proposed 64-bit format. This would be # trivial to do, I have just never come across any data in this format # that could act as a test input. It also does not handle the obscure # 'groups' format. import string from Numeric import * error = 'FITS error' # # Read a FITS image file # # Returns a header, image tuple # The header is returned in two forms, a raw list of header records, and # a parsed dictionary for keyword access. # # By default, the image is returned as floats, using the BSCALE and BZERO # keywords where available. By setting the asFloat keyword to zero it is # returned unscaled and in the (presumably more compact) numeric format # that it was stored in # def Read(filename, asFloat = 1) : file = open(filename, "r") header = {} rawHeader = [] buffer = file.read(2880) if buffer[:6] != 'SIMPLE' : raise error, 'Not a simple fits file' while(1) : for char in range(0,2880,80) : line = buffer[char:char+80] rawHeader.append(line) key = string.strip(line[:8]) if key : val = line[9:] val = string.strip(val) if val : if val[0] == "'" : pos = string.index(val,"'",1) val = val[1:pos] else : pos = string.find(val, '/') if pos != -1 : val = val[:pos] header[key] = val if header.has_key('END') : break buffer = file.read(2880) naxis = string.atoi(header['NAXIS']) shape = [] for i in range(1,naxis+1) : shape.append(string.atoi(header['NAXIS%d' % i])) shape.reverse() numPix = 1 for i in shape : numPix = numPix * i bitpix = string.atoi(header['BITPIX']) if bitpix == 8 : type = UnsignedInt8 elif bitpix == 16 : type = Int16 elif bitpix == 32 : type = Int32 elif bitpix == -32 : type = Float32 bitpix = 32 numByte = numPix * bitpix/8 data = file.read(numByte) data = fromstring(data, type) data.shape = shape if LittleEndian : data = data.byteswapped() if asFloat : bscale = string.atof(header.get('BSCALE', '1.0')) bzero = string.atof(header.get('BZERO', '0.0')) data = data*bscale + bzero return( { 'raw' : rawHeader, 'parsed' : header}, data ) # # Save a numeric array to a FITS image file # # The FITS image dimensions are taken from the array dimensions. # The data is saved in the numeric format of the array, or as 32-bit # floats if a non-FITS data type is passed. # # A list of extra header records can be passed in extraHeader. The # SIMPLE, BITPIX, NAXIS* and END records in this list are ignored. # Header records are padded to 80 characters where necessary. # def Write(data, filename, extraHeader = None) : type = data.typecode() if type == UnsignedInt8 : bitpix = 8 elif type == Int16 : bitpix = 16 elif type == Int32 : bitpix = 32 else : data = data.astype(Float32) bitpix = -32 shape = list(data.shape) shape.reverse() naxis = len(shape) header = [ 'SIMPLE = T', 'BITPIX = %d' % bitpix, 'NAXIS = %d' % naxis ] for i in range(naxis) : keyword = 'NAXIS%d' % (i + 1) keyword = string.ljust(keyword, 8) header.append('%s= %d' % (keyword, shape[i])) if extraHeader != None : for rec in extraHeader : try : key = string.split(rec)[0] except IndexError : pass else : if key != 'SIMPLE' and \ key != 'BITPIX' and \ key[:5] != 'NAXIS' and \ key != 'END' : header.append(rec) header.append('END') header = map(lambda x: string.ljust(x,80)[:80], header) header = string.join(header,'') numBlock = (len(header) + 2880 - 1) / 2880 header = string.ljust(string.join(header,''), numBlock*2880) file = open(filename, 'w') file.write(header) if LittleEndian : data = data.byteswapped() data = data.tostring() file.write(data) numBlock = (len(data) + 2880 - 1) / 2880 padding = ' ' * (numBlock*2880 - len(data)) file.write(padding) From Oliphant.Travis@mayo.edu Mon Nov 23 20:21:45 1998 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Mon, 23 Nov 1998 14:21:45 -0600 (CST) Subject: [Matrix-SIG] Cephes module In-Reply-To: <199811221700.MAA24636@python.org> Message-ID: Hi all, To anyone interested in special functions under numerical python I have released a version of the cephes library wrapped up as ufunc objects (patterned after the umath module). The package is at http://oliphant.netpedia.net/ and it includes all the necessary source-code from the cephes package to create a working cephes module that will allow calculation of bessel functions, elliptic integrals, chi-square distributions, etc. The documentation is pretty sparse but all of the routines are described in the cephes library documentation which is also included. The result, seems to work quite well. Again, I am impressed with the ufunc object. Thanks for any input. Travis ---------------------------------------------------- Travis Oliphant 200 First St SW Rochester MN 55905 Ultrasound Research Lab (507) 286-5923 Mayo Graduate School Oliphant.Travis@mayo.edu From barrett@compass.gsfc.nasa.gov Mon Nov 23 22:27:16 1998 From: barrett@compass.gsfc.nasa.gov (Paul Barrett) Date: Mon, 23 Nov 1998 17:27:16 -0500 (EST) Subject: [Matrix-SIG] Implementing a Record type. Message-ID: <13913.55048.473164.985681@compass.gsfc.nasa.gov> I have been working on implementing a 'record' object for Python for the past few weeks and have run into an implementation problem. I'm hoping someone might lead me in the right direction. This 'record' object that I'm trying to implement is basically a generalization of the 'struct' module which enables packing and unpacking of data in strings. The difference is that a 'record' is an object with array behaviour (as well as a few other features like being able name an item of the record). I also plan to provide a C API for this object, so that I can embed it in another object which is an array of records, like the array object in the numeric module. The problem that I am having is how to deal with the data string. For a single record, this is not much of an issue, but for arrays of records where the data is contiguous, this becomes more difficult, at least from my perspective. For example, suppose we have a 1-D array of records and that we want to manipulate one record (i.e. 'rec' = array_record[n]). When I access this record, should I be accessing a copy of this record or a pointer into the string buffer? The problem that I see is that changing the copy does not change the original which is what I would expect it to do. In addition, the latter implementation appears to be more complicated, since your required to keep track of the entire string buffer in case it disappears at some stage. Any suggestions or thoughts on how best to implement this object type will be most appreciated. Cheers, Paul From Oliphant.Travis@mayo.edu Mon Nov 23 23:52:48 1998 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Mon, 23 Nov 1998 17:52:48 -0600 (CST) Subject: [Matrix-SIG] Cephes module In-Reply-To: <199811191700.MAA20209@python.org> Message-ID: To all interested, I changed the cephes module to version 0.51 and fixed a bug that was causing strange errors when cephes was used with Numeric (umath really) (which of course it's supposed to be used with). Get it at http://oliphant.netpedia.net Sorry to any who downloaded version 0.5 and got the strange errors. ---------------------------------------------------- Travis Oliphant 200 First St SW Rochester MN 55905 Ultrasound Research Lab (507) 286-5923 Mayo Graduate School Oliphant.Travis@mayo.edu From Oliphant.Travis@mayo.edu Tue Nov 24 09:29:26 1998 From: Oliphant.Travis@mayo.edu (Travis E. Oliphant) Date: Tue, 24 Nov 1998 03:29:26 -0600 Subject: [Matrix-SIG] Cephes Module References: <3659EC12.F0B42A38@nmr.mgh.harvard.edu> Message-ID: <365A7C76.4B8B6AE8@mayo.edu> Gary, My testing was a bit premature. My recently reported "fix" for cephes wasn't a true fix for the problem. I have since discovered the real cause of the problem. (I was calling a borrowed subroutine from the umath module---where I took the skeleton code from that I shouldn't have been calling.) I have now corrected the problem and cephes seems to work just fine with Numeric under my Linux i386 platform. I expect it will work with a Sparc 10 as well. Sorry for any inconvenience. cephes-0.52 is now available.... http://oliphant.netpedia.net/packages/cephes-0.52.tgz -Travis -- Travis Oliphant 200 First St SW Rochester MN 55901 Ultrasound Research Lab (507) 286-5923 Mayo Graduate School Oliphant.Travis@mayo.edu From jh@oobleck.tn.cornell.edu Tue Nov 24 18:15:52 1998 From: jh@oobleck.tn.cornell.edu (Joe Harrington) Date: Tue, 24 Nov 1998 13:15:52 -0500 Subject: [Matrix-SIG] advocacy Message-ID: <199811241815.NAA23598@oobleck.tn.cornell.edu> I too have been searching for a free data analysis environment. I currently use the abominable IDL. For the average scientist, IDL is still the best that's out there, and this makes me glum. Python has the promise to be *much* more, but it's not there yet. Mostly this is a problem with the add-ons, not the language itself. When I tried doing a few projects with Numeric, here's what I ran into: Plotting capability is primitive compared to IDL, better plotting might be available but is tedious/difficult to integrate, and there are licensing issues. The breadth of numerical routines is small, and most have very limited implementations. Data types I use (FITS) are poorly supported, if at all. To do anything, you need to do a lot of legwork pulling together disparate packages and integrating them into Python. Documentation is almost nonexistent -- basically you must use the force, read the source. Things are very disorganized; different packages offer overlapping routines that don't work together. To a hacker interested in Python, much of the above is not a big deal. Just do the legwork, read the source code comments for documentation, write what you need. To most scientists, this is unacceptable. They'd rather support a commercial organization they don't generally like (RSI) to make a poor language like IDL have a slick appearance, a broad, well-implemented set of numerical and plotting routines, and good docs. It's the lesson of Windows vs. Mac: to most people the core system doesn't matter, the applications do. A year and a half ago, Paul Barrett and I put in a proposal to NASA with a number of list members as Collaborators. The proposal would have covered some programmers' and doc writers' salaries to make and distribute a coherent package, but it wasn't selected for funding. Given these problems, I sadly put Python aside, though I've been lurking for the past year in hopes of seeing a change or catching an opportunity to do something. Today's digest contained both your message and the announcement of some FITS routines (I wish Paul Barrett would release his class library!). Perhaps this is a good time to step back and ask the community a few questions about where it wants Numerical Python to go. It's great to have the flexibility of modules, but to be successful among non-hacker scientists there has to be a "full version" with all the packages built, a well-organized structure, good docs, plotting capability, and support for all popular data formats. The folks at LLNL have the basics going, but progress at the current rate won't get NumPy to compete with IDL (or several others) any time soon. This is not at all intended as a criticism of LLNL or the list's current level of activity. Making a fleshed-out package will be a lot of work, and will require some real commitment and a lot of late hours from a lot of people. It would mean real coordination, people volunteering time and being tasked with sometimes boring jobs, establishing a means to make overall design decisions and sticking with them, etc. Are list members willing and interested in an organized development effort aimed at creating a competitor to IDL? --jh-- Joe Harrington Cornell University Space Sciences From ffjhl@uaf.edu Tue Nov 24 23:43:06 1998 From: ffjhl@uaf.edu (Jonah Lee) Date: Tue, 24 Nov 1998 14:43:06 -0900 (AKST) Subject: [Matrix-SIG] advocacy In-Reply-To: <199811241815.NAA23598@oobleck.tn.cornell.edu> Message-ID: On Tue, 24 Nov 1998, Joe Harrington wrote: I'd be very happy to contribute. I've started working, on a small scale, some plotting interfaces using Python. At this moment, despite the existence of the SIG, activities are quite disorganized. If we could muster enough core developers we may be able to get a product that we are all proud of that does justice to Python.:-) The KDE and GNOME projects, to me, are quite inspiring. Regards, Jonah PSA Member > I too have been searching for a free data analysis environment. I > currently use the abominable IDL. For the average scientist, IDL is > still the best that's out there, and this makes me glum. Python has > the promise to be *much* more, but it's not there yet. Mostly this is > a problem with the add-ons, not the language itself. When I tried > doing a few projects with Numeric, here's what I ran into: > > Plotting capability is primitive compared to IDL, better plotting > might be available but is tedious/difficult to integrate, and > there are licensing issues. > The breadth of numerical routines is small, and most have very > limited implementations. > Data types I use (FITS) are poorly supported, if at all. > To do anything, you need to do a lot of legwork pulling together > disparate packages and integrating them into Python. > Documentation is almost nonexistent -- basically you must use the > force, read the source. > Things are very disorganized; different packages offer overlapping > routines that don't work together. > > To a hacker interested in Python, much of the above is not a big deal. > Just do the legwork, read the source code comments for documentation, > write what you need. To most scientists, this is unacceptable. > They'd rather support a commercial organization they don't generally > like (RSI) to make a poor language like IDL have a slick appearance, a > broad, well-implemented set of numerical and plotting routines, and > good docs. It's the lesson of Windows vs. Mac: to most people the > core system doesn't matter, the applications do. > > A year and a half ago, Paul Barrett and I put in a proposal to NASA > with a number of list members as Collaborators. The proposal would > have covered some programmers' and doc writers' salaries to make and > distribute a coherent package, but it wasn't selected for funding. > > Given these problems, I sadly put Python aside, though I've been > lurking for the past year in hopes of seeing a change or catching an > opportunity to do something. Today's digest contained both your > message and the announcement of some FITS routines (I wish Paul > Barrett would release his class library!). Perhaps this is a good > time to step back and ask the community a few questions about where it > wants Numerical Python to go. > > It's great to have the flexibility of modules, but to be successful > among non-hacker scientists there has to be a "full version" with all > the packages built, a well-organized structure, good docs, plotting > capability, and support for all popular data formats. The folks at > LLNL have the basics going, but progress at the current rate won't get > NumPy to compete with IDL (or several others) any time soon. This is > not at all intended as a criticism of LLNL or the list's current level > of activity. Making a fleshed-out package will be a lot of work, and > will require some real commitment and a lot of late hours from a lot > of people. It would mean real coordination, people volunteering time > and being tasked with sometimes boring jobs, establishing a means to > make overall design decisions and sticking with them, etc. Are list > members willing and interested in an organized development effort > aimed at creating a competitor to IDL? > > --jh-- > Joe Harrington > Cornell University Space Sciences > > _______________________________________________ > Matrix-SIG maillist - Matrix-SIG@python.org > http://www.python.org/mailman/listinfo/matrix-sig > --------------------------------------------------------------------- | I do not use Microsoft Word. | Linux Rules! www.linux.org | |Please save *.doc file as *.txt | LyX Rules! www.lyx.org | |before sending it as an attachment.| Python Rules! www.python.org | | Many Thanks For Doing This. | | --------------------------------------------------------------------- From ransom@cfa.harvard.edu Wed Nov 25 02:31:18 1998 From: ransom@cfa.harvard.edu (Scott M. Ransom) Date: Wed, 25 Nov 1998 02:31:18 +0000 Subject: [Matrix-SIG] advocacy References: <199811241815.NAA23598@oobleck.tn.cornell.edu> Message-ID: <365B6BF5.E9390625@cfa.harvard.edu> I agree with almost everything you said. And since I am a recent convert from IDL to Python, it is certainly in my interest to see a project such as this succeed. There is quite a serious problem, however. As you stated: > Python has the promise to be *much* more, but it's not there yet. > Mostly this is a problem with the add-ons, not the language itself. and > It's great to have the flexibility of modules, but to be successful > among non-hacker scientists there has to be a "full version" with all > the packages built, a well-organized structure, good docs, plotting > capability, and support for all popular data formats. This is the crux: The module or add-on system is certainly the way to go if you want stability, speed, and well tested code. For example if you want fast FFTs, FFTW is virtually the only solution for fast _and_ portable code. And extremely well-tested, black-box style numerical algotithms are common, freely available, and complete, on NetLib. If we were to try and duplicate these efforts specifically for Python, which a "full version" would probably require, and achieve even half of the successes of the codes we were copying, we would have a _huge_ undertaking. We either have to sacrifice some stability, speed, or bullet-proofness in order to get a tightly integrated set of tools written specifically for the "full version" of SciPy (I kinda like that name, BTW...). Or we figure out some way to embrace the module system and make it work. A couple possibilities for the second option: 1. Have a maintainer for each major platform (Linux, AIX, Cray, HP Unix, Sparc, Irix, DUX, Windows (yuk) etc) who patches, compiles, and links the modules and then provides binary distributions of a working SciPy. An example of this would be NASA's binary distributions of FTOOLS which are notoriously difficult to patch/compile/link. 2. Develop a more robust package building system. I know this is a major topic of discussion for Python in general, but it seems particularly applicable to our problem. Whatever we decide (if anything) I am certainly willing to help. I have switched to Python for the long-haul (and have managed to convince a few of my collaborators to as well ;) Scott Ransom -- Scott M. Ransom Phone: (580) 536-7215 Address: 703 SW Chaucer Cir. email: ransom@cfa.harvard.edu Lawton, OK 73505 PGP Fingerprint: D2 0E D0 10 CD 95 06 DA EF 78 FE 2B CB 3A D3 53 From frohne@alaska.net Wed Nov 25 07:08:48 1998 From: frohne@alaska.net (Ivan Frohne) Date: Tue, 24 Nov 1998 22:08:48 -0900 Subject: [Matrix-SIG] advocacy In-Reply-To: <365B6BF5.E9390625@cfa.harvard.edu> Message-ID: <000001be1842$72fddae0$359170d1@vladimir> -- -- We either have to sacrifice some stability, speed, or bullet-proofness in order to get a tightly integrated set of tools written specifically for the "full version" of SciPy (I kinda like that name, BTW...). Well, I like sPy better. --Ivan Frohne From hinsen@cnrs-orleans.fr Wed Nov 25 13:34:58 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Wed, 25 Nov 1998 14:34:58 +0100 Subject: [Matrix-SIG] advocacy In-Reply-To: <199811241815.NAA23598@oobleck.tn.cornell.edu> (message from Joe Harrington on Tue, 24 Nov 1998 13:15:52 -0500) References: <199811241815.NAA23598@oobleck.tn.cornell.edu> Message-ID: <199811251334.OAA16908@dirac.cnrs-orleans.fr> > a problem with the add-ons, not the language itself. When I tried > doing a few projects with Numeric, here's what I ran into: ... These are indeed serious problems if what you want is a complete "scientific workbench" kind of system. I'd love to see this in Python, and I have no doubt it could be done, but it requires a level of organization that does not seem to be available at the moment. LLNL is taking care of NumPy maintenance, but there is no real organized development activity. On the other hand, for another large group of applications, the current state is sufficient to make it the best alternative. For example my work: computational chemistry. I don't need the majority of operations in the well-known packages (IDL, Matlab, ...), but rather specialized operations that I have to implement myself anyway. Python+NumPy is an excellent basis for such projects, much better than other systems, which may have better standard functionality, but not Python's flexibility for extension. > A year and a half ago, Paul Barrett and I put in a proposal to NASA > with a number of list members as Collaborators. The proposal would > have covered some programmers' and doc writers' salaries to make and > distribute a coherent package, but it wasn't selected for funding. So do it again! I think this is the only way to succeed, there must be people dedicated to organizing a coherent development effort. > Barrett would release his class library!). Perhaps this is a good > time to step back and ask the community a few questions about where it > wants Numerical Python to go. Wishes are one thing, but someone must actually do the work and/or provide the funding to get it done. Like many others, I'd be happy to contribute, but I just can't dedicate enough time to this to do major improvements on my own. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From gward@cnri.reston.va.us Wed Nov 25 14:32:58 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 25 Nov 1998 09:32:58 -0500 Subject: [Matrix-SIG] advocacy In-Reply-To: <365B6BF5.E9390625@cfa.harvard.edu>; from Scott M. Ransom on Wed, Nov 25, 1998 at 02:31:18AM +0000 References: <199811241815.NAA23598@oobleck.tn.cornell.edu> <365B6BF5.E9390625@cfa.harvard.edu> Message-ID: <19981125093258.A13306@thrak.cnri.reston.va.us> Much of this NumPy (SciPy?) 'advocacy' discussion deals with problems that are not specific to "Python for the working scientist", but to "Python for anyone except dedicated Python hackers" in general. In particular, Scott Ransom nailed this point exactly: > 2. Develop a more robust package building system. I know this is a > major topic of discussion for Python in general, but it seems > particularly applicable to our problem. Well, the good news is, you scientific guys are not alone: *everyone* recognizes the need to make building Python add-on modules and extension modules easier. The better news is, something is starting to happen: I ran a session at the recent Python Conference Developer's Day devoted to starting a project along these lines, and from that was born (in concept at least) the Python Module Distribution Utilities. Right now, there's a bit of talk on the meta-sig to get a distutils-sig started and get cracking on the work; if you'd like to lend a hand, then take a look at http://www.foretec.com/python/workshops/1998-11/dd-ward-sum.html where you'll find a summary of that Developer's Day session. I don't know if there's much point in joining the meta-sig -- this topic hasn't generated a lot of heat, and if nothing more happens soon I'll just go ask the appropriate authorities to create the silly sig so we can get on with it. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From HYoon@exchange.ml.com Wed Nov 25 14:37:15 1998 From: HYoon@exchange.ml.com (Yoon, Hoon (CICG - NY Program Trading)) Date: Wed, 25 Nov 1998 09:37:15 -0500 Subject: [Matrix-SIG] advocacy Message-ID: > http://lib.stat.cmu.edu/R/CRAN/ > > Hi, > > I am NOT advocating usage of R at all, but it's nice to see something > already there. So, we can copy and learn from it. > The above site contains R. If you know anything about S or Splus, then > you would find the free package called R very very close. By going this > way, the creator of R takes advantage of huge docs available in S and > Splus. > Frankly, I have my own problem with Splus and will only use R for > regression analysis only; however, if Python ever achieves the state of R > -> Splus then I would never bother using anything else. > It's a good place to find out how statistic code is put together and > also graphics are put together. R preserved Spluses plot, box, hist, and > most interestingly identify (check this out). This is also completely > cross platform on Unix and NT. > In addition, we can probably pick up things like dataframe and higher > statistics, which Python currently lacks. R is based on Scheme. I am sure > if they can do it in Scheme, it would be easy job with Python. > > ************************************************************** > S. Hoon Yoon (Quant) Merrill Lynch > Equity Trading > yelled@yahoo.com hoon@bigfoot.com(w) > "Miracle is always only few standard deviations away, but so is > catastrophe." > * Expressed opinions are often my own, but NOT my employer's. > "I feel like a fugitive from the law of averages." Maudin > ************************************************************** BTW: May be we can just wrap the thing? Plot will be nice to incorporate by any means. From barrett@compass.gsfc.nasa.gov Wed Nov 25 15:43:43 1998 From: barrett@compass.gsfc.nasa.gov (Paul Barrett) Date: Wed, 25 Nov 1998 10:43:43 -0500 (EST) Subject: [Matrix-SIG] advocacy In-Reply-To: <199811241815.NAA23598@oobleck.tn.cornell.edu> References: <199811241815.NAA23598@oobleck.tn.cornell.edu> Message-ID: <13916.8064.216755.648737@compass.gsfc.nasa.gov> Joe Harrington writes: > Data types I use (FITS) are poorly supported, if at all. > > [text deleted] > > ... Today's digest contained both your message and the announcement > of some FITS routines (I wish Paul Barrett would release his class > library!). ... Joe, I'm still working on the PyFITS module. I presented a poster about it at the latest ADASS Conference (Astronomical Data Analysis Software and Systems Conf. VIII; see http://monet.astro.uiuc.edu/adass98/). It's getting close, but still not really alpha yet. I still need to create a Python extension to read arrays of binary records for easy and efficient access to the data. Once this is done I will announce it for flaming, ... or should I say testing. > A year and a half ago, Paul Barrett and I put in a proposal to NASA > with a number of list members as Collaborators. The proposal would > have covered some programmers' and doc writers' salaries to make and > distribute a coherent package, but it wasn't selected for funding. I also resubmitted our proposal again this year to NASA, but it was rejected a second time. I fear NASA management just doesn't get it. I have been advocating Python to the NASA community for two years now with very little success. I did get the acknowledgement from a NASA civil servant that this is somewhat of a chicken-and-egg issue, but they don't think it is their responsibility to help. So much for the new NASA of "Better, Faster, and Cheaper" > Given these problems, I sadly put Python aside, though I've been > lurking for the past year in hopes of seeing a change or catching an > opportunity to do something. Today's digest contained both your > message and the announcement of some FITS routines (I wish Paul > Barrett would release his class library!). Perhaps this is a good > time to step back and ask the community a few questions about where it > wants Numerical Python to go. I agree with your assessment. I should, since we've discussed it before. I now feel compelled to spend those long hours late at night trying to improve the current situation (Who needs sleep anyway? ;-}). I believe that for the astronomical community that the first step is PyFITS, a Python module to manipulate FITS files. The next step is a basic 2-D plotting package. Using Gtk to implement this might be a good solution, since it would eventually be well integrated with GNOME. I feel that in the astronomical community at least Python is reaching critical mass. More and more people that I talk to are interested in using it. (There is even discussion of tossing the IRAF CL and replacing it with Python.) So we need to make a push now to get a basic data analysis system operational, if we are not to loose these people. We need a few more people to step forward and take charge of a few critical issues, for example plotting and documentation, if we are to succeed. -- Paul -- Paul Barrett - Astrophysicist - Universities Space Research Association Compton Observatory Science Support Center NASA/Goddard SFC phone: 301-286-1108 "Guk a 'mzimba, sala 'nhliziyo" Code 660.2, FAX: 301-286-1629 (body grow old, but Greenbelt,MD 20771 barrett@compass.gsfc.nasa.gov heart remain behind) http://lheawww.gsfc.nasa.gov/users/barrett/CV.html From asterian@eecs.umich.edu Wed Nov 25 16:18:05 1998 From: asterian@eecs.umich.edu (Andrew Sterian) Date: Wed, 25 Nov 1998 11:18:05 -0500 (EST) Subject: [Matrix-SIG] Using existing packages as back-ends? Message-ID: <199811251618.LAA14276@dip.eecs.umich.edu> Hi, I'm new to the Matrix-SIG so I apologize if the following has already been dealt with. I looked back a couple of months in the archives and didn't see anything relevant, so here goes. With respect to expanding the functionality of NumPy (modules, plotting, etc.) it seems to me that it would be easiest to leverage existing free software that is already mature and has already dealt with many of these issues in a standalone context. I'm thinking of MATLAB-esque programs like RLab, SciLab, and Octave. Would it not be the quickest path of growth to work on making NumPy an Python interface to one (or more) of these packages rather than redeveloping everything from scratch? Again, I apologize if this has already been answered, but I'd like to know the answer too :) Andrew. asterian@umich.edu | Me: http://www-personal.umich.edu/~asterian ----------------------------+---------------------------------------------- Faith is the willingness to get out of bed in the morning and just show up... -- Richard Thieme From jh@oobleck.tn.cornell.edu Wed Nov 25 16:13:21 1998 From: jh@oobleck.tn.cornell.edu (Joe Harrington) Date: Wed, 25 Nov 1998 11:13:21 -0500 Subject: [Matrix-SIG] advocacy In-Reply-To: <199811251334.OAA16908@dirac.cnrs-orleans.fr> (message from Konrad Hinsen on Wed, 25 Nov 1998 14:34:58 +0100) References: <199811241815.NAA23598@oobleck.tn.cornell.edu> <199811251334.OAA16908@dirac.cnrs-orleans.fr> Message-ID: <199811251613.LAA00158@oobleck.tn.cornell.edu> > > A year and a half ago, Paul Barrett and I put in a proposal to NASA > > with a number of list members as Collaborators. The proposal would > > have covered some programmers' and doc writers' salaries to make and > > distribute a coherent package, but it wasn't selected for funding. > So do it again! I think this is the only way to succeed, there must > be people dedicated to organizing a coherent development effort. We did (see Paul's email). We learned some lessons the first time around, including that there are better programs in NASA than the one we submitted to for this sort of thing. However, both of us have had workload increases since then, and supervising a software project doesn't get you promoted or tenured in this business...unfortunately. So that's why I turned to the list. > > Barrett would release his class library!). Perhaps this is a good > > time to step back and ask the community a few questions about where it > > wants Numerical Python to go. > Wishes are one thing, but someone must actually do the work and/or > provide the funding to get it done. Like many others, I'd be happy > to contribute, but I just can't dedicate enough time to this > to do major improvements on my own. Precisely my point. A few people working full time and in close communication could do this job, but that won't happen without money. On the other hand, there are several good examples (Debian, Gnome, the Linux kernel) of mailing list collaborations that have pulled off projects of this scale to good effect. They have the advantage of staying power -- no single person or group is critical to the endeavor's success. They have the disadvantage of the overhead of communications and the "many cooks" syndrome. Several people have replied that they have small amounts of time to put into such a project. I don't think it's a critical mass yet, but I'm encouraged. People are offering time first, rather than dumping tons of opinion on how to lay out the CD (flame war to that effect just erupted on the Beowulf/Extreme Linux lists). I think that if around 20 people were interested in putting in some real time (say, 100-200 hours over the next year), with testing and help from others, we could do it. Seeing the initial interest, I'll take a semi-formal count in private email. If you are interested in doing some significant work other than just testing, please email me saying so, and include the following snippet and answers: Name: Email: Fields of professional/serious amateur technical experience: Areas most interested in working on: Really good at (X all that apply): Python programming technical writing (list languages) creating web pages programming numerical algorithms (list areas) programming graphics wrapping compiled code for Python C programming C++ programming Fortran programming Unix Windows MacOS other OS Have access to the following hardware/OS combos for testing: Rough number of hours/month available: I will summarize responses, without names, to the list next Wednesday (I'm gone for 11 days following then). --jh-- From spillar@uwyo.edu Wed Nov 25 09:24:36 1998 From: spillar@uwyo.edu (Earl J. Spillar) Date: Wed, 25 Nov 1998 09:24:36 +0000 Subject: [Matrix-SIG] advocacy References: <199811241815.NAA23598@oobleck.tn.cornell.edu> <13916.8064.216755.648737@compass.gsfc.nasa.gov> Message-ID: <365BCCCD.B8ED794E@uwyo.edu> Paul Barrett wrote: > > I feel that in the astronomical community at least Python is reaching > critical mass. More and more people that I talk to are interested in > using it. (There is even discussion of tossing the IRAF CL and > replacing it with Python.) So we need to make a push now to get a > basic data analysis system operational, if we are not to loose these > people. We need a few more people to step forward and take charge of > a few critical issues, for example plotting and documentation, if we > are to succeed. > > -- Paul > I also think that we are reaching critical mass. I think I count at least 4 efforts to work with FITS files. (Mine does N-dimensional FITS files; I don't deal with extensions at all, and my handling of tables is very far from alpha- but I can and do use the simple stuff all the time. ) I am surprised at the number of astronomers in particular who have contributed written to the matrix-sig- there aren't that many of us, or so I thought! This bodes well for our efforts. I think a high priority might be to set up some sort of a web/ftp site where we can share our creations. This would: 1) prevent the re-invention of the wheel (~4 FITS readers) 2) help focus us on what needs to be done 3) show newbies that there are already some tools to work with 4) make it easier to find some of the stuff out there. Does this make sense to folks? I was about to set up a "internal" site for me and some of my collaborators devoted to astronomical image processing in python. It wasn't supposed to be anything elaborate, but we could certainly open it up to the community. There might be files there, and links to other sites. If there is already such a site, I'd like to know about it/them! The other possibility is to start up a separate SIG or something like that; but I don't think we want to steal the thunder of the matrix-sig, since our focus is probably fairly tight. Thoughts? Earl Spillar spillar@uwyo.edu University of Wyoming Director, Wyoming Infrared Observatory 2.3m Occasional Python Hacker on Openstep, Windows, Linux, Mac, etc. From Paul F. Dubois" I have previously said something along these lines and gotten no reply: What am I supposed to do with the increasing torrent of messages saying that "I've got a package that does this for NumPy"? There are often more than one such author per category. I am not in a position to test or evaluate or describe such packages. Therefore, the effort has to be distributed. But, from the point of view of users it has to be centralized. I believe some uniformity of make/install technology will be achieved. We should adopt that when available. I could make a subdirectory on ftp-icf.llnl.gov to hold "NumPyAddons". People can ftp to an "incoming" directory we have and send me email and I can move the tarball over there. The email can also have a *brief* description for me to paste into a README. I'll reply when done, you post to python.announce, et voila. Each tarball should have a README that tells at least who the author is and how to contact them. This model assumes no attempt to supply binaries. I'm not willing to offer that much disk space and effort. Perhaps there is a better way, but this much I think I can manage. Comments? -----Original Message----- From: Earl J. Spillar To: Paul Barrett ; matrix-sig@python.org Cc: norbert pirzkal Date: Wednesday, November 25, 1998 8:28 AM Subject: Re: [Matrix-SIG] advocacy > > >Paul Barrett wrote: > >> >> I feel that in the astronomical community at least Python is reaching >> critical mass. More and more people that I talk to are interested in >> using it. (There is even discussion of tossing the IRAF CL and >> replacing it with Python.) So we need to make a push now to get a >> basic data analysis system operational, if we are not to loose these >> people. We need a few more people to step forward and take charge of >> a few critical issues, for example plotting and documentation, if we >> are to succeed. >> >> -- Paul >> > >I also think that we are reaching critical mass. I think I count >at least 4 efforts to work with FITS files. (Mine does N-dimensional >FITS files; I don't deal with extensions at all, and my handling >of tables is very far from alpha- but I can and do use the simple >stuff all the time. ) I am surprised at the number of >astronomers in particular who have contributed written to the >matrix-sig- there aren't that many of us, or so I thought! This >bodes well for our efforts. > >I think a high priority might be to set up some sort of a web/ftp >site where we can share our creations. This would: > >1) prevent the re-invention of the wheel (~4 FITS readers) >2) help focus us on what needs to be done >3) show newbies that there are already some tools to work with >4) make it easier to find some of the stuff out there. > >Does this make sense to folks? I was about to set up a "internal" >site for me and some of my collaborators devoted to astronomical >image processing in python. It wasn't supposed to be anything >elaborate, but we could certainly open it up to the community. >There might be files there, and links to other sites. > >If there is already such a site, I'd like to know about it/them! > >The other possibility is to start up a separate SIG or something like >that; but I don't think we want to steal the thunder of the matrix-sig, >since our focus is probably fairly tight. > >Thoughts? > >Earl Spillar >spillar@uwyo.edu >University of Wyoming >Director, Wyoming Infrared Observatory 2.3m >Occasional Python Hacker on Openstep, Windows, Linux, Mac, etc. > >_______________________________________________ >Matrix-SIG maillist - Matrix-SIG@python.org >http://www.python.org/mailman/listinfo/matrix-sig > > From gward@cnri.reston.va.us Wed Nov 25 17:53:29 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 25 Nov 1998 12:53:29 -0500 Subject: [Matrix-SIG] advocacy In-Reply-To: <001801be1892$ccc05080$f4160218@c1004579-C.plstn1.sfba.home.com>; from Paul F. Dubois on Wed, Nov 25, 1998 at 08:43:56AM -0800 References: <001801be1892$ccc05080$f4160218@c1004579-C.plstn1.sfba.home.com> Message-ID: <19981125125329.B13357@thrak.cnri.reston.va.us> > I believe some uniformity of make/install technology will be achieved. We > should adopt that when available. I could make a subdirectory on > ftp-icf.llnl.gov to hold "NumPyAddons". People can ftp to an "incoming" > directory we have and send me email and I can move the tarball over there. > The email can also have a *brief* description for me to paste into a README. > I'll reply when done, you post to python.announce, et voila. Paul -- I think this will serve as a good interim measure until something better -- like distutils and Trove -- comes along. It certainly sounds like it will be a great service to the scientific Python community, but I hope for your sake that the flow of NumPy contributions is no more than one or two a week! Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From perry@stsci.edu Wed Nov 25 22:50:32 1998 From: perry@stsci.edu (Perry Greenfield) Date: Wed, 25 Nov 1998 17:50:32 -0500 Subject: [Matrix-SIG] re: advocacy Message-ID: <199811252250.RAA15587@eclipse.stsci.edu> Paul Barrett writes: > I feel that in the astronomical community at least Python is reaching > critical mass. More and more people that I talk to are interested in > using it. (There is even discussion of tossing the IRAF CL and > replacing it with Python.) So we need to make a push now to get a > basic data analysis system operational, if we are not to loose these > people. We need a few more people to step forward and take charge of > a few critical issues, for example plotting and documentation, if we > are to succeed. > Before any wild rumors get started, I'd like to clarify the above reference to IRAF since it is probably based on comments from me. The people that distribute IRAF (National Optical Astronomy Observatories) have no plans to get rid of the IRAF CL any time soon. We at the Space Telescope Science Institute have written much of our software under the IRAF system and are looking for an alternate scripting language for use in place of the IRAF CL. We are investigating the use of Python as an alternate CL that could still run most IRAF (or STSDAS) tasks. We figure that it will take a few months to see if this approach will work. While we are pretty confident that we can get this to work, we would like to see it work before we make any solid commitments to using and supporting Python. We--like many other astronomers apparently--have noted the shortcomings with Python as a substitute for either the IRAF CL or IDL. Our list is pretty much the same but I'd add one or two more items (and briefly repeat the others). These are ones that we feel must be addressed eventually for us to completely switch over to Python: Previously mentioned: 1) Weak plotting support (as compared to IDL) 2) Small number of numerical utilities (as compared to IDL) 3) No support (yet) for FITS (I mean full access to images, keywords, tables,...the whole 9 yards) 4) Installation hassles. 5) Documentation lacking (especially for NumPy...in the works though) To which I would add: 6) Weak interactive environment: It is not always appreciated by non-scientists that interactive mode is where most time is spent looking at data. To that end, the interactive mode needs some conveniences that are currently lacking. Such as: - command (and output) logging. - easy system escapes (e.g., !ls in IRAF, $ls in IDL) - as trivial as it sounds, an intuitive exit command (like 'exit' or 'quit') - an easy way of re-executing moderately long command sequences or blocks without using an editor (e.g., you invoke a loop of several statements repeatedly because you need to tweak some parameters each time). - utility functions to start up help, or examine the namespace for existing data elements (much like IDL does) To these one can raise one more issue. I sense that Python gurus consider IDL users to be programming adverse, but in the spectrum of astronomical users, those that use IDL tend to be the ones that are most interested in programming. We (STScI) also must concern ourselves with the other half (or whatever) that find even IDL too intimidating and are much more comfortable with the more shell-like environment of IRAF. There are a lot of users that would not want to have to quote all strings or put parenthesis around all argument lists. This doesn't have a simple solution but I'm pretty sure that ignoring this issue will preclude use of Python by many astronomers (I won't speak for other fields). 7) Image display convenient for astronomers (either by saoimage or ximtool lookalikes or connections to either of those so that images could be displayed on those tools. After such a list some might wonder why we even consider using Python at all. Essentially the existing situation has even more serious drawbacks that we are trying to escape. Should our initial efforts succeed then we need solutions to the above issues. We make our software available to a wide community for a number of different platforms so it is simply not a matter of solving these problems for our own organization. If what we deliver is too hard to install or is to hard to use or understand in the wider astronomical community it will be a failure (I'm sure some would argue that existing IRAF-based software falls into that category). So it is essential, for example, that there is a standard distribution and installation mechanism. Fortunately, there appears to be serious interest in solving this for the whole Python community. For our initial plans, these issues do not all have to be solved immediately, but we have to see a way to deal with them eventually. Hopefully the FITS issue will be solved soon--I'm eagerly looking forward to Paul's implementation. (And make that ~5 FITS implementations; I did a primitive one for some test programs we were doing here). There is reason to believe that documentation will improve, and we can augment it where necessary. Numerical utilities can be developed as needed. It doesn't seem that one giant effort is needed there. The plotting situation seems to be somewhat muddled and one of my more serious concerns. I'd rather not spend the effort to develop a package from scratch. There a number of graphics packages out there but each has a shortcoming or two that makes us reluctant to use it. Our need is primarily for simple 2-d plotting tasks (x-y, contour, surface, etc.) but that has enough control to produce publication quality output. While we don't currently support Windows platforms, we don't want to rule that out. Nor can we distribute commercial software. We would prefer it to have an object oriented interface as well. This appears to rule out most of what is out there. PyGIST is not available on Windows, DISTLIN is commercial, GTK appears only to provide plotting primitives and doesn't run on Windows, VTK is volume rendering based and has little for 2-d plotting...about the only thing available now appears to be pgplot.) For the moment, I'm waiting to see what David Ascher produces; it sounds promising. Complicating our decisions is trying to decide how Java and JPython will fit into our future. We prefer a solution that will run in JPython as well (It almost sounds like we constraining ourselves out of a solution!) We are working on adding some conveniences to the interpreter by making our own wrapper. I'm not Guido's new IDLE environment will do it for us. I haven't had a chance to take a look at it yet. If we do go with Python it does mean that there will be some institutional support to help supply what is missing in the above areas. Perry Greenfield Space Telescope Science Institute Science Software Group From tss@galstar.com Thu Nov 26 03:05:17 1998 From: tss@galstar.com (Tae-Sung Shin) Date: Wed, 25 Nov 1998 21:05:17 -0600 Subject: [Matrix-SIG] Re: advocacy Message-ID: <002201be18e9$9b4094a0$86230fd0@tae-suns> I am a newbie (and maniac) for Python and JPython. As a developer for numerical/statistical software, I've been dreamed of a numerical software like (but beyond) Splus and Xlispstat with Python and JPython since I met them a couple of months ago. To make story short, I will gradly participate if there is any established development project for statistical/numerical software with (J)Python. Tae-Sung Shin, Ph.D. Statistician/Developer. >make overall design decisions and sticking with them, etc. Are list >members willing and interested in an organized development effort >aimed at creating a competitor to IDL? > >--jh-- >Joe Harrington >Cornell University Space Sciences From: "Yoon, Hoon (CICG - NY Program Trading)" To: matrix-sig@python.org Subject: Re: [Matrix-SIG] advocacy Date: Wed, 25 Nov 1998 09:37:15 -0500 >> >> I am NOT advocating usage of R at all, but it's nice to see something >> already there. So, we can copy and learn from it. >> The above site contains R. If you know anything about S or Splus, then >> you would find the free package called R very very close. By going this >> way, the creator of R takes advantage of huge docs available in S and >> Splus. >> Frankly, I have my own problem with Splus and will only use R for >> regression analysis only; however, if Python ever achieves the state of R >> -> Splus then I would never bother using anything else. >> It's a good place to find out how statistic code is put together and >> also graphics are put together. R preserved Spluses plot, box, hist, and >> most interestingly identify (check this out). This is also completely >> cross platform on Unix and NT. >> In addition, we can probably pick up things like dataframe and higher >> statistics, which Python currently lacks. R is based on Scheme. I am sure >> if they can do it in Scheme, it would be easy job with Python. >> From pearu@ioc.ee Thu Nov 26 07:47:58 1998 From: pearu@ioc.ee (Pearu Peterson) Date: Thu, 26 Nov 1998 09:47:58 +0200 (EET) Subject: [Matrix-SIG] generate set S^g Message-ID: Hi! For those how are interested in genereting a set S^g find below one of many ways how to do it in python. For example, {1,2}^2 is {{1,1}, {1,2}, {2,1}, {2,2}} ------------> combination.py <------------ #!/usr/bin/env python """ Generate combinations. Pearu Peterson, November 25, 1998 """ def comb(S,g): """ comb(S,g) returns a set S^g as a list of lists. s is list, g is positive integer. """ a=b=t='' p=[] if g<1: return p for j in range(g): a='%s\n%sfor i%d in %s:'%(a,t,j,`S`) b='%s,i%d'%(b,j) t='%s\t'%t exec '%s\n%s p.append([%s])\n'%(a,t,b[1:]) return p if __name__ == "__main__": print comb([1,2,3],2) print comb([1,2],3) --------> end of combination.py <------------ Pearu Peterson , MSc, Researcher Department of Mechanics and Applied Mathematics http://koer.ioc.ee/~pearu/ Institute of Cybernetics at Tallinn Technical University Phone: (+3722) 527907 Akadeemia rd. 21, 12618 Tallinn ESTONIA Fax: (+372) 6397039 *** the nonvalidity of rigorous causality is necessary and not just consistently possible (Heisenberg, 1925) *** From hinsen@cnrs-orleans.fr Thu Nov 26 19:17:49 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 26 Nov 1998 20:17:49 +0100 Subject: [Matrix-SIG] re: advocacy In-Reply-To: <199811252250.RAA15587@eclipse.stsci.edu> (perry@stsci.edu) References: <199811252250.RAA15587@eclipse.stsci.edu> Message-ID: <199811261917.UAA16336@dirac.cnrs-orleans.fr> > 6) Weak interactive environment: It is not always appreciated by > non-scientists that interactive mode is where most time is spent > looking at data. To that end, the interactive mode needs some conveniences The best interactive environment for Python work is Emacs, in my opinion. It does: > - command (and output) logging. Trivial: everything is in a buffer anyway, just save it. > - easy system escapes (e.g., !ls in IRAF, $ls in IDL) Requires just one keypress to get into a shell buffer. > - as trivial as it sounds, an intuitive exit command (like 'exit' or 'quit') Just kill the Python buffer. > - an easy way of re-executing moderately long command sequences or blocks > without using an editor (e.g., you invoke a loop of several statements > repeatedly because you need to tweak some parameters each time). Done with Ctrl-! > - utility functions to start up help, or examine the namespace for > existing data elements (much like IDL does) That one may be missing. There's also ptui and Guido's early release of IDLE, both worth checking out. > 7) Image display convenient for astronomers (either by saoimage or ximtool > lookalikes or connections to either of those so that images could be > displayed on those tools. PIL is good enough for me, but I am not a heavy image user. > Hopefully the FITS issue will be solved soon--I'm eagerly looking forward to > Paul's implementation. (And make that ~5 FITS implementations; I did a Just out of curiosity - could someone give a short explanation for non-astronomers what FITS really is? > from scratch. There a number of graphics packages out there but each > has a shortcoming or two that makes us reluctant to use it. Our need That seems to be the major problem; these things have been discussed on the Plot-SIG for a while. Nothing is really good enough, and worse, nothing available seems to be written well enough to permit easy extension. Moreover, cross-platform portability is a big problem with all plotting libraries, and would be a problem even with starting from scratch. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From hinsen@cnrs-orleans.fr Thu Nov 26 19:20:34 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 26 Nov 1998 20:20:34 +0100 Subject: [Matrix-SIG] Using existing packages as back-ends? In-Reply-To: <199811251618.LAA14276@dip.eecs.umich.edu> (message from Andrew Sterian on Wed, 25 Nov 1998 11:18:05 -0500 (EST)) References: <199811251618.LAA14276@dip.eecs.umich.edu> Message-ID: <199811261920.UAA13466@dirac.cnrs-orleans.fr> > With respect to expanding the functionality of NumPy (modules, plotting, etc.) > it seems to me that it would be easiest to leverage existing free software > that is already mature and has already dealt with many of these issues > in a standalone context. I'm thinking of MATLAB-esque programs like > RLab, SciLab, and Octave. The idea seems obvious, but doing it is another matter. What can be wrapped up is libraries, not complete programs (well, it's not entirely impossible, but never a good solution). And there are already some NumPy wrappers for various mathematical libraries out there. There just aren't any good libraries for many applications. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From perry@stsci.edu Mon Nov 30 22:03:53 1998 From: perry@stsci.edu (Perry Greenfield) Date: Mon, 30 Nov 1998 17:03:53 -0500 Subject: [Matrix-SIG] re: advocacy Message-ID: <199811302203.RAA23085@eclipse.stsci.edu> Konrad Hinsen writes > > The best interactive environment for Python work is Emacs, in my > opinion. It does: [many useful emacs comments deleted] Well, it is true that many of these things are available from emacs, but telling people they have to use emacs to get these features turns a lot of users off. Emacs users (and I am one) around here are a distinct minority; we simply cannot tell people that they should use it if we want them to use our software. > > Hopefully the FITS issue will be solved soon--I'm eagerly looking forward to > > Paul's implementation. (And make that ~5 FITS implementations; I did a > > Just out of curiosity - could someone give a short explanation > for non-astronomers what FITS really is? > FITS (Flexible Image Transport System) was a format originally designed to be 1) a standard interchange format for astronomical data and 2) an archival format as well. It was originally intended for use in writing tapes. The original format was fairly simple; data files consisted of a header section followed by a data block. The header section consisted of some number of 80 character ascii records with most having some sort of = pair. In this way, all the revelvant information for observations was stored in the header along with information about the size, data type and structure of the data itself. Later, the standard was extended to allow multiple header/data sets within one file, and to allow other types of data such as tabular data. Since this format became widely used, many now want programs to operate on disk files of this format directly. Its primary benefit is that it is standardized amongst astronomers. But it lacks the flexibility and generality of more modern data format such as HDF (though that particular format does not appear to serve as a good archival format). This inflexibility (or clumsiness) causes us problems in trying to store more complex data sets using FITS format. It is not clear to me how useful it would be outside of astronomy. There are a number of libraries available that can access such data. This url gives links to detailed specifications for the format: http://fits.gsfc.nasa.gov/fits_intro.html Perry Greenfield From miller5@uiuc.edu Mon Nov 30 22:43:46 1998 From: miller5@uiuc.edu (Mike Miller) Date: 30 Nov 1998 16:43:46 -0600 Subject: [Matrix-SIG] re: advocacy In-Reply-To: Konrad Hinsen's message of "Thu, 26 Nov 1998 20:17:49 +0100" References: <199811252250.RAA15587@eclipse.stsci.edu> <199811261917.UAA16336@dirac.cnrs-orleans.fr> Message-ID: >>>>> "Konrad" == Konrad Hinsen writes: >> - command (and output) logging. > Trivial: everything is in a buffer anyway, just save it. If the python interpreter is compiled with readline, it only needs a few lines added to use the associated history library. Some once pointed out that it only needs a python module around libhistory to be very flexible. Mike -- Michael A. Miller miller5@uiuc.edu Department of Physics, University of Illinois, Urbana-Champaign