From eric at scipy.org Wed Jan 2 18:00:36 2002 From: eric at scipy.org (eric) Date: Wed, 2 Jan 2002 18:00:36 -0500 Subject: [SciPy-dev] pre-release peek of compiler... and I need help with a name Message-ID: <039701c193e1$4a384190$777ba8c0@ericlaptop> Hey folks, Happy new year to all. In celebration, I had planned on releasing a new version of the scipy.compiler module, but as a stand alone package since it is useful outside the scientific community. So, the module was to be released as 'compiler'. Unfortunately, at the 11th hour, I just noticed that name is already taken in 2.2. Doh! So, I need a new name. The module has three main functions (or modules). inline() -- allows you to inline C/C++ code in Python. blitz() -- translates python Numeric expressions into Blitz++ (C++) expressions for fast execution. ext_tools -- Module useful for building Python extension modules. You can look here at the temporary site (until I get the real name...) http://www.scipy.org/site_content/compiler/ Please let me know if you have a suitable name. Travis V. has the front runner: cwrap Putting that along side cow.py is a little much though... so we're still looking. Also, there are still tons of typos, etc. in the docs(Trav suggests running a find/replace on your/you're would help. :| ) Let me know how the tests fair if anyone gets a chance to try it, and we'll hopefully have things shaped up with a new name for release by the end of the week. eric From prabhu at aero.iitm.ernet.in Wed Jan 2 22:11:59 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 3 Jan 2002 08:41:59 +0530 Subject: [SciPy-dev] pre-release peek of compiler... and I need help with a name In-Reply-To: <039701c193e1$4a384190$777ba8c0@ericlaptop> References: <039701c193e1$4a384190$777ba8c0@ericlaptop> Message-ID: <15411.52223.166321.116325@monster.linux.in> >>>>> "eric" == eric writes: eric> Hey folks, Happy new year to all. In celebration, I had eric> planned on releasing a new version of the scipy.compiler eric> module, but as a stand alone package since it is useful eric> outside the scientific community. So, the module was to be eric> released as 'compiler'. Unfortunately, at the 11th hour, I eric> just noticed that name is already taken in 2.2. Doh! eric> Please let me know if you have a suitable name. Travis eric> V. has the front runner: eric> cwrap How about wrapc. cembed. embedc. cipy -- C in Python or C inline Python. cip -- ditto. sup -- speed up python. supuc -- speed up python using c. bicep -- blitz inline c embedded in python. inliner. zipper. blitzer. embedder. Are there plans to support other languages apart from C/C++ ? is it possible to embed others even? prabhu From eric at scipy.org Thu Jan 3 00:23:15 2002 From: eric at scipy.org (eric) Date: Thu, 3 Jan 2002 00:23:15 -0500 Subject: [SciPy-dev] pre-release peek of compiler... and I need help with a name References: <039701c193e1$4a384190$777ba8c0@ericlaptop> <15411.52223.166321.116325@monster.linux.in> Message-ID: <03c001c19416$bf157390$777ba8c0@ericlaptop> Hey Prabhu, > embedc embed was already pretty high on the list. Either of these would work fine. > bicep -- blitz inline c embedded in python. Hey, I kinda like that. Nice little double meaning there. > Are there plans to support other languages apart from C/C++ ? No plans right now. I'd like to make this robust and get a few other features in first. Biggest missing features in my mind right now are (a) support for VTK (b) support for array broadcasting in blitz(), and (c) support for general math functions in blitz(). > is it possible to embed others even? PyInline is another module that offers a similar capability, but comes at the problem from a different angle. I think it aims at being able to support multiple lanaguages, so it is possible. As far as whether compiler would ever be able to do it, I'm not sure. It would probably be pretty easy to embed Fortran right now supporting Numeric arrays, scalars, and strings, but I'm not sure that will excite many people. As far as Perl, Ruby, Tcl and friends, I don't really have a feeling for them. I think they would take quite a bit more work. It'd probably be easy to embed Java in Jython, but I haven't played with it much. For all I know, this may already be possible. thanks again for the names, eric From prabhu at aero.iitm.ernet.in Thu Jan 3 02:26:10 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 3 Jan 2002 12:56:10 +0530 Subject: [SciPy-dev] pre-release peek of compiler... and I need help with a name In-Reply-To: <03c001c19416$bf157390$777ba8c0@ericlaptop> References: <039701c193e1$4a384190$777ba8c0@ericlaptop> <15411.52223.166321.116325@monster.linux.in> <03c001c19416$bf157390$777ba8c0@ericlaptop> Message-ID: <15412.1938.232288.17357@cyberwaveindia.com> hi, >>>>> "ej" == eric writes: >> bicep -- blitz inline c embedded in python. ej> Hey, I kinda like that. Nice little double meaning there. Minor modification: blitzed inlined c embedded in python. My vote is on bicep. :) >> Are there plans to support other languages apart from C/C++ ? ej> No plans right now. I'd like to make this robust and get a ej> few other features in first. Biggest missing features in my ej> mind right now are (a) support for VTK (b) support for array ej> broadcasting in blitz(), and (c) support for general math ej> functions in blitz(). Sounds great. >> is it possible to embed others even? ej> PyInline is another module that offers a similar capability, ej> but comes at the problem from a different angle. I think it ej> aims at being able to support multiple lanaguages, so it is ej> possible. ej> As far as whether compiler would ever be able to do it, I'm ej> not sure. It would probably be pretty easy to embed Fortran ej> right now supporting Numeric arrays, scalars, and strings, but ej> I'm not sure that will excite many people. As far as Perl, ej> Ruby, Tcl and friends, I don't really have a feeling for them. ej> I think they would take quite a bit more work. Forget perl, ruby, tcl, java and friends. At the moment I see Python as the best prototyping language. The biggest problem with it in certain situations is speed. For instance I could rapidly prototype CFD code in Python but wouldn't want to make production runs with it. Dylan is really nice for what it aims to be and do but the open versions aren't mature enough, IMHO. So, I see compiler/psyco filling in this gap. Writing everything in C++ is something I am growing to strongly dislike. Fortran is also a nice possibility for raw speed but I dont know how easy it would be to use/write from Python. As I understand, compiler allows you to speed up parts of code by allowing you to embed c++/c in Python. Therefore, I am not interested in seeing support for any language that does not concern itself with speed. Therefore, other scripting languages need not be supported. Its things like C/C++ and Fortran that need to be supported. Of course, if Python were inherently fast, it would be wonderful but that is asking for way too much. Maybe Python should become somewhat dylanic and allow optional specification of types. It could also allow one to stop dynamic name lookups etc. when unnecessary. A proper python compiler might never be possible but it should be possible to create pure python modules that can be fine tuned to be blazingly fast without the programmer having to switch to another language. Maybe a smaller subset of the Python language can be used to do this. I guess this is what Numeric etc. do but they do it outside the language. It would be nice if one could do everything in Python (or a subset of Python). I dont know if any of this makes sense but I'm just thinking aloud. prabhu From fperez at pizero.colorado.edu Thu Jan 3 04:31:24 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Thu, 3 Jan 2002 02:31:24 -0700 (MST) Subject: [SciPy-dev] pre-release peek of compiler... and I need help with a name In-Reply-To: <03c001c19416$bf157390$777ba8c0@ericlaptop> Message-ID: > > > bicep -- blitz inline c embedded in python. > > Hey, I kinda like that. Nice little double meaning there. > how about biceps -- blitz inline c embedded in python Scripts ^ ^^^^^^^ biceps is one of those words you rarely hear in singuar, and the extra s seems natural enough. cheers, f. From eric at scipy.org Fri Jan 4 16:48:49 2002 From: eric at scipy.org (eric) Date: Fri, 4 Jan 2002 16:48:49 -0500 Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation Message-ID: <062601c19569$984f67d0$777ba8c0@ericlaptop> Hello group, I'm pretty close to releasing weave 0.2, a tool that helps in combining C/C++ with Python code. There are basically three ways to use it. inline() offers inline C/C++ in Python. blitz() converts Python Numeric expression to C++ for fast execution. And, ext_tools offer a couple of classes that build extension classes. If you have a few cycles to spare, I'd appreciate a few eye balls on the documentation page and source. Also, if people could download the zip/exe/tar.gz files and let me know of any failures that would be helpful. The website provides info on how to test (it's very simple). Also, success reports on platforms would be good . W2K, RH 7.1, Debian are about all that has been tested. Here is the link: http://www.scipy.org/site_content/weave thanks, eric From fperez at pizero.colorado.edu Fri Jan 4 18:16:40 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Fri, 4 Jan 2002 16:16:40 -0700 (MST) Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation In-Reply-To: <062601c19569$984f67d0$777ba8c0@ericlaptop> Message-ID: > If you have a few cycles to spare, I'd appreciate a few eye balls on the > documentation page and source. Also, if people could download the > zip/exe/tar.gz > files and let me know of any failures that would be helpful. The website > provides > info on how to test (it's very simple). Also, success reports on platforms > would > be good . W2K, RH 7.1, Debian are about all that has been tested. Quick comment on the documentation page http://www.scipy.org/site_content/weave/weave/tutorial.html. It seems to have a hard-coded page width, and lines don't wrap properly (tested with Netscape 4.x and Opera 6.0 under Linux, maybe ok under windows). So it's a pain in the ass to read b/c you have to constantly scroll horizontally or open a hughe window so it all fits (and reading overly long lines is tiring). You may want to fix this. I'll report on running the tests later today. thanks! f. From fperez at pizero.colorado.edu Fri Jan 4 18:43:13 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Fri, 4 Jan 2002 16:43:13 -0700 (MST) Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation In-Reply-To: <062601c19569$984f67d0$777ba8c0@ericlaptop> Message-ID: Hi Eric, Failure report with weave. Platform: [~]> python Python 2.1.1 (#1, Aug 30 2001, 17:36:05) [GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.61mdk)] on linux-i386 I installed it with the standard routine (setup.py install) with apparently no problems. When running weave.test(), I get: [snipped early part which seems ok] No test suite found for weave.sequence_spec .. Expression: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1]+ b[1:-1,2:] + b[1:-1,:-2]) / 5. Run: (10, 10) f EEE....EEEE....EEE....EE....EEEE....EEE......EEEEEE..........................EEEwarning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations ...warning: specified build_dir '..' does not exist or is or is not writable. Trying default locations .warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations ...warning: specified build_dir '..' does not exist or is or is not writable. Trying default locations ............................................................................................... ====================================================================== ERROR: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.1/site-packages/weave/tests/test_blitz_tools.py", line 148, in check_5point_avg_2d self.generic_2d(expr) File "/usr/lib/python2.1/site-packages/weave/tests/test_blitz_tools.py", line 124, in generic_2d standard,compiled = self.generic_test(expr,arg_dict,type,size) File "/usr/lib/python2.1/site-packages/weave/tests/test_blitz_tools.py", line 81, in generic_test blitz_tools.blitz(expr,arg_dict,{},verbose=0) File "/usr/lib/python2.1/site-packages/weave/blitz_tools.py", line 93, in blitz type_factories = blitz_type_factories) File "/usr/lib/python2.1/site-packages/weave/inline_tools.py", line 407, in compile_function exec 'import ' + module_name File "", line 1, in ? ImportError: /usr/local/home/fperez/tmp/test_files/sc_5599df30197fe981824ad8ec934a784e3.so: undefined symbol: cerr ====================================================================== ERROR: check_int_return (test_scalar_spec.test_unix_int_specification) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.1/site-packages/weave/tests/test_scalar_spec.py", line 110, in check_int_return exec 'from ' + mod_name + ' import test' File "", line 1, in ? ImportError: /usr/local/home/fperez/tmp/test_files/sc_504aaf25fe1bf17daaf243c7c8f713d93.so: undefined symbol: cerr [snipped lots more of similar error messages, all failing because cerr isn't available]. I tested my c++ libraries and cerr is perfectly accessible for me with g++ with a simple #include , so I don't think my gcc/g++ setup is the problem. At the end after all the snipped stuff I get: ====================================================================== ERROR: check_call_function (test_common_spec.test_callable_specification) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.1/site-packages/weave/tests/test_common_spec.py", line 56, in check_call_function actual = inline_tools.inline(code,['func','search_str','sub_str']) File "/usr/lib/python2.1/site-packages/weave/inline_tools.py", line 316, in inline auto_downcast = auto_downcast, File "/usr/lib/python2.1/site-packages/weave/inline_tools.py", line 407, in compile_function exec 'import ' + module_name File "", line 1, in ? ImportError: /usr/local/home/fperez/tmp/test_files/sc_263056b76227e75db905e6c70b5397fd1.so: undefined symbol: cerr ---------------------------------------------------------------------- Ran 184 tests in 260.053s FAILED (errors=28) I hope this helps you, ask me if you want more details. Very much looking forward to weave being fully functional. Cheers, f. From eric at scipy.org Fri Jan 4 18:10:01 2002 From: eric at scipy.org (eric) Date: Fri, 4 Jan 2002 18:10:01 -0500 Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation References: Message-ID: <066501c19574$f00970a0$777ba8c0@ericlaptop> Hey Fernando, The fixed width problem occurs for me too. I haven't found the cause yet. I'll keep looking later tonight. The problem with cerr was supposed to go away when I hard coded the inclusion of the stdc++ library in build_tools.py. Hmmm. Looks like I did this in a pretty hap-hazard way: if sys.platform[:-1] == 'linux': libraries = kw.get('libraries',[]) kw['libraries'] = ['stdc++'] + libraries Looks like this is guaranteed to fail for most cases. Does mandrake define sys.platform differently than RH? Anyway, I've changed this to: if compiler_name != 'msvc': libraries = kw.get('libraries',[]) kw['libraries'] = ['stdc++'] + libraries so it should add the library for everything except microsoft vc++. Until I figure out how to do this right (like force g++), hopefully this will work. I've uploaded new source files. Please see if this fixes the problem. thanks, eric ----- Original Message ----- From: "Fernando P?rez" To: Sent: Friday, January 04, 2002 6:43 PM Subject: Re: [SciPy-dev] weave -- inline C/C++ in Python, an implementation > Hi Eric, > > Failure report with weave. Platform: > > [~]> python > Python 2.1.1 (#1, Aug 30 2001, 17:36:05) > [GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.61mdk)] on linux-i386 > > I installed it with the standard routine (setup.py install) with apparently no > problems. When running weave.test(), I get: > > [snipped early part which seems ok] > > No test suite found for weave.sequence_spec > .. > Expression: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1]+ > b[1:-1,2:] + b[1:-1,:-2]) / 5. > Run: (10, 10) f > EEE....EEEE....EEE....EE....EEEE....EEE......EEEEEE......................... .EEEwarning: > specified build_dir '_bad_path_' does not exist or is or is not writable. > Trying default locations > ...warning: specified build_dir '..' does not exist or is or is not writable. > Trying default locations > .warning: specified build_dir '_bad_path_' does not exist or is or is not > writable. Trying default locations > ...warning: specified build_dir '..' does not exist or is or is not writable. > Trying default locations > ............................................................................ ................... > ====================================================================== > ERROR: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib/python2.1/site-packages/weave/tests/test_blitz_tools.py", > line 148, in check_5point_avg_2d > self.generic_2d(expr) > File "/usr/lib/python2.1/site-packages/weave/tests/test_blitz_tools.py", > line 124, in generic_2d > standard,compiled = self.generic_test(expr,arg_dict,type,size) > File "/usr/lib/python2.1/site-packages/weave/tests/test_blitz_tools.py", > line 81, in generic_test blitz_tools.blitz(expr,arg_dict,{},verbose=0) > File "/usr/lib/python2.1/site-packages/weave/blitz_tools.py", line 93, in > blitz > type_factories = blitz_type_factories) > File "/usr/lib/python2.1/site-packages/weave/inline_tools.py", line 407, in > compile_function > exec 'import ' + module_name > File "", line 1, in ? > ImportError: > /usr/local/home/fperez/tmp/test_files/sc_5599df30197fe981824ad8ec934a784e3.s o: > undefined symbol: cerr > ====================================================================== > ERROR: check_int_return (test_scalar_spec.test_unix_int_specification) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib/python2.1/site-packages/weave/tests/test_scalar_spec.py", > line 110, in check_int_return > exec 'from ' + mod_name + ' import test' > File "", line 1, in ? > ImportError: > /usr/local/home/fperez/tmp/test_files/sc_504aaf25fe1bf17daaf243c7c8f713d93.s o: > undefined symbol: cerr > > > [snipped lots more of similar error messages, all failing because cerr isn't > available]. I tested my c++ libraries and cerr is perfectly accessible for me > with g++ with a simple #include , so I don't think my gcc/g++ > setup is the problem. > > At the end after all the snipped stuff I get: > > ====================================================================== > ERROR: check_call_function (test_common_spec.test_callable_specification) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib/python2.1/site-packages/weave/tests/test_common_spec.py", > line 56, in check_call_function > actual = inline_tools.inline(code,['func','search_str','sub_str']) > File "/usr/lib/python2.1/site-packages/weave/inline_tools.py", line 316, in > inline > auto_downcast = auto_downcast, > File "/usr/lib/python2.1/site-packages/weave/inline_tools.py", line 407, in > compile_function > exec 'import ' + module_name > File "", line 1, in ? > ImportError: > /usr/local/home/fperez/tmp/test_files/sc_263056b76227e75db905e6c70b5397fd1.s o: > undefined symbol: cerr > ---------------------------------------------------------------------- > Ran 184 tests in 260.053s > > FAILED (errors=28) > > > > I hope this helps you, ask me if you want more details. > > Very much looking forward to weave being fully functional. > > Cheers, > > f. > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From fperez at pizero.colorado.edu Fri Jan 4 19:30:21 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Fri, 4 Jan 2002 17:30:21 -0700 (MST) Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation In-Reply-To: <066501c19574$f00970a0$777ba8c0@ericlaptop> Message-ID: On Fri, 4 Jan 2002, eric wrote: > Hey Fernando, > > The fixed width problem occurs for me too. I haven't found the cause yet. > I'll keep looking later tonight. good > > The problem with cerr was supposed to go away when I hard coded the > inclusion of the stdc++ library > in build_tools.py. Hmmm. Looks like I did this in a pretty hap-hazard way: > > if sys.platform[:-1] == 'linux': > libraries = kw.get('libraries',[]) > kw['libraries'] = ['stdc++'] + libraries > > Looks like this is guaranteed to fail for most cases. Does mandrake define > sys.platform differently than > RH? Anyway, I've changed this to: yes, this is Mandrake: >>> sys.platform 'linux-i386' Something like: if sys.platform.lower().find('linux') > -1: ... might be much more robust. Or just force compilation with g++ (why don't you?) > > I've uploaded new source files. Please see if this fixes the problem. > I'll let you know. cheers, f From fperez at pizero.colorado.edu Fri Jan 4 19:39:44 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Fri, 4 Jan 2002 17:39:44 -0700 (MST) Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation Message-ID: Still failing on a Mandrake box: [snip early part] No test suite found for weave.sequence_spec .. Expression: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1]+ b[1:-1,2:] + b[1:-1,:-2]) / 5. Run: (10, 10) f 1st run(Numeric,compiled,speed up): 0.0026, 30.1661, 0.0001 2nd run(Numeric,compiled,speed up): 0.0011, 0.0047, 0.2285 Run: (50, 50) f 1st run(Numeric,compiled,speed up): 0.0014, 0.0048, 0.2860 2nd run(Numeric,compiled,speed up): 0.0017, 0.0048, 0.3601 Run: (100, 100) f 1st run(Numeric,compiled,speed up): 0.0030, 0.0054, 0.5426 2nd run(Numeric,compiled,speed up): 0.0027, 0.0058, 0.4589 Run: (500, 500) f 1st run(Numeric,compiled,speed up): 0.0931, 0.0260, 3.5852 2nd run(Numeric,compiled,speed up): 0.0955, 0.0266, 3.5889 Run: (1000, 1000) f 1st run(Numeric,compiled,speed up): 0.3852, 0.0866, 4.4467 2nd run(Numeric,compiled,speed up): 0.8154, 0.1647, 4.9497 Run: (10, 10) d Abort And I'm dumped out at the system prompt. ??? (this is with the updated code which fixes the cerr prob.) cheers, f. From travis at scipy.org Fri Jan 4 22:58:46 2002 From: travis at scipy.org (Travis N. Vaught) Date: Fri, 4 Jan 2002 21:58:46 -0600 Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation In-Reply-To: Message-ID: > -----Original Message----- > From: scipy-dev-admin at scipy.org [mailto:scipy-dev-admin at scipy.org]On > Behalf Of Fernando Perez > Sent: Friday, January 04, 2002 5:17 PM > To: scipy-dev at scipy.org > Subject: Re: [SciPy-dev] weave -- inline C/C++ in Python, an > implementation > > ....... ....... > Quick comment on the documentation page > http://www.scipy.org/site_content/weave/weave/tutorial.html. > > It seems to have a hard-coded page width, and lines don't wrap properly > (tested with Netscape 4.x and Opera 6.0 under Linux, maybe ok > under windows). > So it's a pain in the ass to read b/c you have to constantly scroll > horizontally or open a hughe window so it all fits (and reading > overly long > lines is tiring). You may want to fix this. > This is caused by the tag. It never wraps (at least with all browsers I've used). Therefore, your page width is minimally the longest line of "". I'll try to break them up manually to a reasonable minimum width. Thanks for the feedback. Travis V. From eric at scipy.org Fri Jan 4 23:46:11 2002 From: eric at scipy.org (eric) Date: Fri, 4 Jan 2002 23:46:11 -0500 Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation References: Message-ID: <069d01c195a3$e63d7600$777ba8c0@ericlaptop> Hey Fernando, > Still failing on a Mandrake box: > > Abort > > And I'm dumped out at the system prompt. ??? Oooo. Thats no good. This looks like a problem in the blitz() stuff. I just tested on another Linux system, and everything worked there (though it took 15 minutes...).Here's all the pertinent information for that machine: [ej at beowulf1 ~]$ gcc -v Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-85) [ej at beowulf1 ~]$ python Python 2.1.1 (#2, Sep 6 2001, 03:17:03) [GCC 2.96 20000731 (Red Hat Linux 7.1 2.96-85)] on linux2 Type "copyright", "credits" or "license" for more information. >>> Numeric.__version__ '20.1.1' How does this compare to your setup? Is there anything else we should compare? Unfortunately I don't have access to a Mandrake box to check things -- but I thought they were very similar to RH in their setup. You might get a little more of a clue what is breaking by changing line 81 of weave/tests/test_blitz_tools.py from: blitz_tools.blitz(expr,arg_dict,{},verbose=0) to blitz_tools.blitz(expr,arg_dict,{},verbose=2) This will print out all the compilation steps. However, I have a feeling they are working, and the abort occurs when trying to import the compiled library. Still, maybe we can glean something. Also, to test everything except for blitz, you can comment out the test() and test_suite() functions at the bottom of weave.blitz_tools.py. This will allow you to test whether non-blitz/Numeric stuff is working. thanks, eric ----- Original Message ----- From: "Fernando P?rez" To: Sent: Friday, January 04, 2002 7:39 PM Subject: Re: [SciPy-dev] weave -- inline C/C++ in Python, an implementation > Still failing on a Mandrake box: > > [snip early part] > > No test suite found for weave.sequence_spec > .. > Expression: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1]+ > b[1:-1,2:] + b[1:-1,:-2]) > / 5. > Run: (10, 10) f > 1st run(Numeric,compiled,speed up): 0.0026, 30.1661, 0.0001 > 2nd run(Numeric,compiled,speed up): 0.0011, 0.0047, 0.2285 > Run: (50, 50) f > 1st run(Numeric,compiled,speed up): 0.0014, 0.0048, 0.2860 > 2nd run(Numeric,compiled,speed up): 0.0017, 0.0048, 0.3601 > Run: (100, 100) f > 1st run(Numeric,compiled,speed up): 0.0030, 0.0054, 0.5426 > 2nd run(Numeric,compiled,speed up): 0.0027, 0.0058, 0.4589 > Run: (500, 500) f > 1st run(Numeric,compiled,speed up): 0.0931, 0.0260, 3.5852 > 2nd run(Numeric,compiled,speed up): 0.0955, 0.0266, 3.5889 > Run: (1000, 1000) f > 1st run(Numeric,compiled,speed up): 0.3852, 0.0866, 4.4467 > 2nd run(Numeric,compiled,speed up): 0.8154, 0.1647, 4.9497 > Run: (10, 10) d > Abort > > > And I'm dumped out at the system prompt. ??? > > (this is with the updated code which fixes the cerr prob.) > > cheers, > > f. > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From fperez at pizero.colorado.edu Sat Jan 5 03:31:28 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Sat, 5 Jan 2002 01:31:28 -0700 (MST) Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation In-Reply-To: <069d01c195a3$e63d7600$777ba8c0@ericlaptop> Message-ID: > > How does this compare to your setup? Is there anything else we should > compare? Unfortunately I don't have access to a Mandrake box to check > things -- but I thought they were very similar to RH in their setup. let's see. First, some info on my setup: root[site-packages]# gcc -v Reading specs from /usr/lib/gcc-lib/i586-mandrake-linux/2.96/specs gcc version 2.96 20000731 (Linux-Mandrake 8.0 2.96-0.48mdk) In [4]: Numeric.__version__ Out[4]= '20.2.1' In [5]: print sys.version 2.1.1 (#1, Aug 30 2001, 17:36:05) [GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.61mdk)] > > You might get a little more of a clue what is breaking by changing line 81 > of weave/tests/test_blitz_tools.py from: > blitz_tools.blitz(expr,arg_dict,{},verbose=0) > to > blitz_tools.blitz(expr,arg_dict,{},verbose=2) > > This will print out all the compilation steps. However, I have a feeling > they are working, and the abort occurs when trying to import the compiled > library. Still, maybe we can glean something. > > Also, to test everything except for blitz, you can comment out the test() > and test_suite() functions at the bottom of weave.blitz_tools.py. This will > allow you to test whether non-blitz/Numeric stuff is working. Well, did both of the above and the result is even less informative: [snipped early part] No test suite found for weave.sequence_spec .......Abort [~/test]> I don't get *any* compilation warnings whatsoever, and the root of the problem doesn't seem to be in the blitz stuff since it's commented out. Any ideas? One thing I noticed, and I don't know if it may cause problems: your files have DOS line endings (\r\n). In emacs these show up as ugly ^M characters at the end of every line, but I don't know if they can confuse python. If nothing else, do you know if there's a way to have distutils clean up the line endings for each platform? I'll try to put some error messages somewhere, but since I don't know the code at all I can't make any promises. My setup is fairly normal and I haven't seen any problems with any other python or numeric packages, so I'm kind of lost here. I'll let you know. cheers, f From fperez at pizero.colorado.edu Sat Jan 5 03:52:05 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Sat, 5 Jan 2002 01:52:05 -0700 (MST) Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation In-Reply-To: <069d01c195a3$e63d7600$777ba8c0@ericlaptop> Message-ID: Hi Eric, I'm pretty lost. Here's a summary: - with the blitz stuff on, and the verbosity at 2, here's what I get: No test suite found for weave.sequence_spec .. Expression: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1]+ b[1:-1,2:] + b[1:-1,:-2]) / 5. Run: (10, 10) f running build_ext building 'sc_5599df30197fe981824ad8ec934a784e0' extension gcc -O3 -fomit-frame-pointer -pipe -mcpu=pentiumpro -march=i586 -ffast-math -fno-strength-reduce -fPIC -I/usr/lib/python2.1/site-packages/weave -I/usr/lib/python2.1/site-packages/weave/blitz-20001213 -I/usr/include/python2.1 -c /usr/local/home/fperez/tmp/test_files/sc_5599df30197fe981824ad8ec934a784e0.cpp -o /usr/local/home/fperez/.python21_compiled/temp/sc_5599df30197fe981824ad8ec934a784e0.o gcc -O3 -fomit-frame-pointer -pipe -mcpu=pentiumpro -march=i586 -ffast-math -fno-strength-reduce -fPIC -I/usr/lib/python2.1/site-packages/weave -I/usr/lib/python2.1/site-packages/weave/blitz-20001213 -I/usr/include/python2.1 -c /usr/lib/python2.1/site-packages/weave/CXX/IndirectPythonInterface.cxx -o /usr/local/home/fperez/.python21_compiled/temp/IndirectPythonInterface.o gcc -O3 -fomit-frame-pointer -pipe -mcpu=pentiumpro -march=i586 -ffast-math -fno-strength-reduce -fPIC -I/usr/lib/python2.1/site-packages/weave -I/usr/lib/python2.1/site-packages/weave/blitz-20001213 -I/usr/include/python2.1 -c /usr/lib/python2.1/site-packages/weave/CXX/cxxextensions.c -o /usr/local/home/fperez/.python21_compiled/temp/cxxextensions.o gcc -O3 -fomit-frame-pointer -pipe -mcpu=pentiumpro -march=i586 -ffast-math -fno-strength-reduce -fPIC -I/usr/lib/python2.1/site-packages/weave -I/usr/lib/python2.1/site-packages/weave/blitz-20001213 -I/usr/include/python2.1 -c /usr/lib/python2.1/site-packages/weave/CXX/cxxsupport.cxx -o /usr/local/home/fperez/.python21_compiled/temp/cxxsupport.o gcc -O3 -fomit-frame-pointer -pipe -mcpu=pentiumpro -march=i586 -ffast-math -fno-strength-reduce -fPIC -I/usr/lib/python2.1/site-packages/weave -I/usr/lib/python2.1/site-packages/weave/blitz-20001213 -I/usr/include/python2.1 -c /usr/lib/python2.1/site-packages/weave/CXX/cxx_extensions.cxx -o /usr/local/home/fperez/.python21_compiled/temp/cxx_extensions.o gcc -shared /usr/local/home/fperez/.python21_compiled/temp/sc_5599df30197fe981824ad8ec934a784e0.o /usr/local/home/fperez/.python21_compiled/temp/IndirectPythonInterface.o /usr/local/home/fperez/.python21_compiled/temp/cxxextensions.o /usr/local/home/fperez/.python21_compiled/temp/cxxsupport.o /usr/local/home/fperez/.python21_compiled/temp/cxx_extensions.o -lstdc++ -o /usr/local/home/fperez/tmp/test_files/sc_5599df30197fe981824ad8ec934a784e0.so 1st run(Numeric,compiled,speed up): 0.0011, 57.3475, 0.0000 2nd run(Numeric,compiled,speed up): 0.0011, 0.0048, 0.2251 Run: (50, 50) f 1st run(Numeric,compiled,speed up): 0.0014, 0.0050, 0.2753 2nd run(Numeric,compiled,speed up): 0.0014, 0.0050, 0.2870 Run: (100, 100) f 1st run(Numeric,compiled,speed up): 0.0026, 0.0058, 0.4457 2nd run(Numeric,compiled,speed up): 0.0026, 0.0059, 0.4471 Run: (500, 500) f 1st run(Numeric,compiled,speed up): 0.1011, 0.0268, 3.7749 2nd run(Numeric,compiled,speed up): 0.1007, 0.0265, 3.7987 Run: (1000, 1000) f 1st run(Numeric,compiled,speed up): 0.4165, 0.0969, 4.2990 2nd run(Numeric,compiled,speed up): 0.3538, 0.0899, 3.9337 Run: (10, 10) d Abort So the run craps out when first trying a double run (I think). - Now, if I turn the blitz stuff off, things die almost at the same place: No test suite found for weave.sequence_spec .......Abort If you tell me where in the code this is happening, I can try to put some informative statements to see what's going on. But I don't know the code and could spend a week blind hunting there. Let me know and I'll try to help. Cheers, f. From pearu at cens.ioc.ee Sat Jan 5 03:53:43 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sat, 5 Jan 2002 10:53:43 +0200 (EET) Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation In-Reply-To: Message-ID: On Sat, 5 Jan 2002, Fernando P?rez wrote: > Any ideas? One thing I noticed, and I don't know if it may cause problems: > your files have DOS line endings (\r\n). In emacs these show up as ugly ^M > characters at the end of every line, but I don't know if they can confuse > python. If nothing else, do you know if there's a way to have distutils clean > up the line endings for each platform? I had similar problem with DOS line endings, in particular with executing setup.py script from a command line: ./setup.py .... that failed with strange messages (not reletated to python) until I changed the line endings with the `flip -u *` command. However, in my debian box I don't experience the problems that you do, so the DOS line endings should not be the problem for linux python. Pearu From pearu at cens.ioc.ee Sat Jan 5 07:30:32 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sat, 5 Jan 2002 14:30:32 +0200 (EET) Subject: [SciPy-dev] Automated get_version Message-ID: Hi! I have commited a new function get_version to scipy_distutils that provides automatic versioning of packages developed within CVS repositories. Here is the signature of get_version: def get_version(release_level='alpha', path='.', major=None): """ Return version string calculated from CVS tree or found in /__version__.py. Automatically update /__version__.py if the version is changed. An attempt is made to guarantee that version is increasing in time. This function always succeeds. None is returned if no version information is available. Version string is in the form ..-- and its items have the following meanings: serial - shows cumulative changes in all files in the CVS repository micro - a number that is equivalent to the number of files minor - indicates the changes in micro value (files are added or removed) release_level - is alpha, beta, canditate, or final major - indicates changes in release_level. """ Here is an example of the process of automatic versioning that is executed inside scipy_distutils directory: >>> from misc_util import get_version >>> get_version ('alpha') No module named __version__ updating version: None -> 0.1.19-alpha-45 '0.1.19-alpha-45' >>> get_version ('alpha') '0.1.19-alpha-45' >>> get_version ('beta') updating version: 0.1.19-alpha-45 -> 0.1.19-beta-45 '0.1.19-beta-45' >>> get_version ('final') updating version: 0.1.19-beta-45 -> 0.1.19-final-45 '0.1.19-final-45' >>> get_version ('final') '0.1.19-final-45' >>> get_version ('alpha') updating version: 0.1.19-final-45 -> 1.1.19-alpha-45 '1.1.19-alpha-45' >>> get_version ('alpha') '1.1.19-alpha-45' >>> The function get_version should be used only in setup.py files. See scipy_distutils/setup.py for example. Comments are most welcome. In particular, opinions about the major=None argument are appreciated. It is there for the case if packagers wish to have a manual control over the major number (then changes in release_level are just ignored), but is this actually desired? Should I remove major=None argument or leave it there? Regards, Pearu From rob at pythonemproject.com Sat Jan 5 11:08:20 2002 From: rob at pythonemproject.com (Rob) Date: Sat, 05 Jan 2002 08:08:20 -0800 Subject: [SciPy-dev] tried new weave compiler, no-go Message-ID: <3C3724F3.27F0C8EF@pythonemproject.com> Hi All, Weave built and installed OK on FreeBSD, but when I try to use it Python complains that weave has no blitz() member. e.g. import weave foo="a*b" weave.blitz(foo) Comes back with the error. Rob. -- The Numeric Python EM Project www.pythonemproject.com From travis at scipy.org Sat Jan 5 12:08:02 2002 From: travis at scipy.org (Travis N. Vaught) Date: Sat, 5 Jan 2002 11:08:02 -0600 Subject: [SciPy-dev] tried new weave compiler, no-go In-Reply-To: <3C3724F3.27F0C8EF@pythonemproject.com> Message-ID: Does this fail as well: >>> from Numeric import * >>> import weave >>> a = ones((5,5),Float32) >>> b = ones((5,5),Float32) >>> weave.blitz("a = a + b") # I get the following for 'a'... >>> a array([[ 2., 2., 2., 2., 2.], [ 2., 2., 2., 2., 2.], [ 2., 2., 2., 2., 2.], [ 2., 2., 2., 2., 2.], [ 2., 2., 2., 2., 2.]],'f') > -----Original Message----- > From: scipy-dev-admin at scipy.org [mailto:scipy-dev-admin at scipy.org]On > Behalf Of Rob > Sent: Saturday, January 05, 2002 10:08 AM > To: scipy-dev at scipy.org > Subject: [SciPy-dev] tried new weave compiler, no-go > > > Hi All, > > Weave built and installed OK on FreeBSD, but when I try to use it Python > complains that weave has no blitz() member. > > e.g. > import weave > foo="a*b" > weave.blitz(foo) > > Comes back with the error. > > Rob. > -- > The Numeric Python EM Project > > www.pythonemproject.com > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev From eric at scipy.org Sat Jan 5 11:23:49 2002 From: eric at scipy.org (eric) Date: Sat, 5 Jan 2002 11:23:49 -0500 Subject: [SciPy-dev] tried new weave compiler, no-go References: <3C3724F3.27F0C8EF@pythonemproject.com> Message-ID: <070501c19605$5b603880$777ba8c0@ericlaptop> Hey Rob, In order to allow people with Numeric installed on there machine to use weave, I put the following code in weave/__init__.py: try: from blitz_tools import blitz except ImportError: pass # Numeric wasn't available It looks like blitz_tools is failing to import for some reason. Poking around, I noticed that all of weave's modules that depend on Numeric also import fastumath so that they are compatible with SciPy. I've changed every import of fastumath to be: # The following try/except so that non-SciPy users can still use blitz try: from fastumath import * except: pass # fastumath not available This should allow non-SciPy users to use weave as well. By the way, did you have SciPy installed when you tried out weave? The changes are in CVS, and I'll upload the changed tar-balls, etc within the next 10 minutes. Please grab them and try again. Another thing you could try just to find out where the error is occuring is to get rid of the try/except around blitz_tools in the __init__.py file. This should through an error telling us what went wrong. eric ----- Original Message ----- From: "Rob" To: Sent: Saturday, January 05, 2002 11:08 AM Subject: [SciPy-dev] tried new weave compiler, no-go > Hi All, > > Weave built and installed OK on FreeBSD, but when I try to use it Python > complains that weave has no blitz() member. > > e.g. > import weave > foo="a*b" > weave.blitz(foo) > > Comes back with the error. > > Rob. > -- > The Numeric Python EM Project > > www.pythonemproject.com > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From prabhu at aero.iitm.ernet.in Sat Jan 5 12:45:20 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sat, 5 Jan 2002 23:15:20 +0530 Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation In-Reply-To: <062601c19569$984f67d0$777ba8c0@ericlaptop> References: <062601c19569$984f67d0$777ba8c0@ericlaptop> Message-ID: <15415.15280.164483.718499@monster.linux.in> >>>>> "eric" == eric writes: eric> balls on the documentation page and source. Also, if people eric> could download the zip/exe/tar.gz files and let me know of eric> any failures that would be helpful. The website provides Here are my test results. I'm running this on a woody debian machine. Python 2.1.1 (built from source), $ gcc -v Reading specs from /usr/lib/gcc-lib/i386-linux/2.95.4/specs gcc version 2.95.4 20011006 (Debian prerelease) scipy CVS tree (absolutely *the* latest) an older version of the scipy CVS tree is also installed (i.e. pre-weave). I tested like so: $ cd cvs/scipy/ $ python >>> import weave >>> weave.test() It works for a while and then gives up with this error. [snip] repairing catalog by removing key Erepairing catalog by removing key F.........F.......warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations ....warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations .................................................................................................. ====================================================================== ERROR: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_blitz_tools.py", line 148, in check_5point_avg_2d self.generic_2d(expr) File "weave/tests/test_blitz_tools.py", line 124, in generic_2d standard,compiled = self.generic_test(expr,arg_dict,type,size) File "weave/tests/test_blitz_tools.py", line 81, in generic_test blitz_tools.blitz(expr,arg_dict,{},verbose=0) File "weave/blitz_tools.py", line 100, in blitz function_catalog.add_function(expr,func,module_dir) File "weave/catalog.py", line 525, in add_function File "/usr/local/lib/python2.1/dumbdbm.py", line 129, in __delitem__ del self._index[key] KeyError: __path__ff ====================================================================== FAIL: Test persisting a function in the default catalog ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_catalog.py", line 280, in check_add_function_persistent1 assert(i in pfuncs) AssertionError ====================================================================== FAIL: Shouldn't get a single file from the temp dir. ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_catalog.py", line 207, in check_get_existing_files2 assert(len(files) == 1) AssertionError ---------------------------------------------------------------------- Ran 184 tests in 423.602s FAILED (failures=2, errors=2) >>> I dont have the time to track it down any more. I just hope this information is helpful. Travis and Robs test cases seem to work perfectly well for me. >>> from Numeric import * >>> import weave >>> a = ones((5,5),Float32) >>> b = ones((5,5),Float32) >>> weave.blitz("a = a + b") >>> a array([[ 2., 2., 2., 2., 2.], [ 2., 2., 2., 2., 2.], [ 2., 2., 2., 2., 2.], [ 2., 2., 2., 2., 2.], [ 2., 2., 2., 2., 2.]],'f') >>> a=1 >>> b = 100.0 >>> weave.blitz(foo) Both work fine. prabhu From prabhu at aero.iitm.ernet.in Sat Jan 5 12:50:10 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sat, 5 Jan 2002 23:20:10 +0530 Subject: [SciPy-dev] Automated get_version In-Reply-To: References: Message-ID: <15415.15570.95637.888919@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> I have commited a new function get_version to scipy_distutils PP> that provides automatic versioning of packages developed PP> within CVS repositories. Here is the signature of get_version: PP> def get_version(release_level='alpha', path='.', major=None): PP> """ Return version string calculated from CVS tree or found in PP> /__version__.py. Automatically update PP> /__version__.py if the version is changed. An attempt PP> is made to guarantee that version is increasing in time. This PP> function always succeeds. None is returned if no version PP> information is available. Sorry for the dumb question but I'm afraid I really dont understand this too well. Does get_version automatically change the version? So if I were to rebuild the sources the version would increase? If get_version changes the version maybe the name should be set_version and not get_version. Or maybe you should have two separate functions for each. prabhu From eric at scipy.org Sat Jan 5 12:15:03 2002 From: eric at scipy.org (eric) Date: Sat, 5 Jan 2002 12:15:03 -0500 Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation References: <062601c19569$984f67d0$777ba8c0@ericlaptop> <15415.15280.164483.718499@monster.linux.in> Message-ID: <071f01c1960c$83a11330$777ba8c0@ericlaptop> Thanks Prabhu, This looks like a problem in how functions are cataloged. I look into it. eric ----- Original Message ----- From: "Prabhu Ramachandran" To: Sent: Saturday, January 05, 2002 12:45 PM Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation > >>>>> "eric" == eric writes: > > eric> balls on the documentation page and source. Also, if people > eric> could download the zip/exe/tar.gz files and let me know of > eric> any failures that would be helpful. The website provides > > Here are my test results. I'm running this on a woody debian machine. > > Python 2.1.1 (built from source), > > $ gcc -v > Reading specs from /usr/lib/gcc-lib/i386-linux/2.95.4/specs > gcc version 2.95.4 20011006 (Debian prerelease) > > scipy CVS tree (absolutely *the* latest) > > an older version of the scipy CVS tree is also installed > (i.e. pre-weave). > > I tested like so: > > $ cd cvs/scipy/ > $ python > >>> import weave > >>> weave.test() > > It works for a while and then gives up with this error. > > [snip] > repairing catalog by removing key > Erepairing catalog by removing key > F.........F.......warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations > ....warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations > ............................................................................ ...................... > ====================================================================== > ERROR: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "weave/tests/test_blitz_tools.py", line 148, in check_5point_avg_2d > self.generic_2d(expr) > File "weave/tests/test_blitz_tools.py", line 124, in generic_2d > standard,compiled = self.generic_test(expr,arg_dict,type,size) > File "weave/tests/test_blitz_tools.py", line 81, in generic_test > blitz_tools.blitz(expr,arg_dict,{},verbose=0) > File "weave/blitz_tools.py", line 100, in blitz > function_catalog.add_function(expr,func,module_dir) > File "weave/catalog.py", line 525, in add_function > File "/usr/local/lib/python2.1/dumbdbm.py", line 129, in __delitem__ > del self._index[key] > KeyError: __path__ff > ====================================================================== > FAIL: Test persisting a function in the default catalog > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "weave/tests/test_catalog.py", line 280, in check_add_function_persistent1 > assert(i in pfuncs) > AssertionError > ====================================================================== > FAIL: Shouldn't get a single file from the temp dir. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "weave/tests/test_catalog.py", line 207, in check_get_existing_files2 > assert(len(files) == 1) > AssertionError > ---------------------------------------------------------------------- > Ran 184 tests in 423.602s > > FAILED (failures=2, errors=2) > > >>> > > I dont have the time to track it down any more. I just hope this > information is helpful. > > Travis and Robs test cases seem to work perfectly well for me. > > >>> from Numeric import * > >>> import weave > >>> a = ones((5,5),Float32) > >>> b = ones((5,5),Float32) > >>> weave.blitz("a = a + b") > >>> a > array([[ 2., 2., 2., 2., 2.], > [ 2., 2., 2., 2., 2.], > [ 2., 2., 2., 2., 2.], > [ 2., 2., 2., 2., 2.], > [ 2., 2., 2., 2., 2.]],'f') > > >>> a=1 > >>> b = 100.0 > >>> weave.blitz(foo) > > Both work fine. > > prabhu > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From pearu at cens.ioc.ee Sat Jan 5 13:22:51 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sat, 5 Jan 2002 20:22:51 +0200 (EET) Subject: [SciPy-dev] Automated get_version In-Reply-To: <15415.15570.95637.888919@monster.linux.in> Message-ID: On Sat, 5 Jan 2002, Prabhu Ramachandran wrote: > >>>>> "PP" == Pearu Peterson writes: > PP> def get_version(release_level='alpha', path='.', major=None): > PP> """ Return version string calculated from CVS tree or found in > PP> /__version__.py. Automatically update > PP> /__version__.py if the version is changed. An attempt > PP> is made to guarantee that version is increasing in time. This > PP> function always succeeds. None is returned if no version > PP> information is available. > > Sorry for the dumb question but I'm afraid I really dont understand > this too well. Does get_version automatically change the version? So > if I were to rebuild the sources the version would increase? If > get_version changes the version maybe the name should be set_version > and not get_version. Or maybe you should have two separate functions > for each. get_version updates the version only after modified files are commited to the CVS repository (as a result these files get new revision numbers). Building the sources or doing any other operations that does not change the CVS revision numbers in CVS/Entries file(s) will _not_ alter the version. get_version is "get a version" in that sense that it returns the version string from __version__.py file. But before that it checks that __version__.py is up to date by calculating the version from the CVS revision numbers. May be update_version would be a better name because the purpose of this function is neither to get or set the version, but to ensure that the version numbers in __version__.py reflect the actual state of the CVS repository. It is just convinient that it returns the version string that can be used in the version keyword argument of the setup() function. Pearu From prabhu at aero.iitm.ernet.in Sat Jan 5 13:28:01 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sat, 5 Jan 2002 23:58:01 +0530 Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation In-Reply-To: <071f01c1960c$83a11330$777ba8c0@ericlaptop> References: <062601c19569$984f67d0$777ba8c0@ericlaptop> <15415.15280.164483.718499@monster.linux.in> <071f01c1960c$83a11330$777ba8c0@ericlaptop> Message-ID: <15415.17841.136421.215483@monster.linux.in> >>>>> "eric" == eric writes: eric> Thanks Prabhu, This looks like a problem in how functions eric> are cataloged. I look into it. I just glanced thru catalog.py and it sounds really cool. weave itself seems really nice. The interface seems very easy to use. Of course, I've only looked superficially. Hopefully I'll use some of this stuff during the CFD course I'm sitting thru this sem. Maybe even a CFD module might come out of it. We'll see how it goes. :) BTW I se a few temp files and directories in /tmp. Are you deleting unused files later or is this done on purpose for debugging or does it happen because some tests failed? thanks, prabhu From eric at scipy.org Sat Jan 5 22:56:46 2002 From: eric at scipy.org (eric) Date: Sat, 5 Jan 2002 22:56:46 -0500 Subject: [SciPy-dev] weave -- inline C/C++ in Python, an implementation References: Message-ID: <083801c19666$29b28340$777ba8c0@ericlaptop> Pearu and Fernando, I prefer Unix line endings also. I messed around with fixing this, and only found one file with line ending problems (ast_tools.py). The others looked fine to me when I checked them out of CVS and looked at them on a Sun machine. Wait. Are yall having problems with the CVS version of the files or the tar balls? The tar balls are made on a Windows machine and that is probably that is the problem. CVS must do the line conversion automatically on my Windows machine and convert the Unix line endings in CVS to Windows line endings when they are check out. Then, the tar balls created have the wrong line endings also. bummer. I'll try and figure out an automated fix for this. For now, they'll still be DOS endings in the tar ball. Like Pearu said though, this shouldn't be a problem, it's just annoying. eric ----- Original Message ----- From: "Pearu Peterson" To: Sent: Saturday, January 05, 2002 3:53 AM Subject: Re: [SciPy-dev] weave -- inline C/C++ in Python, an implementation > > On Sat, 5 Jan 2002, Fernando P?rez wrote: > > > Any ideas? One thing I noticed, and I don't know if it may cause problems: > > your files have DOS line endings (\r\n). In emacs these show up as ugly ^M > > characters at the end of every line, but I don't know if they can confuse > > python. If nothing else, do you know if there's a way to have distutils clean > > up the line endings for each platform? > > I had similar problem with DOS line endings, in particular with executing > setup.py script from a command line: > ./setup.py .... > that failed with strange messages (not reletated to python) until I > changed the line endings with the `flip -u *` command. > However, in my debian box I don't experience the problems that you do, so > the DOS line endings should not be the problem for linux python. > > Pearu > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From prabhu at aero.iitm.ernet.in Sun Jan 6 21:58:26 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 7 Jan 2002 08:28:26 +0530 Subject: [SciPy-dev] Automated get_version In-Reply-To: References: <15415.15570.95637.888919@monster.linux.in> Message-ID: <15417.3794.691758.303749@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> get_version updates the version only after modified files are PP> commited to the CVS repository (as a result these files get PP> new revision numbers). Building the sources or doing any PP> other operations that does not change the CVS revision numbers PP> in CVS/Entries file(s) will _not_ alter the version. OK. That clarifies it. PP> May be update_version would be a better name because the PP> purpose of this function is neither to get or set the version, I would think update_version should check and change the version if necessary and get_version should simply return the current version. But maybe it does not matter. prabhu From eric at scipy.org Mon Jan 7 01:48:54 2002 From: eric at scipy.org (eric) Date: Mon, 7 Jan 2002 01:48:54 -0500 Subject: [SciPy-dev] new version of weave Message-ID: <097101c19747$6026c3f0$777ba8c0@ericlaptop> Hey folks, I cleaned up a few issues with weave over the weekend, and the new files are available for testing. Among the things fixed: * Files in tar ball now have Unix line endings * Fixed a few platform specific issues with the catalog found by Prabhu * Added support for Solaris with gcc -- all tests pass weave now works (for me) under RH 7.1 (gcc 2.96) and Solaris (gcc 2.95.1). On windows, gcc and MSVC both pass. I haven't been able to figure out the "Abort" issue reported by Fernando on Mandrake. I'll try and talk with him more this week and try to ferret out the issue. Files at: http://www.scipy.org/site_content/weave Please download and test! I hope to release by mid-week. thanks, eric From pearu at cens.ioc.ee Mon Jan 7 02:59:46 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 7 Jan 2002 09:59:46 +0200 (EET) Subject: [SciPy-dev] Automated get_version In-Reply-To: <15417.3794.691758.303749@monster.linux.in> Message-ID: On Mon, 7 Jan 2002, Prabhu Ramachandran wrote: > I would think update_version should check and change the version if > necessary and get_version should simply return the current version. > But maybe it does not matter. Not really indeed. Nevertheless, I renamed get_version to update_version and introduced get_version that does not change the __version__.py file on the disk, just warns if the version information is changed. In addition, update_version has a new keyword argument version_template = '%(major)d.%(minor)d.%(micro)d-%(release_level)s-%(serial)d', so that one has a full control over how the version string should look like. Regards, Pearu From prabhu at aero.iitm.ernet.in Mon Jan 7 03:44:42 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 7 Jan 2002 14:14:42 +0530 Subject: [SciPy-dev] new version of weave In-Reply-To: <097101c19747$6026c3f0$777ba8c0@ericlaptop> References: <097101c19747$6026c3f0$777ba8c0@ericlaptop> Message-ID: <15417.24570.448897.692367@monster.linux.in> >>>>> "eric" == eric writes: eric> Hey folks, I cleaned up a few issues with weave over the eric> weekend, and the new files are available for testing. Among eric> the things fixed: eric> * Files in tar ball now have Unix line endings * Fixed a eric> few platform specific issues with the catalog found by eric> Prabhu * Added support for Solaris with gcc -- all tests eric> pass No luck still. :( I just updated my cvs copy and tried weave.test. I even deleted my older ~/.python21_compiled/ directory. I get the following error: [snip] repairing catalog by removing key Frepairing catalog by removing key F.................warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations ....warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations .................................................................................................. ====================================================================== ERROR: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_blitz_tools.py", line 152, in check_5point_avg_2d self.generic_2d(expr) File "weave/tests/test_blitz_tools.py", line 128, in generic_2d standard,compiled = self.generic_test(expr,arg_dict,type,size) File "weave/tests/test_blitz_tools.py", line 85, in generic_test blitz_tools.blitz(expr,arg_dict,{},verbose=0) File "weave/blitz_tools.py", line 100, in blitz function_catalog.add_function(expr,func,module_dir) File "weave/catalog.py", line 549, in add_function self.add_function_persistent(code,function) File "weave/catalog.py", line 577, in add_function_persistent function_list = [function] + cat.get(code,[]) File "/usr/local/lib/python2.1/shelve.py", line 66, in get return self[key] File "/usr/local/lib/python2.1/shelve.py", line 71, in __getitem__ return Unpickler(f).load() UnpicklingError: pickle data was truncated ====================================================================== FAIL: check_add_function_ordered (test_catalog.test_catalog) ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_catalog.py", line 361, in check_add_function_ordered assert(funcs1 == [string.lower,string.upper]) AssertionError ====================================================================== FAIL: Test persisting a function in the default catalog ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_catalog.py", line 281, in check_add_function_persistent1 assert(i in pfuncs) AssertionError ---------------------------------------------------------------------- Ran 184 tests in 491.451s FAILED (failures=2, errors=1) I hope I'm not doing something stupid here. prabhu From eric at scipy.org Mon Jan 7 09:48:07 2002 From: eric at scipy.org (eric) Date: Mon, 7 Jan 2002 09:48:07 -0500 Subject: [SciPy-dev] new version of weave References: <097101c19747$6026c3f0$777ba8c0@ericlaptop> <15417.24570.448897.692367@monster.linux.in> Message-ID: <09f801c1978a$51ecd0c0$777ba8c0@ericlaptop> One possibility is that you have some corrupt stuff left over from the previous test runs, since you were playing with this earlier. Can you remove the ~/.python_compiled directory in your home and everything under it. Also, as you noted earlier, there could be some files left laying around in your /tmp directory. Trash these if you see them also. Then run the test again -- hopefully that helped. In the mean time, I'll try to track down why corrupt files actually matter (they shouldn't break things) eric ----- Original Message ----- From: "Prabhu Ramachandran" To: Sent: Monday, January 07, 2002 3:44 AM Subject: [SciPy-dev] new version of weave > >>>>> "eric" == eric writes: > > eric> Hey folks, I cleaned up a few issues with weave over the > eric> weekend, and the new files are available for testing. Among > eric> the things fixed: > > eric> * Files in tar ball now have Unix line endings * Fixed a > eric> few platform specific issues with the catalog found by > eric> Prabhu * Added support for Solaris with gcc -- all tests > eric> pass > > No luck still. :( I just updated my cvs copy and tried weave.test. I > even deleted my older ~/.python21_compiled/ directory. I get the > following error: > > [snip] > repairing catalog by removing key > Frepairing catalog by removing key > F.................warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations > ....warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations > ............................................................................ ...................... > ====================================================================== > ERROR: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "weave/tests/test_blitz_tools.py", line 152, in check_5point_avg_2d > self.generic_2d(expr) > File "weave/tests/test_blitz_tools.py", line 128, in generic_2d > standard,compiled = self.generic_test(expr,arg_dict,type,size) > File "weave/tests/test_blitz_tools.py", line 85, in generic_test > blitz_tools.blitz(expr,arg_dict,{},verbose=0) > File "weave/blitz_tools.py", line 100, in blitz > function_catalog.add_function(expr,func,module_dir) > File "weave/catalog.py", line 549, in add_function > self.add_function_persistent(code,function) > File "weave/catalog.py", line 577, in add_function_persistent > function_list = [function] + cat.get(code,[]) > File "/usr/local/lib/python2.1/shelve.py", line 66, in get > return self[key] > File "/usr/local/lib/python2.1/shelve.py", line 71, in __getitem__ > return Unpickler(f).load() > UnpicklingError: pickle data was truncated > ====================================================================== > FAIL: check_add_function_ordered (test_catalog.test_catalog) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "weave/tests/test_catalog.py", line 361, in check_add_function_ordered > assert(funcs1 == [string.lower,string.upper]) > AssertionError > ====================================================================== > FAIL: Test persisting a function in the default catalog > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "weave/tests/test_catalog.py", line 281, in check_add_function_persistent1 > assert(i in pfuncs) > AssertionError > ---------------------------------------------------------------------- > Ran 184 tests in 491.451s > > FAILED (failures=2, errors=1) > > > > I hope I'm not doing something stupid here. > > prabhu > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From prabhu at aero.iitm.ernet.in Mon Jan 7 11:08:37 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 7 Jan 2002 21:38:37 +0530 Subject: [SciPy-dev] new version of weave In-Reply-To: <09f801c1978a$51ecd0c0$777ba8c0@ericlaptop> References: <097101c19747$6026c3f0$777ba8c0@ericlaptop> <15417.24570.448897.692367@monster.linux.in> <09f801c1978a$51ecd0c0$777ba8c0@ericlaptop> Message-ID: <15417.51205.897078.969051@monster.linux.in> >>>>> "eric" == eric writes: eric> One possibility is that you have some corrupt stuff left eric> over from the previous test runs, since you were playing eric> with this earlier. eric> Can you remove the ~/.python_compiled directory in your home eric> and everything under it. Also, as you noted earlier, there eric> could be some files left laying around in your /tmp eric> directory. Trash these if you see them also. eric> Then run the test again -- hopefully that helped. well, even earlier I had removed my .python21_compiled directory. I dont think I removed the stuff in /tmp. This time I did remove things in /tmp. However, I still get the same error. :( Also, the tests fail very badly when I use ipython. prabhu From eric at scipy.org Mon Jan 7 10:16:40 2002 From: eric at scipy.org (eric) Date: Mon, 7 Jan 2002 10:16:40 -0500 Subject: [SciPy-dev] new version of weave References: <097101c19747$6026c3f0$777ba8c0@ericlaptop><15417.24570.448897.692367@monster.linux.in><09f801c1978a$51ecd0c0$777ba8c0@ericlaptop> <15417.51205.897078.969051@monster.linux.in> Message-ID: <0a3d01c1978e$4eb5c4d0$777ba8c0@ericlaptop> > eric> One possibility is that you have some corrupt stuff left > eric> over from the previous test runs, since you were playing > eric> with this earlier. > > eric> Can you remove the ~/.python_compiled directory in your home > eric> and everything under it. Also, as you noted earlier, there > eric> could be some files left laying around in your /tmp > eric> directory. Trash these if you see them also. > > eric> Then run the test again -- hopefully that helped. > > well, even earlier I had removed my .python21_compiled directory. I > dont think I removed the stuff in /tmp. This time I did remove things > in /tmp. However, I still get the same error. :( bummer. Ok. We'll work through this off line. > > Also, the tests fail very badly when I use ipython. What is ipython? eric From fperez at pizero.colorado.edu Mon Jan 7 15:09:53 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Mon, 7 Jan 2002 13:09:53 -0700 (MST) Subject: [SciPy-dev] new version of weave In-Reply-To: <15417.51205.897078.969051@monster.linux.in> Message-ID: > > Also, the tests fail very badly when I use ipython. > Hi Prabhu, load IPython with -nodeep_reload. After seeing that I made it the new default (you can permanently set it in your .ipython/ipythonrc as deep_reload 0). Making that change at least I get the exact same failure with ipython as with regular python. Let me know if that's not enough. I definitely want ipython to reproduce 100% of python's normal behavior with valid code. Cheers, f From fperez at pizero.colorado.edu Mon Jan 7 15:13:25 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Mon, 7 Jan 2002 13:13:25 -0700 (MST) Subject: [SciPy-dev] new version of weave In-Reply-To: <0a3d01c1978e$4eb5c4d0$777ba8c0@ericlaptop> Message-ID: > > Also, the tests fail very badly when I use ipython. > > What is ipython? > That's my baby :) Check it out at www-hep.colorado.edu/~fperez/ipython. It's basically a python interactive interpreter on steroids (input/output history, dynamic object introspection, shell access, lots more). Let me know if you like it. Cheers, f From eric at scipy.org Mon Jan 7 14:40:06 2002 From: eric at scipy.org (eric) Date: Mon, 7 Jan 2002 14:40:06 -0500 Subject: [SciPy-dev] SciPy CVS disruption Message-ID: <0b4701c197b3$1ca79750$777ba8c0@ericlaptop> Hey group, I'm making some pretty big changes to the build process, so the CVS version may be broken for a little while. I'll let you know when changes are finished. eric From prabhu at aero.iitm.ernet.in Mon Jan 7 20:49:53 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Tue, 8 Jan 2002 07:19:53 +0530 Subject: [SciPy-dev] new version of weave In-Reply-To: References: <15417.51205.897078.969051@monster.linux.in> Message-ID: <15418.20545.187001.297457@monster.linux.in> >>>>> "FP" == Fernando P?rez writes: >> Also, the tests fail very badly when I use ipython. FP> load IPython with -nodeep_reload. After seeing that I made it FP> the new default (you can permanently set it in your FP> .ipython/ipythonrc as deep_reload 0). Making that change at FP> least I get the exact same failure with ipython as with FP> regular python. Yes, that seems to do it, thanks. prabhu From rob at pythonemproject.com Tue Jan 8 15:59:12 2002 From: rob at pythonemproject.com (Rob) Date: Tue, 08 Jan 2002 12:59:12 -0800 Subject: [SciPy-dev] my second try at weave :) Message-ID: <3C3B5DA0.47D3AF48@pythonemproject.com> Here is the ouput upon trying to install the latest tarball: Script started on Tue Jan 8 12:56:06 2002 dep: ['scipy_distutils', 'scipy_test'] Traceback (most recent call last): File "setup.py", line 44, in ? stand_alone_package(with_dependencies) File "setup.py", line 23, in stand_alone_package config_dict = package_config(primary,dependencies) File "scipy_distutils/misc_util.py", line 174, in package_config config.extend([get_package_config(x) for x in primary]) File "scipy_distutils/misc_util.py", line 161, in get_package_config mod = __import__('setup_'+package_name) ImportError: No module named setup_weave Script done on Tue Jan 8 12:56:06 2002 -- The Numeric Python EM Project www.pythonemproject.com From rlytle at tqs.com Wed Jan 9 09:28:56 2002 From: rlytle at tqs.com (Lytle, Robert TQO) Date: Wed, 9 Jan 2002 06:28:56 -0800 Subject: [SciPy-dev] new Weave package Message-ID: <81E1D2E15CCBD311A74700A0C9E1CC8E03D9FBCB@chunky.tqs.com> Excellent, maybe it will install on FreeBSD without 20 hacks being made to the source :) I'll try it out as soon as possible this weekend as I've already got code that worked with the old compiler. Rob. From nbecker at fred.net Wed Jan 9 13:11:50 2002 From: nbecker at fred.net (nbecker at fred.net) Date: 09 Jan 2002 13:11:50 -0500 Subject: [SciPy-dev] SciPy-0.1 failure Message-ID: I just tried SciPy-0.1 + Numeric-20.3 + Python-2.1. I have this result. Any hints? Python 2.1.1 (#1, Aug 13 2001, 19:37:40) [GCC 2.96 20000731 (Red Hat Linux 7.1 2.96-96)] on linux2 Type "copyright", "credits" or "license" for more information. >>> import scipy Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.1/site-packages/scipy/__init__.py", line 41, in ? from handy import * File "/usr/lib/python2.1/site-packages/scipy/handy.py", line 1, in ? import Numeric File "/usr/lib/python2.1/site-packages/Numeric/Numeric.py", line 119, in ? arrayrange = multiarray.arange AttributeError: 'multiarray' module has no attribute 'arange' >>> From eric at scipy.org Wed Jan 9 17:46:09 2002 From: eric at scipy.org (eric) Date: Wed, 9 Jan 2002 17:46:09 -0500 Subject: [SciPy-dev] new buld process in CVS Message-ID: <01ab01c1995f$7021f120$6b01a8c0@ericlaptop> SciPy's build process has been revamped. Thanks to Pearu for tons of help in this area on getting this all setup. Please grab the CVS and test on your given architecture so we can find out what has broken. Please report both success and failure. You'll need the latest snapshot of f2py (see below). Also, you'll need to set the following line in setup.py to point to your atlas libraries: scipy_distutils.atlas_info.library_path = ['home/ej/lib/atlas'] Then: python setup.py build python setup.py install python >>> import scipy >>> scipy.test() The main purpose of the reorg was (1) to modularize the build process so that it is easier to maintain, (2) make it easy for others to build packages that can be dropped into SciPy with minimum fuss, and (3) Allow for building and installing individual modules outside of SciPy without changing any install scripts. There are notes about how to do (2) at the bottom of this file. thanks, eric new dependencies: f2py2e (http://cens.ioc.ee/projects/f2py2e/) The latest snap shot is needed and is here: http://cens.ioc.ee/projects/f2py2e/rel-5.x/ This must be installed to build SciPy -- it has a setup.py script so it is easy. old dependencies: You still need these. atlas fftw changes: * scipy_distutils is a new package (installed by scipy, but as a separate package) that provides support for Fortran, f2py, and most of the other customizations that scipy uses for distutils. It is also used by Pearu's f2py package, and he's made a ton of contributions here. * setup.py is now very compact. Most setup work is delegated to sub-packages. * each sub-package has a setup_foo.py file in it that has a configuration() method. This returns a dictionary of key word arguments that can be passed to setup.py to build the package. A simple example can be found in fastumath/setup_fastumath.py. More complex examples are xplt/setup_xplt.py and linalg/setup_linalg.py. * scipy_distutils.atlas_info attempts to find your atlas library installation. It is pretty dumb right now. It has a variable called library_path that you can use to set the path to your atlas directory. Note that it takes a list, not a string as the argument: import scipy_distutils.atlas_info scipy_distutils.atlas_info.library_path = ['/home/ej/lib/atlas'] fftw_info.py serves the same purpose for fftw. Still need to add library_path to it. * scipy.compiler has been renamed to scipy.weave. It has been significantly enhanced in its capability. (still a few bugs here though.) * Numeric is no longer installed by the setup.py script. You'll have to install it separately (we'll continue to include it in the windows executables at point releases) * fastumath is installed as a separate module instead of being dropped in the Numeric directory. * scipy_test.py is now a package (do to distutils issues) and installed as a separate stand-alone package. * SciPy now builds with compilers other than g77 on Linux. Not much testing has been done yet with these: * Intel Fortran (by Pearu Peterson) * Absoft 5.0 (compiles, but still problems with upper/lowercase and underscore issues) ------------------ To include the package foo within SciPy, you'll need to place the foo directory within the SciPy directory, and create a setup_foo.py script within that directory: scipy/ foo/ foo1_source.c foo2_source.c __init__.py foo_script.py setup_foo.py setup_foo.py should have a single method in it called configuration() that looks like: #setup_foo.py import os from scipy_distutils.core import Extension from scipy_distutils.misc_util import get_path, default_config_dict def configuration(parent_package=''): # if we have a parent module, append a '.' to it # to simplify adding it to package names, etc. if parent_package: parent_package += '.' # find the local path of this module (/home/eric/scipy/foo) local_path = get_path(__name__) # grab a default configuration dictionary. We'll fill this # with this packages specifics. config = default_config_dict() # Set the name of this package. config['packages'].append(parent_package+'foo') # build the first extension module sources = ['foo1_source.c'] sources = [os.path.join(local_path,x) for x in sources] ext = Extension(parent_package+'foo.foo1',sources) config['ext_modules'].append(ext) # build the second extension module sources = ['foo2_source.c'] sources = [os.path.join(local_path,x) for x in sources] ext = Extension(parent_package+'foo.foo2',sources) config['ext_modules'].append(ext) # return the configuration information. return config Second, you have to add the name of the module 'foo' to one of the installation lists in setup.py. # setup.py (in scipy directory) . . . # standard modules standard_packages = ['io','linalg','special','signal','stats', 'interpolate','integrate','optimize', 'cluster','cow','ga','weave',] . . . That is about it. The build process should detect your package and install it as a sub-package of scipy. If you'd like to bundle your package with SciPy, but want to have it installed as a stand-alone package, put it in the following list of packages: # setup.py (in scipy directory) . . . # these packages aren't nested under scipy separate_packages = ['gui_thread','scipy_test','scipy_distutils', 'fastumath'] . . . The default_config_dict function looks like this: # scipy_distuils/misc_util.py . . . list_keys = ['packages', 'ext_modules', 'data_files', 'include_dirs', 'libraries', 'fortran_libraries', 'headers'] dict_keys = ['package_dir'] def default_config_dict(): d={} for key in list_keys: d[key] = [] for key in dict_keys: d[key] = {} return d . . . Let me know if the list of key words needs to be augmented to support something specail you are doing. From rob at pythonemproject.com Wed Jan 9 19:31:24 2002 From: rob at pythonemproject.com (Rob) Date: Wed, 09 Jan 2002 16:31:24 -0800 Subject: [SciPy-dev] Sorry for my sarcastic post from work Message-ID: <3C3CE0DC.3EA22092@pythonemproject.com> See Eric, I'm in a much better mood when I get home :) Rob. -- The Numeric Python EM Project www.pythonemproject.com From pearu at cens.ioc.ee Thu Jan 10 03:39:25 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Thu, 10 Jan 2002 10:39:25 +0200 (EET) Subject: [SciPy-dev] new buld process in CVS In-Reply-To: <01ab01c1995f$7021f120$6b01a8c0@ericlaptop> Message-ID: Hi, > To include the package foo within SciPy, you'll need to place the foo > > def configuration(parent_package=''): > # if we have a parent module, append a '.' to it > # return the configuration information. > return config To achive a better modularization of scipy, I would suggest the following addition to the setup_foo.py files: if __name__ == "__main__": from scipy_distutils.core import setup setup(**configuration()) so that when setup_foo.py build is run from foo/ directory, it must build and the resulting module must be usable without scipy. I think it makes it also easier to test submodules of scipy from their directories rather than editing the top level setup.py to disable other packages in order to just build the one module under testing. I realize that this might not be possible for all modules (due to scipy leveling scheme) but the lowest level modules as the most important ones should have more convinient testing facilities. What do you think? Regards, Pearu From prabhu at aero.iitm.ernet.in Thu Jan 10 07:06:13 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 10 Jan 2002 17:36:13 +0530 Subject: [SciPy-dev] new buld process in CVS In-Reply-To: <01ab01c1995f$7021f120$6b01a8c0@ericlaptop> References: <01ab01c1995f$7021f120$6b01a8c0@ericlaptop> Message-ID: <15421.33717.327232.805427@monster.linux.in> >>>>> "eric" == eric writes: eric> SciPy's build process has been revamped. Thanks to Pearu eric> for tons of help in this area on getting this all setup. eric> Please grab the CVS and test on your given architecture so eric> we can find out what has broken. Please report both success eric> and failure. eric> You'll need the latest snapshot of f2py (see below). Also, eric> you'll need to set the following line in setup.py to point eric> to your atlas libraries: eric> scipy_distutils.atlas_info.library_path = eric> ['home/ej/lib/atlas'] The new build seems to run fine and I didn't need to change the above flag. I also dont like this setting since you'll have trouble with CVS versioning when you change setup.py. The next time setup.py is changed in the CVS tree you risk the complication of a conflict. I'd suggest placing user configs in a separate file that is not in the CVS tree. prabhu From eric at scipy.org Thu Jan 10 09:31:12 2002 From: eric at scipy.org (eric) Date: Thu, 10 Jan 2002 09:31:12 -0500 Subject: [SciPy-dev] new buld process in CVS References: Message-ID: <002a01c199e3$73d37b70$6b01a8c0@ericlaptop> Hey Pearu, > if __name__ == "__main__": > from scipy_distutils.core import setup > setup(**configuration()) > > > > What do you think? I think it is good. I'll add it. I have usually just added setup.py module to the packages that does exactly this. That is just because most people expect the "setup.py" to be the name of the setup script. Either (or both) are good ideas. eric From eric at scipy.org Thu Jan 10 09:37:17 2002 From: eric at scipy.org (eric) Date: Thu, 10 Jan 2002 09:37:17 -0500 Subject: [SciPy-dev] new buld process in CVS References: <01ab01c1995f$7021f120$6b01a8c0@ericlaptop> <15421.33717.327232.805427@monster.linux.in> Message-ID: <003001c199e4$4daf4590$6b01a8c0@ericlaptop> > The new build seems to run fine and I didn't need to change the above > flag. I also dont like this setting since you'll have trouble with > CVS versioning when you change setup.py. The next time setup.py is > changed in the CVS tree you risk the complication of a conflict. I'd > suggest placing user configs in a separate file that is not in the CVS > tree. Good idea. Maybe the setup.cfg file is a good choice. I'm hoping that the atlas_info script gets more itelligent -- perhaps asking you for the path if it can't find it. It could then generate the file for you. If it didn't find the library, in the best scenarios, it would download it from the scipy website. This is probably a little ways out though. eric From prabhu at aero.iitm.ernet.in Thu Jan 10 11:19:40 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 10 Jan 2002 21:49:40 +0530 Subject: [SciPy-dev] new buld process in CVS In-Reply-To: <003001c199e4$4daf4590$6b01a8c0@ericlaptop> References: <01ab01c1995f$7021f120$6b01a8c0@ericlaptop> <15421.33717.327232.805427@monster.linux.in> <003001c199e4$4daf4590$6b01a8c0@ericlaptop> Message-ID: <15421.48924.58063.901822@monster.linux.in> >>>>> "eric" == eric writes: >> complication of a conflict. I'd suggest placing user configs >> in a separate file that is not in the CVS tree. eric> Good idea. Maybe the setup.cfg file is a good choice. eric> I'm hoping that the atlas_info script gets more itelligent eric> -- perhaps asking you for the path if it can't find it. It eric> could then generate the file for you. If it didn't find the That sounds like a good idea. eric> library, in the best scenarios, it would download it from eric> the scipy website. This is probably a little ways out eric> though. Maybe once ciphon (http://sourceforge.net/projects/pythonsiphon/) gets good enough we could use it. Also, I noticed on the weave documentation page that you were looking for a way to check if given an xterm if there is an X11 display available. The easiest way to do this is to check for the DISPLAY env variable. In [1]: import os In [2]: print os.environ['DISPLAY'] :0.0 On a remote telnet session this will cause a KeyError. $ python -c "import os; print os.environ.get('DISPLAY')" None Eric, weave.blitz is fantastic! Great job!! I haven't really checked out weave.inline yet but like blitz a lot. :) One thing I thought I'd mention. I'm sure you are aware of this but I think it worthy of mention. When you use blitz for an expression the computation is different from that when done by numeric. blitz will use the computed values immediately (Gauss-Siedel) while numeric will make a copy and do the computation (Gauss-Jordan). This means that sometimes the result of a computation when done by exec(expr) will be different from that of blitz(expr). Here is an example. # 4 point average. >>> expr = "u[1:-1, 1:-1] = (u[0:-2, 1:-1] + u[2:, 1:-1] + "\ ... "u[1:-1,0:-2] + u[1:-1, 2:])*0.25" >>> u = zeros((5, 5), 'd'); u[0,:] = 100 >>> exec (expr) >>> u array([[ 100., 100., 100., 100., 100.], [ 0., 25., 25., 25., 0.], [ 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0.]]) >>> u = zeros((5, 5), 'd'); u[0,:] = 100 >>> weave.blitz (expr) >>> u array([[ 100. , 100. , 100. , 100. , 100. ], [ 0. , 25. , 31.25 , 32.8125 , 0. ], [ 0. , 6.25 , 9.375 , 10.546875 , 0. ], [ 0. , 1.5625 , 2.734375 , 3.3203125, 0. ], [ 0. , 0. , 0. , 0. , 0. ]]) This is not a bad thing at all but worth knowing, IMHO. prabhu From prabhu at aero.iitm.ernet.in Thu Jan 10 11:31:17 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 10 Jan 2002 22:01:17 +0530 Subject: [SciPy-dev] Nightly tarballs? Message-ID: <15421.49621.12026.741288@monster.linux.in> hi, I think it would be a good idea to make nightly (or atleast frequent) snapshots of scipy available. At my lab we are behind a firewall and there is no way I can get scipy off CVS. I'm sure there are others in a similar situation. Ofcourse, at home I can get the CVS tree and make an sdist but I thought it would be nice if this were available of scipy.org for the benefit of those behind a firewall. thanks, prabhu From eric at scipy.org Thu Jan 10 10:57:07 2002 From: eric at scipy.org (eric) Date: Thu, 10 Jan 2002 10:57:07 -0500 Subject: [SciPy-dev] new buld process in CVS References: <01ab01c1995f$7021f120$6b01a8c0@ericlaptop><15421.33717.327232.805427@monster.linux.in><003001c199e4$4daf4590$6b01a8c0@ericlaptop> <15421.48924.58063.901822@monster.linux.in> Message-ID: <007701c199ef$7518fd50$6b01a8c0@ericlaptop> Hey Prabhu, > > eric> library, in the best scenarios, it would download it from > eric> the scipy website. This is probably a little ways out > eric> though. > > Maybe once ciphon (http://sourceforge.net/projects/pythonsiphon/) gets > good enough we could use it. Yes, it might. Has anyone played with it at all? On the other hand, Python's ftp/http support is so good, that it only takes a few lines of code to handle the download. The architecture/CPU detection is more difficult, but not to bad. Still, if ciphon handles this, then know need to duplicate. > Also, I noticed on the weave documentation page that you were looking > for a way to check if given an xterm if there is an X11 display > available. The easiest way to do this is to check for the DISPLAY env > variable. > > In [1]: import os > In [2]: print os.environ['DISPLAY'] > :0.0 > > On a remote telnet session this will cause a KeyError. > > $ python -c "import os; print os.environ.get('DISPLAY')" > None Good. Thanks. I'll try and work this into the code in the needed places. > > Eric, weave.blitz is fantastic! Great job!! I haven't really checked > out weave.inline yet but like blitz a lot. :) Glad it is working for you. > One thing I thought I'd mention. I'm sure you are aware of this but I > think it worthy of mention. When you use blitz for an expression the > computation is different from that when done by numeric. blitz will > use the computed values immediately (Gauss-Siedel) while numeric will > make a copy and do the computation (Gauss-Jordan). This means that > sometimes the result of a computation when done by exec(expr) will be > different from that of blitz(expr). Here is an example. > This is true. I definitely need to mention this in the documentation somewhere. It can happen if values on the RHS of the equation are also on the LHS of the equation -- but only in certain circumstances like your example. Thanks for pointing this out. eric From pearu at cens.ioc.ee Thu Jan 10 12:22:43 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Thu, 10 Jan 2002 19:22:43 +0200 (EET) Subject: [SciPy-dev] new buld process in CVS In-Reply-To: <007701c199ef$7518fd50$6b01a8c0@ericlaptop> Message-ID: On Thu, 10 Jan 2002, eric wrote: > > In [1]: import os > > In [2]: print os.environ['DISPLAY'] > > :0.0 > > > > On a remote telnet session this will cause a KeyError. > > > > $ python -c "import os; print os.environ.get('DISPLAY')" > > None > > Good. Thanks. I'll try and work this into the code in the needed places. However, there must be some better way for that as checking DISPLAY is not very reliable. For example, one can have DISPLAY variable not set also in X terminals: pearu at localhost:~$ python -c "import os;print repr(os.environ.get('DISPLAY'))" ':0.0' pearu at localhost:~$ unset DISPLAY pearu at localhost:~$ python -c "import os;print repr(os.environ.get('DISPLAY'))" None Regards, Pearu From prabhu at aero.iitm.ernet.in Thu Jan 10 12:31:56 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 10 Jan 2002 23:01:56 +0530 Subject: [SciPy-dev] new buld process in CVS In-Reply-To: References: <007701c199ef$7518fd50$6b01a8c0@ericlaptop> Message-ID: <15421.53260.896075.447513@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> However, there must be some better way for that as checking PP> DISPLAY is not very reliable. For example, one can have PP> DISPLAY variable not set also in X terminals: I think that it is not too bad if DISPLAY is not set. (1) In such a case you cannot run an X application properly so you cant run the tests anyway. well, technically you could by passing -display but seldom do folks do it. $ unset DISPLAY $ display # an X11 app display: Unable to connect to X server () [Function not implemented]. (2) Even if you ssh to a remote host and enable X11 forwarding the DISPLAY variable is automatically set. (3) I dont know why one would want to unset DISPLAY. :) prabhu From prabhu at aero.iitm.ernet.in Thu Jan 10 12:50:42 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 10 Jan 2002 23:20:42 +0530 Subject: [SciPy-dev] Trouble with sdist. Message-ID: <15421.54386.287100.32297@monster.linux.in> hi, I am unable to create an sdist with the latest CVS tree. :( $ python setup.py sdist -d /tmp [snip] File "scipy_distutils/command/sdist.py", line 93, in make_distribution self.make_release_tree(base_dir, self.filelist.files) File "scipy_distutils/command/sdist.py", line 32, in make_release_tree dest_files = remove_common_base(files) File "scipy_distutils/command/sdist.py", line 126, in remove_common_base results = [string.replace(file,base,'') for file in files] File "/usr/local/lib/python2.1/string.py", line 369, in replace return s.replace(old, new, maxsplit) ValueError: empty pattern string I tried fixing scipy_distutils/command/sdist.py to do this: results = [] if base: results = [string.replace(file,base,'') for file in files] But then I got a trivially small tarball (341 bytes -- I've heard of good compression but this is too much ;). I'm unable to figure out why the problem occurs. prabhu From prabhu at aero.iitm.ernet.in Fri Jan 11 10:44:27 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Fri, 11 Jan 2002 21:14:27 +0530 Subject: [SciPy-dev] scipy.weave versus simple C++. Message-ID: <15423.2139.753600.106641@monster.linux.in> hi, I've been checking out Python, Numeric, Weave recently for serious numeric computations. I've started off with a very easy sample problem that translates well to Numeric expressions. The results are very interesting. I'm comparing weave.blitz with simple C++ code (no classes or virtual functions etc, just plain arrays and structs.) The problem is to solve the Laplace equation on a square grid with a given boundary condition using a four point average. Here is what I found. The core loop in C++ (I use double for all the computations). double **u = g->u; // g is a simple structure. for (int i=1; i Message-ID: <012f01c19ab3$231d22a0$6b00a8c0@ericlaptop> Hey Prabhu, Thanks for the comparison. This kind of benchmarking is very helpful for figuring out what work still need to be done. Send me your code, and I'll either make an inline() version of it, or try and figure out how we can do the slow stuff with blitz(). eric ----- Original Message ----- From: "Prabhu Ramachandran" To: Sent: Friday, January 11, 2002 10:44 AM Subject: [SciPy-dev] scipy.weave versus simple C++. > hi, > > I've been checking out Python, Numeric, Weave recently for serious > numeric computations. I've started off with a very easy sample > problem that translates well to Numeric expressions. The results are > very interesting. > > I'm comparing weave.blitz with simple C++ code (no classes or virtual > functions etc, just plain arrays and structs.) The problem is to > solve the Laplace equation on a square grid with a given boundary > condition using a four point average. Here is what I found. > > The core loop in C++ (I use double for all the computations). > > double **u = g->u; // g is a simple structure. > > for (int i=1; i for (int j=1; j tmp = u[i][j]; > u[i][j] = ((u[i-1][j] + u[i+1][j])*dy2 + > (u[i][j-1] + u[i][j+1])*dx2)*dnr_inv; > err += SQR(u[i][j] - tmp); // SQR is a simple inline function. > } > return sqrt(err); > > The same core code in Python is next. I've labelled different > sections for convenience. > > u = self.grid.u # a (500x500 NumPy array). > # Step 1 > self.grid.old_u = u.copy() # yes, memory inefficient. > > # Step 2 > expr = "u[1:-1, 1:-1] = ((u[0:-2, 1:-1] + u[2:, 1:-1])*dy2 + "\ > "(u[1:-1,0:-2] + u[1:-1, 2:])*dx2)*dnr_inv" > scipy.weave.blitz(expr, check_size=0) > > # Step 3 > v = (u - self.grid.old_u).flat > return sqrt(dot(v,v)) > > > I compiled the c++ code with the -O3 flag. I then time the programs > for a 500x500 grid for 100 iterations. > > As it stands, here is the comparison. > > Weave.Blitz/Python: 10.68 s > Numeric/Python: 31.80 s > C++: 2.71 s > Best/Ratio: ~4.0 > > For other runsI get different numbers but the ratio seems set on about > 4. This is pretty darned good. Please remember that the Python code > does the 100 iterations in pure Python along with a few while/if > conditions to check for the error and number of iterations and other > things . Only the core loop has been optimized with weave/Numeric. > > If you look at the Python code you will notice that Step 1 and Step 3 > are pretty bad steps. Just for the sake of comparison I remove Step 1 > alone. I recompute the C++ code again just to take care in case of > other processes on my system etc. > > Weave.Blitz/Python: 7.0 s > C++: 2.47 s > Ratio: 2.83 > > You can see some definite improvement. Now I go further and replace > Step 3 with 'return 1.0'. Here are the results: > > Weave.Blitz/Python: 2.17 s > C++: 2.41 s > Ratio: 0.90 s > > This is an amazing result and shows why blitz++ and consequently > weave.blitz is so cool -- its faster than simple C/C++. I also inter > changed the order of the for loops in the C code and it only gets > slower (so thisis the best possible way to do it I guess). > > This shows that weave holds tremendous promise. However, I have a few > questions/issues. > > (1) Is there a better way to speed up the copy (step 1) and remove > the old_u array? Numeric is nice but not perfect. > > (2) I tried using blitz to speed up the error computation (Step 3) > and thecopy (step 1) but no go. I couldnt get it to work. > > (3) Whats the best way to deal with stuff that belongs in the > innermost loop (like the error computation that I do) ? Would > weave.inline do the job? I'd really like an inline example that > does something like my C++ for loop above using Numeric or Python > arrays. It would be very illustrative. > > All in all weave looks to be *very* promising!! I hope it grows and > gets even better. :) > > Please let me know if you want to look at my code. Also, let me know > if I'mdoing something brain dead here. > > prabhu > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From eric at scipy.org Fri Jan 11 12:50:43 2002 From: eric at scipy.org (eric) Date: Fri, 11 Jan 2002 12:50:43 -0500 Subject: Fw: [SciPy-dev] scipy.weave versus simple C++. Message-ID: <017901c19ac8$7e0ea430$6b00a8c0@ericlaptop> Hey Prabhu, Here is a near verbatim conversion of your laplace to use Python/Inline. I didn't spend much time, and there are errors. The results don't match. You'll get the general idea from it though. I used blitz arrays because they're indexing is kinda like double** indexing so the conversion was easy. Standard Numeric arrays don't have the pointer arrays to the rows that C uses, so your code wouldn't translate easily to it without macros or calculating offsets -- you could do this though. I'm not sure that indexing blitz arrays is as fast as indexing normal arrays -- this example would suggest not: 500x500, 100 iterations, PII 850 laptop, W2K, gcc-2.95.2 laplace.py: 3.47 seconds laplace.cxx: 2.34 seconds The Python calling overhead should be close to nil, so there is no reason why the python version should be slower, other than the fact that I used blitz arrays and indexing. But the current implementation still pays a 50% penalty for using Python. eric --------------------------------------------------------- # laplace.py import time try: from scipy import weave from scipy import * except: import weave from Numeric import * from weave.blitz_tools import blitz_type_factories def BC(x,y): return x**2-y**2 class Grid: def __init__(self,shape,typecode=Float64): self.shape = shape # should really handle typecode here. self.dx = 1.0 / shape[0] - 1 self.dy = 1.0 / shape[1] - 1 self.u = zeros(shape,typecode) def setBCFunc(self,f): xmin, ymin, xmax, ymax = 0.0, 0.0, 1.0, 1.0 x = arange(self.shape[0])*self.dy y = arange(self.shape[1])*self.dy self.u[0 ,:] = f(xmin,y) self.u[-1,:] = f(xmax,y) self.u[:, 0] = f(x,ymin) self.u[:,-1] = f(x,ymax) class LaplaceSolver: def __init__(self,grid): self.g = grid def timeStep(self, dt = 0.0): dx2 = self.g.dx**2 dy2 = self.g.dy**2 dnr_inv = .5/(dx2+dy2) nx, ny = self.g.shape u = self.g.u code = """ #line 39 "laplace.py" double tmp, err, diff; for (int i=1; i eps: if n_iter and (count > n_iter): return err; err = self.timeStep() #print err count += 1 return count if __name__ == "__main__": grid = Grid((500,500)) grid.setBCFunc(BC) s = LaplaceSolver(grid) t1 = time.time() err = s.solve(20) t2 = time.time() print "Iterations took ", t2 - t1, " seconds." print "Error: ", err From prabhu at aero.iitm.ernet.in Fri Jan 11 22:57:40 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sat, 12 Jan 2002 09:27:40 +0530 Subject: Fw: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: <017901c19ac8$7e0ea430$6b00a8c0@ericlaptop> References: <017901c19ac8$7e0ea430$6b00a8c0@ericlaptop> Message-ID: <15423.46132.813185.515758@monster.linux.in> Hi, >>>>> "eric" == eric writes: eric> Here is a near verbatim conversion of your laplace to use eric> Python/Inline. I didn't spend much time, and there are eric> errors. The results don't match. You'll get the general eric> idea from it though. This is wonderful! Thanks *very* much. I spent the last hour and a half looking over this code over a text terminal (since my desktop's keyboard is a pain to use). I fixed the errors. I didnt realize that it would be so easy to inline the innermost loop. The Numeric/copy/transpose example got me confused a bit. Later on today I'll try to create a simple Python version of your and my code and maybe write up a small document on the different ways to solve this problem using Python (i.e. slow, numeric, inline and blitz). I'll remove all the fancy stuff and clean up the Python code and keep the sample c++ example the way it is. I think it should make for a decent benchmark and introductory document to anyone interested in Python and speed. A comparison with c++ would really prove the point as to why developing with Python is a great idea. :) eric> I used blitz arrays because they're indexing is kinda like eric> double** indexing so the conversion was easy. Standard eric> Numeric arrays don't have the pointer arrays to the rows eric> that C uses, so your code wouldn't translate easily to it eric> without macros or calculating offsets -- you could do this eric> though. I'm not sure that indexing blitz arrays is as fast eric> as indexing normal arrays -- this example would suggest not: Yes, at this point in time I'm happy with the simplest solution. The inline code is pretty much next to trivial -- just a translation to straight C++. Thats really cool! Great job, Eric!! eric> 500x500, 100 iterations, PII 850 laptop, W2K, gcc-2.95.2 eric> laplace.py: 3.47 seconds laplace.cxx: 2.34 seconds Wow! Thats not bad at all! Considering that the example is completely scriptable and trivially extendible with a zillion other Python tools (the standard lib, scipy.plt, gracePlot, mayavi whatnot). eric> The Python calling overhead should be close to nil, so there eric> is no reason why the python version should be slower, other eric> than the fact that I used blitz arrays and indexing. But eric> the current implementation still pays a 50% penalty for eric> using Python. Hmm, maybe it has something to do with the way we are looping. I'll experiment with all this later and mail the list. However, a 50% penalty isnt so bad at all, I guess most folks can live with it. I have more generic questions which I'll leave for later. For now I have enough to do and think about. Eric, my hearty congratulations to you on weave! Great job!! prabhu From pearu at cens.ioc.ee Sat Jan 12 04:34:31 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sat, 12 Jan 2002 11:34:31 +0200 (EET) Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: <017901c19ac8$7e0ea430$6b00a8c0@ericlaptop> Message-ID: Hi Prabhu and Eric, Just from a curiousity I tried your test also using Fortran and here are the results (now comparing only cxx and Fortran): 500x500, 100 iterations, PII 400 laptop, Linux, gcc-2.95.4 Iterations took 6.12530398369 seconds. Fortran iterations took 3.15447306633 seconds. So, using Fortran you get speed up approx. _two_ times compared to cxx!!! If one would use optimized blas and laplack functions, the speed up could be even greater. Below is the testing code that I used. Here is how I used it, step by step, to illustrate that using Fortran from python is very simple: 1) Build fortran wrapper: f2py -c flaplace.f -m flaplace # here you need f2py from CVS As a result you'll get flaplace.so in the current directory. 2) Run tests: $ python laplace.py Iterations took 6.51088702679 seconds. Error: 38047.7567709 Fortran iterations took 3.22211503983 seconds. Error: 21778.5117188 Regards, Pearu -------------------- c File flaplace.f subroutine timestep(u,n,m,dx,dy,error) double precision u(n,m) double precision dx,dy,dx2,dy2,dnr_inv,tmp,err,diff integer n,m,i,j cf2py intent(in) :: dx,dy cf2py intent(in,out) :: u cf2py intent(out) :: error cf2py intent(hide) :: n,m dx2 = dx*dx dy2 = dy*dy dnr_inv = 0.5d0 / (dx2+dy2) error = 0 do 200,i=2,n-1 do 100,j=2,m-1 tmp = u(i,j) u(i,j) = ((u(i-1,j) + u(i+1,j))*dy2+ & (u(i,j-1) + u(i,j+1))*dx2)*dnr_inv diff = u(i,j) - tmp error = error + diff*diff 100 continue 200 continue error = sqrt(error) end ------------------- # modifications to laplace.py import flaplace # New method to LaplaceSolver class def ftimeStep(self,dt = 0.0): u,err = flaplace.timestep(self.g.u,self.g.dx,self.g.dy) return err # Slightly modified solve method: def solve(self, n_iter=0, eps=1e-16,fortran = 0): if fortran: timeStep = self.ftimeStep else: timeStep = self.timeStep err = timeStep() #print err count = 1 while err > eps: if n_iter and (count > n_iter): return err; err = timeStep() #print err count += 1 return count # Slightly modified runner: t1 = time.time() err = s.solve(100,fortran) t2 = time.time() print "Iterations took ", t2 - t1, " seconds." print "Error: ", err t1 = time.time() err = s.solve(100,fortran=1) t2 = time.time() print "Fortran iterations took ", t2 - t1, " seconds." print "Error: ", err --------------- EOF message From rob at pythonemproject.com Sat Jan 12 10:39:37 2002 From: rob at pythonemproject.com (Rob) Date: Sat, 12 Jan 2002 07:39:37 -0800 Subject: [SciPy-dev] try 3 at weave Message-ID: <3C4058B9.2A019B99@pythonemproject.com> I found that the weave tarball doesn't contain many of the files needed for weave to work. So I started downloading them off of CVS, but there were so many, I think I will just get the whole CVS weave distribution. I'll have to hone up on my CVS checkout skills. Rob. -- The Numeric Python EM Project www.pythonemproject.com From prabhu at aero.iitm.ernet.in Sat Jan 12 12:38:30 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sat, 12 Jan 2002 23:08:30 +0530 Subject: [SciPy-dev] try 3 at weave In-Reply-To: <3C4058B9.2A019B99@pythonemproject.com> References: <3C4058B9.2A019B99@pythonemproject.com> Message-ID: <15424.29846.549657.143076@monster.linux.in> >>>>> "Rob" == rob writes: Rob> I found that the weave tarball doesn't contain many of the Rob> files needed for weave to work. So I started downloading Rob> them off of CVS, but there were so many, I think I will just Rob> get the whole CVS weave distribution. I'll have to hone up Rob> on my CVS checkout skills. Rob. Yes, checking out from CVS is possibly the best possible approach and is also is pretty easy to do. The first time round the command is a little complex but later on its easy to do it. There are very useful instructions on the scipy page. http://www.scipy.org/site_content/tutorials/CVSInstruct prabhu From prabhu at aero.iitm.ernet.in Sat Jan 12 13:31:27 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sun, 13 Jan 2002 00:01:27 +0530 Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: References: <017901c19ac8$7e0ea430$6b00a8c0@ericlaptop> Message-ID: <15424.33023.15187.479609@monster.linux.in> Hi, >>>>> "PP" == Pearu Peterson writes: PP> Just from a curiousity I tried your test also using Fortran PP> and here are the results (now comparing only cxx and Fortran): [snip] PP> So, using Fortran you get speed up approx. _two_ times PP> compared to cxx!!! If one would use optimized blas and PP> laplack functions, the speed up could be even greater. PP> Below is the testing code that I used. Here is how I used it, PP> step by step, to illustrate that using Fortran from python is PP> very simple: Sorry for the delayed response. I was out for most of the day. Your example is pretty awesome! Thanks! f2py is really very cool! I'll include this example also in my test script. One point to note is that my cxx example was pretty straight forward. I'm pretty sure that there might be ways to improve the speed. I'll try and see if I can get that done. I think it would be very nice if there were a way to write inline fortran code and use f2py to generate the code on the fly and then also use weave's catalog etc. to handle this transparently. I guess that this should not be hard to do since f2py seems to do most of the dirty work. If Eric could do it for C++ I guess Fortran should be very easy for him. :) I know too little of f2py or weave to actually write this but FWIW here is some pseudo code that is a start. I looked at inline briefly and it looks like the following does something like what it does. def magic_generate_cf2py(in_=[], out=[], in_out=[], hide=[]): d = {'in': in_, 'out':out, 'in,out': in_out, 'hide':hide} cf2py = "" for key, val in d.items(): if val: cf2py += 'cf2py intent(%s) :: '%key for i in val: cf2py += i + ', ' cf2py += '\n' return cf2py def inlinef(code, arg_names=[], local_dict=None, global_dict=None in_=[], out=[], in_out=[], hide=[]): # check if already catalogued. If so simply return catalogued # func. function = magic_check_catalog(code) if function: return function # figure out the type of the args and add the declarations at the # top. declare = magic_declare_headers() # get local dict and global dict local_dict, global_dict = magic_code() # get arg list. args = "" for name in arg_names: args += name + ", " # get cf2py stuff cf2py = magic_generate_cf2py(in_, out, in_out, hide) f_code = """c Inline Fortran code. subroutine func_call(%(args)s) %(declare)s %(cf2py)s %(code)s"""%locals() # write fortran code to file and run f2py on it. file = open(file_name, "w") print >> file, header os.system("f2py -c %s -m %s"%(file_name, mod_name)) # load the module and get the function. function = magic_load_func() # catalog the function for future use. magic_catalog(code, function) return function prabhu From eric at scipy.org Sat Jan 12 14:00:00 2002 From: eric at scipy.org (eric) Date: Sat, 12 Jan 2002 14:00:00 -0500 Subject: [SciPy-dev] try 3 at weave References: <3C4058B9.2A019B99@pythonemproject.com> Message-ID: <028e01c19b9b$599cf7e0$6b00a8c0@ericlaptop> > I found that the weave tarball doesn't contain many of the files needed > for weave to work. Which files were you missing? I just downloaded the websites .tar.gz file and didn't find any problems -- all tests passed on my machine. Can you send me the errors you are getting from it? There may be some sort of search path issue inside weave that is OS dependent. eric From eric at scipy.org Sat Jan 12 15:20:06 2002 From: eric at scipy.org (eric) Date: Sat, 12 Jan 2002 15:20:06 -0500 Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. References: Message-ID: <02bc01c19ba6$86c7c500$6b00a8c0@ericlaptop> > Iterations took 6.12530398369 seconds. > Fortran iterations took 3.15447306633 seconds. > > So, using Fortran you get speed up approx. _two_ times compared to cxx!!! Hey! That is larger than I would have expected. Fortran often has a slight advantage ( a few percent), but not a factor of 2. Several things could be in play here. (1) Prabhu and I have no clue how to set compiler switches in gcc for best results. (very possible) (2) Your using a better optimizing Fortran compiler (wonder what the Intel or KAI compiler would do on the C++ code). Which one are you using? If it is g77, then this is exhibit A for supporting (1). (3) If comparisons are made against Prabhu's C++ code instead of my extension module which (mis) uses blitz++ arrays for indexing, would only show a 33% improvement. Correct? This is starting to be more what I'd expect. Anyway, I'd be surprised if the difference is this great in the final analysis -- I'll bet less than 10% difference either way. > If one would use optimized blas and laplack functions, the speed up could > be even greater. Not sure about the difference here. Which blas functions would you use for this? Also, you can use optimized blas/lapack from C just as easily. Keep in mind that ATLAS is pretty much as fast as it gets for BLAS/LAPACK and it is all written in C with special assembly for some CPUs (Intel/AMD/PowerPC). Still, in the end, I would like to have a version of inline that works for Fortran. It's not a large project, but also not one I have time for now. If anyone is interested, drop me a line, and I'll point out how to do it. On the usability side of inline Fortran, the array transpose issue is confusing to people (including me), and, for speed, you have to leave the details of this up to the user. We're just stuck with this. eric > > Below is the testing code that I used. Here is how I used it, step by > step, to illustrate that using Fortran from python is very simple: > > 1) Build fortran wrapper: > f2py -c flaplace.f -m flaplace # here you need f2py from CVS > > As a result you'll get flaplace.so in the current directory. > > 2) Run tests: > $ python laplace.py > Iterations took 6.51088702679 seconds. > Error: 38047.7567709 > Fortran iterations took 3.22211503983 seconds. > Error: 21778.5117188 > > Regards, > Pearu > > -------------------- > c File flaplace.f > subroutine timestep(u,n,m,dx,dy,error) > double precision u(n,m) > double precision dx,dy,dx2,dy2,dnr_inv,tmp,err,diff > integer n,m,i,j > cf2py intent(in) :: dx,dy > cf2py intent(in,out) :: u > cf2py intent(out) :: error > cf2py intent(hide) :: n,m > dx2 = dx*dx > dy2 = dy*dy > dnr_inv = 0.5d0 / (dx2+dy2) > error = 0 > do 200,i=2,n-1 > do 100,j=2,m-1 > tmp = u(i,j) > u(i,j) = ((u(i-1,j) + u(i+1,j))*dy2+ > & (u(i,j-1) + u(i,j+1))*dx2)*dnr_inv > diff = u(i,j) - tmp > error = error + diff*diff > 100 continue > 200 continue > error = sqrt(error) > end > > ------------------- > # modifications to laplace.py > > import flaplace > > # New method to LaplaceSolver class > def ftimeStep(self,dt = 0.0): > u,err = flaplace.timestep(self.g.u,self.g.dx,self.g.dy) > return err > > # Slightly modified solve method: > def solve(self, n_iter=0, eps=1e-16,fortran = 0): > if fortran: > timeStep = self.ftimeStep > else: > timeStep = self.timeStep > err = timeStep() > #print err > count = 1 > while err > eps: > if n_iter and (count > n_iter): > return err; > err = timeStep() > #print err > count += 1 > return count > > # Slightly modified runner: > t1 = time.time() > err = s.solve(100,fortran) > t2 = time.time() > print "Iterations took ", t2 - t1, " seconds." > print "Error: ", err > t1 = time.time() > err = s.solve(100,fortran=1) > t2 = time.time() > print "Fortran iterations took ", t2 - t1, " seconds." > print "Error: ", err > > --------------- > EOF message > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev From eric at scipy.org Sat Jan 12 16:03:59 2002 From: eric at scipy.org (eric) Date: Sat, 12 Jan 2002 16:03:59 -0500 Subject: [SciPy-dev] New tarballs for weave Message-ID: <02fc01c19bac$a8407000$6b00a8c0@ericlaptop> Hello group, New weave tarballs available on: http://www.scipy.org/site_content/weave These tarballs include some attempts at fixes for Prabhu on Debian. They have been tested on Solaris (2.95.1), W2K (gcc 2.95.2), and RH 7.1 (gcc 2.96) and all tests pass. I still haven't found any reason for the "Abort" error reported on Mandrake (bummer). I'd been interested in reports of success or failure of weave.test() on any and all architectures. Instructions for testing are on the page. There is one minor change likely tomorrow and then I'll announce on comp.python.lang on Monday to get a wider testing audience. Thanks to those who have already spent time testing. eric From pearu at cens.ioc.ee Sat Jan 12 18:30:32 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sun, 13 Jan 2002 01:30:32 +0200 (EET) Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: <02bc01c19ba6$86c7c500$6b00a8c0@ericlaptop> Message-ID: On Sat, 12 Jan 2002, eric wrote: > > Iterations took 6.12530398369 seconds. > > Fortran iterations took 3.15447306633 seconds. > > Hey! That is larger than I would have expected. Fortran often has a slight > advantage ( a few percent), but not a factor of 2. Several things could be > in play here. > > (1) Prabhu and I have no clue how to set compiler switches in gcc for best > results. (very possible) > (2) Your using a better optimizing Fortran compiler (wonder what the Intel > or KAI compiler would do on the C++ code). Which one are you using? If it > is g77, then this is exhibit A for supporting (1). > (3) If comparisons are made against Prabhu's C++ code instead of my > extension module which (mis) uses blitz++ arrays for indexing, would only > show a 33% improvement. Correct? This is starting to be more what I'd > expect. I used the same code (laplace.py) that was in your message to this list. > > Anyway, I'd be surprised if the difference is this great in the final > analysis -- I'll bet less than 10% difference either way. I used g77 in these tests. But indeed, also with heavy optimizations switches turned on. Here are other results with different optimization flags enabled: 1) g77 -02 (standard) Iterations took 6.34938693047 seconds. Error: 38047.7567709 Fortran iterations took 3.69506299496 seconds. Error: 21778.5117188 2) g77 (no optimization) Iterations took 6.44155204296 seconds. Error: 38047.7567709 Fortran iterations took 3.82356095314 seconds. Error: 21778.5117188 3) g77 -O3 -funroll-loops -march=i686 -malign-double (machine depend.) Iterations took 6.34858202934 seconds. Error: 38047.7567709 Fortran iterations took 3.29500699043 seconds. Error: 21778.5117188 4) ifc -O3 -tpp6 -xi (this is Intel Fortran 90 compiler for Linux) Iterations took 6.38630092144 seconds. Error: 38047.7567709 Fortran iterations took 2.84432804585 seconds. Error: 21779.0996094 Note that even without any optimization and with g77, the factor is still quite high: 1.6. And with Intel compiler this is 2.2. So, I would not be so pessimistic about Fortran capabilities. I guess that these numbers also depend on the particular test, and the current one just fits for Fortran. Anyway, I don't think that people would use Fortran if the gain in speed would be only few percent. It must be higher than that. Regards, Pearu From eric at scipy.org Sat Jan 12 21:55:44 2002 From: eric at scipy.org (eric) Date: Sat, 12 Jan 2002 21:55:44 -0500 Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. References: Message-ID: <03ea01c19bdd$cb6d2510$6b00a8c0@ericlaptop> Hey Pearu, > > > Iterations took 6.12530398369 seconds. > > > Fortran iterations took 3.15447306633 seconds. I've rewritten the code in C++ as quickly as I know how and added a few optimizations. It has the same errors as the previous example, but, again, is useful for benchmarking. (maybe Prabhu will fix my bugs in this code also. :) The results are as follows: Blitz indexing: Iterations took 3.54500007629 seconds. After re-write: iterations took 1.77300000191 seconds. With the re-write, C++ is in the same performance range as the fortran code. My timestep loop is included below. It uses pointer math instead of blitz arrays to reduce the effort of array element lookups down to incrementing pointers in the inner loop. It isn't exactly pretty, but we were looking for speed, not beauty. I expect there are some ways to clean it up and still get about the same speed. I used the following optimizations: extra_compile_args = ['-O3','-malign-double','-funroll-loops'] The difference between using and not using -funroll-loops was significant (maybe 25%). > Anyway, I don't think that people would use Fortran if > the gain in speed would be only few percent. It must be higher than that. I think it used to be, but the recent years have been kinder to C than Fortran. Among other things, market forces have pushed C/C++ compilers to evolve more quickly than Fortran compilers simply because more people are interested in fast C/C++ compilers. The result is that C/C++ can be made to execute algorithms at close to the same speed as Fortran in most cases -- at least that has been my experience. As for gcc and g77, I think only the front ends of the compilers are different. The same back-end is shared by both. Still, there are lots of good reasons besides speed for scientist to use Fortran: 1. It is the language you know. (probably the main reason many still use it) 2. There are tons of scientific libraries available. 3. All your labs legacy code is in Fortran. 4. You don't have to mess with memory management. (debugging dangling pointers stinks) 5. You don't have to resort to tricks for speed. 6. You just like the language better. Of course, there are reasons not to use it also ( number 4 above can go on both the "for" and "against" lists), but all the above reasons are valid. Scientist are interested in the fastest way to results -- not in writing elegant, full featured, re-usable, scriptable programs. For one or more of the above reasons, some still consider Fortran their best choice to optimize this interest. Other scientist choose Matlab or IDL or even C. Hopefully we can increase the share that see Python+Numeric+SciPy+f2py+weave as a good choice also. This thread definitely needs to get summarized and put into a web page. see ya, eric ---------------------------------------------------------------------------- ------------------------------ # just replace the timeStep algorithm from the previous post with this one. def timeStep(self, dt = 0.0): dx2 = self.g.dx**2 dy2 = self.g.dy**2 dnr_inv = .5/(dx2+dy2) nx, ny = self.g.shape u = self.g.u code = """ #line 44 "laplace.py" double tmp, err, diff; double *uc, *uu, *ud, *ul, *ur; for (int i=1; i References: <017901c19ac8$7e0ea430$6b00a8c0@ericlaptop> Message-ID: <15425.1929.990936.228227@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> 1) Build fortran wrapper: f2py -c flaplace.f -m flaplace # PP> here you need f2py from CVS I've been having a hard time getting my version of f2py (2.5.391) to do this but cant seem to get it right. Does f2py 2.5.393 have this feature or is the CVS version absolutely necessary? PP> As a result you'll get flaplace.so in the current directory. Also what about the transposing of the arrays?? Eric was talking about transposing the array. I quickly read some of your examples and internally you seem to do the necessary adjustments (swapping n and m) to get the right array, right? If so, technically the result of your fortran code should be identical to the c++ version? Please remember that the order of execution is important, i.e. the array must be accessed in the same sequence if not the intermediate step results will be different. However, both versions will converge but I'm just curious about the intermediate steps too. My apologies if these questions sound dumb but I come from a C background with very little fortran experience. I'll learn fortran soon though, not because its faster but because I believe that its worth knowing. prabhu From prabhu at aero.iitm.ernet.in Sat Jan 12 23:31:52 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sun, 13 Jan 2002 10:01:52 +0530 Subject: [SciPy-dev] New tarballs for weave In-Reply-To: <02fc01c19bac$a8407000$6b00a8c0@ericlaptop> References: <02fc01c19bac$a8407000$6b00a8c0@ericlaptop> Message-ID: <15425.3512.238919.487083@monster.linux.in> >>>>> "eric" == eric writes: eric> I'd been interested in reports of success or failure of eric> weave.test() on any and all architectures. Instructions for eric> testing are on the page. Here are more tests with the latest cvs tree. $ cd cvs/scipy/weave/tests; python [snip] >>> import test_catalog >>> test_catalog.test() ..........repairing catalog by removing key repairing catalog by removing key repairing catalog by removing key repairing catalog by removing key repairing catalog by removing key repairing catalog by removing key FE.............. ====================================================================== ERROR: Test persisting a function in the default catalog ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_catalog.py", line 275, in check_add_function_persistent1 q.add_function_persistent('code',i) File "weave/catalog.py", line 579, in add_function_persistent NameError: global name 'pickle' is not defined ====================================================================== FAIL: check_add_function_ordered (test_catalog.test_catalog) ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_catalog.py", line 330, in check_add_function_ordered assert(funcs1[:2] == [string.lower,string.upper]) AssertionError ---------------------------------------------------------------------- Ran 26 tests in 1.254s FAILED (failures=1, errors=1) The first error I think can be fixed with an import pickle added to catalog.py. I did that (I just committed it) and then I get the same old error. ====================================================================== ERROR: Test persisting a function in the default catalog ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_catalog.py", line 275, in check_add_function_persistent1 q.add_function_persistent('code',i) File "/skratch/prabhu/scipy/cvs/scipy/weave/catalog.py", line 578, in add_function_persistent function_list = function_list + cat.get(code,[]) File "/usr/local/lib/python2.1/shelve.py", line 66, in get return self[key] File "/usr/local/lib/python2.1/shelve.py", line 71, in __getitem__ return Unpickler(f).load() UnpicklingError: pickle data was truncated ====================================================================== FAIL: check_add_function_ordered (test_catalog.test_catalog) ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_catalog.py", line 330, in check_add_function_ordered assert(funcs1[:2] == [string.lower,string.upper]) AssertionError ---------------------------------------------------------------------- Ran 26 tests in 1.216s FAILED (failures=1, errors=1) Why is this happening? Shouldnt the try block catch it? I'll try to take a look at this. prabhu From pearu at cens.ioc.ee Sun Jan 13 03:02:39 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sun, 13 Jan 2002 10:02:39 +0200 (EET) Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: <15425.1929.990936.228227@monster.linux.in> Message-ID: On Sun, 13 Jan 2002, Prabhu Ramachandran wrote: > >>>>> "PP" == Pearu Peterson writes: > > PP> 1) Build fortran wrapper: f2py -c flaplace.f -m flaplace # > PP> here you need f2py from CVS > > I've been having a hard time getting my version of f2py (2.5.391) to > do this but cant seem to get it right. Does f2py 2.5.393 have this > feature or is the CVS version absolutely necessary? Yes, currently it is absolutely necessary. The corresponding code in 2.5.x is broken due to the recent fast changes in scipy_distutils. In fact, you'll need also get scipy_distutils from its CVS then. After I will check some things, I'll make a new snapshot available soon. > PP> As a result you'll get flaplace.so in the current directory. > > Also what about the transposing of the arrays?? Eric was talking > about transposing the array. I quickly read some of your examples and > internally you seem to do the necessary adjustments (swapping n and m) > to get the right array, right? If so, technically the result of your > fortran code should be identical to the c++ version? Please remember > that the order of execution is important, i.e. the array must be > accessed in the same sequence if not the intermediate step results > will be different. However, both versions will converge but I'm just > curious about the intermediate steps too. I think that the intermediate steps are different (this shows also in different error results). I just ignored transposing stuff as the test problem is symmetric, right?. It's a mess and a headache to deal with different array orderings, even if you are fully aware about the issue. Somekind of rules need to be worked out, double-checked, and documented in order to ease the pain ;) And I didn't care about bugs also as we are studying the speed differences and here only the number of operations are relevant - they should be same in all test cases for different approaches so that the results will be comparable. So, I think we should count operations, not bugs, though it would be nice to get those fixed as well. Pearu From prabhu at aero.iitm.ernet.in Sun Jan 13 03:04:03 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sun, 13 Jan 2002 13:34:03 +0530 Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: <03ea01c19bdd$cb6d2510$6b00a8c0@ericlaptop> References: <03ea01c19bdd$cb6d2510$6b00a8c0@ericlaptop> Message-ID: <15425.16243.404926.567668@monster.linux.in> >>>>> "eric" == eric writes: eric> I've rewritten the code in C++ as quickly as I know how and eric> added a few optimizations. It has the same errors as the eric> previous example, but, again, is useful for eric> benchmarking. (maybe Prabhu will fix my bugs in this code eric> also. :) The results are as follows: Yes, I'll try to fix this one also. I'll stop with this though. I hope no one decides to send in an inline assembler comparison. ;) eric> Blitz indexing: Iterations took 3.54500007629 seconds. eric> After re-write: iterations took 1.77300000191 seconds. eric> I think it used to be, but the recent years have been kinder eric> to C than Fortran. Among other things, market forces have eric> pushed C/C++ compilers to evolve more quickly than Fortran eric> compilers simply because more people are interested in fast eric> C/C++ compilers. The result is that C/C++ can be made to eric> execute algorithms at close to the same speed as Fortran in eric> most cases -- at least that has been my experience. As for eric> gcc and g77, I think only the front ends of the compilers eric> are different. The same back-end is shared by both. I also think that it is a known fact that if one uses something like blitz++ can acheive speed as good if not better than Fortran. eric> the above reasons are valid. Scientist are interested in eric> the fastest way to results -- not in writing elegant, full eric> featured, re-usable, scriptable programs. For one or more eric> of the above reasons, some still consider Fortran their best eric> choice to optimize this interest. Other scientist choose eric> Matlab or IDL or even C. Hopefully we can increase the eric> share that see Python+Numeric+SciPy+f2py+weave as a good eric> choice also. Yes. Unfortunately, dylan is not as complete or mature as Python is. To me it seems the best possible approach where you can rapidly prototype and later speed up your code all in the same language. eric> This thread definitely needs to get summarized and put into eric> a web page. Well, I'll work on a draft sometime next week and send it to you folks. Meanwhile I'm stuck pretty bad on f2py. Yeah, this discussion might not belong here but I need to get Pearu's example working if I am to include it. More on that later. prabhu From prabhu at aero.iitm.ernet.in Sun Jan 13 03:27:21 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sun, 13 Jan 2002 13:57:21 +0530 Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: References: <15425.1929.990936.228227@monster.linux.in> Message-ID: <15425.17641.157009.76331@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> Yes, currently it is absolutely necessary. The corresponding PP> code in 2.5.x is broken due to the recent fast changes in PP> scipy_distutils. In fact, you'll need also get PP> scipy_distutils from its CVS then. After I will check some PP> things, I'll make a new snapshot available soon. I just checked out f2py2e from cvs. I had goofed earlier. I have a local install of Python and also a global one (both the same version). I accidentally installed f2py as root somewhere else. It works beautifully now. There was just one hitch. I had to link cvs/f2py2e/scipy_distutils to cvs/scipy/scipy_distutils Once I did that f2py2e installed just fine now it also builds flaplace.so very nicely. [Prabhu on transposing arrays ...] PP> I think that the intermediate steps are different (this shows PP> also in different error results). I just ignored transposing PP> stuff as the test problem is symmetric, right?. It's a mess PP> and a headache to deal with different array orderings, even if PP> you are fully aware about the issue. Somekind of rules need PP> to be worked out, double-checked, and documented in order to PP> ease the pain ;) And I didn't care about bugs also as we are Yes, that is what I was asking about. I think we'll do this instead. I'll work on a quick draft document -- "The newbie guide to scipy.weave" or something. I'll try to cover what I've done with the test problem and try to illustrate numeric, weave, inline and f2py. I'll let the experts correct it. So maybe you can write a section on array transposing and stuff. PP> studying the speed differences and here only the number of PP> operations are relevant - they should be same in all test PP> cases for different approaches so that the results will be PP> comparable. So, I think we should count operations, not bugs, PP> though it would be nice to get those fixed as well. Of course, I wasnt saying that it was a bug. Actually if you look at my code the numeric version of timeStep will not re-use computed variables. In fact there is no way to do this in pure numeric. Only inline, weave, f2py and pure Python let you do this. i.e. once u(i,j) is computed the next (i or j) computation will re-use this latest value. Numeric uses temporaries so you cant do it. So, I know that the numeric results *will* be different from that of the rest. However, I only wanted clarifications on f2py and what one must worry about when one uses f2py. Anyway, my question is u(i,j) as referred to in either Python (u[i,j]) or in blitz or f2py -- do they mean the same thing? Or is it that when I write a loop in fortran that uses a numeric array I must make sure that u(i,j) == u[j,i] ?? I ask this because there is hope that someday inline fortran is possible and it looks to me that inline fortran is just as easy (if not easier) to write as inline c++. Also, if we are sure of getting more speed with easier to write/understand code, its worth while writing the loops in fortran. I'd really like to know what one should watch out for when one writes fortran code that is used from Python. Thanks, prabhu From prabhu at aero.iitm.ernet.in Sun Jan 13 03:37:11 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sun, 13 Jan 2002 14:07:11 +0530 Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: <02bc01c19ba6$86c7c500$6b00a8c0@ericlaptop> References: <02bc01c19ba6$86c7c500$6b00a8c0@ericlaptop> Message-ID: <15425.18231.672645.722422@monster.linux.in> >>>>> "eric" == eric writes: eric> Still, in the end, I would like to have a version of inline eric> that works for Fortran. It's not a large project, but also eric> not one I have time for now. If anyone is interested, drop eric> me a line, and I'll point out how to do it. On the I'm not volunteering, but it is sure is tempting. Why dont you post the "howto" here? If someone has enough time they could do it. I know that it most probably is painful for you to write it up at this time but I guess writing about it now when you are fresh from coding is the best time to write the doc. :) Also, I am not familiar with f2py or fortran so I'll most probably stumble quite often (if not all the time). I think we need to get together on this. Pearu is obviously the f2py expert and you the weave expert (code weaver?). I'm the only(?) tester so far. If it won't take long I dont mind blowing a day on it helping (with coding/testing whatever) but no more. eric> usability side of inline Fortran, the array transpose issue eric> is confusing to people (including me), and, for speed, you eric> have to leave the details of this up to the user. We're eric> just stuck with this. Yes, maybe a nice document with a few examples on what happens and what to watch out for would be good. Let me first get that draft on the thread done. prabhu From pearu at cens.ioc.ee Sun Jan 13 04:07:07 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sun, 13 Jan 2002 11:07:07 +0200 (EET) Subject: [SciPy-dev] New F2PY snapshot available In-Reply-To: Message-ID: Hi! I made a new snapshot of F2PY available. It is beta version of the 5th F2PY release. It includes many updates, bug fixes, the latest scipy_distutils etc. See NEWS.txt for details. The most important new feature is *** New -c switch that will run the Fortran wrapping process from Alpha to Omega. For example, if foo.f contains subroutine bar, then running f2py -c foo.f -m car will create car.so in the current directory that can be readily imported to Python: >>> import car >>> print car.bar.__doc__ The snapshot is available for download in http://cens.ioc.ee/projects/f2py2e/2.x/ Regards, Pearu From prabhu at aero.iitm.ernet.in Sun Jan 13 04:45:32 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sun, 13 Jan 2002 15:15:32 +0530 Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: <03ea01c19bdd$cb6d2510$6b00a8c0@ericlaptop> References: <03ea01c19bdd$cb6d2510$6b00a8c0@ericlaptop> Message-ID: <15425.22332.464372.457459@monster.linux.in> hi, >>>>> "eric" == eric writes: eric> Hey Pearu, >> > > Iterations took 6.12530398369 seconds. > > Fortran >> iterations took 3.15447306633 seconds. eric> I've rewritten the code in C++ as quickly as I know how and eric> added a few optimizations. It has the same errors as the eric> previous example, but, again, is useful for eric> benchmarking. (maybe Prabhu will fix my bugs in this code eric> also. :) The results are as follows: I've added all the stuff to my version of the code -- I still have to remove the mayavi dependency and maybe clean it up a bit but here are some prelim results. I have the following solution procedures. (a) slow Python - normal for loops and stuff. (b) numeric -- just there for testing the speed the results will be different since it uses temporaries. (c) blitz -- this uses the silly way to compute the error. (d) inline -- thanks to Eric's code slightly modfied. (e) fastinline -- Eric's new version (the only bug was that err should be set to zero before the loop). (f) fortran -- This is Pearu's version pretty much verbatim. (1) I made pretty sure that the output of weave/inline/fastinline/fortran are all very much alike. I found that there is some loss of precision in the fortan code but the results are pretty much identical. The errors were different by very small numbers. For a 500 by 500 grid I computed a few iterations using weave/inline/fastinline and compared them with the fortan results and they are the same to within 1 part in 1e-15 or something. (2) Here are the results of speed tests. I create a 500x500 grid and run it for 100 iterations. I do two runs and average the results. Just for the heck of it I ran the slow time step for 5 iterations multiplied the result by 20 to get total time for 100 iterations. slow: 1847.60 s == (time for 5 iterations)*20 numeric: 29.5 s blitz: 9.75 s inline: 4.60 s fortran: 3.24 s fastinline: 2.92 s Fastinline is about 630 times faster than the normal slow for loop! Wow! unbelievable!! Obviously, more speedup can be achieved but its clear that blitz, inline, fortran and fastinline are big winners and that they are all reasonably close. inline and fortran are about 40% off which is not bad at all. fastinline is about 10% faster than fortran but the code is dirty. prabhu From fperez at pizero.colorado.edu Sun Jan 13 18:13:25 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Sun, 13 Jan 2002 16:13:25 -0700 (MST) Subject: [SciPy-dev] Weave Mandrake report. In-Reply-To: <03ea01c19bdd$cb6d2510$6b00a8c0@ericlaptop> Message-ID: Hi Eric, here's a report from Mandrake again (a Mandrake 8.0 laptop with 512 Mb of ram, and stock packages for the most part -- python updated to 2.1.1 from Mandrake 8.1). Some system info: [~/scipy]> uname -a Linux maqroll 2.4.3-20mdk #1 Sun Apr 15 23:03:10 CEST 2001 i686 unknown [~/scipy]> gcc -v Reading specs from /usr/lib/gcc-lib/i586-mandrake-linux/2.96/specs gcc version 2.96 20000731 (Linux-Mandrake 8.0 2.96-0.48mdk) [~/scipy]> python Python 2.1.1 (#1, Aug 30 2001, 17:36:05) [GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.61mdk)] on linux-i386 Type "copyright", "credits" or "license" for more information. First, I tried using the updated tarballs, and failed with: dep: ['scipy_distutils', 'scipy_test'] Traceback (most recent call last): File "setup.py", line 44, in ? stand_alone_package(with_dependencies) File "setup.py", line 23, in stand_alone_package config_dict = package_config(primary,dependencies) File "scipy_distutils/misc_util.py", line 174, in package_config config.extend([get_package_config(x) for x in primary]) File "scipy_distutils/misc_util.py", line 161, in get_package_config mod = __import__('setup_'+package_name) ImportError: No module named setup_weave Ron reported the same problem before with no answer, so I went for the cvs files instead. It might be worth fixing this though, as I suspect a lot of people are reluctant to use cvs. Now, using the cvs setup, I may well be doing something wrong. I just grabbed the whole scipy cvs tree, cd'ed into weave and tried import weave;weave.test() after removing my .python21_* stuff and all files in ~/tmp/ related to this. I made sure no other weave is available in sys.path to me. I don't get the 'Abort' I was getting before anymore, so that's good news :) But it still fails, though. Here's a summary: First, I get the following error message a zillion times: E/usr/local/home/fperez/.python21_compiled/44731/ext_string_and_int.cpp: In function `FILE *convert_to_file (PyObject *, char *)': /usr/local/home/fperez/.python21_compiled/44731/ext_string_and_int.cpp:53: `handle_conversion_error_type' undeclared (first use this function) /usr/local/home/fperez/.python21_compiled/44731/ext_string_and_int.cpp:53: (Each undeclared identifier is reported only once for each function it appears in.) This happens for many many files. I don't know how critical this particular problem is, and what incidence it has on the later stuff. Later, I get: Ewarning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations ....warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations .................................................................................................. ====================================================================== ERROR: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] ---------------------------------------------------------------------- Traceback (most recent call last): File "weave/tests/test_blitz_tools.py", line 153, in check_5point_avg_2d self.generic_2d(expr) File "weave/tests/test_blitz_tools.py", line 127, in generic_2d mod_location) File "weave/tests/test_blitz_tools.py", line 84, in generic_test blitz_tools.blitz(expr,arg_dict,{},verbose=0) File "weave/blitz_tools.py", line 99, in blitz type_factories = blitz_type_factories) File "weave/inline_tools.py", line 402, in compile_function verbose=verbose, **kw) File "weave/ext_tools.py", line 347, in compile verbose = verbose, **kw) File "weave/build_tools.py", line 176, in build_extension setup(name = module_name, ext_modules = [ext],verbose=verb) File "/usr/lib/python2.1/distutils/core.py", line 157, in setup raise SystemExit, "error: " + str(msg) CompileError: error: command 'gcc' failed with exit status 1 >From here on it's a long string of ERROR:... all of which end with a 'failed gcc' message. Things end with: ---------------------------------------------------------------------- Ran 184 tests in 267.801s FAILED (errors=28) I hope this is more useful than what I had before, and let me know if I can help more, or what I may be doing wrong. Since I now have the cvs tree I can easily update as needed and rerun the tests. I'm really looking forward to weave stabilizing, as I suspect I might have immediate need for it in my thesis. So I'll try to help as much as possible. Cheers, F. From pearu at cens.ioc.ee Sun Jan 13 19:42:35 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 14 Jan 2002 02:42:35 +0200 (EET) Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: <15425.17641.157009.76331@monster.linux.in> Message-ID: On Sun, 13 Jan 2002, Prabhu Ramachandran wrote: > Anyway, my question is u(i,j) as referred to in either Python (u[i,j]) > or in blitz or f2py -- do they mean the same thing? Or is it that > when I write a loop in fortran that uses a numeric array I must make > sure that u(i,j) == u[j,i] ?? I am currently reconsidering the Fortran/C array issues. It is looking promising what I have worked out: u[i,j] in Python and u(i,j) in Fortran will be the same. And one will not need to think about how matrices are stored in memory, as a first instance at least. However, if you are aware about the fact that Fortran and C arrays have different storage orders, then you can be clever by creating arrays with a proper storage order that will be good for performance. The approach will be similar to the one in Pyfort but I think it can be more efficient. Let me say that with this new approach there will be no performance hit compared to the current f2py approach. Just the appearance of Fortran and C multidimensional arrays in Python will be unified and that is a very important step as then matrices in mathematical sense will coincide. However, to code this approach efficiently I need a PyArray_TransposedContiguousFromObject function that does not exists in Numeric. If an Object is PyArray already, then I can easily write that part. However, if an Object is a List or any other Sequence then one needs to repeat lots of code in Numeric. Basically, Array_FromSequence needs to be rewritten to TransposedArray_FromSequence which requires quite good knowledge of Numeric internals which I lack. A good thing is that PyArrayObject needs not to be changed at all. If someone has already coded all that in above or at least parts of it, I would appreciate code examples very much. Pearu From eric at scipy.org Sun Jan 13 22:01:03 2002 From: eric at scipy.org (eric) Date: Sun, 13 Jan 2002 22:01:03 -0500 Subject: [SciPy-dev] Weave Mandrake report. References: Message-ID: <04fc01c19ca7$b66c8000$6b00a8c0@ericlaptop> Hey Fernando, You caught the CVS in a transitional state -- that is the error your getting with the FILE stuff. I started fixing an issue with exception handling that was bugging me this afternoon. It will be completed tonight. Also, instead of running from the scipy/weave directory of the CVS, try running from the scipy/ directory so that the weave package is in the current directory. Without installing weave explicitly, this is the best way to test it. Of course, wait till the above changes are finished before trying again (I announce this). > dep: ['scipy_distutils', 'scipy_test'] > Traceback (most recent call last): > File "setup.py", line 44, in ? > stand_alone_package(with_dependencies) > File "setup.py", line 23, in stand_alone_package > config_dict = package_config(primary,dependencies) > File "scipy_distutils/misc_util.py", line 174, in package_config > config.extend([get_package_config(x) for x in primary]) > File "scipy_distutils/misc_util.py", line 161, in get_package_config > mod = __import__('setup_'+package_name) > ImportError: No module named setup_weave Hmmm. I'll look into this. I haven't had the problem when installing from the tar-ball, but *can* think of a reason that it might happen. I'll do a little investigation -- especially since I'd like to open it up to wider inspection tomorrow. > > I hope this is more useful than what I had before, and let me know if I can > help more, or what I may be doing wrong. Since I now have the cvs tree I can > easily update as needed and rerun the tests. I'll bet a nickle your not getting the Abort because the test that was causing the Abort fails even earlier, and doesn't allow the Abort to happen. Call me a pessimist... :) > > I'm really looking forward to weave stabilizing, as I suspect I might have > immediate need for it in my thesis. So I'll try to help as much as possible. I'd like to work this out also. The cross platform issues are proving to be a headache. Also, last night we learned that the Python 2.1.1 version of dumbdbm (which Prabhu was using on Debian) is buggy. The new code hopefully solves these issues, but I'll bet it remains a trouble spot. Two things could help as far as Mandrake is concerned. (1) If I had a shell temporary shell account on a Mandrake machine, I could diagnose the issue the quickest. Barring that (2) we could set up a time to chat via Yahoo tomorrow or the next day so that you and I could have a co-debugging session. Let me know what works best for you. eric From prabhu at aero.iitm.ernet.in Mon Jan 14 00:58:19 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 14 Jan 2002 11:28:19 +0530 Subject: Fortran comparison Re: [SciPy-dev] scipy.weave versus simple C++. In-Reply-To: References: <15425.17641.157009.76331@monster.linux.in> Message-ID: <15426.29563.520859.229101@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> I am currently reconsidering the Fortran/C array issues. It is PP> looking promising what I have worked out: u[i,j] in Python and PP> u(i,j) in Fortran will be the same. And one will not need to PP> think about how matrices are stored in memory, as a first PP> instance at least. However, if you are aware about the fact PP> that Fortran and C arrays have different storage orders, then PP> you can be clever by creating arrays with a proper storage PP> order that will be good for performance. The approach will be PP> similar to the one in Pyfort but I think it can be more PP> efficient. PP> Let me say that with this new approach there will be no PP> performance hit compared to the current f2py approach. Just PP> the appearance of Fortran and C multidimensional arrays in PP> Python will be unified and that is a very important step as PP> then matrices in mathematical sense will coincide. Sounds great! Unfortunately I dont have much experience at all with the internals of Numeric or its C API. Good luck! prabhu From eric at scipy.org Mon Jan 14 02:01:12 2002 From: eric at scipy.org (eric) Date: Mon, 14 Jan 2002 02:01:12 -0500 Subject: [SciPy-dev] new weave tar balls -- multiple fixes Message-ID: <005001c19cc9$40c27590$777ba8c0@ericlaptop> Hey group, There are new weave tar balls available. http://www.scipy.org/site_content/weave The CVS version is also up to date. I think we have cleaned up *most* outstanding issues. Please try the tar-balls (preferred over CVS to find any packaging mistakes) as soon as possible. I'll likely announce to c.l.p by noon if no major problems are found. Several major fixes/changes are in the code: * setup_weave.py not found error shouldn't be happening on any platform now. Installation with setup.py should work as expected. * Most errors occuring on other platforms had to do with bugs in dumbdbm in the Python 2.1.x distribution. weave now ships with its own version of dumbdbm and a shelve file that uses it. Hopefully this will get rid of most cross-platform issues. * A change to how exceptions are handled in inline ( no one is likely to notice this one, but it does make a few places in the tutorial out of date. Other things to note: * Testing takes maybe 75 MB of disk space, so if you have a low quota, some tests will fail. * there is no cross platform way to cleanup after the tests, so most of the 75 MB of test files will remain on your harddrive. The main problem here is that windows does not allow you to delete a DLL that is currently loaded into memory. Versions of Unix may have the same problem. Platforms Reports: * W2K, MSVC, gcc-2.95.2 (mingw32) (passed 185 tests) * RH 7.1, gcc-2.96 (passed 185 tests) * Solaris gcc-2.95.1 (3 failures out of 185 tests -- should work for inline and blitz) * Debian (something like 5 failures out of 185 tests -- should work) ( Thanks to Prahbu) thanks, eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From prabhu at aero.iitm.ernet.in Mon Jan 14 04:15:18 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 14 Jan 2002 14:45:18 +0530 Subject: [SciPy-dev] new weave tar balls -- multiple fixes In-Reply-To: <005001c19cc9$40c27590$777ba8c0@ericlaptop> References: <005001c19cc9$40c27590$777ba8c0@ericlaptop> Message-ID: <15426.41382.733554.770617@monster.linux.in> >>>>> "eric" == eric writes: eric> Hey group, There are new weave tar balls available. eric> http://www.scipy.org/site_content/weave eric> The CVS version is also up to date. I think we have cleaned eric> up *most* outstanding issues. Please try the tar-balls eric> (preferred over CVS to find any packaging mistakes) as soon eric> as possible. I'll likely announce to c.l.p by noon if no eric> major problems are found. Sorry, I didnt get the tarball but am testing this off CVS. Here is what I get. cd cvs/scipy; python $ python Python 2.1.1 (#1, Nov 12 2001, 19:01:44) [GCC 2.95.4 20011006 (Debian prerelease)] on linux2 Type "copyright", "credits" or "license" for more information. >>> import weave >>> weave.test() [snip] Run: (1000, 1000) D 1st run(Numeric,compiled,speed up): 1.8170, 0.7485, 2.4274 2nd run(Numeric,compiled,speed up): 1.8238, 0.7478, 2.4388 .................................................a:2 .............................../home/prabhu/.python21_compiled/sc_9a25bc84add18fe6c75501f6b01bd84e1.cpp: In function `struct PyObject * compiled_func(PyObject *, PyObject *)': /home/prabhu/.python21_compiled/sc_9a25bc84add18fe6c75501f6b01bd84e1.cpp:456: no match for `Py::String & < int' /skratch/prabhu/scipy/cvs/scipy/weave/CXX/Objects.hxx:390: candidates are: bool Py::Object::operator <(const Py::Object &) const /skratch/prabhu/scipy/cvs/scipy/weave/CXX/Objects.hxx:1433: bool Py::operator <(const Py::SeqBase::const_iterator &, const Py::SeqBase::const_iterator &) /skratch/prabhu/scipy/cvs/scipy/weave/CXX/Objects.hxx:1426: bool Py::operator <(const Py::SeqBase::iterator &, const Py::SeqBase::iterator &) .warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations ....warning: specified build_dir '_bad_path_' does not exist or is or is not writable. Trying default locations .................................................................................................. ---------------------------------------------------------------------- Ran 185 tests in 643.650s OK >>> Here is the code segment in sc_9a25bc84add18fe6c75501f6b01bd84e1.cpp 446 if(!PyArg_ParseTuple(args,"OO:compiled_func",&py__locals,&py__globa 446 ls)) 447 return NULL; 448 try 449 { 450 PyObject* raw_locals = py_to_raw_dict(py__locals,"_locals"); 451 PyObject* raw_globals = py_to_raw_dict(py__globals,"_globals"); 452 /* argument conversion code */ 453 Py::String a = convert_to_string(get_variable("a",raw_locals,ra 453 w_globals),"a"); 454 /* inline code */ 455 456 if (a < 2) 457 Py::ValueError("the variable 'a' should not 457 be less than 2"); 458 return_val = Py::new_reference_to(Py::Int(a+1)); 459 /*I would like to fill in changed locals and globals here... 459 */ 460 461 } I guess the third test a = 'string' is failing (as expected). I think this test comes from test_inline.py::test_inline.check_exceptions. (line 34). The test doesnt seem to have failed either. Also maybe you can use self.assertRaises(except_type, func, args) for the exception tests? Otherwise everything else seems to run just fine! All those wierd catalog errors are gone! prabhu From prabhu at aero.iitm.ernet.in Mon Jan 14 09:56:16 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 14 Jan 2002 20:26:16 +0530 Subject: [SciPy-dev] Weave introductory document. Message-ID: <15426.61840.319405.210359@monster.linux.in> hi, I finished a simple text formatted document called "A beginners guide to using Python for performance computing" I've attached it along with the following files. If you find it useful put up at scipy by all means. laplace.py -- all the Python stuff (slow, numeric, blitz, inline, fortran, fastinline). flaplace.f -- the fortran subroutine (thanks to Pearu). laplace.cxx -- the c++ version of the solver, compile it like so: g++ laplace.cxx -O3 -o laplace Here are some timing results for a 500x500 grid for 100 iterations. Slow python estimate: 1897.0 s Numeric: 29.0 s Blitz: 10.2 s Inline: 4.30 s Fast Inline: 2.9 s Python/Fortran: 3.2 s Pure C++: 2.4 s prabhu From prabhu at aero.iitm.ernet.in Mon Jan 14 09:58:15 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 14 Jan 2002 20:28:15 +0530 Subject: [SciPy-dev] Weave introductory document. (Attachments) Message-ID: <15426.61959.86732.792300@monster.linux.in> hi, Oops. I forgot to attach the doc and the code. prabhu -------------- next part -------------- A non-text attachment was scrubbed... Name: report Type: application/octet-stream Size: 15036 bytes Desc: A beginners guide to using Python for performance computing URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: laplace.py Type: application/octet-stream Size: 9757 bytes Desc: laplace.py URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: flaplace.f Type: application/octet-stream Size: 671 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: laplace.cxx Type: application/octet-stream Size: 3369 bytes Desc: not available URL: From pearu at cens.ioc.ee Mon Jan 14 11:10:28 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 14 Jan 2002 18:10:28 +0200 (EET) Subject: [SciPy-dev] Python/Fortran multi-dimensional array issues Message-ID: Hi! In this long posting I will describe a new idea how to overcome with the difficulties caused by different storage-orders in C and Fortran multi-dimensional arrays. I have send this message also to scipy developers list because I think the issue here is also very relevant in SciPy project. It should improve the quality of the lapack and blas wrappers and reduce lots of code there. I'll greately appreciate if you could review what follows. Comments and criticism are most welcome. Of cource, real code contributions are most welcome ;) Regards, Pearu -------------------------------------------------------------- _____________________________________________________________ / Proposed internal structure for f2py generated extension \ < modules regarding the issues with different storage-orders > \ of multi-dimensional matrices in Fortran and C. / ============================================================= Author: Pearu Peterson Date: 14 January, 2001 Definitions: ============ In the following I will use the following definitions: 1) A matrix is a mathematical object that represents a collection of objects (elements), usually visualized in a table form, and one can define a set of various (algebraic,etc) operations for matrices. One can think of a matrix as a defintion of a certain mapping: (i) |--> A(i) where i belongs to the set of indices (an index itself can be a sequence of objects, for example, a sequence of integers) and A(i) is an element from a specified set, for example, a set of fruits. Symbol A then denotes a matrix of fruits. 2) An array is a storage object that represents a collection of objects stored in a certain systematic way, for example, as an ordered sequence of array elements in computer memory. In order to manipulate matrices using computers, one must store matrix elements in computer memory. In the following, I will assume that the elements of a matrix are stored as an array. There is no unique way in which order one should save matrix elements in the array. However, in C and Fortran programming languages, two, unfortunately different, conventions are used. Aim: ==== The purpose of this writing is to work out an interface for Python language so that C and Fortran routines can be called without bothering about how multi-dimensional matrices are stored in memory. For example, accessing a matrix element A[i,j] in Python will be equivalent to accessing the same matrix in C, using A[i][j], or in Fortran, using A(i,j). External conditions: ==================== In C programming language, it is custom to think that matrices are stored in the so-called row-major order, that is, a matrix is stored row by row, each row is as a contiguous array in computer memory. In Fortran programming language, matrices are stored in the column-major order: each column is a contiguous array in computer memory. In Python programming language, matrices can be stored using Python Numeric array() function that uses internally C approach, that is, elements of matrices are stored in row-major order. For example, A = array([[1,2,3],[4,5,6]]) represents a 2-by-3 matrix / 1 2 3 \ | | \ 4 5 6 / and its elements are stored in computer memory as the following array: 1 2 3 4 5 6 The same matrix, if used in Fortran, would be stored in computer memory as the following array: 1 4 2 5 3 6 Problem and solution: ===================== A problem arises if one wants to use the same matrix both in C and in Fortran functions. Then the difference in storage order of a matrix elements must be taken into account. This technical detail can be very confusing even for an experienced programmer. This is because when passing a matrix to a Fortran subroutine, you must (mentally or programmically) transpose the matrix and when the subroutine returns, you must transpose it back. As will be discussed below, there is a way to overcome these difficulties in Python by creating an interface between Python and Fortran code layers that takes care of this transition internally. So that if you will read the manual pages of the Fortran codes, you need not to think about how matrices are actually stored, the storage order will be the same, seemingly. Python / C / Fortran interface: =============================== The interface between Python and Fortran codes will use the following Python Numeric feature: transposing a Numeric array does not involve copying of its data but just permuting the dimensions and strides of the array (the so-called lazy transpose). However, when passing a Numeric array data pointer to Fortran or C function, the data must be contiguous in memory. If it is not, then data is rearranged inplace. I don't think that it can be avoided. This is certainly a penalty hit to performance. However, one can easily avoid it by creating a Numeric array with the right storage order, so that after transposing, the array data will be contiguous in memory and the data pointer can safely passed on to the Fortran subroutine. This lazy-transpose operation will be done within the interface and users need not to bother about this detail anymore (that is, after they initialize Numeric array with matrix elements using the proper order. Of course, the proper order depends on the target function: C or Fortran). The interface should be smart enough to minimize the need of real-transpose operations and the need to additional memory storage as well. Statement of the problem: ========================= Consider a M-by-N matrix A of integers, where M and N are the number A rows and columns, respectively. In Fortran language, the storing array of this matrix can be defined as follows: integer A(M,N) in C: int A[M][N]; and in Python: A = Numeric.zeros((M,N),'i') Consider also the corresponding Fortran and C functions that use matrix arguments: Fortran: subroutine FUN(A,M,N) integer A(M,N) ... end C: void cun(int *a,int m,int n) { ... } and the corresponding Python interface signatures: def py_fun(a): ... def py_cun(a): ... Main goal: ========== Our goal is to generate Python C/API functions py_fun and py_cun such that their usage in Python would be identical. The cruical part of their implementation is in functions that take a PyObject and return a PyArrayObject such that it is contiguous and its data pointer is suitable for passing on to the arguments of C or Fortran functions. The prototypes of these functions are: PyArrayObject* fortran_array_from_pyobj( int typecode, int *dims, int rank, int intent, PyObject *obj); and PyArrayObject* c_array_from_pyobj( int typecode, int *dims, int rank, int intent, PyObject *obj); when wrapping Fortran and C functions, respectively. Pseudo-code for fortran_array_from_pyobj: ========================================= if type(obj) is ArrayType: #raise not check(len(ravel(obj)) >= dims[0]*dims[1]*...*dims[rank-1]) if obj.typecode is typecode: if is_contiguous(obj): transpose_data_inplace(obj) # real-transpose set_transpose_strides(obj) # lazy-transpose Py_INCREF(obj); return obj set_transpose_strides(obj) if is_contiguous(obj): set_transpose_strides(obj) Py_INCREF(obj); return obj else: tmp_obj = PyArray_ContiguousFromObject(obj,typecode,0,0) swap_datapointer_and_typeinfo(obj,tmp_obj) Py_DECREF(tmp_obj); set_transpose_strides(obj) Py_INCREF(obj); return obj else: tmp_obj = PyArray_FromDims(rank,dims,typecode) set_transpose_strides(tmp_obj) if intent in [in,inout]: copy_ND_array(obj,tmp_obj) swap_datapointer_and_typeinfo(obj,tmp_obj) Py_DECREF(tmp_obj); Py_INCREF(obj); return obj elif obj is None: # happens when only intent is 'hide' tmp_obj = PyArray_FromDims(rank,dims,typecode) if intent is out: set_transpose_strides(tmp_obj) # otherwise tmp_obj->data is used as a work array Py_INCREF(tmp_obj) return tmp_obj else: tmp_obj = PyArray_ContiguousFromObject(obj,typecode,0,0) #raise not check(len(ravel(obj)) >= dims[0]*dims[1]*...*dims[rank-1]) set_transpose_strides(tmp_obj) transpose_data_inplace(tmp_obj) Py_INCREF(tmp_obj) return tmp_obj Notes: 1) CPU expensive tasks are in transpose_data_inplace and copy_ND_array, PyArray_ContiguousFromObject. 2) Memory expensive tasks are in PyArray_FromDims, PyArray_ContiguousFromObject 3) Side-effects are expected when set_transpose_strides and transpose_data_inplace are used. For example: >>> a = Numeric([[1,2,3],[4,5,6]],'d') >>> a.is_contiguous() 1 >>> py_fun(a) >>> a.typecode() 'i' >>> a.is_contiguous() 0 >>> transpose(a).is_contiguous() 1 Pseudo-code for c_array_from_pyobj: =================================== if type(obj) is ArrayType: #raise not check(len(ravel(obj)) >= dims[0]*dims[1]*...*dims[rank-1]) if obj.typecode is typecode: if is_contiguous(obj): Py_INCREF(obj); return obj else: tmp_obj = PyArray_ContiguousFromObject(obj,typecode,0,0) swap_datapointer_and_typeinfo(obj,tmp_obj) Py_DECREF(tmp_obj); Py_INCREF(obj); return obj else: tmp_obj = PyArray_FromDims(rank,dims,typecode) if intent in [in,inout]: copy_ND_array(obj,tmp_obj) swap_datapointer_and_typeinfo(obj,tmp_obj) Py_DECREF(tmp_obj); Py_INCREF(obj); return obj elif obj is None: # happens when only intent is 'hide' tmp_obj = PyArray_FromDims(rank,dims,typecode) Py_INCREF(tmp_obj) return tmp_obj else: tmp_obj = PyArray_ContiguousFromObject(obj,typecode,0,0) #raise not check(len(ravel(obj)) >= dims[0]*dims[1]*...*dims[rank-1]) Py_INCREF(tmp_obj) return tmp_obj 14 January, 2002 Pearu Peterson From eric at scipy.org Mon Jan 14 10:21:11 2002 From: eric at scipy.org (eric) Date: Mon, 14 Jan 2002 10:21:11 -0500 Subject: [SciPy-dev] Weave introductory document. (Attachments) References: <15426.61959.86732.792300@monster.linux.in> Message-ID: <002c01c19d0f$1a57da80$6b01a8c0@ericlaptop> Prabhu, This is *excellent*. Thanks for taking the time to put together a real world example showing so many different ways of doing things. Its a great introduction to multiple concepts. It should definitely go in SciPy. I'll find a place for it soon. For now, I'll post it on the web-site. Thanks again for all your help debugging things and for the nice document, eric ----- Original Message ----- From: "Prabhu Ramachandran" To: Sent: Monday, January 14, 2002 9:58 AM Subject: [SciPy-dev] Weave introductory document. (Attachments) > hi, > > Oops. I forgot to attach the doc and the code. > > prabhu > > From eric at scipy.org Mon Jan 14 10:35:38 2002 From: eric at scipy.org (eric) Date: Mon, 14 Jan 2002 10:35:38 -0500 Subject: [SciPy-dev] new weave tar balls -- multiple fixes References: <005001c19cc9$40c27590$777ba8c0@ericlaptop> <15426.41382.733554.770617@monster.linux.in> Message-ID: <003601c19d11$1f696410$6b01a8c0@ericlaptop> Hey Prabhu, > >>> import weave > >>> weave.test() > [snip] > ---------------------------------------------------------------------- > Ran 185 tests in 643.650s > > OK > Very good. > The test doesnt seem to have failed either. Also maybe you can use > > self.assertRaises(except_type, func, args) > > for the exception tests? We could -- is there a particular benefit? I haven't look at this at all, and am unlikely to run through all the tests to make the changes now. :) But maybe for future tests. > > Otherwise everything else seems to run just fine! All those wierd > catalog errors are gone! good deal. I'll try to incorporate your tutorial thing and then announce to c.l.p within an hour or so. eric From prabhu at aero.iitm.ernet.in Mon Jan 14 11:51:26 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 14 Jan 2002 22:21:26 +0530 Subject: [SciPy-dev] Weave introductory document. (Attachments) In-Reply-To: <002c01c19d0f$1a57da80$6b01a8c0@ericlaptop> References: <15426.61959.86732.792300@monster.linux.in> <002c01c19d0f$1a57da80$6b01a8c0@ericlaptop> Message-ID: <15427.3214.949731.819739@monster.linux.in> >>>>> "eric" == eric writes: eric> Prabhu, This is *excellent*. Thanks for taking the time to eric> put together a real world example showing so many different eric> ways of doing things. Its a great introduction to multiple eric> concepts. It should definitely go in SciPy. I'll find a eric> place for it soon. For now, I'll post it on the web-site. Thanks! Since I'd spent time learning things I thought it would be easy for others to get the hang of it if I summarized it all and gave a working example. I learnt quite a bit. We also got to benchmark things with c++ and fortran. I have one more question. How good is pyMPI? I noticed that you are one of the developers on the sourceforge project. There isnt a proper web page or docs or any examples. ScientificPython has an MPI module. How good are these in terms of performance/capabilities and ease of use? Its obvious as to what direction I'm going. If pyMPI works well, then based on the preliminary results we have with weave, f2py etc. its pretty clear that one can do really *serious* high performance computing using Python. That would be simply amazing! The power of the Python language with very decent performance. This would make it possible to create complex CFD (or other highly compute intensive) applications using Python. The possibilities are immense! eric> Thanks again for all your help debugging things and for the eric> nice document, eric Thank you for weave!! This is the least I could do. :) prabhu From prabhu at aero.iitm.ernet.in Mon Jan 14 12:00:28 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 14 Jan 2002 22:30:28 +0530 Subject: [SciPy-dev] new weave tar balls -- multiple fixes In-Reply-To: <003601c19d11$1f696410$6b01a8c0@ericlaptop> References: <005001c19cc9$40c27590$777ba8c0@ericlaptop> <15426.41382.733554.770617@monster.linux.in> <003601c19d11$1f696410$6b01a8c0@ericlaptop> Message-ID: <15427.3756.674777.481542@monster.linux.in> >>>>> "eric" == eric writes: >> The test doesnt seem to have failed either. Also maybe you can >> use >> >> self.assertRaises(except_type, func, args) >> >> for the exception tests? eric> We could -- is there a particular benefit? I haven't look Smaller code. Its easier writing something like this self.assertRaises(AssertionError, func, arg1, arg2) instead of: try: func(arg1, arg2) except AssertionError: pass else: raise AssertionError, \ "Test should have raised an exception." # or something You could look at the em_viz tests or the gui_thread tests (there is only one of these in gui_thread though) for concrete examples. eric> at this at all, and am unlikely to run through all the tests eric> to make the changes now. :) But maybe for future tests. Sure, I was just suggesting. And this is useful only when you want to make sure that something raises an error. prabhu From eric at scipy.org Mon Jan 14 11:13:10 2002 From: eric at scipy.org (eric) Date: Mon, 14 Jan 2002 11:13:10 -0500 Subject: [SciPy-dev] Weave introductory document. (Attachments) References: <15426.61959.86732.792300@monster.linux.in> Message-ID: <008501c19d16$5c9451b0$6b01a8c0@ericlaptop> Prabhu's docs are now on the web. They are linked off the main weave page: http://www.scipy.org/site_content/weave/ eric ----- Original Message ----- From: "Prabhu Ramachandran" To: Sent: Monday, January 14, 2002 9:58 AM Subject: [SciPy-dev] Weave introductory document. (Attachments) > hi, > > Oops. I forgot to attach the doc and the code. > > prabhu > > From fperez at pizero.colorado.edu Mon Jan 14 12:18:13 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Mon, 14 Jan 2002 10:18:13 -0700 (MST) Subject: [SciPy-dev] Weave Mandrake report. In-Reply-To: <04fc01c19ca7$b66c8000$6b00a8c0@ericlaptop> Message-ID: On Sun, 13 Jan 2002, eric wrote: > Hey Fernando, > > You caught the CVS in a transitional state -- that is the error your getting > with the FILE stuff. I started fixing an issue with exception handling that > was bugging me this afternoon. It will be completed tonight. Ok. Updated cvs now. >> > dep: ['scipy_distutils', 'scipy_test'] > > Traceback (most recent call last): > > File "setup.py", line 44, in ? > > stand_alone_package(with_dependencies) > > File "setup.py", line 23, in stand_alone_package > > config_dict = package_config(primary,dependencies) > > File "scipy_distutils/misc_util.py", line 174, in package_config > > config.extend([get_package_config(x) for x in primary]) > > File "scipy_distutils/misc_util.py", line 161, in get_package_config > > mod = __import__('setup_'+package_name) > > ImportError: No module named setup_weave > > Hmmm. I'll look into this. I haven't had the problem when installing from > the tar-ball, but *can* think of a reason that it might happen. I'll do a > little investigation -- especially since I'd like to open it up to wider > inspection tomorrow. The new tarballs are fine, this problem is now gone. > I'll bet a nickle your not getting the Abort because the test that was > causing the Abort fails even earlier, and doesn't allow the Abort to happen. > Call me a pessimist... :) But a correct pessimist, unfortunately! Now that the tarballs are fine, the Abort is back! > I'd like to work this out also. The cross platform issues are proving to be > a headache. Also, last night we learned that the Python 2.1.1 version of > dumbdbm (which Prabhu was using on Debian) is buggy. The new code hopefully > solves these issues, but I'll bet it remains a trouble spot. Two things > could help as far as Mandrake is concerned. (1) If I had a shell temporary > shell account on a Mandrake machine, I could diagnose the issue the > quickest. Barring that (2) we could set up a time to chat via Yahoo > tomorrow or the next day so that you and I could have a co-debugging > session. Let me know what works best for you. Unfortunately my office machine (on the net) is an alpha with python 1.5, and on my laptop I only have modem access. But Yahoo is fine, and if you want to chance ssh over a modem, I'll give you an account on my laptop :). Maybe this evening we could try something on yahoo? I live in Colorado (Mountain time), and anything after 5 or 6 pm would be fine by me. Cheers, f. From pearu at cens.ioc.ee Mon Jan 14 12:24:13 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 14 Jan 2002 19:24:13 +0200 (EET) Subject: [SciPy-dev] Weave introductory document. (Attachments) In-Reply-To: <008501c19d16$5c9451b0$6b01a8c0@ericlaptop> Message-ID: On Mon, 14 Jan 2002, eric wrote: > Prabhu's docs are now on the web. They are linked off the main weave page: > > http://www.scipy.org/site_content/weave/ That's great! However, note that the end of Prabhu's docs is quite difficult to read due to formatting failures. It shows up in my browser as a continuous text, including all code fragments. Pearu From rossini at blindglobe.net Mon Jan 14 12:39:32 2002 From: rossini at blindglobe.net (A.J. Rossini) Date: 14 Jan 2002 09:39:32 -0800 Subject: [SciPy-dev] Weave introductory document. (Attachments) In-Reply-To: <15427.3214.949731.819739@monster.linux.in> References: <15426.61959.86732.792300@monster.linux.in> <002c01c19d0f$1a57da80$6b01a8c0@ericlaptop> <15427.3214.949731.819739@monster.linux.in> Message-ID: <87zo3g7rxn.fsf@jeeves.blindglobe.net> >>>>> "PR" == Prabhu Ramachandran writes: >>>>> "eric" == eric writes: PR> I have one more question. How good is pyMPI? I noticed that PR> you are one of the developers on the sourceforge project. PR> There isnt a proper web page or docs or any examples. PR> ScientificPython has an MPI module. How good are these in PR> terms of performance/capabilities and ease of use? On a completely separate (but related note), I've updated (fixed) PyPVM to actually work, if you care for PVM's API. See http://software.biostat.washington.edu/statsoft/snake/pypvm (yes, I know that there are reasons for MPI, and it's nice, and I can give lots of reasons for preferring it; but until all of the MPI2 changes make it down, and cross-platform is easy, PVM works for me...). best, -tony -- A.J. Rossini Rsrch. Asst. Prof. of Biostatistics U. of Washington Biostatistics rossini at u.washington.edu FHCRC/SCHARP/HIV Vaccine Trials Net rossini at scharp.org -------------- http://software.biostat.washington.edu/ ---------------- FHCRC: M: 206-667-7025 (fax=4812)|Voicemail is pretty sketchy/use Email UW: Th: 206-543-1044 (fax=3286)|Change last 4 digits of phone to FAX --- I'm 40% time until March 1st. Try email the other 3 days.... ----- From prabhu at aero.iitm.ernet.in Mon Jan 14 12:40:26 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Mon, 14 Jan 2002 23:10:26 +0530 Subject: [SciPy-dev] Weave introductory document. (Attachments) In-Reply-To: References: <008501c19d16$5c9451b0$6b01a8c0@ericlaptop> Message-ID: <15427.6154.678304.467047@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> On Mon, 14 Jan 2002, eric wrote: >> Prabhu's docs are now on the web. They are linked off the main >> weave page: >> >> http://www.scipy.org/site_content/weave/ PP> That's great! However, note that the end of Prabhu's docs is PP> quite difficult to read due to formatting failures. It shows PP> up in my browser as a continuous text, including all code PP> fragments. Hmm, yeah thats right, on mozilla I cant see anything after the inline part. I dont think I formatted the document too correctly. The biggest problem being the code fragments. I didnt want to indent the code too much for fear of it becoming even harder to read. Also there is trouble with the laplace.py file i.e. http://www.scipy.org/site_content/weave/laplace.html the code after inlineTimeStep is all mangled up. Ahh I get it its the <> marks. I need to use < etc. Should I mail you an updated document? I'm sorry if for all these formatting errors. I guess this type of document is a little painful to write/edit if you use simple formatted text. Do you want me to create the document in proper HTML by using Mozilla composer or something and then mail it across? prabhu From prabhu at aero.iitm.ernet.in Mon Jan 14 13:36:26 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Tue, 15 Jan 2002 00:06:26 +0530 Subject: [SciPy-dev] Weave introductory document. (Attachments) In-Reply-To: <87zo3g7rxn.fsf@jeeves.blindglobe.net> References: <15426.61959.86732.792300@monster.linux.in> <002c01c19d0f$1a57da80$6b01a8c0@ericlaptop> <15427.3214.949731.819739@monster.linux.in> <87zo3g7rxn.fsf@jeeves.blindglobe.net> Message-ID: <15427.9514.686771.497554@monster.linux.in> >>>>> "AJR" == A J Rossini writes: AJR> On a completely separate (but related note), I've updated AJR> (fixed) PyPVM to actually work, if you care for PVM's API. AJR> See AJR> http://software.biostat.washington.edu/statsoft/snake/pypvm Oh, good. Thanks. Will keep this also in mind. prabhu From eric at scipy.org Mon Jan 14 13:01:05 2002 From: eric at scipy.org (eric) Date: Mon, 14 Jan 2002 13:01:05 -0500 Subject: [SciPy-dev] Weave introductory document. (Attachments) References: Message-ID: <00c401c19d25$74a907f0$6b01a8c0@ericlaptop> > > That's great! > However, note that the end of Prabhu's docs is quite difficult to read > due to formatting failures. It shows up in my browser as a continuous > text, including all code fragments. Can you check and see if it is fixed now? I made a couple changes, and it looks fine here -- well at least readable. eric From pearu at cens.ioc.ee Mon Jan 14 14:05:42 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 14 Jan 2002 21:05:42 +0200 (EET) Subject: [SciPy-dev] Weave introductory document. (Attachments) In-Reply-To: <00c401c19d25$74a907f0$6b01a8c0@ericlaptop> Message-ID: On Mon, 14 Jan 2002, eric wrote: > Can you check and see if it is fixed now? I made a couple changes, and it > looks fine here -- well at least readable. Looks good to me. Thanks, Pearu From prabhu at aero.iitm.ernet.in Mon Jan 14 14:19:02 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Tue, 15 Jan 2002 00:49:02 +0530 Subject: [SciPy-dev] Python/Fortran multi-dimensional array issues In-Reply-To: References: Message-ID: <15427.12070.76694.378319@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> Hi! PP> In this long posting I will describe a new idea how to PP> overcome with the difficulties caused by different PP> storage-orders in C and Fortran multi-dimensional arrays. Sounds great, but I couldnt understand the algorithms (pseudo code). I'll look carefully later on. prabhu From prabhu at aero.iitm.ernet.in Mon Jan 14 14:17:43 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Tue, 15 Jan 2002 00:47:43 +0530 Subject: [SciPy-dev] Weave introductory document. (Attachments) In-Reply-To: <00c401c19d25$74a907f0$6b01a8c0@ericlaptop> References: <00c401c19d25$74a907f0$6b01a8c0@ericlaptop> Message-ID: <15427.11991.531815.170717@monster.linux.in> >>>>> "eric" == eric writes: >> That's great! However, note that the end of Prabhu's docs is >> quite difficult to read due to formatting failures. It shows up >> in my browser as a continuous text, including all code >> fragments. eric> Can you check and see if it is fixed now? I made a couple eric> changes, and it looks fine here -- well at least readable. Yeah thats much better now, you also need to fix laplace.html similarly I think there are two <'s that need to become <. I just finished editing two different HTML versions of the document with a few spelling mistakes fixed. One was done by hand using Xemacs (this is clean HTML and called report.html) and the other with Mozilla (this is horrible to edit and called Performance_Python.html). I've attached both. Use it if you find it useful. prabhu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu at cens.ioc.ee Mon Jan 14 15:38:12 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 14 Jan 2002 22:38:12 +0200 (EET) Subject: [SciPy-dev] Python/Fortran multi-dimensional array issues In-Reply-To: <15427.12070.76694.378319@monster.linux.in> Message-ID: On Tue, 15 Jan 2002, Prabhu Ramachandran wrote: > >>>>> "PP" == Pearu Peterson writes: > > PP> Hi! > > PP> In this long posting I will describe a new idea how to > PP> overcome with the difficulties caused by different > PP> storage-orders in C and Fortran multi-dimensional arrays. > > Sounds great, but I couldnt understand the algorithms (pseudo code). > I'll look carefully later on. If it helps, then fortran|c_array_from_pyobj() functions will replace the current arr_from_pyobj() function in f2py generated code. I can guess that difficult parts of the pseudo code are the functions transpose_data_inplace, etc. Let me add here few comments on those: transpose_data_inplace(obj) - obj is PyArrayObject that holds various attributes, eg data - that is pointer to actual data array, dimensions, rank, etc. The purpose of transpose_data_inplace(obj) is best described by the following Python statement: obj = transpose(obj) but it will not consume extra memory for that. The transpose operation is carried out only for data attribute. Other attributes are not changed. set_transpose_strides(obj) - again, obj is PyArrayObject. This function behaves like Numeric.transpose, it permutes the dimensions and strides of the array object, but does not touch data attribute. swap_datapointer_and_typeinfo(obj1,obj2) - will swap the data attributes of Numeric arrays, dimensions and strides are untouched. Since now data will store different elements, also obj1 and obj2 typeinfo attributes needs to be swapped. (Travis O., can you give a hand here to get this function right?) copy_ND_array(obj1,obj2) - this is equivalent to python statement obj1[:] = obj2 where obj1,obj2 can be also noncontiguous arrays. intent - this argument is used to check whether a given variable had certain intent(..) attribute set or not. For example, in C/API code we'll have if (intent & F2PY_INTENT_IN) { ... The typecode,dims, and rank arguments in *_array_from_pyobj() functions correspond to expected properties of arguments. For example, if in Fortran A is defined as integer a(m,n) then in C/API module, typecode = PyArray_INT; dims[0] = m; dims[1] = n; rank = 2; will be set. Please don't hesitate to ask more clarifications. I will be happy to explain the algorithm as eventually it must become perfect;) Regards, Pearu From oliphant.travis at ieee.org Mon Jan 14 18:35:02 2002 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 14 Jan 2002 16:35:02 -0700 Subject: [SciPy-dev] Strange fastumath behavior. Message-ID: After recently installing the new SciPy, I'm getting very strange behavior with the function sqrt() in fastumath.so for floating point numbers. With an old installation I get >>> sqrt(10) 3.1622776601683795 However, with the new version I get >>> sqrt(10) 9.1706675698104778 I have no idea what changed in fastumath to cause this. It seems more like compiler flags that changed and are now causing incorrect results. I'm just not sure. Any ideas what could have caused this? -Travis From jochen at jochen-kuepper.de Mon Jan 14 18:58:58 2002 From: jochen at jochen-kuepper.de (Jochen =?iso-8859-1?q?K=FCpper?=) Date: 14 Jan 2002 18:58:58 -0500 Subject: [SciPy-dev] new buld process in CVS In-Reply-To: <15421.53260.896075.447513@monster.linux.in> References: <007701c199ef$7518fd50$6b01a8c0@ericlaptop> <15421.53260.896075.447513@monster.linux.in> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, 10 Jan 2002 23:01:56 +0530 Prabhu Ramachandran wrote: Prabhu> (3) I dont know why one would want to unset DISPLAY. :) 'cause you don't want programs to spawn their X-interface over a modem line. Numerous (X)Emacs sessions from home come to mind... Greetings, Jochen - -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: 44BCCD8E Sex, drugs and rock-n-roll -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8Q3DCiJ/aUUS8zY4RAkkaAKCAwfHtXmC/GtvcPvVYuh42YvMlNQCgkVaJ 1mIzPux6CbAUfXWgyneOwXE= =5qvc -----END PGP SIGNATURE----- From prabhu at aero.iitm.ernet.in Mon Jan 14 20:55:26 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Tue, 15 Jan 2002 07:25:26 +0530 Subject: [SciPy-dev] new buld process in CVS In-Reply-To: References: <007701c199ef$7518fd50$6b01a8c0@ericlaptop> <15421.53260.896075.447513@monster.linux.in> Message-ID: <15427.35854.105604.969034@monster.linux.in> >>>>> "JK" == Jochen K?pper writes: Prabhu> (3) I dont know why one would want to unset DISPLAY. :) JK> 'cause you don't want programs to spawn their X-interface over JK> a modem line. Numerous (X)Emacs sessions from home come to JK> mind... Yes, but would you want scipy to test the gui related things over that same modem line? My guess is no. Therefore using $DISPLAY to check if X is up or not and then decide to run the GUI tests seems to be a good idea. prabhu From rob at pythonemproject.com Mon Jan 14 20:55:29 2002 From: rob at pythonemproject.com (Rob) Date: Mon, 14 Jan 2002 17:55:29 -0800 Subject: [SciPy-dev] Weave now works OK on FreeBSD Message-ID: <3C438C11.50F3BCC@pythonemproject.com> The latest tarball worked AOK, Rob. -- The Numeric Python EM Project www.pythonemproject.com From prabhu at aero.iitm.ernet.in Mon Jan 14 21:00:33 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Tue, 15 Jan 2002 07:30:33 +0530 Subject: [SciPy-dev] Strange fastumath behavior. In-Reply-To: References: Message-ID: <15427.36161.810723.663995@monster.linux.in> >>>>> "TO" == Travis Oliphant writes: TO> After recently installing the new SciPy, I'm getting very TO> strange behavior with the function sqrt() in fastumath.so for TO> floating point numbers. Well, FWIW, my install seems to be doing fine on this (and I installed this just yesterday evening). scipy.sqrt, scipy.fastumath.sqrt all give me the same (correct) result. prabhu From eric at scipy.org Mon Jan 14 21:18:51 2002 From: eric at scipy.org (eric) Date: Mon, 14 Jan 2002 21:18:51 -0500 Subject: [SciPy-dev] Weave now works OK on FreeBSD References: <3C438C11.50F3BCC@pythonemproject.com> Message-ID: <001e01c19d6b$05d71dd0$777ba8c0@ericlaptop> Hey Rob, Excellent news. Can you send me the vitals of your setup (OS/Python version/Numeric version/gcc version) and I'll add it to the supported list. eric ----- Original Message ----- From: "Rob" To: Sent: Monday, January 14, 2002 8:55 PM Subject: [SciPy-dev] Weave now works OK on FreeBSD > The latest tarball worked AOK, Rob. > -- > The Numeric Python EM Project > > www.pythonemproject.com > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From jochen at jochen-kuepper.de Mon Jan 14 23:09:14 2002 From: jochen at jochen-kuepper.de (Jochen =?iso-8859-1?q?K=FCpper?=) Date: 14 Jan 2002 23:09:14 -0500 Subject: [SciPy-dev] new buld process in CVS In-Reply-To: <15427.35854.105604.969034@monster.linux.in> References: <007701c199ef$7518fd50$6b01a8c0@ericlaptop> <15421.53260.896075.447513@monster.linux.in> <15427.35854.105604.969034@monster.linux.in> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 15 Jan 2002 07:25:26 +0530 Prabhu Ramachandran wrote: >>>>>> "JK" == Jochen K?pper writes: Prabhu> (3) I dont know why one would want to unset DISPLAY. :) JK> 'cause you don't want programs to spawn their X-interface over JK> a modem line. Numerous (X)Emacs sessions from home come to JK> mind... Prabhu> Yes, but would you want scipy to test the gui related things over that Prabhu> same modem line? My guess is no. Oh, I actually fully agree, point taken! Don't know what I was thinking before. Greetings, Jochen - -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: 44BCCD8E Sex, drugs and rock-n-roll -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8Q6tqiJ/aUUS8zY4RAmgUAKCOtwpZG33T2DIYCHuGJ559vyKOEACguEV1 mcSPXChHKRU0cZh9k1SO2y8= =l9VV -----END PGP SIGNATURE----- From rob at pythonemproject.com Tue Jan 15 09:43:47 2002 From: rob at pythonemproject.com (Rob) Date: Tue, 15 Jan 2002 06:43:47 -0800 Subject: [SciPy-dev] FreeBSD with Weave particulars Message-ID: <3C444023.8D18A372@pythonemproject.com> Running FreeBSD 4.0-STABLE, (i.e. 4.4 equiv), Python 2.1, Gcc 2.95. Rob. -- The Numeric Python EM Project www.pythonemproject.com From pearu at cens.ioc.ee Wed Jan 16 14:38:59 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Wed, 16 Jan 2002 21:38:59 +0200 (EET) Subject: [SciPy-dev] Python/Fortran multi-dimensional array issues In-Reply-To: Message-ID: Hi! I have implemented a novel multi-dimensional array support for f2py. The snapshot is available in http://cens.ioc.ee/projects/f2py2e/2.x/ Basically, from now on there is no visible difference for arrays in Python or Fortran. f2py generated modules will deal with this rather confusing issue (that is due the difference how matrices are stored in arrays for C and Fortran programming languages) in a very efficient way. Note that this is quite a big news for f2py as this new approach finally solves the long lasted dilemma of Fortran and C storage orders and how to deal with it from Python. An example of a signature file with comments below illustrates all that with source files included to the attachment. Enjoy! Pearu !%f90 -*- f90 -*- ! Example: ! Using f2py for wrapping multi-dimensional Fortran and C arrays ! [NEW APPROACH, use it with f2py higher than 2.9.x] ! See also files foo.f and bar.c. ! ! $Id: fun.pyf,v 1.2 2002/01/16 18:57:33 pearu Exp $ ! Usage (with gcc compiler): ! f2py -c fun.pyf foo.f bar.c python module fun ! in interface ! in :fun ! >>> from Numeric import * ! >>> import fun ! >>> a=array([[1,2,3],[4,5,6]]) subroutine foo(a,m,n) ! in :fun:foo.f integer dimension(m,n) :: a intent(in,out) :: a integer optional,check(shape(a,0)==m),depend(a) :: m=shape(a,0) integer optional,check(shape(a,1)==n),depend(a) :: n=shape(a,1) end subroutine foo ! >>> print fun.foo.__doc__ ! foo - Function signature: ! a = foo(a,[m,n]) ! Required arguments: ! a : input rank-2 array('i') with bounds (m,n) ! Optional arguments: ! m := shape(a,0) input int ! n := shape(a,1) input int ! Return objects: ! a : rank-2 array('i') with bounds (m,n) ! >>> print fun.foo(a) ! F77: ! m= 2, n= 3 ! Row 1: ! a(i= 1,j= 1) = 1 ! a(i= 1,j= 2) = 2 ! a(i= 1,j= 3) = 3 ! Row 2: ! a(i= 2,j= 1) = 4 ! a(i= 2,j= 2) = 5 ! a(i= 2,j= 3) = 6 ! [[77777 2 3] ! [ 4 5 6]] subroutine bar(a,m,n) intent(c) intent(c) bar integer dimension(m,n) :: a intent(in,out) :: a integer optional,check(shape(a,0)==m),depend(a) :: m=shape(a,0) integer optional,check(shape(a,1)==n),depend(a) :: n=shape(a,1) intent(in) m,n end subroutine bar ! >>> print fun.bar.__doc__ ! bar - Function signature: ! a = bar(a,[m,n]) ! Required arguments: ! a : input rank-2 array('i') with bounds (m,n) ! Optional arguments: ! m := shape(a,0) input int ! n := shape(a,1) input int ! Return objects: ! a : rank-2 array('i') with bounds (m,n) ! >>> print fun.bar(a) ! C:m=2, n=3 ! Row 1: ! a(i=0,j=0)=1 ! a(i=0,j=1)=2 ! a(i=0,j=2)=3 ! Row 2: ! a(i=1,j=0)=4 ! a(i=1,j=1)=5 ! a(i=1,j=2)=6 ! [[7777 2 3] ! [ 4 5 6]] end interface end python module fun -------------- next part -------------- !%f90 -*- f90 -*- ! Example: ! Using f2py for wrapping multi-dimensional Fortran and C arrays ! [NEW APPROACH, use it with f2py higher than 2.8.x] ! $Id: fun.pyf,v 1.2 2002/01/16 18:57:33 pearu Exp $ ! Usage (with gcc compiler): ! f2py -c fun.pyf foo.f bar.c python module fun ! in interface ! in :fun ! >>> from Numeric import * ! >>> import fun ! >>> a=array([[1,2,3],[4,5,6]]) subroutine foo(a,m,n) ! in :fun:foo.f integer dimension(m,n) :: a intent(in,out) :: a integer optional,check(shape(a,0)==m),depend(a) :: m=shape(a,0) integer optional,check(shape(a,1)==n),depend(a) :: n=shape(a,1) end subroutine foo ! >>> print fun.foo.__doc__ ! foo - Function signature: ! a = foo(a,[m,n]) ! Required arguments: ! a : input rank-2 array('i') with bounds (m,n) ! Optional arguments: ! m := shape(a,0) input int ! n := shape(a,1) input int ! Return objects: ! a : rank-2 array('i') with bounds (m,n) ! >>> print fun.foo(a) ! F77: ! m= 2, n= 3 ! Row 1: ! a(i= 1,j= 1) = 1 ! a(i= 1,j= 2) = 2 ! a(i= 1,j= 3) = 3 ! Row 2: ! a(i= 2,j= 1) = 4 ! a(i= 2,j= 2) = 5 ! a(i= 2,j= 3) = 6 ! [[77777 2 3] ! [ 4 5 6]] subroutine bar(a,m,n) intent(c) intent(c) bar integer dimension(m,n) :: a intent(in,out) :: a integer optional,check(shape(a,0)==m),depend(a) :: m=shape(a,0) integer optional,check(shape(a,1)==n),depend(a) :: n=shape(a,1) intent(in) m,n end subroutine bar ! >>> print fun.bar.__doc__ ! bar - Function signature: ! a = bar(a,[m,n]) ! Required arguments: ! a : input rank-2 array('i') with bounds (m,n) ! Optional arguments: ! m := shape(a,0) input int ! n := shape(a,1) input int ! Return objects: ! a : rank-2 array('i') with bounds (m,n) ! >>> print fun.bar(a) ! C:m=2, n=3 ! Row 1: ! a(i=0,j=0)=1 ! a(i=0,j=1)=2 ! a(i=0,j=2)=3 ! Row 2: ! a(i=1,j=0)=4 ! a(i=1,j=1)=5 ! a(i=1,j=2)=6 ! [[7777 2 3] ! [ 4 5 6]] end interface end python module fun ! This file was auto-generated with f2py (version:2.9.166). ! See http://cens.ioc.ee/projects/f2py2e/ -------------- next part -------------- subroutine foo(a,m,n) integer a(m,n), m,n,i,j print*, "F77:" print*, "m=",m,", n=",n do 100,i=1,m print*, "Row ",i,":" do 50,j=1,n print*, "a(i=",i,",j=",j,") = ",a(i,j) 50 continue 100 continue if (m*n.gt.0) a(1,1) = 77777 end -------------- next part -------------- A non-text attachment was scrubbed... Name: bar.c Type: text/x-csrc Size: 279 bytes Desc: URL: From eric at scipy.org Wed Jan 16 13:51:00 2002 From: eric at scipy.org (eric) Date: Wed, 16 Jan 2002 13:51:00 -0500 Subject: [SciPy-dev] Python/Fortran multi-dimensional array issues References: Message-ID: <00cc01c19ebe$be004c70$6b01a8c0@ericlaptop> Hey Pearu, This is big news! Could you post the wrapper code that f2py generates for this? thanks, eric ----- Original Message ----- From: "Pearu Peterson" To: Cc: Sent: Wednesday, January 16, 2002 2:38 PM Subject: Re: [SciPy-dev] Python/Fortran multi-dimensional array issues > > Hi! > > I have implemented a novel multi-dimensional array support for f2py. The > snapshot is available in > > http://cens.ioc.ee/projects/f2py2e/2.x/ > > Basically, from now on there is no visible difference for arrays in > Python or Fortran. f2py generated modules will deal with this rather > confusing issue (that is due the difference how matrices are stored in > arrays for C and Fortran programming languages) in a very efficient way. > > Note that this is quite a big news for f2py as this new approach finally > solves the long lasted dilemma of Fortran and C storage orders and how to > deal with it from Python. > > An example of a signature file with comments below illustrates all that > with source files included to the attachment. > > Enjoy! > Pearu > > !%f90 -*- f90 -*- > > ! Example: > ! Using f2py for wrapping multi-dimensional Fortran and C arrays > ! [NEW APPROACH, use it with f2py higher than 2.9.x] > ! See also files foo.f and bar.c. > ! > ! $Id: fun.pyf,v 1.2 2002/01/16 18:57:33 pearu Exp $ > > ! Usage (with gcc compiler): > ! f2py -c fun.pyf foo.f bar.c > > python module fun ! in > interface ! in :fun > > ! >>> from Numeric import * > ! >>> import fun > ! >>> a=array([[1,2,3],[4,5,6]]) > > subroutine foo(a,m,n) ! in :fun:foo.f > integer dimension(m,n) :: a > intent(in,out) :: a > integer optional,check(shape(a,0)==m),depend(a) :: m=shape(a,0) > integer optional,check(shape(a,1)==n),depend(a) :: n=shape(a,1) > end subroutine foo > > ! >>> print fun.foo.__doc__ > ! foo - Function signature: > ! a = foo(a,[m,n]) > ! Required arguments: > ! a : input rank-2 array('i') with bounds (m,n) > ! Optional arguments: > ! m := shape(a,0) input int > ! n := shape(a,1) input int > ! Return objects: > ! a : rank-2 array('i') with bounds (m,n) > > ! >>> print fun.foo(a) > ! F77: > ! m= 2, n= 3 > ! Row 1: > ! a(i= 1,j= 1) = 1 > ! a(i= 1,j= 2) = 2 > ! a(i= 1,j= 3) = 3 > ! Row 2: > ! a(i= 2,j= 1) = 4 > ! a(i= 2,j= 2) = 5 > ! a(i= 2,j= 3) = 6 > ! [[77777 2 3] > ! [ 4 5 6]] > > > subroutine bar(a,m,n) > intent(c) > intent(c) bar > integer dimension(m,n) :: a > intent(in,out) :: a > integer optional,check(shape(a,0)==m),depend(a) :: m=shape(a,0) > integer optional,check(shape(a,1)==n),depend(a) :: n=shape(a,1) > intent(in) m,n > end subroutine bar > > ! >>> print fun.bar.__doc__ > ! bar - Function signature: > ! a = bar(a,[m,n]) > ! Required arguments: > ! a : input rank-2 array('i') with bounds (m,n) > ! Optional arguments: > ! m := shape(a,0) input int > ! n := shape(a,1) input int > ! Return objects: > ! a : rank-2 array('i') with bounds (m,n) > > ! >>> print fun.bar(a) > ! C:m=2, n=3 > ! Row 1: > ! a(i=0,j=0)=1 > ! a(i=0,j=1)=2 > ! a(i=0,j=2)=3 > ! Row 2: > ! a(i=1,j=0)=4 > ! a(i=1,j=1)=5 > ! a(i=1,j=2)=6 > ! [[7777 2 3] > ! [ 4 5 6]] > > end interface > end python module fun > > From pearu at cens.ioc.ee Wed Jan 16 15:31:45 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Wed, 16 Jan 2002 22:31:45 +0200 (EET) Subject: [SciPy-dev] Python/Fortran multi-dimensional array issues In-Reply-To: <00cc01c19ebe$be004c70$6b01a8c0@ericlaptop> Message-ID: On Wed, 16 Jan 2002, eric wrote: > Hey Pearu, > > This is big news! Could you post the wrapper code that f2py generates for > this? It is not small (23K) and you could easily generate it with latest the f2py, but if you insist, then you can find it attached. The new thing there is array_from_pyobj() function that implements interfacing Fortran and C arrays with Python arrays. Pearu -------------- next part -------------- A non-text attachment was scrubbed... Name: funmodule.c Type: text/x-csrc Size: 24110 bytes Desc: URL: From prabhu at aero.iitm.ernet.in Wed Jan 16 22:31:01 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 17 Jan 2002 09:01:01 +0530 Subject: [SciPy-dev] Python/Fortran multi-dimensional array issues In-Reply-To: References: Message-ID: <15430.17781.6579.745068@monster.linux.in> Hi, Thats great. >>>>> "PP" == Pearu Peterson writes: PP> ! >>> print fun.foo(a) ! F77: ! m= 2, n= 3 ! Row 1: ! a(i= PP> 1,j= 1) = 1 ! a(i= 1,j= 2) = 2 ! a(i= 1,j= 3) = 3 ! Row 2: PP> ! a(i= 2,j= 1) = 4 ! a(i= 2,j= 2) = 5 ! a(i= 2,j= 3) = 6 ! PP> [[77777 2 3] ! [ 4 5 6]] ^^^^^^^^^ I just hope that is a typo. :) prabhu From pearu at cens.ioc.ee Thu Jan 17 00:18:18 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Thu, 17 Jan 2002 07:18:18 +0200 (EET) Subject: [SciPy-dev] Python/Fortran multi-dimensional array issues In-Reply-To: <15430.17781.6579.745068@monster.linux.in> Message-ID: On Thu, 17 Jan 2002, Prabhu Ramachandran wrote: > >>>>> "PP" == Pearu Peterson writes: > > PP> ! >>> print fun.foo(a) ! F77: ! m= 2, n= 3 ! Row 1: ! a(i= > PP> 1,j= 1) = 1 ! a(i= 1,j= 2) = 2 ! a(i= 1,j= 3) = 3 ! Row 2: > PP> ! a(i= 2,j= 1) = 4 ! a(i= 2,j= 2) = 5 ! a(i= 2,j= 3) = 6 ! > PP> [[77777 2 3] ! [ 4 5 6]] > ^^^^^^^^^ > > I just hope that is a typo. :) No, it is not a typo. The example is real, not typed in. If you look at the Fortran file foo.f then you see that after printing the data to standard output, the first element is set to 77777. It just for demostrating data flow from Fortran to Python. Pearu From prabhu at aero.iitm.ernet.in Thu Jan 17 00:36:19 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 17 Jan 2002 11:06:19 +0530 Subject: [SciPy-dev] Python/Fortran multi-dimensional array issues In-Reply-To: References: <15430.17781.6579.745068@monster.linux.in> Message-ID: <15430.25299.182315.651003@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> On Thu, 17 Jan 2002, Prabhu Ramachandran wrote: PP> ! a(i= 2,j= 1) = 4 ! a(i= 2,j= 2) = 5 ! a(i= 2,j= 3) = 6 ! PP> [[77777 2 3] ! [ 4 5 6]] >> ^^^^^^^^^ >> >> I just hope that is a typo. :) PP> No, it is not a typo. The example is real, not typed in. Sorry my bad. prabhu From eric at scipy.org Fri Jan 18 02:07:40 2002 From: eric at scipy.org (eric) Date: Fri, 18 Jan 2002 02:07:40 -0500 Subject: [SciPy-dev] updates to weave in the CVS Message-ID: <00ac01c19fee$d18dab80$777ba8c0@ericlaptop> Hey group, The problems with weave on Mandrake have been largely solved -- blitz support is still broken, but everything else *should* work. If you are following the CVS, please grab the latest version and test weave to make sure the fixes didn't break it for your architecture... thanks, eric -- Eric Jones Enthought, Inc. [www.enthought.com and www.scipy.org] (512) 536-1057 From prabhu at aero.iitm.ernet.in Fri Jan 18 04:51:22 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Fri, 18 Jan 2002 15:21:22 +0530 Subject: [SciPy-dev] updates to weave in the CVS In-Reply-To: <00ac01c19fee$d18dab80$777ba8c0@ericlaptop> References: <00ac01c19fee$d18dab80$777ba8c0@ericlaptop> Message-ID: <15431.61466.64723.18001@monster.linux.in> >>>>> "eric" == eric writes: eric> Hey group, The problems with weave on Mandrake have been eric> largely solved -- blitz support is still broken, but eric> everything else *should* work. If you are following the eric> CVS, please grab the latest version and test weave to make eric> sure the fixes didn't break it for your architecture... Under debian: [snip] ---------------------------------------------------------------------- Ran 185 tests in 536.717s OK :) prabhu From pearu at cens.ioc.ee Fri Jan 18 05:39:54 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 18 Jan 2002 12:39:54 +0200 (EET) Subject: [SciPy-dev] lapack and blas Message-ID: Hi! I have finished with introducing many new features to f2py and made the latest f2py snapshot available. Now I would like to re-work with lapack and blas modules in scipy. Currently they are included in linalg package but I am thinking of separating lapack and blas from there to stand-alone packages into scipy/ tree. Here are my reasons: 1) I don't want to break linalg package. 2) lapack and blas packages are valuable also outside scipy environment. Later, linalg would start to use these new lapack and blas packages as they should become easier to maintain and more efficient when using the latest features in f2py. Please, let me know if you have any objections or comments or ideas regarding the plans above. Regards, Pearu From eric at scipy.org Fri Jan 18 12:23:37 2002 From: eric at scipy.org (eric) Date: Fri, 18 Jan 2002 12:23:37 -0500 Subject: [SciPy-dev] lapack and blas References: Message-ID: <008601c1a044$e100a260$6b01a8c0@ericlaptop> Hey Pearu, > Now I would like to re-work with lapack and blas modules in > scipy. Currently they are included in linalg package but I am thinking of > separating lapack and blas from there to stand-alone packages into scipy/ > tree. Here are my reasons: > 1) I don't want to break linalg package. > 2) lapack and blas packages are valuable also outside scipy environment. I think some reworking would be beneficial. Can you give a brief outline of your plans? There is hope that the current wrapper generator will also be able to generate Scalapack wrappers from the same set of generic_lapack.pyf definitions. I want to keep this possibility open. We also need to keep both c and fortran interfaces so that people using the respective languages will have a "native" ordering package to use for the fastest possible computations. Also, I have a small set of routines that overlap with some in linalg that we use on some high performance (but serial) applications. They're interface is necessarily differnt from the current linalg stuff so that uncessary copying, etc. can be avoided (they have and overwrite=1 flag and things like that). I'd like us to revisit the linalg interfaces in the hopes that we can come up with a design that provides convenience and still allows for high performance computations. > > Later, linalg would start to use these new lapack and blas packages as > they should become easier to maintain and more efficient when using the > latest features in f2py. Is it possible to just branch the current package in the CVS? I've never done this (the one time I tried, I screwed everything up). We can add an alias in the modules file so that cvs co linalg just grabs the linear algebra package. Also, the current build infrastructure makes it possible build sub-packages in scipy either within SciPy or standalone. So, if there are no scipy dependencies in the linalg module, then it should work fine elsewhere. If we can find a way to do it, I'd like to do it within the current package. That keeps from having two of them floating around. Also, it will keep everybod interested in the re-write involved in it. I'm all for the re-write though, and am currently interested in the same topic. Lets get to it. eric From pearu at cens.ioc.ee Fri Jan 18 15:14:17 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 18 Jan 2002 22:14:17 +0200 (EET) Subject: [SciPy-dev] lapack and blas In-Reply-To: <008601c1a044$e100a260$6b01a8c0@ericlaptop> Message-ID: On Fri, 18 Jan 2002, eric wrote: > Hey Pearu, > > > Now I would like to re-work with lapack and blas modules in > > scipy. Currently they are included in linalg package but I am thinking of > > separating lapack and blas from there to stand-alone packages into scipy/ > > tree. Here are my reasons: > > 1) I don't want to break linalg package. > > 2) lapack and blas packages are valuable also outside scipy environment. > > I think some reworking would be beneficial. Can you give a brief outline of > your plans? There is hope that the current wrapper generator will also be > able to generate Scalapack wrappers from the same set of generic_lapack.pyf > definitions. I want to keep this possibility open. We also need to keep > both c and fortran interfaces so that people using the respective languages > will have a "native" ordering package to use for the fastest possible > computations. I was thinking of creating two extension modules: flapack.so and clapack.so. One would be efficient for column-major ordered arrays, the other for row-major ordered arrays, respectively. Both should have identical interfaces for Python: if one is not available then other should serve the same needs, just with less efficency due to ordering issues. Now, flapack.so and clapack.so are used from a Python module lapack.py that is interface for lapack users. Basically, lapack.py implements the selection mechanism for choosing Lapack functions from flapack.so or clapack.so based on the array types and ordering (btw, f2py provides an efficent way how to decide if the array is in column-major orderd). Also, if one of the extension modules is not available (as it may happen with flapack.so on Windows), then it would automatically use clapack.so instead. (Btw, clapack has also functions for column-major arrays, we could also consider supporting those functions). The same scheme applies also for fblas and cblas. If scalapack has also a C version available, then we can wrap it in the similar way. Note that I do see advantages in the current wrapper generator, I was thinking that some of its code should re-used. Basically the -stuff and may be info-stuff also. Many things like reordering axes are dealt by f2py generated modules. In order to deal with C versions of lapack subroutines, that are actually C functions, I will introduce intent(nowrap) attribute for function names so that f2py will not generate wrappers for them. In this way we get rid of global --no-wrap-functions flag. (Or if subroutine has already intent(c) attribute, then f2py should not create a wrapper for it?) > Also, I have a small set of routines that overlap with some in linalg that > we use on some high performance (but serial) applications. They're > interface is necessarily differnt from the current linalg stuff so that > uncessary copying, etc. can be avoided (they have and overwrite=1 flag and > things like that). I'd like us to revisit the linalg interfaces in the > hopes that we can come up with a design that provides convenience and still > allows for high performance computations. This issue is also dealt by f2py. There is now intent(copy) attribute for array variables. By default, a copy of the array is made before passing it (acctually its copy) to the wrapped function. f2py generates also keyword arguments for such cases that can be used to disable the copying for efficency. For example, consider subroutine foo (a,..) integer intent(in,out,copy):: a(m,n) ! do nothing end Its Python signature is a = foo(a,copy_a=1) If using the proper type arrays for arguments then the following holds: id(foo(a)) != id(a) # a copy of `a' is passed to foo and # returned to Python id(foo(a,copy_a=0)) == id(a) # no copy is made, the same array is # return to Python So, I hope this solves the problems of efficient memory usage in a flexible way. > > Later, linalg would start to use these new lapack and blas packages as > > they should become easier to maintain and more efficient when using the > > latest features in f2py. > > Is it possible to just branch the current package in the CVS? I've never > done this (the one time I tried, I screwed everything up). We can add an > alias in the modules file so that cvs co linalg just grabs the linear > algebra package. Also, the current build infrastructure makes it possible > build sub-packages in scipy either within SciPy or standalone. So, if there > are no scipy dependencies in the linalg module, then it should work fine > elsewhere. I haven't tried to branch CVS repositories either. > If we can find a way to do it, I'd like to do it within the current package. > That keeps from having two of them floating around. Also, it will keep > everybod interested in the re-write involved in it. It is fine for me to do it in inside linalg as scipy already has so many directories with different stuff. However, I would like to work without breaking current linalg until we have wrappers to all these functions that linalg currently uses. It shouldn't take too long but it would be safer for now. I would suggest separate sub-directories for lapack,blas,scalapack,etc in linalg/. Also, I would like keep the set of wrapped functions minimal as a first instance. We should choose some representatives and test the re-write on them. After some experience we can wrap other functions as well, but then with less pain. > I'm all for the re-write though, and am currently interested in the same > topic. Lets get to it. Very good, lets do it. Pearu From jochen at unc.edu Fri Jan 18 15:22:59 2002 From: jochen at unc.edu (Jochen =?iso-8859-1?q?K=FCpper?=) Date: 18 Jan 2002 15:22:59 -0500 Subject: [SciPy-dev] scipy on cygwin Message-ID: <86g053mmsc.fsf@bock.chem.unc.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi All, I managed to compile scipy (cvs) on cygwin (latest). I had to do the following changes to cephes because log2 is defined as a macro in /usr/include/math.h ,---- | Index: special/cephes.h | =================================================================== | RCS file: /home/cvsroot/world/scipy/special/cephes.h,v | retrieving revision 1.1 | diff -u -r1.1 cephes.h | --- special/cephes.h 2001/06/20 05:29:41 1.1 | +++ special/cephes.h 2002/01/18 20:14:48 | @@ -102,7 +102,9 @@ | /* | extern int levnsn ( int n, double r[], double a[], double e[], double refl[] ); | */ | +#ifndef log2 | extern double log2 ( double x ); | +#endif | /* | extern long lrand ( void ); | extern long lsqrt ( long x ); | Index: special/cephes/protos.h | =================================================================== | RCS file: /home/cvsroot/world/scipy/special/cephes/protos.h,v | retrieving revision 1.2 | diff -u -r1.2 protos.h | --- special/cephes/protos.h 2001/07/03 08:32:00 1.2 | +++ special/cephes/protos.h 2002/01/18 20:14:48 | @@ -101,7 +101,9 @@ | extern int levnsn ( int n, double r[], double a[], double e[], double refl[] ); | extern double log ( double x ); | extern double log10 ( double x ); | +#ifndef log2 | extern double log2 ( double x ); | +#endif | extern long lrand ( void ); | extern long lsqrt ( long x ); | extern int minv ( double A[], double X[], int n, double B[], int IPS[] ); `---- I am not sure whether we want the macro to be used or to be undefined for this module... However, the scipy.test() falls on almost everything, it looks like that is is mostly due to import issues. Interactively I get the following behaviour, for example: ,---- | python | Python 2.2 (#1, Dec 31 2001, 15:21:18) | [GCC 2.95.3-5 (cygwin special)] on cygwin | Type "help", "copyright", "credits" or "license" for more information. | >>> import scipy | gnuplot: not found | >>> from scipy import scipy_version | Traceback (most recent call last): | File "", line 1, in ? | ImportError: cannot import name scipy_version | >>> import scipy.scipy_version | >>> `---- Anybody seen that? Hints? Greetings, Jochen - -- University of North Carolina phone: +1-919-962-4403 Department of Chemistry phone: +1-919-962-1579 Venable Hall CB#3290 (Kenan C148) fax: +1-919-843-6041 Chapel Hill, NC 27599, USA GnuPG key: 44BCCD8E -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6-cygwin-fcn-1 (Cygwin) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8SIQkiJ/aUUS8zY4RAiVrAJ96fcAA49xBlBjjKBVrRNvEwFMM1ACfTyf2 Uy3gXURjohJAHvHWzO0UumA= =4DvG -----END PGP SIGNATURE----- From pearu at cens.ioc.ee Fri Jan 18 19:04:30 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sat, 19 Jan 2002 02:04:30 +0200 (EET) Subject: [SciPy-dev] lapack and blas In-Reply-To: Message-ID: Hi! Here is a sample pyf file that shows how one can wrap both Fortran LAPACK and ATLAS LAPACK functions with almost the same signatures. Note that callstatement is the latest feature in f2py and right now only available in CVS. The callstatement is quite powerful and allows one to design a wrapper with almost arbitrary Python signature. Here we needed it because sgesv functions in lapack and atlas lapack have different signatures (consider the 'info' argument, for example). Here are the Python signatures of these wrappers: A,ipiv,B,info = f_sgesv(A,B,copy_A=1,copy_B=1) A,ipiv,B,info = c_sgesv(A,B,copy_A=1,copy_B=1) However, a bad news is that lapack and atlas lapack functions are not compatible. For example, >>> LU,ipiv,X,info=flapack.c_sgesv([[1,2],[3,4]],[1,1]) >>> print LU [[ 2. 0.5] [ 4. 1. ]] >>> print X [-0.5 0.5] >>> LU,ipiv,X,info=flapack.f_sgesv([[1,2],[3,4]],[1,1]) >>> print LU [[ 3. 4. ] [ 0.33333334 0.66666669]] >>> print X [-0.99999994 0.99999994] Just thought it would be useful you to know some of the latest features in f2py and issues in wrapping lapack and atlas lapack functions. Regards, Pearu !%f90 -*- f90 -*- python module flapack interface subroutine f_sgesv(n,nrhs,A,lda,ipiv,B,ldb,info) fortranname sgesv integer depend(A),intent(hide):: n = shape(A,0) integer depend(B),intent(hide):: nrhs = shape(B,1) real dimension(n,n),check(shape(A,0)==shape(A,1)), intent(in,out,copy) :: A integer intent(hide),depend(n) :: lda=n integer dimension(n),depend(n),intent(out) :: ipiv real dimension(n,nrhs),check(shape(A,0)==shape(B,0)),depend(n),intent(in,out,copy) :: B integer intent(hide),depend(n) :: ldb=n integer intent(out):: info end subroutine sgesv subroutine c_sgesv(n,nrhs,A,lda,ipiv,B,ldb,info) ! ! Here starts ATLAS specific stuff ! fortranname clapack_sgesv intent(c) intent(c) :: c_sgesv callstatement {extern int clapack_sgesv(); info =clapack_sgesv(102,n,nrhs,A,lda,ipiv,B,ldb); } ! ! Here ends ATLAS specific stuff, what follows is identical to above ! integer depend(A),intent(hide):: n = shape(A,0) integer depend(B),intent(hide):: nrhs = shape(B,1) real dimension(n,n),check(shape(A,0)==shape(A,1)), intent(in,out,copy) :: A integer intent(hide),depend(n) :: lda=n integer dimension(n),depend(n),intent(out) :: ipiv real dimension(n,nrhs),check(shape(A,0)==shape(B,0)),depend(n),intent(in,out,copy) :: B integer intent(hide),depend(n) :: ldb=n integer intent(out):: info end subroutine c_sgesv end interface end python module flapack From prabhu at aero.iitm.ernet.in Fri Jan 18 22:39:24 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sat, 19 Jan 2002 09:09:24 +0530 Subject: [SciPy-dev] scipy on cygwin In-Reply-To: <86g053mmsc.fsf@bock.chem.unc.edu> References: <86g053mmsc.fsf@bock.chem.unc.edu> Message-ID: <15432.60012.543319.58795@monster.linux.in> >>>>> "JK" == Jochen K?pper writes: JK> not found | >>> from scipy import scipy_version | Traceback JK> (most recent call last): | File "", line 1, in ? | JK> ImportError: cannot import name scipy_version | >>> import JK> scipy.scipy_version | >>> `---- JK> Anybody seen that? Hints? I get the same scipy_version problem. It happens thanks to this in __init__.py #get version import scipy_version __version__ = scipy_version.scipy_version del scipy_version Instead you should try to print scipy.__version__ prabhu From pearu at cens.ioc.ee Sat Jan 19 13:07:39 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sat, 19 Jan 2002 20:07:39 +0200 (EET) Subject: [SciPy-dev] New clapack and flapack In-Reply-To: Message-ID: Hi Eric, I have finished new interfaces to both ATLAS C LAPACK and Fortran LAPACK. The generic pyf files and the reduced interface_gen.py are included in attachements (its is much faster now). Here is how to build the extension modules from scratch (you'll need the latest f2py, though): 1. $ python interface_gen.py # this creates clapack.pyf and flapack.pyf 2. $ f2py -c clapack.pyf -L/usr/local/lib/atlas -llapack -lf77blas -lcblas -latlas -lg2c $ f2py -c flapack.pyf -L/usr/local/lib/atlas -llapack -lf77blas -lcblas -latlas -lg2c This builds clapack.so and flapack.so. 3. In python: >>> import clapack,flapack >>> print clapack.__doc__ This module 'clapack' is auto-generated with f2py (version:2.11.174-1131). Functions: LU,piv,X,info = sgesv(A,B,rowmajor=1,copy_A=1,copy_B=1) LU,piv,X,info = dgesv(A,B,rowmajor=1,copy_A=1,copy_B=1) LU,piv,X,info = cgesv(A,B,rowmajor=1,copy_A=1,copy_B=1) LU,piv,X,info = zgesv(A,B,rowmajor=1,copy_A=1,copy_B=1) LU,piv,info = sgetrf(A,rowmajor=1,copy_A=1) LU,piv,info = dgetrf(A,rowmajor=1,copy_A=1) LU,piv,info = cgetrf(A,rowmajor=1,copy_A=1) LU,piv,info = zgetrf(A,rowmajor=1,copy_A=1) .... >>> print flapack.__doc__ This module 'flapack' is auto-generated with f2py (version:2.11.174-1131). Functions: LU,piv,X,info = sgesv(A,B,copy_A=1,copy_B=1) LU,piv,X,info = dgesv(A,B,copy_A=1,copy_B=1) LU,piv,X,info = cgesv(A,B,copy_A=1,copy_B=1) LU,piv,X,info = zgesv(A,B,copy_A=1,copy_B=1) LU,piv,info = sgetrf(A,copy_A=1) LU,piv,info = dgetrf(A,copy_A=1) LU,piv,info = cgetrf(A,copy_A=1) LU,piv,info = zgetrf(A,copy_A=1) .... Few remarks: 1) The sources to these extension modules are quite big (more than 5000 lines in both). Should we factor these modules to cslapack.so cclapack.so cdlapack.so czlapack.so fslapack.so fclapack.so fdlapack.so fzlapack.so (any better name convention?). What do you think? 2) How shall I commit all this to the scipy CVS repository? In a subdirectory of linalg? Eventually it should be in linalg directory, I think. Regards, Pearu -------------- next part -------------- #!/usr/bin/env python import string,re,os def all_subroutines(interface_in): # remove comments comment_block_exp = re.compile(r'/\*(?:\s|.)*?\*/') subroutine_exp = re.compile(r'subroutine (?:\s|.)*?end subroutine.*') function_exp = re.compile(r'function (?:\s|.)*?end function.*') interface = comment_block_exp.sub('',interface_in) subroutine_list = subroutine_exp.findall(interface) function_list = function_exp.findall(interface) #function_list = [] subroutine_list = subroutine_list + function_list subroutine_list = map(lambda x: string.strip(x),subroutine_list) return subroutine_list def real_convert(val_string): return val_string def complex_convert(val_string): return '(' + val_string + ',0.)' def convert_types(interface_in,converter): regexp = re.compile(r'') interface = interface_in[:] while 1: sub = regexp.search(interface) if sub is None: break converted = converter(sub.group(1)) interface = string.replace(interface,sub.group(),converted) return interface def generic_expand(generic_interface): generic_types ={'s' :('real', 'real', real_convert, 'real'), 'd' :('double precision','double precision',real_convert, 'double precision'), 'c' :('complex', 'complex',complex_convert, 'real'), 'z' :('double complex', 'double complex',complex_convert, 'double precision'), 'sc':('complex', 'real',real_convert, 'real'), 'dz':('double complex', 'double precision',real_convert, 'double precision'), 'cs':('real', 'complex',complex_convert, 'real'), 'zd':('double precision','double complex', complex_convert, 'double precision')} #2. get all subroutines subs = all_subroutines(generic_interface) print len(subs) #loop through the subs type_exp = re.compile(r'') interface = '' for sub in subs: #3. Find the typecodes to use: m = type_exp.search(sub) if m is None: interface = interface + '\n\n' + sub continue type_chars = m.group(1) # get rid of spaces type_chars = string.replace(type_chars,' ','') # get a list of the characters (or character pairs) type_chars = string.split(type_chars,',') #print type_chars # Now get rid of the special tag that contained the types sub = re.sub(type_exp,'',sub) sub_generic = string.strip(sub) for char in type_chars: type_in,type_out,converter, rtype_in = generic_types[char] sub = convert_types(sub_generic,converter) function_def = string.replace(sub,'',char) function_def = string.replace(function_def,'',type_in) #function_def = string.replace(function_def,'',rtype_in) #function_def = string.replace(function_def,'',type_out) interface = interface + '\n\n' + function_def return interface #def interface_to_module(interface_in,module_name,include_list,sdir='.'): def interface_to_module(interface_in,module_name): pre_prefix = "!%f90 -*- f90 -*-\n" # includes = '' # for file in include_list: # f = open(os.path.join(sdir,file)) # includes += f.read(-1) # f.close() # heading and tail of the module definition. file_prefix = "\npython module " + module_name +" ! in\n" \ " interface \n" file_suffix = "\n end interface\n" \ "end module %s" % module_name #return pre_prefix + includes + file_prefix + interface_in + file_suffix return pre_prefix + file_prefix + interface_in + file_suffix def generate_clapack(sdir,output_path='.'): print "generating clapack interface" f = open(os.path.join(sdir,'generic_clapack.pyf')) module_name = 'clapack' generic_interface = f.read(-1) f.close() generic_interface = generic_expand(generic_interface) module_def = interface_to_module(generic_interface,module_name) f = open(os.path.join(output_path,module_name+'.pyf'),'w') f.write(module_def) f.close() def generate_flapack(sdir,output_path='.'): print "generating flapack interface" f = open(os.path.join(sdir,'generic_flapack.pyf')) module_name = 'flapack' generic_interface = f.read(-1) f.close() generic_interface = generic_expand(generic_interface) module_def = interface_to_module(generic_interface,module_name) f = open(os.path.join(output_path,module_name+'.pyf'),'w') f.write(module_def) f.close() def process_all(): # process the standard files. generate_clapack('.') generate_flapack('.') if __name__ == "__main__": process_all() -------------- next part -------------- !%f90 -*- f90 -*- python module generic_flapack interface subroutine gesv(n,nrhs,A,piv,B,info) ! LU,piv,X,info = gesv(A,B,copy_A=1,copy_B=1) ! Solve A * X = B. ! A = P * L * U ! U is upper diagonal triangular, L is unit lower triangular, ! piv pivots columns. callstatement (*f2py_func)(&n,&nrhs,A,&n,piv,B,&n,&info) integer depend(A),intent(hide):: n = shape(A,0) integer depend(B),intent(hide):: nrhs = shape(B,1) dimension(n,n),check(shape(A,0)==shape(A,1)) :: A integer dimension(n),depend(n),intent(out) :: piv dimension(n,nrhs),check(shape(A,0)==shape(B,0)),depend(n) :: B integer intent(out)::info intent(in,out,copy,out=X) B intent(in,out,copy,out=LU) A end subroutine gesv subroutine getrf(m,n,A,piv,info) ! LU,piv,info = getrf(A,copy_A=1) ! Compute an LU factorization of a general M-by-N matrix A. ! A = P * L * U callstatement (*f2py_func)(&m,&n,A,&m,piv,&info) integer depend(A),intent(hide):: m = shape(A,0) integer depend(A),intent(hide):: n = shape(A,1) dimension(m,n),intent(in,out,copy,out=LU) :: A integer dimension((mgetrf subroutine getrs(n,nrhs,LU,piv,B,info,trans) ! X,info = getrs(LU,piv,B,trans=0,copy_B=1) ! Solve A * X = B if trans=0 ! Solve A^T * X = B if trans=1 ! Solve A^H * X = B if trans=2 ! A = P * L * U callstatement (*f2py_func)((trans?(trans==2?'C':'T'):'N'),&n,&nrhs,LU,&n,piv,B,&n,&info) integer optional,intent(in),check(trans>=0 && trans <=2) :: trans = 0 integer depend(LU),intent(hide):: n = shape(LU,0) integer depend(B),intent(hide):: nrhs = shape(B,1) dimension(n,n),intent(in) :: LU check(shape(LU,0)==shape(LU,1)) :: LU integer dimension(n),intent(in),depend(n) :: piv dimension(n,nrhs),intent(in,out,copy,out=X),depend(n),check(shape(LU,0)==shape(B,0)) :: B integer intent(out):: info end subroutine getrs subroutine getri(n,LU,piv,work,lwork,info) ! invA,info = getri(LU,piv,copy_LU=1) ! Find A inverse A^-1. ! A = P * L * U callstatement (*f2py_func)(&n,LU,&n,piv,work,&lwork,&info) integer depend(LU),intent(hide):: n = shape(LU,0) dimension(n,n),intent(in,out,copy,out=invA) :: LU check(shape(LU,0)==shape(LU,1)) :: LU integer dimension(n),intent(in),depend(n) :: piv integer intent(out):: info integer optional,intent(in),depend(n),check(lwork>=n) :: lwork=3*n dimension(lwork),intent(hide),depend(lwork) :: work end subroutine getri subroutine posv(n,nrhs,A,B,info,lower) ! C,X,info = posv(A,B,lower=0,copy_A=1,copy_B=1) ! Solve A * X = B. ! A is symmetric positive defined ! A = U^T * U, C = U if lower = 0 ! A = L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. callstatement (*f2py_func)((lower?'L':'U'),&n,&nrhs,A,&n,B,&n,&info) integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(A),intent(hide):: n = shape(A,0) integer depend(B),intent(hide):: nrhs = shape(B,1) dimension(n,n),intent(in,out,copy,out=C) :: A check(shape(A,0)==shape(A,1)) :: A dimension(n,nrhs),intent(in,out,copy,out=X),depend(n):: B check(shape(A,0)==shape(B,0)) :: B integer intent(out) :: info end subroutine posv subroutine potrf(n,A,info,lower) ! C,info = potrf(A,lower=0=1,copy_A=1) ! Compute Cholesky decomposition of symmetric positive defined matrix: ! A = U^T * U, C = U if lower = 0 ! A = L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. callstatement (*f2py_func)((lower?'L':'U'),&n,A,&n,&info) integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(A),intent(hide):: n = shape(A,0) dimension(n,n),intent(in,out,copy,out=C) :: A check(shape(A,0)==shape(A,1)) :: A integer intent(out) :: info end subroutine potrf subroutine potrs(n,nrhs,C,B,info,lower) ! X,info = potrs(C,B,lower=0=1,copy_B=1) ! Solve A * X = B. ! A is symmetric positive defined ! A = U^T * U, C = U if lower = 0 ! A = L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. callstatement (*f2py_func)((lower?'L':'U'),&n,&nrhs,C,&n,B,&n,&info) integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(C),intent(hide):: n = shape(C,0) integer depend(B),intent(hide):: nrhs = shape(B,1) dimension(n,n),intent(in) :: C check(shape(C,0)==shape(C,1)) :: C dimension(n,nrhs),intent(in,out,copy,out=X),depend(n):: B check(shape(C,0)==shape(B,0)) :: B integer intent(out) :: info end subroutine potrs subroutine potri(n,C,info,lower) ! invA,info = potri(C,lower=0,copy_C=1) ! Compute A inverse A^-1. ! A = U^T * U, C = U if lower = 0 ! A = L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. callstatement (*f2py_func)((lower?'L':'U'),&n,C,&n,&info) integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(C),intent(hide):: n = shape(C,0) dimension(n,n),intent(c,in,out,copy,out=invA) :: C check(shape(C,0)==shape(C,1)) :: C integer intent(out) :: info end subroutine potri subroutine lauum(n,C,info,lower) ! A,info = lauum(C,lower=0,copy_C=1) ! Compute product ! U^T * U, C = U if lower = 0 ! L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. callstatement (*f2py_func)((lower?'L':'U'),&n,C,&n,&info) integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(C),intent(hide):: n = shape(C,0) dimension(n,n),intent(in,out,copy,out=A) :: C check(shape(C,0)==shape(C,1)) :: C integer intent(out) :: info end subroutine lauum subroutine trtri(n,C,info,lower,unitdiag) ! invC,info = trtri(C,lower=0,unitdiag=1,copy_C=1) ! Compute C inverse C^-1 where ! C = U if lower = 0 ! C = L if lower = 1 ! C is non-unit triangular matrix if unitdiag = 0 ! C is unit triangular matrix if unitdiag = 1 callstatement (*f2py_func)((lower?'L':'U'),(unitdiag?'U':'N'),&n,C,&n,&info) integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer optional,intent(in),check(unitdiag==0||unitdiag==1) :: unitdiag = 0 integer depend(C),intent(hide):: n = shape(C,0) dimension(n,n),intent(in,out,copy,out=invC) :: C check(shape(C,0)==shape(C,1)) :: C integer intent(out) :: info end subroutine trtri end interface end python module generic_flapack ! This file was auto-generated with f2py (version:2.10.173). ! See http://cens.ioc.ee/projects/f2py2e/ -------------- next part -------------- !%f90 -*- f90 -*- python module generic_clapack interface function gesv(n,nrhs,A,piv,B,info,rowmajor) ! LU,piv,X,info = gesv(A,B,rowmajor=1,copy_A=1,copy_B=1) ! Solve A * X = B. ! A * P = L * U ! U is unit upper diagonal triangular, L is lower triangular, ! piv pivots columns. fortranname clapack_gesv integer intent(c,hide) :: gesv callstatement info = (*f2py_func)(102-rowmajor,n,nrhs,A,n,piv,B,n) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer depend(A),intent(hide):: n = shape(A,0) integer depend(B),intent(hide):: nrhs = shape(B,1) dimension(n,n),check(shape(A,0)==shape(A,1)) :: A integer dimension(n),depend(n),intent(out) :: piv dimension(n,nrhs),check(shape(A,0)==shape(B,0)),depend(n) :: B integer intent(out)::info intent(in,out,copy,out=X) B intent(c,in,out,copy,out=LU) A end function gesv function getrf(m,n,A,piv,info,rowmajor) ! LU,piv,info = getrf(A,rowmajor=1,copy_A=1) ! Compute an LU factorization of a general M-by-N matrix A. ! A * P = L * U fortranname clapack_getrf integer intent(c,hide) :: getrf callstatement info = (*f2py_func)(102-rowmajor,m,n,A,(rowmajor?n:m),piv) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer depend(A),intent(hide):: m = shape(A,0) integer depend(A),intent(hide):: n = shape(A,1) dimension(m,n),intent(c,in,out,copy,out=LU) :: A integer dimension((mgetrf function getrs(n,nrhs,LU,piv,B,info,trans,rowmajor) ! X,info = getrs(LU,piv,B,trans=0,rowmajor=1,copy_B=1) ! Solve A * X = B if trans=0 ! Solve A^T * X = B if trans=1 ! Solve A^H * X = B if trans=2 ! A * P = L * U fortranname clapack_getrs integer intent(c,hide) :: getrs callstatement info = (*f2py_func)(102-rowmajor,110+trans,n,nrhs,LU,n,piv,B,n) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer optional,intent(in),check(trans>=0 && trans <=2) :: trans = 0 integer depend(LU),intent(hide):: n = shape(LU,0) integer depend(B),intent(hide):: nrhs = shape(B,1) dimension(n,n),intent(c,in) :: LU check(shape(LU,0)==shape(LU,1)) :: LU integer dimension(n),intent(in),depend(n) :: piv dimension(n,nrhs),intent(in,out,copy,out=X),depend(n),check(shape(LU,0)==shape(B,0)) :: B integer intent(out):: info end function getrs function getri(n,LU,piv,info,rowmajor) ! invA,info = getri(LU,piv,rowmajor=1,copy_LU=1) ! Find A inverse A^-1. ! A * P = L * U fortranname clapack_getri integer intent(c,hide) :: getri callstatement info = (*f2py_func)(102-rowmajor,n,LU,n,piv) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer depend(LU),intent(hide):: n = shape(LU,0) dimension(n,n),intent(c,in,out,copy,out=invA) :: LU check(shape(LU,0)==shape(LU,1)) :: LU integer dimension(n),intent(in),depend(n) :: piv integer intent(out):: info end function getri function posv(n,nrhs,A,B,info,lower,rowmajor) ! C,X,info = posv(A,B,lower=0,rowmajor=1,copy_A=1,copy_B=1) ! Solve A * X = B. ! A is symmetric positive defined ! A = U^T * U, C = U if lower = 0 ! A = L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. fortranname clapack_posv integer intent(c,hide) :: posv callstatement info = (*f2py_func)(102-rowmajor,121+lower,n,nrhs,A,n,B,n) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(A),intent(hide):: n = shape(A,0) integer depend(B),intent(hide):: nrhs = shape(B,1) dimension(n,n),intent(c,in,out,copy,out=C) :: A check(shape(A,0)==shape(A,1)) :: A dimension(n,nrhs),intent(in,out,copy,out=X),depend(n):: B check(shape(A,0)==shape(B,0)) :: B integer intent(out) :: info end function posv function potrf(n,A,info,lower,rowmajor) ! C,info = potrf(A,lower=0,rowmajor=1,copy_A=1) ! Compute Cholesky decomposition of symmetric positive defined matrix: ! A = U^T * U, C = U if lower = 0 ! A = L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. fortranname clapack_potrf integer intent(c,hide) :: potrf callstatement info = (*f2py_func)(102-rowmajor,121+lower,n,A,n) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(A),intent(hide):: n = shape(A,0) dimension(n,n),intent(c,in,out,copy,out=C) :: A check(shape(A,0)==shape(A,1)) :: A integer intent(out) :: info end function potrf function potrs(n,nrhs,C,B,info,lower,rowmajor) ! X,info = potrs(C,B,lower=0,rowmajor=1,copy_B=1) ! Solve A * X = B. ! A is symmetric positive defined ! A = U^T * U, C = U if lower = 0 ! A = L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. fortranname clapack_potrs integer intent(c,hide) :: potrs callstatement info = (*f2py_func)(102-rowmajor,121+lower,n,nrhs,C,n,B,n) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(C),intent(hide):: n = shape(C,0) integer depend(B),intent(hide):: nrhs = shape(B,1) dimension(n,n),intent(c,in) :: C check(shape(C,0)==shape(C,1)) :: C dimension(n,nrhs),intent(in,out,copy,out=X),depend(n):: B check(shape(C,0)==shape(B,0)) :: B integer intent(out) :: info end function potrs function potri(n,C,info,lower,rowmajor) ! invA,info = potri(C,lower=0,rowmajor=1,copy_C=1) ! Compute A inverse A^-1. ! A = U^T * U, C = U if lower = 0 ! A = L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. fortranname clapack_potri integer intent(c,hide) :: potri callstatement info = (*f2py_func)(102-rowmajor,121+lower,n,C,n) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(C),intent(hide):: n = shape(C,0) dimension(n,n),intent(c,in,out,copy,out=invA) :: C check(shape(C,0)==shape(C,1)) :: C integer intent(out) :: info end function potri function lauum(n,C,info,lower,rowmajor) ! A,info = lauum(C,lower=0,rowmajor=1,copy_C=1) ! Compute product ! U^T * U, C = U if lower = 0 ! L * L^T, C = L if lower = 1 ! C is triangular matrix of the corresponding Cholesky decomposition. fortranname clapack_lauum integer intent(c,hide) :: lauum callstatement info = (*f2py_func)(102-rowmajor,121+lower,n,C,n) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer depend(C),intent(hide):: n = shape(C,0) dimension(n,n),intent(c,in,out,copy,out=A) :: C check(shape(C,0)==shape(C,1)) :: C integer intent(out) :: info end function lauum function trtri(n,C,info,lower,unitdiag,rowmajor) ! invC,info = trtri(C,lower=0,unitdiag=0,rowmajor=1,copy_C=1) ! Compute C inverse C^-1 where ! C = U if lower = 0 ! C = L if lower = 1 ! C is non-unit triangular matrix if unitdiag = 0 ! C is unit triangular matrix if unitdiag = 1 fortranname clapack_trtri integer intent(c,hide) :: trtri callstatement info = (*f2py_func)(102-rowmajor,121+lower,131+unitdiag,n,C,n) integer optional,intent(in),check(rowmajor==1||rowmajor==0) :: rowmajor = 1 integer optional,intent(in),check(lower==0||lower==1) :: lower = 0 integer optional,intent(in),check(unitdiag==0||unitdiag==1) :: unitdiag = 0 integer depend(C),intent(hide):: n = shape(C,0) dimension(n,n),intent(c,in,out,copy,out=invC) :: C check(shape(C,0)==shape(C,1)) :: C integer intent(out) :: info end function trtri end interface end python module generic_clapack ! This file was auto-generated with f2py (version:2.10.173). ! See http://cens.ioc.ee/projects/f2py2e/ From eric at scipy.org Sat Jan 19 14:10:53 2002 From: eric at scipy.org (eric) Date: Sat, 19 Jan 2002 14:10:53 -0500 Subject: [SciPy-dev] New clapack and flapack References: Message-ID: <008601c1a11d$07af5860$777ba8c0@ericlaptop> Hey Pearu, Jeeze. You work fast. I haven't even had a chance to grab the new f2py yet... I'm guessing I need to update f2py from CVS to try all this out. I'll try and spend some time with this today, but it may be tomorrow before I have a chance to take a look. > 1. > $ python interface_gen.py # this creates clapack.pyf and flapack.pyf > > 2. > $ f2py -c lapack.pyf -L/usr/local/lib/atlas -llapack -lf77blas -lcblas -latlas -lg2c > > $ f2py -c flapack.pyf -L/usr/local/lib/atlas -llapack -lf77blas -lcblas -latlas -lg2c > > This builds clapack.so and flapack.so. Looks easy enough. f2py is really gotten things pretty automated. > 3. > LU,piv,X,info = sgesv(A,B,rowmajor=1,copy_A=1,copy_B=1) > LU,piv,info = sgetrf(A,rowmajor=1,copy_A=1) A few points about naming. 1. SciPy has settled on all lower case convention for consistency across interfaces. So using: lu,piv,x,info = sgesv(a,b,rowmajor=1,copy_a=1,copy_b=1) will fit the scheme. 2. Second, is "copy_A" a f2py generated name or can we specify it? I prefer overwrite = 0 to copy=1 because it is more explicit about what is happening. It really warns people that they are clobbering data when they choose to set the flag. Also, the "A" part of copy_A will be ambiguous during usage. In the worst case for the sgesv example, imagine someone does the following: >>> a = rhs_vector >>> b = lhs_matrix >>> lu,piv,x,info = sgesv(b,a,copy_a=1,copy_b=0) If the person is trying to save their variable "a" here, they didn't do it, because the "copy_a" is refering to the function's argument "a", not the variable bound to it which is "b" in this example. I ran into this conundrum once when writing an interface generator in the past. It be better if the names somehow were more descriptive. For sgetrf, the following seems fine: lu,piv,info = sgetrf(a,overwrite=0,rowmajor=1) For sgesv, the naming isn't quite as obvious. Maybe, lu,piv,x,info = sgesv(a,b,overwrite_rhs=0,overwrite_lhs=0) 3. In the C API's, I think the overwrite argument will be used much more than the rowmajor flag. It should come first in the argument list. In fact, in the "old" wrappers, rowmajor was just hardcoded to 1 because the flalpack routines were considered the colummajor functions. Keeping it is fine, it just should go to the end. >>> lu,piv,info = sgetrf(a, overwrite=0, rowmajor=1) Not sure how hard these changes are -- I expect they might require some enhancements to f2py. I'm not opposed to having Python wrappers around the routines that help with this stuff. We used to have to do that for the clapack wrappers (until your additions to f2py's capabilities made this unnecessary). If it is easy to get f2py to do this stuff, then, of course, that is better. Comment on testing. There was a fairly complete set of tests written for the *interfaces* of the blas routines. These guys only tested how the interfaces worked and didn't do much testing of the functions accuracy. We need both. I don't remember if the LAPACK tests were as complete -- I certainly didn't write them, but Travis O. may have. Based on the errors we saw a week or two ago in the CLAPACK functions, we really do need to get some tests implemented for this stuff. You and I probably handle writing the interface tests (very tedious, but maybe not as bad with all your changes to f2py), but we really should find an expert to put the linear algebra routines through their paces. We can write some "smoke tests" though that make sure LU and solvers, etc. work for general problems. I'd stick this way up there as one of the most important packages in SciPy, so we should really try and make sure it works right! Maybe there is a test set of matrices laying around somewhere to help us out in all this. Anyone know of such a thing? -- perhaps in the LAPACK distribution itself. > Few remarks: > 1) The sources to these extension modules are quite big (more than 5000 > lines in both). Should we factor these modules to > cslapack.so cclapack.so cdlapack.so czlapack.so > fslapack.so fclapack.so fdlapack.so fzlapack.so > (any better name convention?). What do you think? Well, I think in practice, they'll rarely be needed separately. Having big wrappers isn't that big a deal (heck, I've had SWIG wrappers that were 60000 lines...). I'd vote for keeping them all together unless there is a compelling reason besides size to split them. This keeps things a little simpler (import once, get everything). > 2) How shall I commit all this to the scipy CVS repository? In a > subdirectory of linalg? Eventually it should be in linalg directory, I > think. We'll since we don't know how to do CVS branches reliably... :| Why don't you make a directory called linalg2. We'll work in there until it gets stable. Is there a CVS expert out there that can help us out on this? Pearu, this all looking very promising. Thanks a ton. eric From eric at scipy.org Sat Jan 19 14:57:15 2002 From: eric at scipy.org (eric) Date: Sat, 19 Jan 2002 14:57:15 -0500 Subject: [SciPy-dev] lapack and blas References: Message-ID: <00d601c1a123$8137ce50$777ba8c0@ericlaptop> Hey Pearu, > However, a bad news is that lapack and atlas lapack functions are not > compatible. > For example, > >>> LU,ipiv,X,info=flapack.c_sgesv([[1,2],[3,4]],[1,1]) > >>> print X > [-0.5 0.5] For some reason, this is solving using the transpose of the matrix. 1 3 x1 = 1 --> x1 = -0.5 2 4 x2 1 x2 = 0.5 Is this a problem with c_sgesv, or is f2py doing something incorrectly with the matrix transpose stuff? I need to look at the magic your doing under the covers to handle all this, since I haven't figured out how you'd do it completely transparently in a way that never leads to ambiguity and also maintains efficiency. Still, this is wrapping the C version of the function, so it should be the "trivial" case since nothing special needs to be done to the matrix. > >>> LU,ipiv,X,info=flapack.f_sgesv([[1,2],[3,4]],[1,1]) > >>> print X > [-0.99999994 0.99999994] As you know, this is the correct answer. eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu at cens.ioc.ee Sat Jan 19 16:02:44 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sat, 19 Jan 2002 23:02:44 +0200 (EET) Subject: [SciPy-dev] New clapack and flapack In-Reply-To: <008601c1a11d$07af5860$777ba8c0@ericlaptop> Message-ID: On Sat, 19 Jan 2002, eric wrote: > Hey Pearu, > > Jeeze. You work fast. I haven't even had a chance to grab the new f2py > yet... I'm guessing I need to update f2py from CVS to try all this out. > I'll try and spend some time with this today, but it may be tomorrow before > I have a chance to take a look. I made also a tarball available with the latest changes. > > 3. > > LU,piv,X,info = sgesv(A,B,rowmajor=1,copy_A=1,copy_B=1) > > LU,piv,info = sgetrf(A,rowmajor=1,copy_A=1) > > A few points about naming. > > 1. > SciPy has settled on all lower case convention for consistency across > interfaces. So using: > > lu,piv,x,info = sgesv(a,b,rowmajor=1,copy_a=1,copy_b=1) > > will fit the scheme. Ok, that's not a problem. It is easy to fix by query-replace in emacs... Just thought that capitalized arguments will look more like matrices. > 2. > Second, is "copy_A" a f2py generated name or can we specify it? I prefer > overwrite = 0 to copy=1 because it is more explicit about what is happening. > It really warns people that they are clobbering data when they choose to set > the flag. Also, the "A" part of copy_A will be ambiguous during usage. In > the worst case for the sgesv example, imagine someone does the following: > > >>> a = rhs_vector > >>> b = lhs_matrix > >>> lu,piv,x,info = sgesv(b,a,copy_a=1,copy_b=0) > > If the person is trying to save their variable "a" here, they didn't do it, > because the "copy_a" is refering to the function's argument "a", not the > variable bound to it which is "b" in this example. I ran into this > conundrum once when writing an interface generator in the past. > > It be better if the names somehow were more descriptive. For sgetrf, the > following seems fine: > > lu,piv,info = sgetrf(a,overwrite=0,rowmajor=1) > > For sgesv, the naming isn't quite as obvious. Maybe, > > lu,piv,x,info = sgesv(a,b,overwrite_rhs=0,overwrite_lhs=0) copy_A is a f2py generated name. overwrite certainly is more suitable to warn users of an in situ changes (I assume that intent(copy) is not used for arguments that are not changed in situ by the wrapped function). However, intent(copy), or should I call it now intent(overwrite), may have wider applications than wrapping lapack. There overwrite arguments may be used differently, that is, for one argument this flag is set to 0, and for the other one, to 1. So, I would like to keep overwrite_ rather than having a global overwrite keyword argument. > 3. > In the C API's, I think the overwrite argument will be used much more than > the rowmajor flag. It should come first in the argument list. In fact, in > the "old" wrappers, rowmajor was just hardcoded to 1 because the flalpack > routines were considered the colummajor functions. Keeping it is fine, it > just should go to the end. > > >>> lu,piv,info = sgetrf(a, overwrite=0, rowmajor=1) > > Not sure how hard these changes are -- I expect they might require some > enhancements to f2py. I'm not opposed to having Python wrappers around the > routines that help with this stuff. We used to have to do that for the > clapack wrappers (until your additions to f2py's capabilities made this > unnecessary). If it is easy to get f2py to do this stuff, then, of course, > that is better. Since overwrite argument is f2py generated and rowmajor is specified in the routine signature, you see that rowmajor becomes first (f2py generated arguments are alway appended to the end of argument list). I agree that rowmajor will be probably rarely used but I don't have any good ideas how to tell f2py about user-specified argument ordering. I looked to it and found that the required changes are not just quick hacks. On the other hand. I don't think that lapack or blas routines will be often used in interactive work. They will be more like to be used in nice interfaces such as solve() etc that can have arguments ordered as we want. In using f2py wrappers, I would always recommend >>> lu,piv,info = sgetrf(a, overwrite_a=0) instead of >>> lu,piv,info = sgetrf(a, 0) that is less informative. > Comment on testing. > There was a fairly complete set of tests written for the *interfaces* of the > blas routines. These guys only tested how the interfaces worked and didn't > do much testing of the functions accuracy. We need both. I don't remember > if the LAPACK tests were as complete -- I certainly didn't write them, but > Travis O. may have. > > Based on the errors we saw a week or two ago in the CLAPACK functions, we > really do need to get some tests implemented for this stuff. You and I > probably handle writing the interface tests (very tedious, but maybe not as > bad with all your changes to f2py), but we really should find an expert to > put the linear algebra routines through their paces. We can write some > "smoke tests" though that make sure LU and solvers, etc. work for general > problems. I'd stick this way up there as one of the most important packages > in SciPy, so we should really try and make sure it works right! > > Maybe there is a test set of matrices laying around somewhere to help us out > in all this. Anyone know of such a thing? -- perhaps in the LAPACK > distribution itself. Yes, testing is important. I don't dare to claim that f2py is bug free ;-) but testing whether the interface works should not be the main goal for linalg, indeed. More important is that the interface acts according to its documentation. I mean here things like whether getrf really returns mathematically correct LU decomposition (leaving aside rounding errors). If it would return, for example, a transpose LU (which would be incorrect), then passing LU to getrs (that may have the same bug), the final result may be correct. I looked already the tests in the lapack distribution but they seem rather difficult to use for us. These tests seem to be "hardcoded" to the Fortran files and extracting the test data from there is not simple. Testing linalg stuff is not a small or trivial project. Any one wants to volunteer? > > Few remarks: > > 1) The sources to these extension modules are quite big (more than 5000 > > lines in both). Should we factor these modules to > > cslapack.so cclapack.so cdlapack.so czlapack.so > > fslapack.so fclapack.so fdlapack.so fzlapack.so > > (any better name convention?). What do you think? > > Well, I think in practice, they'll rarely be needed separately. Having big > wrappers isn't that big a deal (heck, I've had SWIG wrappers that were 60000 > lines...). I'd vote for keeping them all together unless there is a > compelling reason besides size to split them. This keeps things a little > simpler (import once, get everything). That issue is then solved. Good. > > 2) How shall I commit all this to the scipy CVS repository? In a > > subdirectory of linalg? Eventually it should be in linalg directory, I > > think. > > We'll since we don't know how to do CVS branches reliably... :| Why don't > you make a directory called linalg2. We'll work in there until it gets > stable. Ok. > Pearu, this all looking very promising. Thanks a ton. Your are very welcome. Pearu From eric at scipy.org Sat Jan 19 15:02:09 2002 From: eric at scipy.org (eric) Date: Sat, 19 Jan 2002 15:02:09 -0500 Subject: [SciPy-dev] scipy on cygwin References: <86g053mmsc.fsf@bock.chem.unc.edu> <15432.60012.543319.58795@monster.linux.in> Message-ID: <00e201c1a124$2d824050$777ba8c0@ericlaptop> Prabhu, If you know the fix, go ahead and make it. eric ----- Original Message ----- From: "Prabhu Ramachandran" To: Sent: Friday, January 18, 2002 10:39 PM Subject: [SciPy-dev] scipy on cygwin > >>>>> "JK" == Jochen K?pper writes: > > JK> not found | >>> from scipy import scipy_version | Traceback > JK> (most recent call last): | File "", line 1, in ? | > JK> ImportError: cannot import name scipy_version | >>> import > JK> scipy.scipy_version | >>> `---- > > JK> Anybody seen that? Hints? > > I get the same scipy_version problem. It happens thanks to this in > __init__.py > > #get version > import scipy_version > __version__ = scipy_version.scipy_version > del scipy_version > > Instead you should try to print scipy.__version__ > > prabhu > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From pearu at cens.ioc.ee Sat Jan 19 16:13:57 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sat, 19 Jan 2002 23:13:57 +0200 (EET) Subject: [SciPy-dev] lapack and blas In-Reply-To: <00d601c1a123$8137ce50$777ba8c0@ericlaptop> Message-ID: Hi Eric, On Sat, 19 Jan 2002, eric wrote: > > However, a bad news is that lapack and atlas lapack functions are not > > compatible. > > > For example, > > >>> LU,ipiv,X,info=flapack.c_sgesv([[1,2],[3,4]],[1,1]) > > >>> print X > > [-0.5 0.5] > > For some reason, this is solving using the transpose of the matrix. > > 1 3 x1 = 1 --> x1 = -0.5 > 2 4 x2 1 x2 = 0.5 > > Is this a problem with c_sgesv, or is f2py doing something incorrectly > with the matrix transpose stuff? I need to look at the magic your doing > under the covers to handle all this, since I haven't figured out how > you'd do it completely transparently in a way that never leads to > ambiguity and also maintains efficiency. I'd appreciate if you could look at the magic that f2py generates. I welcome any critisism and I am happy to explain the reasons why I did this or that. You may find there something that I have missed... However, this issue should be solved in the latest clapack and flapack interfaces (the problem was related to sgesv handeling the rhs matrix in the column-major storage order). Here is an example: >>> clapack.dgesv([[1,2],[3,4]],[[1,2,1],[2,0,2]])[2] array([[ 0. , -4. , 0. ], [ 0.5, 3. , 0.5]]) >>> flapack.dgesv([[1,2],[3,4]],[[1,2,1],[2,0,2]])[2] array([[ 0. , -4. , 0. ], [ 0.5, 3. , 0.5]]) that show the solution matrix X for [ 1 2 ] [ 1 2 1 ] | | * X = | | [ 3 4 ] [ 2 0 2 ] calculated using atlas C lapack and Fortran lapack, respectively. Regards, Pearu From pearu at cens.ioc.ee Sat Jan 19 17:31:23 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sun, 20 Jan 2002 00:31:23 +0200 (EET) Subject: [SciPy-dev] New clapack and flapack In-Reply-To: Message-ID: > On Sat, 19 Jan 2002, eric wrote: > > We'll since we don't know how to do CVS branches reliably... :| Why don't > > you make a directory called linalg2. We'll work in there until it gets > > stable. I have commited linalg2 with the new clapack and flapack pyf files to scipy CVS repository. You'll need to get also the latest f2py that contains the change copy_=1 -> overwrite_=0 for variables that has intent(copy) or intent(overwrite) attributes defined. Regards, Pearu From prabhu at aero.iitm.ernet.in Sat Jan 19 23:15:57 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sun, 20 Jan 2002 09:45:57 +0530 Subject: [SciPy-dev] scipy on cygwin In-Reply-To: <00e201c1a124$2d824050$777ba8c0@ericlaptop> References: <86g053mmsc.fsf@bock.chem.unc.edu> <15432.60012.543319.58795@monster.linux.in> <00e201c1a124$2d824050$777ba8c0@ericlaptop> Message-ID: <15434.17533.18750.926925@monster.linux.in> >>>>> "eric" == eric writes: eric> Prabhu, If you know the fix, go ahead and make it. I'm not sure that it is a bug. The del statement in scipy's __init__ makes the behaviour somewhat funny. All of the following work: In [1]: import scipy In [2]: from scipy.scipy_version import scipy_version In [3]: scipy_version Out[3]= '0.2.0' In [7]: print scipy.__version__ 0.2.0 The following wont work though In [8]: print scipy.scipy_version --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) ? AttributeError: 'scipy' module has no attribute 'scipy_version' The reason is that the attribute is deleted in __init__.py. Why the attribute is deleted I dont know but if the version is all that is required scipy.__version__ seems to make more sense that scipy.scipy_version etc. So this is not a bug afaik. :) prabhu From eric at scipy.org Sun Jan 20 05:35:44 2002 From: eric at scipy.org (eric) Date: Sun, 20 Jan 2002 05:35:44 -0500 Subject: [SciPy-dev] Info on what needs to be changed in the stats module Message-ID: <00eb01c1a19e$3784cf80$777ba8c0@ericlaptop> Hey group, Questions on the stats module keep coming up, and it really needs some work. I don't have time to do it, but I've tried to outline what changes need to be made. They are detailed here: http://www.scipy.org/Members/eric/stats_notes.html If someone picked this up, the world would be a better place. Any takers? eric -- Eric Jones Enthought, Inc. [www.enthought.com and www.scipy.org] (512) 536-1057 From eric at scipy.org Mon Jan 21 00:08:44 2002 From: eric at scipy.org (eric) Date: Mon, 21 Jan 2002 00:08:44 -0500 Subject: [SciPy-dev] Anyone out there have the latest ATLAS (3.1.13) for windows/PIII Message-ID: <006b01c1a239$b389e980$777ba8c0@ericlaptop> Hey group, Some errors have turned up in older versions of ATLAS, with the linalg module. Pearu (or was it Travis O.?) reported that they went away with 3.1.13. Problem, is the build for this version fails on my machine. https://sourceforge.net/projects/math-atlas/ I was wondering if anyone else out there has a successful build on a Windows machine with a Pentium III that links with SciPy. Or, if you have this architecture and cygwin, could you try downloading and building ATLAS? I'd like to find out if it is my machine setup or the ATLAS build process. thanks, eric -- Eric Jones Enthought, Inc. [www.enthought.com and www.scipy.org] (512) 536-1057 From a.schmolck at gmx.net Mon Jan 21 10:05:00 2002 From: a.schmolck at gmx.net (A.Schmolck) Date: 21 Jan 2002 15:05:00 +0000 Subject: [SciPy-dev] Patch for broken scipy.plt.plot_objects.py Message-ID: Hi, I didn't find anything specific on the www.scipy.org on how to submit patches, so I hope submitting them here is OK. I hit upon a bug in plt.imagesc, that took me quite a long time to track down. Basically, it won't display any non-square matrices correctly. However, reshaping a flat array (which I happen to mostly work with), incorrectly (i.e. reshape(a.flat,(col, row)) instead of reshape(a.flat, (row, col))), will result in the plot one expected. I was therefore quite dumbfounded when I found that plt.imagesc(reshape(a.flat,(col, row))) worked but plt.imagesc(transpose(reshape(a.flat,(col,row)))) produced garbage. Here is a fix (plus anoter small change), which I hope should work. -------------- next part -------------- A non-text attachment was scrubbed... Name: plot_objects.py.PATCH Type: text/x-patch Size: 1867 bytes Desc: patch for plot_objects.py URL: -------------- next part -------------- alex -- Alexander Schmolck Postgraduate Research Student Department of Computer Science University of Exeter A.Schmolck at gmx.net http://www.dcs.ex.ac.uk/people/aschmolc/ From jochen at jochen-kuepper.de Mon Jan 21 22:06:33 2002 From: jochen at jochen-kuepper.de (Jochen =?iso-8859-1?q?K=FCpper?=) Date: 21 Jan 2002 22:06:33 -0500 Subject: [SciPy-dev] Anyone out there have the latest ATLAS (3.1.13) for windows/PIII In-Reply-To: <006b01c1a239$b389e980$777ba8c0@ericlaptop> References: <006b01c1a239$b389e980$777ba8c0@ericlaptop> Message-ID: On Mon, 21 Jan 2002 00:08:44 -0500 eric wrote: Eric> Some errors have turned up in older versions of ATLAS, with the Eric> linalg module. Pearu (or was it Travis O.?) reported that they Eric> went away with 3.1.13. Problem, is the build for this version Eric> fails on my machine. I assume you're talking about 3.3.13. Builds fine on an up2date generic Cygwin system (Win2000) on an Athlon. Just tested it earlier today. I can send you the libs if you want. Greetings, Jochen -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: 44BCCD8E Sex, drugs and rock-n-roll From eric at scipy.org Mon Jan 21 23:58:57 2002 From: eric at scipy.org (eric) Date: Mon, 21 Jan 2002 23:58:57 -0500 Subject: [SciPy-dev] Anyone out there have the latest ATLAS (3.1.13) for windows/PIII References: <006b01c1a239$b389e980$777ba8c0@ericlaptop> Message-ID: <001a01c1a301$7fccfd80$777ba8c0@ericlaptop> > On Mon, 21 Jan 2002 00:08:44 -0500 eric wrote: > > Eric> Some errors have turned up in older versions of ATLAS, with the > Eric> linalg module. Pearu (or was it Travis O.?) reported that they > Eric> went away with 3.1.13. Problem, is the build for this version > Eric> fails on my machine. > > I assume you're talking about 3.3.13. Builds fine on an up2date > generic Cygwin system (Win2000) on an Athlon. Just tested it earlier > today. Ok. Maybe I need to update cygwin -- I'll try that. I don't think that is the problem though. > I can send you the libs if you want. Since they are Athlon, won't they have the 3DNow stuff in them? If so, I don't think they'll run on my PIII. Thanks though. Maybe someone else has the same for a PIII? It'd be better if updating cygwin solves my problem. see ya, eric From jochen at jochen-kuepper.de Tue Jan 22 01:46:33 2002 From: jochen at jochen-kuepper.de (Jochen =?iso-8859-1?q?K=FCpper?=) Date: 22 Jan 2002 01:46:33 -0500 Subject: [SciPy-dev] Anyone out there have the latest ATLAS (3.1.13) for windows/PIII In-Reply-To: <001a01c1a301$7fccfd80$777ba8c0@ericlaptop> References: <006b01c1a239$b389e980$777ba8c0@ericlaptop> <001a01c1a301$7fccfd80$777ba8c0@ericlaptop> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon, 21 Jan 2002 23:58:57 -0500 eric wrote: Eric> Since they are Athlon, won't they have the 3DNow stuff in them? No, I go for correct vs. "even faster" -- never enable the 3DNow for that kind of stuff... Eric> It'd be better if updating cygwin solves my problem. Sure. Greetings, Jochen - -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: 44BCCD8E Sex, drugs and rock-n-roll -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8TQrJiJ/aUUS8zY4RAkoOAJ4yQ1e2QsbBy9QlzNDh/rifERwZzACgoDLR 7jD7v9EC8sTmsoSDdQiMGtg= =mX/Z -----END PGP SIGNATURE----- From andreas.frankfurt at gmx.de Tue Jan 22 13:15:52 2002 From: andreas.frankfurt at gmx.de (Andreas Lorenz) Date: Tue, 22 Jan 2002 19:15:52 +0100 Subject: [SciPy-dev] Problems using Weave Message-ID: <200201221812.g0MICHP20496@scipy.org> Hi everyone, I have installed Numeric + Weave. But when I Try to run the binary_search.py example I get the following error: Binary search for 50000 items in 100000 length list of integers: speed in python: 1.38124608994 speed of bisect: 1.17039096355 speed up: 1.18 running build_ext building 'sc_2c6cc4f1f91a224f83f8465d7fb0b46a4' extension gcc -D_FILE_OFFSET_BITS=64 -DHAVE_LARGEFILE_SUPPORT -O2 -mcpu=i486 -march=i486 - fPIC -I/home/ude6/Python/weave-0.2.1/weave -I/usr/include/python2.1 -c /home/ude 6/.python21_compiled/sc_2c6cc4f1f91a224f83f8465d7fb0b46a4.cpp -o /tmp/python21_i ntermediate/sc_2c6cc4f1f91a224f83f8465d7fb0b46a4.o /home/ude6/.python21_compiled/sc_2c6cc4f1f91a224f83f8465d7fb0b46a4.cpp: In funct ion `struct PyObject * compiled_func(PyObject *, PyObject *)': /home/ude6/.python21_compiled/sc_2c6cc4f1f91a224f83f8465d7fb0b46a4.cpp:530: impl icit declaration of function `int convert_to_list(...)' Traceback (most recent call last): File "binary_search.py", line 161, in ? search_compare(a,n) File "binary_search.py", line 126, in search_compare c_int_search(a,i) File "binary_search.py", line 56, in c_int_search return inline_tools.inline(code,['seq','t'],verbose = 2) File "../inline_tools.py", line 321, in inline auto_downcast = auto_downcast, File "../inline_tools.py", line 415, in compile_function verbose=verbose, **kw) File "../ext_tools.py", line 349, in compile verbose = verbose, **kw) File "../build_tools.py", line 194, in build_extension setup(name = module_name, ext_modules = [ext],verbose=verb) File "/var/tmp/python-root//usr/lib/python2.1/distutils/core.py", line 157, in setup build_tools.CompileError: error: command 'gcc' failed with exit status 1 Any suggestions??? Andreas Lorenz From eric at scipy.org Tue Jan 22 12:17:53 2002 From: eric at scipy.org (eric) Date: Tue, 22 Jan 2002 12:17:53 -0500 Subject: [SciPy-dev] Patch for broken scipy.plt.plot_objects.py References: Message-ID: <00b001c1a368$b9faeb40$6b01a8c0@ericlaptop> Hey Alex, > I didn't find anything specific on the www.scipy.org on how to submit patches, > so I hope submitting them here is OK. Seems as good a place as any for now. > I hit upon a bug in plt.imagesc, that took me quite a long time to track > down. Basically, it won't display any non-square matrices correctly. However, > reshaping a flat array (which I happen to mostly work with), incorrectly > (i.e. reshape(a.flat,(col, row)) instead of reshape(a.flat, (row, col))), will > result in the plot one expected. I was therefore quite dumbfounded when I > found that plt.imagesc(reshape(a.flat,(col, row))) worked but > plt.imagesc(transpose(reshape(a.flat,(col,row)))) produced garbage. > > Here is a fix (plus anoter small change), which I hope should work. All looks good. Thanks. I've added the patch to the current CVS. eric From eric at scipy.org Tue Jan 22 13:19:24 2002 From: eric at scipy.org (eric) Date: Tue, 22 Jan 2002 13:19:24 -0500 Subject: [SciPy-dev] Problems using Weave References: <200201221812.g0MICHP20496@scipy.org> Message-ID: <00f901c1a371$52195b70$6b01a8c0@ericlaptop> Thanks Andreas. Looks like some lists escaped some of the changes that were made in the last release. I added the necessary macro definitions to cxx_info.py in the CVS. I'll make new tar-balls and upload them to the web-site shortly. eric ----- Original Message ----- From: "Andreas Lorenz" To: Sent: Tuesday, January 22, 2002 1:15 PM Subject: [SciPy-dev] Problems using Weave > Hi everyone, > I have installed Numeric + Weave. But when I Try to run the binary_search.py > example I get the following error: > Binary search for 50000 items in 100000 length list of integers: > speed in python: 1.38124608994 > speed of bisect: 1.17039096355 > speed up: 1.18 > running build_ext > building 'sc_2c6cc4f1f91a224f83f8465d7fb0b46a4' extension > gcc -D_FILE_OFFSET_BITS=64 -DHAVE_LARGEFILE_SUPPORT -O2 -mcpu=i486 > -march=i486 - > fPIC -I/home/ude6/Python/weave-0.2.1/weave -I/usr/include/python2.1 -c > /home/ude > 6/.python21_compiled/sc_2c6cc4f1f91a224f83f8465d7fb0b46a4.cpp -o > /tmp/python21_i > ntermediate/sc_2c6cc4f1f91a224f83f8465d7fb0b46a4.o > /home/ude6/.python21_compiled/sc_2c6cc4f1f91a224f83f8465d7fb0b46a4.cpp: In > funct > ion `struct PyObject * compiled_func(PyObject *, PyObject *)': > /home/ude6/.python21_compiled/sc_2c6cc4f1f91a224f83f8465d7fb0b46a4.cpp:530: > impl > icit declaration of function `int convert_to_list(...)' > Traceback (most recent call last): > File "binary_search.py", line 161, in ? > search_compare(a,n) > File "binary_search.py", line 126, in search_compare > c_int_search(a,i) > File "binary_search.py", line 56, in c_int_search > return inline_tools.inline(code,['seq','t'],verbose = 2) > File "../inline_tools.py", line 321, in inline > auto_downcast = auto_downcast, > File "../inline_tools.py", line 415, in compile_function > verbose=verbose, **kw) > File "../ext_tools.py", line 349, in compile > verbose = verbose, **kw) > File "../build_tools.py", line 194, in build_extension > setup(name = module_name, ext_modules = [ext],verbose=verb) > File "/var/tmp/python-root//usr/lib/python2.1/distutils/core.py", line 157, > in > setup > build_tools.CompileError: error: command 'gcc' failed with exit status 1 > > > Any suggestions??? > > Andreas Lorenz > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From pearu at cens.ioc.ee Tue Jan 22 16:00:39 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Tue, 22 Jan 2002 23:00:39 +0200 (EET) Subject: [SciPy-dev] fblas1 Message-ID: Hi! I have commited a complete fblas1 and partially implemented fblas2 interfaces to scipy CVS repository. I also have troubles with cblas1 from atlas 3.3.13. Just too many functions from atlas just crash when I am trying to wrap them. However, using identical (well, almost) wrappers for fortran blas, they work fine. So, I don't think that these crashes are due to the interface but more like atlas generated (or may be just me;). Here is a simple example: File foo_axpy.c: /* gcc -g foo_axpy.c -L/usr/local/lib/atlas -lcblas -lf77blas -latlas -DDMALLOC -ldmalloc */ #include //#include extern cblas_saxpy(); extern saxpy_(); int main () { float x[2] = {6,2}; float *y = (float *)malloc(2*sizeof(float)); int n = 2; int incx = 1; int incy = 1; float a = 5.4; y[0] = 4.2; y[1] = 4.5; //cblas_saxpy(n,a,x,incx,y,incy); saxpy_(&n,&a,x,&incx,y,&incy); printf("y[0]=%f\n",y[0]); printf("y[0]=%f\n",y[1]); free(y); } Could someone try to compile and run this program, both with atlas and fortran version of saxpy? Do they work for you? If I am using cblas_saxpy, then the program always crashes. Can you tell what I am doing wrong here? Regards, Pearu From prabhu at aero.iitm.ernet.in Wed Jan 23 03:17:30 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Wed, 23 Jan 2002 13:47:30 +0530 Subject: [SciPy-dev] fblas1 In-Reply-To: References: Message-ID: <15438.29082.410502.61267@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> File foo_axpy.c: /* gcc -g foo_axpy.c -L/usr/local/lib/atlas PP> -lcblas -lf77blas -latlas -DDMALLOC -ldmalloc */ [snip] FWIW this does not compile. $ gcc -g foo_axpy.c -lcblas -lf77blas -latlas -DDMALLOC -ldmalloc /usr/lib/libf77blas.so: undefined reference to `e_wsfe' /usr/lib/libf77blas.so: undefined reference to `do_fio' /usr/lib/libf77blas.so: undefined reference to `s_stop' /usr/lib/libf77blas.so: undefined reference to `s_wsfe' collect2: ld returned 1 exit status prabhu From jochen at unc.edu Wed Jan 23 09:42:41 2002 From: jochen at unc.edu (Jochen =?iso-8859-1?q?K=FCpper?=) Date: 23 Jan 2002 09:42:41 -0500 Subject: [SciPy-dev] fblas1 In-Reply-To: <15438.29082.410502.61267@monster.linux.in> References: <15438.29082.410502.61267@monster.linux.in> Message-ID: <86r8ohjfha.fsf@bock.chem.unc.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 23 Jan 2002 13:47:30 +0530 Prabhu Ramachandran wrote: Prabhu> $ gcc -g foo_axpy.c -lcblas -lf77blas -latlas -DDMALLOC -ldmalloc Try g77 or include libg2c... Should be done when generating the shard object, though. Greetings, Jochen - -- University of North Carolina phone: +1-919-962-4403 Department of Chemistry phone: +1-919-962-1579 Venable Hall CB#3290 (Kenan C148) fax: +1-919-843-6041 Chapel Hill, NC 27599, USA GnuPG key: 44BCCD8E -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6-cygwin-fcn-1 (Cygwin) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8TsvhiJ/aUUS8zY4RApr2AKCRz0Ef+p+LmqI7UKpOPBDjgwSaawCgsCeP OFc5DqNCYVzHfPd4+nfDQjA= =2FIz -----END PGP SIGNATURE----- From prabhu at aero.iitm.ernet.in Wed Jan 23 10:04:38 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Wed, 23 Jan 2002 20:34:38 +0530 Subject: [SciPy-dev] fblas1 In-Reply-To: <86r8ohjfha.fsf@bock.chem.unc.edu> References: <15438.29082.410502.61267@monster.linux.in> <86r8ohjfha.fsf@bock.chem.unc.edu> Message-ID: <15438.53510.787566.235967@monster.linux.in> >>>>> "JK" == Jochen K?pper writes: JK> On Wed, 23 Jan 2002 13:47:30 +0530 Prabhu Ramachandran wrote: Prabhu> $ gcc -g foo_axpy.c -lcblas -lf77blas -latlas -DDMALLOC Prabhu> -ldmalloc JK> Try g77 or include libg2c... Should be done when generating JK> the shard object, though. Ahh, thanks, that worked. So here is what I got. $ g77 -g foo_axpy.c -lcblas -lf77blas -latlas -DDMALLOC -ldmalloc $ ./a.out y[0]=36.599998 y[0]=15.300000 $ This is atlas version: 3.2.1-12. prabhu From andreas.frankfurt at gmx.de Thu Jan 24 08:54:08 2002 From: andreas.frankfurt at gmx.de (Andreas Lorenz) Date: Thu, 24 Jan 2002 14:54:08 +0100 Subject: [SciPy-dev] Can I call Python-functions from """code"""? Message-ID: <200201241347.g0ODlEP11452@scipy.org> Is there a way to cll my self-defined Python-functions from the embeded C++ code?? And if so, how??? Andreas Lorenz From prabhu at aero.iitm.ernet.in Thu Jan 24 12:19:51 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 24 Jan 2002 22:49:51 +0530 Subject: [SciPy-dev] Can I call Python-functions from """code"""? In-Reply-To: <200201241347.g0ODlEP11452@scipy.org> References: <200201241347.g0ODlEP11452@scipy.org> Message-ID: <15440.16951.803169.855174@monster.linux.in> >>>>> "AL" == Andreas Lorenz writes: AL> Is there a way to cll my self-defined Python-functions from AL> the embeded C++ code?? Well, that is not possible currently and even if it were it would apparently be quite hard (or painful) to do. Of course, Eric will correct me if I am wrong about this. prabhu From eric at scipy.org Thu Jan 24 12:57:56 2002 From: eric at scipy.org (eric) Date: Thu, 24 Jan 2002 12:57:56 -0500 Subject: [SciPy-dev] Can I call Python-functions from """code"""? References: <200201241347.g0ODlEP11452@scipy.org> Message-ID: <010801c1a500$a70cedf0$777ba8c0@ericlaptop> Hey Andreas, > Is there a way to cll my self-defined Python-functions from the embeded C++ > code?? You can. Since code is all C++ code, and your wanting to call a Python function, you have to do two things: 1. Pass the function you want to call as an argument to inline. 2. Call the function using the Python API. Take note that the call back into Python exacts a large performance toll, so you probably won't see much speed up. > > And if so, how??? > Here is an example from weave/examples/functional.py that is similar to Python's builtin map(). There is a 2nd "prettier" version in functional.py, but it is half the speed of the standard map() function. def c_list_map2(func,seq): """ Uses Python API more than CXX to implement a simple map-like function. It does not provide any error checking. """ code = """ Py::Tuple args(1); Py::List result(seq.length()); PyObject* item = NULL; PyObject* this_result = NULL; for(int i = 0; i < seq.length();i++) { item = PyList_GET_ITEM(seq.ptr(),i); PyTuple_SetItem(args.ptr(),0,item); this_result = PyEval_CallObject(func,args.ptr()); PyList_SetItem(result.ptr(),i,this_result); } return_val = Py::new_reference_to(result); """ return inline_tools.inline(code,['func','seq']) seq = ['aadasdf'] * 5 result = c_list_map2(len,seq) If you can re-write the function you need to call as a C function, then you can call it directly in C to get the speed back. There are many cases where this isn't feasible, but some where it is. Use the support_code key variable for this. I think there are some examples of this in weave/examples/fibonacci.py Hope that helps, eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From prabhu at aero.iitm.ernet.in Thu Jan 24 21:50:54 2002 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Fri, 25 Jan 2002 08:20:54 +0530 Subject: [SciPy-dev] Can I call Python-functions from """code"""? In-Reply-To: <010801c1a500$a70cedf0$777ba8c0@ericlaptop> References: <200201241347.g0ODlEP11452@scipy.org> <010801c1a500$a70cedf0$777ba8c0@ericlaptop> Message-ID: <15440.51214.642690.860326@monster.linux.in> >>> "eric" == "eric" writes: eric> Hey Andreas, >> Is there a way to cll my self-defined Python-functions from the >> = eric> embeded C++=20 >> code?? eric> You can. Since code is all C++ code, and your wanting to eric> call a Python = function, you have to do two things: Umm, sorry folks for misleading you all. I must have misunderstood something when I talked with Eric. My bad. But I sure am glad that we can do this. It opens up so many possibilities. :) Thanks Eric! prabhu From eric at scipy.org Thu Jan 24 21:05:41 2002 From: eric at scipy.org (eric) Date: Thu, 24 Jan 2002 21:05:41 -0500 Subject: [SciPy-dev] Can I call Python-functions from """code"""? References: <200201241347.g0ODlEP11452@scipy.org><010801c1a500$a70cedf0$777ba8c0@ericlaptop> <15440.51214.642690.860326@monster.linux.in> Message-ID: <01b301c1a544$d03cf6e0$777ba8c0@ericlaptop> No prob. The mechanism for doing it isn't as transparent or obvious as I'd like. Still, possible is better than impossible... eric. ----- Original Message ----- From: "Prabhu Ramachandran" To: Sent: Thursday, January 24, 2002 9:50 PM Subject: Re: [SciPy-dev] Can I call Python-functions from """code"""? > >>> "eric" == "eric" writes: > eric> Hey Andreas, > > >> Is there a way to cll my self-defined Python-functions from the > >> = > eric> embeded C++=20 > >> code?? > > eric> You can. Since code is all C++ code, and your wanting to > eric> call a Python = function, you have to do two things: > > Umm, sorry folks for misleading you all. I must have misunderstood > something when I talked with Eric. My bad. But I sure am glad that > we can do this. It opens up so many possibilities. :) > > Thanks Eric! > > prabhu > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From pearu at cens.ioc.ee Fri Jan 25 08:35:38 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 25 Jan 2002 15:35:38 +0200 (EET) Subject: [SciPy-dev] fblas1 In-Reply-To: <15438.53510.787566.235967@monster.linux.in> Message-ID: On Wed, 23 Jan 2002, Prabhu Ramachandran wrote: > >>>>> "JK" == Jochen K?pper writes: > > JK> On Wed, 23 Jan 2002 13:47:30 +0530 Prabhu Ramachandran wrote: > > Prabhu> $ gcc -g foo_axpy.c -lcblas -lf77blas -latlas -DDMALLOC > Prabhu> -ldmalloc > > JK> Try g77 or include libg2c... Should be done when generating > JK> the shard object, though. > > Ahh, thanks, that worked. So here is what I got. > > $ g77 -g foo_axpy.c -lcblas -lf77blas -latlas -DDMALLOC -ldmalloc > $ ./a.out > y[0]=36.599998 > y[0]=15.300000 > $ > > This is atlas version: 3.2.1-12. To be sure that you tested atlas cblas, did you uncomment the line //cblas_saxpy(n,a,x,incx,y,incy); ? If yes, then I just have to wait until they fix atlas as with the 3.2.1 there were problems with clapack and now, with 3.3.13, I got these craches with cblas. Has anyone else tried this test, but now preferably with the latest atlas? Regards, Pearu From jochen at unc.edu Fri Jan 25 10:30:31 2002 From: jochen at unc.edu (Jochen =?iso-8859-1?q?K=FCpper?=) Date: 25 Jan 2002 10:30:31 -0500 Subject: [SciPy-dev] fblas1 In-Reply-To: References: Message-ID: On Fri, 25 Jan 2002 15:35:38 +0200 (EET) Pearu Peterson wrote: Pearu> On Wed, 23 Jan 2002, Prabhu Ramachandran wrote: >> $ g77 -g foo_axpy.c -lcblas -lf77blas -latlas -DDMALLOC -ldmalloc >> $ ./a.out >> y[0]=36.599998 >> y[0]=15.300000 >> $ >> >> This is atlas version: 3.2.1-12. Pearu> To be sure that you tested atlas cblas, did you uncomment the line Pearu> //cblas_saxpy(n,a,x,incx,y,incy); Pearu> Has anyone else tried this test, but now preferably with the latest atlas? latest Cygwin, ATLAS-3.3.13: f77blas works, cblas crashes in ATL_saxpy. Greetings, Jochen -- University of North Carolina phone: +1-919-962-4403 Department of Chemistry phone: +1-919-962-1579 Venable Hall CB#3290 (Kenan C148) fax: +1-919-843-6041 Chapel Hill, NC 27599, USA GnuPG key: 44BCCD8E From andreas.frankfurt at gmx.de Wed Jan 30 09:51:41 2002 From: andreas.frankfurt at gmx.de (Andreas Lorenz) Date: Wed, 30 Jan 2002 15:51:41 +0100 Subject: [SciPy-dev] Problems with weave Message-ID: <200201301444.g0UEigP18316@scipy.org> Hi! I worte an inline function using weave but when I try to run it it compiles but the I get this error: running build_ext building 'sc_4c24e4aef4fbfb5310eb79f279bcaaaf1' extension gcc -D_FILE_OFFSET_BITS=64 -DHAVE_LARGEFILE_SUPPORT -O2 -mcpu=i486 -march=i486 -fPIC -I/usr/lib/python2.1/site-packages/weave -I/usr/lib/python2.1/site-packages/weave/blitz-20001213 -I/usr/include/python2.1 -c /root/.python21_compiled/sc_4c24e4aef4fbfb5310eb79f279bcaaaf1.cpp -o /tmp/python21_intermediate/sc_4c24e4aef4fbfb5310eb79f279bcaaaf1.o /root/.python21_compiled/sc_4c24e4aef4fbfb5310eb79f279bcaaaf1.cpp:550: warning: `convert_to_numpy' redefined /root/.python21_compiled/sc_4c24e4aef4fbfb5310eb79f279bcaaaf1.cpp:549: warning: this is the location of the previous definition skipping /usr/lib/python2.1/site-packages/weave/CXX/IndirectPythonInterface.cxx (/tmp/python21_intermediate/IndirectPythonInterface.o up-to-date) skipping /usr/lib/python2.1/site-packages/weave/CXX/cxxextensions.c (/tmp/python21_intermediate/cxxextensions.o up-to-date) skipping /usr/lib/python2.1/site-packages/weave/CXX/cxxsupport.cxx (/tmp/python21_intermediate/cxxsupport.o up-to-date) skipping /usr/lib/python2.1/site-packages/weave/CXX/cxx_extensions.cxx (/tmp/python21_intermediate/cxx_extensions.o up-to-date) g++ -shared /tmp/python21_intermediate/sc_4c24e4aef4fbfb5310eb79f279bcaaaf1.o /tmp/python21_intermediate/IndirectPythonInterface.o /tmp/python21_intermediate/cxxextensions.o /tmp/python21_intermediate/cxxsupport.o /tmp/python21_intermediate/cxx_extensions.o -o /root/.python21_compiled/sc_4c24e4aef4fbfb5310eb79f279bcaaaf1.so Traceback (most recent call last): File "MultiLayer0.1.2.py", line 1520, in ? evolvepop(200,38,"FUTURE+1",5) File "MultiLayer0.1.2.py", line 1473, in evolvepop n.calcerror() File "MultiLayer0.1.2.py", line 1070, in calcerror verbose = 2) File "/usr/lib/python2.1/site-packages/weave/inline_tools.py", line 331, in inline results = attempt_function_call(code,local_dict,global_dict) File "/usr/lib/python2.1/site-packages/weave/inline_tools.py", line 374, in attempt_function_call raise ValueError, 'function with correct signature not found' ValueError: function with correct signature not found Anyone any idea??? Regards, Andreas Lorenz From andreas.frankfurt at gmx.de Wed Jan 30 11:04:48 2002 From: andreas.frankfurt at gmx.de (Andreas Lorenz) Date: Wed, 30 Jan 2002 17:04:48 +0100 Subject: [SciPy-dev] Problem solved ---- Sorry! Message-ID: <200201301557.g0UFvnP20364@scipy.org> I calld inline with arguments taht were not used in the code... That caused the error .... :-( Bug or feature? :-) Andreas From eric at scipy.org Wed Jan 30 12:58:59 2002 From: eric at scipy.org (eric) Date: Wed, 30 Jan 2002 12:58:59 -0500 Subject: [SciPy-dev] Problem solved ---- Sorry! References: <200201301557.g0UFvnP20364@scipy.org> Message-ID: <020f01c1a9b7$ce86e1e0$6b01a8c0@ericlaptop> Hey Andreas, > I calld inline with arguments taht were not used in the code... > That caused the error .... :-( > > Bug or feature? :-) The way weave is written, it views the C code as a continuation of the Python code, not really a function call, so the same variables are used in both. There might be a way around it. It would incur an efficiency hit, though I think very small. A bigger issue is that it could be confusing. For example: a,b,c,d,e = 1,1,1,1,1 code = """ return_val = Py::new_reference_to(Py::Int(a+b+c+d)); """ weave.inline(code,['a','b','c','d']) weave.inline(code,['b','c','d','e']) In the second version, are the argument names substituted positionally or by name? If we go with positional, then the idea of the C/C++ code being a continuation of the python code is "broken", and this becomes more of a function call. Of course the current behavior was confusing to you, so maybe it is 6 one way and half a dozen the other... :) Eventually, I'd like to remove the need to specify the argument list. This would require some C/C++ parsing of the code string to find the necessary arguments. If your interested in more of a function calling approach, for now, take a look at weave.ext_tools. It builds extension functions that are called in the "normal" way. see ya, eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez at pizero.colorado.edu Wed Jan 30 14:12:45 2002 From: fperez at pizero.colorado.edu (=?ISO-8859-1?Q?Fernando_P=E9rez?=) Date: Wed, 30 Jan 2002 12:12:45 -0700 (MST) Subject: [SciPy-dev] Problem solved ---- Sorry! In-Reply-To: <020f01c1a9b7$ce86e1e0$6b01a8c0@ericlaptop> Message-ID: On Wed, 30 Jan 2002, eric wrote: > code = """ > return_val = Py::new_reference_to(Py::Int(a+b+c+d)); > """ > weave.inline(code,['a','b','c','d']) > weave.inline(code,['b','c','d','e']) > > In the second version, are the argument names substituted positionally or > by name? If we go with positional, then the idea of the C/C++ code being > a continuation of the python code is "broken", and this becomes more of a > function call. I hope you're not considering 'going positional'. It would make weave's use very unnatural and bizarre, IMHO. The way it currently works seems perfectly natural and logical to me (hopefully it's not that confusing to others once they realize how to use it once). > Eventually, I'd like to remove the need to specify the argument list. > This would require some C/C++ parsing of the code string to find the > necessary arguments. Please follow this route *only* if you can be positive that it's really solid. It sounds tricky to make it really robust and a fair bit of work, and the 'inconvenience' it addresses seems reasonably minor to me. I also kind of like the fact that the argumnent list gives me a known interface between the two layers of code, so if I care to define an 'i' counter in C even if I had one in Python, it's not a problem. I don't know, maybe others feel differently but from the possible areas for weave to improve (and where your time and effort can go), to me this one doesn't really feel that critical. Cheers, f. From eric at scipy.org Wed Jan 30 13:55:48 2002 From: eric at scipy.org (eric) Date: Wed, 30 Jan 2002 13:55:48 -0500 Subject: [SciPy-dev] Problem solved ---- Sorry! References: Message-ID: <023101c1a9bf$be5c6b20$6b01a8c0@ericlaptop> > On Wed, 30 Jan 2002, eric wrote: > > > code = """ > > return_val = Py::new_reference_to(Py::Int(a+b+c+d)); > > """ > > weave.inline(code,['a','b','c','d']) > > weave.inline(code,['b','c','d','e']) > > > > In the second version, are the argument names substituted positionally or > > by name? If we go with positional, then the idea of the C/C++ code being > > a continuation of the python code is "broken", and this becomes more of a > > function call. > > I hope you're not considering 'going positional'. It would make weave's use > very unnatural and bizarre, IMHO. The way it currently works seems perfectly > natural and logical to me (hopefully it's not that confusing to others once > they realize how to use it once). inline was intended not to be positional and ext_tools provides function like behavior. But Andreas getting confused about inline's behavior, and the fact that I have thought about positional arguments before, does mean that there isn't a cut-and-dried "obvious" behavior. However, I'm fairly comfortable with the current behavior, and without some very good reasons to change it, it'll stay that way. > > > Eventually, I'd like to remove the need to specify the argument list. > > This would require some C/C++ parsing of the code string to find the > > necessary arguments. > > Please follow this route *only* if you can be positive that it's really solid. > It sounds tricky to make it really robust and a fair bit of work, and the > 'inconvenience' it addresses seems reasonably minor to me. I think its possible to make it solid, but it would be a big project and the benefit versus level of effort is really low. So, like you, I see it as pretty far down the list of desired features. I tend to go for the low hanging fruit while it is still there... > I also kind of like > the fact that the argumnent list gives me a known interface between the two > layers of code, so if I care to define an 'i' counter in C even if I had one > in Python, it's not a problem. Well, that kinda breaks the notion of the C code as a continuation of the Python code where all variables are shared between the two. Your right, it can be useful to specify what crosses the C/Python boundary, but it also isn't also isn't so hard to rename variables that clash. But see the previous comment as to how likely this is to change in the near future. > I don't know, maybe others feel differently but > from the possible areas for weave to improve (and where your time and effort > can go), to me this one doesn't really feel that critical. Agreed. eric From 66zj9yq13s6r at aol.com Wed Jan 23 04:09:40 2002 From: 66zj9yq13s6r at aol.com (Willie Maddox) Date: Wed, 23 Jan 02 09:09:40 GMT Subject: [SciPy-dev] ~~Find a Mortgage Loan... Refinance, 2nd, Purchase, Home Improvement unywd Message-ID: <7jk$wdsox23vcr7p20$86@dne.4x15e3175j> An HTML attachment was scrubbed... URL: From y9bz4p4zaix7 at yahoo.com Fri Jan 25 11:37:39 2002 From: y9bz4p4zaix7 at yahoo.com (Dee Young) Date: Fri, 25 Jan 02 16:37:39 GMT Subject: [SciPy-dev] Find a Mortgage Loan... Refinance, 2nd, Purchase, Home Improvement BB gtw Message-ID: An HTML attachment was scrubbed... URL: