From neal@metaslash.com Tue Apr 1 12:32:39 2003 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 01 Apr 2003 07:32:39 -0500 Subject: [Compiler-sig] ast branch Message-ID: <20030401123239.GH16288@epoch.metaslash.com> The AST branch seems to be making pretty good progress. I just wanted to collect the known issues so we don't overlap. Brett Cannon had mentioned that he was interested in the new compiler, so perhaps this will provide some insight into the status. Known problems: * class is not handled * need more error checking on memory alloc failures, etc. * use PyOS_snprintf instead of sprintf * the following code at the module scope core dumps: if 1: try: from posix import _exit except ImportError: pass there is an extra POP_TOP at offset 10 after SETUP_EXCEPT the initial JUMP_IF_FALSE jumps into the try, not after The new files are in Python: asdl.c, ast.c newcompile.c. Python/symtable.c has been modified to use the new AST too. I'm sure there's a ton of things I forgot or don't know. Neal From nas@python.ca Tue Apr 1 14:55:00 2003 From: nas@python.ca (Neil Schemenauer) Date: Tue, 1 Apr 2003 06:55:00 -0800 Subject: [Compiler-sig] ast branch In-Reply-To: <20030401123239.GH16288@epoch.metaslash.com> References: <20030401123239.GH16288@epoch.metaslash.com> Message-ID: <20030401145500.GA4790@glacier.arctrix.com> Neal Norwitz wrote: > * class is not handled I implemented 'class' at pycon. It might even work. Method calls are broken. The following code does not work: class A: def f(self): print 'hello' A().f() I haven't dug into what's wrong. Neil From jeremy@alum.mit.edu Tue Apr 1 15:44:41 2003 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: 01 Apr 2003 10:44:41 -0500 Subject: [Compiler-sig] ast branch In-Reply-To: <20030401123239.GH16288@epoch.metaslash.com> References: <20030401123239.GH16288@epoch.metaslash.com> Message-ID: <1049211881.14152.9.camel@slothrop.zope.com> On Tue, 2003-04-01 at 07:32, Neal Norwitz wrote: > The AST branch seems to be making pretty good progress. I just wanted > to collect the known issues so we don't overlap. Brett Cannon had > mentioned that he was interested in the new compiler, so perhaps this > will provide some insight into the status. I fixed a wild right in assemble_jump_offsets() yesterday. That eliminate the code dumps I had been seeing during compilation of site.py and the modules it depends on. I still get a core dump somewhere during the startup, but the failure is the result of generating bad code. Much more progress on that front. We've still got lots of niggling problems. I also fixed a problem with expressions containing more than one binary operator. 1 + 2 + 3 was being turned into BinOp(1, +, 2) by the ast converter, instead of BinOp(BinOp(1, +, 2), + 3). It's still helpful to try trivial one-line expressions at the interpreter prompt and see what happens. Jeremy From jeremy@alum.mit.edu Tue Apr 1 18:48:34 2003 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: 01 Apr 2003 13:48:34 -0500 Subject: [Compiler-sig] ast branch In-Reply-To: <20030401123239.GH16288@epoch.metaslash.com> References: <20030401123239.GH16288@epoch.metaslash.com> Message-ID: <1049222914.14149.36.camel@slothrop.zope.com> On Tue, 2003-04-01 at 07:32, Neal Norwitz wrote: > The AST branch seems to be making pretty good progress. I just wanted > to collect the known issues so we don't overlap. Brett Cannon had > mentioned that he was interested in the new compiler, so perhaps this > will provide some insight into the status. Here are two important tasks: - Compute the stack depth from the basic blocks. - Create a valid lnotab. I think the former problem is what's prevent us from importing os.py. Jeremy From neal@metaslash.com Tue Apr 1 22:25:46 2003 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 01 Apr 2003 17:25:46 -0500 Subject: [Compiler-sig] ast branch In-Reply-To: <1049222914.14149.36.camel@slothrop.zope.com> References: <20030401123239.GH16288@epoch.metaslash.com> <1049222914.14149.36.camel@slothrop.zope.com> Message-ID: <20030401222546.GK16288@epoch.metaslash.com> On Tue, Apr 01, 2003 at 01:48:34PM -0500, Jeremy Hylton wrote: > On Tue, 2003-04-01 at 07:32, Neal Norwitz wrote: > > The AST branch seems to be making pretty good progress. I just wanted > > to collect the known issues so we don't overlap. Brett Cannon had > > mentioned that he was interested in the new compiler, so perhaps this > > will provide some insight into the status. > > Here are two important tasks: > - Compute the stack depth from the basic blocks. > - Create a valid lnotab. Not sure about the first, but the second would be very helpful when the interpreter crashes. I ran with valgrind, there were a few checkins as a result. There are still some memory leaks and a problem with raise statements IIRC. ==3872== Invalid read of size 2 ==3872== at 0x8101E3F: ast_for_expr (Python/ast.c:700) ==3872== by 0x8102D2A: ast_for_flow_stmt (Python/ast.c:1009) ==3872== by 0x810425E: ast_for_stmt (Python/ast.c:1432) ==3872== by 0x81036BF: ast_for_suite (Python/ast.c:1195) ==3872== by 0x810135B: ast_for_funcdef (Python/ast.c:426) ==3872== by 0x810437A: ast_for_stmt (Python/ast.c:1460) ==3872== by 0x8100877: PyAST_FromNode (Python/ast.c:113) ==3872== by 0x80C2483: PyParser_ASTFromFile (Python/pythonrun.c:1102) ==3872== by 0x80B6E63: parse_source_module (Python/import.c:660) ==3872== by 0x80B7115: load_source_module (Python/import.c:782) ==3872== 12 bytes in 1 blocks are definitely lost in loss record 4 of 241 ==3872== at 0x40168A70: malloc (vg_clientfuncs.c:100) ==3872== by 0x806C3A7: PyObject_Malloc (Objects/obmalloc.c:854) ==3872== by 0x81002C5: asdl_seq_new (Python/asdl.c:7) ==3872== by 0x8101777: ast_for_listcomp (Python/ast.c:527) ==3872== by 0x81019C6: ast_for_atom (Python/ast.c:579) ==3872== by 0x8102261: ast_for_expr (Python/ast.c:779) ==3872== by 0x81026B8: ast_for_testlist (Python/ast.c:872) ==3872== by 0x8102C8B: ast_for_flow_stmt (Python/ast.c:1000) ==3872== by 0x810425E: ast_for_stmt (Python/ast.c:1432) ==3872== by 0x81036BF: ast_for_suite (Python/ast.c:1195) ==3872== by 0x8103D76: ast_for_except_clause (Python/ast.c:1337) ==3872== by 0x8103F65: ast_for_try_stmt (Python/ast.c:1369) ==3872== by 0x8104366: ast_for_stmt (Python/ast.c:1458) ==3872== by 0x81036BF: ast_for_suite (Python/ast.c:1195) ==3872== by 0x810135B: ast_for_funcdef (Python/ast.c:426) ==3872== 12 bytes in 1 blocks are definitely lost in loss record 6 of 241 ==3872== at 0x40168A70: malloc (vg_clientfuncs.c:100) ==3872== by 0x806C3A7: PyObject_Malloc (Objects/obmalloc.c:854) ==3872== by 0x81002C5: asdl_seq_new (Python/asdl.c:7) ==3872== by 0x8102A84: ast_for_exprlist (Python/ast.c:953) ==3872== by 0x81016B7: ast_for_listcomp (Python/ast.c:515) ==3872== by 0x81019C6: ast_for_atom (Python/ast.c:579) ==3872== by 0x8102261: ast_for_expr (Python/ast.c:779) ==3872== by 0x81026B8: ast_for_testlist (Python/ast.c:872) ==3872== by 0x8102C8B: ast_for_flow_stmt (Python/ast.c:1000) ==3872== by 0x810425E: ast_for_stmt (Python/ast.c:1432) ==3872== by 0x81036BF: ast_for_suite (Python/ast.c:1195) ==3872== by 0x8103D76: ast_for_except_clause (Python/ast.c:1337) ==3872== by 0x8103F65: ast_for_try_stmt (Python/ast.c:1369) ==3872== by 0x8104366: ast_for_stmt (Python/ast.c:1458) ==3872== 100 bytes in 5 blocks are definitely lost in loss record 106 of 241 ==3872== at 0x40168A70: malloc (vg_clientfuncs.c:100) ==3872== by 0x81059DF: PyNode_New (Parser/node.c:10) ==3872== by 0x80DBEBA: PyParser_New (Parser/parser.c:85) ==3872== by 0x8054F77: parsetok (Parser/parsetok.c:100) ==3872== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81) ==3872== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099) ==3872== by 0x80B6E63: parse_source_module (Python/import.c:660) ==3872== by 0x80B7115: load_source_module (Python/import.c:782) ==3872== by 0x80B7EDC: load_module (Python/import.c:1455) ==3872== by 0x80B95A5: import_submodule (Python/import.c:1994) From jeremy@alum.mit.edu Tue Apr 1 23:46:35 2003 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: 01 Apr 2003 18:46:35 -0500 Subject: [Compiler-sig] ast branch In-Reply-To: <20030401222546.GK16288@epoch.metaslash.com> References: <20030401123239.GH16288@epoch.metaslash.com> <1049222914.14149.36.camel@slothrop.zope.com> <20030401222546.GK16288@epoch.metaslash.com> Message-ID: <1049240794.14146.57.camel@slothrop.zope.com> I guess I better figure out how to use Valgrind, because it diagnoses problems faster than I do :-). The PyObject_Free(mod) fails in a debug build. file_input containing 38 statements Debug memory block at address p=0x8144f28: 3688618971 bytes originally requested The 4 pad bytes at p-4 are not all FORBIDDENBYTE (0xfb): at p-4: 0x11 *** OUCH at p-3: 0x00 *** OUCH at p-2: 0x00 *** OUCH at p-1: 0x00 *** OUCH Because memory is corrupted at the start, the count of bytes requested may be bogus, and checking the trailing pad bytes may segfault. The 4 pad bytes at tail=0xe3f02b03 are Segmentation fault (core dumped) Also, the compiler_use_next_block() after a jump is not the right answer in the long run. The next part means that there is a control flow from the previous block to the next block. If the previous block ends with a jump, there's clearly no implicit control flow to the next block. There must be some other way to fix the problem, but I'm not sure what it is. Jeremy From neal@metaslash.com Wed Apr 2 00:02:57 2003 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 01 Apr 2003 19:02:57 -0500 Subject: [Compiler-sig] ast branch In-Reply-To: <1049240794.14146.57.camel@slothrop.zope.com> References: <20030401123239.GH16288@epoch.metaslash.com> <1049222914.14149.36.camel@slothrop.zope.com> <20030401222546.GK16288@epoch.metaslash.com> <1049240794.14146.57.camel@slothrop.zope.com> Message-ID: <20030402000257.GN16288@epoch.metaslash.com> On Tue, Apr 01, 2003 at 06:46:35PM -0500, Jeremy Hylton wrote: > I guess I better figure out how to use Valgrind, because it diagnoses > problems faster than I do :-). :-) It's very easy. Build, install, run: valgrind --show-reachable=yes --num-callers=15 --leak-check=yes ./python I run under script so I can catch the output, redirecting doesn't work if there's a crash because valgrind does some funny stuff. > The PyObject_Free(mod) fails in a debug build. > file_input containing 38 statements > Debug memory block at address p=0x8144f28: > 3688618971 bytes originally requested > The 4 pad bytes at p-4 are not all FORBIDDENBYTE (0xfb): > at p-4: 0x11 *** OUCH > at p-3: 0x00 *** OUCH > at p-2: 0x00 *** OUCH > at p-1: 0x00 *** OUCH > Because memory is corrupted at the start, the count of bytes > requested > may be bogus, and checking the trailing pad bytes may segfault. > The 4 pad bytes at tail=0xe3f02b03 are Segmentation fault (core > dumped) Hmmm, I'm also using debug, but don't get the problem when running under valgrind or not. I have no outstanding changes. But it could be the read from uninitialized memory is behaving differently on our boxes too. > Also, the compiler_use_next_block() after a jump is not the right answer > in the long run. The next part means that there is a control flow from > the previous block to the next block. If the previous block ends with a > jump, there's clearly no implicit control flow to the next block. There > must be some other way to fix the problem, but I'm not sure what it is. Ok. I just made it consistent. It was 2 and 2 I think. I think there is also a lot more memory that stays in use after it is needed. ==7289== 37348 bytes in 7274 blocks are still reachable in loss record 237 of 241 ==7289== at 0x40168A70: malloc (vg_clientfuncs.c:100) ==7289== by 0x8055036: parsetok (Parser/parsetok.c:128) ==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81) ==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099) ==7289== by 0x80B6E63: parse_source_module (Python/import.c:660) ==7289== by 0x80B7115: load_source_module (Python/import.c:782) ==7289== by 0x80B7EDC: load_module (Python/import.c:1455) ==7289== by 0x80B95A5: import_submodule (Python/import.c:1994) ==7289== by 0x80B8D47: load_next (Python/import.c:1850) ==7289== by 0x80B869A: import_module_ex (Python/import.c:1701) ==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742) ==7289== by 0x809355C: builtin___import__ (Python/bltinmodule.c:40) ==7289== by 0x80F9A6D: PyCFunction_Call (Objects/methodobject.c:80) ==7289== by 0x80E07EE: PyObject_Call (Objects/abstract.c:1688) ==7289== by 0x80A6E24: PyEval_CallObjectWithKeywords (Python/ceval.c:3026) ==7289== ==7289== 41560 bytes in 2078 blocks are still reachable in loss record 238 of 241 ==7289== at 0x40168A70: malloc (vg_clientfuncs.c:100) ==7289== by 0x40168FF0: realloc (vg_clientfuncs.c:265) ==7289== by 0x8105B20: PyNode_AddChild (Parser/node.c:35) ==7289== by 0x80DBF99: shift (Parser/parser.c:112) ==7289== by 0x80DC356: PyParser_AddToken (Parser/parser.c:244) ==7289== by 0x80550AB: parsetok (Parser/parsetok.c:148) ==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81) ==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099) ==7289== by 0x80B6E63: parse_source_module (Python/import.c:660) ==7289== by 0x80B7115: load_source_module (Python/import.c:782) ==7289== by 0x80B7EDC: load_module (Python/import.c:1455) ==7289== by 0x80B95A5: import_submodule (Python/import.c:1994) ==7289== by 0x80B8D47: load_next (Python/import.c:1850) ==7289== by 0x80B869A: import_module_ex (Python/import.c:1701) ==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742) ==7289== ==7289== 124740 bytes in 1576 blocks are still reachable in loss record 239 of 241 ==7289== at 0x40169079: realloc (vg_clientfuncs.c:273) ==7289== by 0x8105B20: PyNode_AddChild (Parser/node.c:35) ==7289== by 0x80DC00B: push (Parser/parser.c:126) ==7289== by 0x80DC2E0: PyParser_AddToken (Parser/parser.c:234) ==7289== by 0x80550AB: parsetok (Parser/parsetok.c:148) ==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81) ==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099) ==7289== by 0x80B6E63: parse_source_module (Python/import.c:660) ==7289== by 0x80B7115: load_source_module (Python/import.c:782) ==7289== by 0x80B7EDC: load_module (Python/import.c:1455) ==7289== by 0x80B95A5: import_submodule (Python/import.c:1994) ==7289== by 0x80B8D47: load_next (Python/import.c:1850) ==7289== by 0x80B869A: import_module_ex (Python/import.c:1701) ==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742) ==7289== by 0x809355C: builtin___import__ (Python/bltinmodule.c:40) ==7289== ==7289== 141480 bytes in 1859 blocks are still reachable in loss record 240 of 241 ==7289== at 0x40169079: realloc (vg_clientfuncs.c:273) ==7289== by 0x8105B20: PyNode_AddChild (Parser/node.c:35) ==7289== by 0x80DBF99: shift (Parser/parser.c:112) ==7289== by 0x80DC356: PyParser_AddToken (Parser/parser.c:244) ==7289== by 0x80550AB: parsetok (Parser/parsetok.c:148) ==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81) ==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099) ==7289== by 0x80B6E63: parse_source_module (Python/import.c:660) ==7289== by 0x80B7115: load_source_module (Python/import.c:782) ==7289== by 0x80B7EDC: load_module (Python/import.c:1455) ==7289== by 0x80B95A5: import_submodule (Python/import.c:1994) ==7289== by 0x80B8D47: load_next (Python/import.c:1850) ==7289== by 0x80B869A: import_module_ex (Python/import.c:1701) ==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742) ==7289== by 0x809355C: builtin___import__ (Python/bltinmodule.c:40) ==7289== ==7289== 466920 bytes in 23346 blocks are still reachable in loss record 241 of 241 ==7289== at 0x40168A70: malloc (vg_clientfuncs.c:100) ==7289== by 0x40168FF0: realloc (vg_clientfuncs.c:265) ==7289== by 0x8105B20: PyNode_AddChild (Parser/node.c:35) ==7289== by 0x80DC00B: push (Parser/parser.c:126) ==7289== by 0x80DC2E0: PyParser_AddToken (Parser/parser.c:234) ==7289== by 0x80550AB: parsetok (Parser/parsetok.c:148) ==7289== by 0x8054F51: PyParser_ParseFileFlags (Parser/parsetok.c:81) ==7289== by 0x80C246A: PyParser_ASTFromFile (Python/pythonrun.c:1099) ==7289== by 0x80B6E63: parse_source_module (Python/import.c:660) ==7289== by 0x80B7115: load_source_module (Python/import.c:782) ==7289== by 0x80B7EDC: load_module (Python/import.c:1455) ==7289== by 0x80B95A5: import_submodule (Python/import.c:1994) ==7289== by 0x80B8D47: load_next (Python/import.c:1850) ==7289== by 0x80B869A: import_module_ex (Python/import.c:1701) ==7289== by 0x80B89DF: PyImport_ImportModuleEx (Python/import.c:1742) From jeremy@alum.mit.edu (Jeremy Hylton) Wed Apr 2 04:30:32 2003 From: jeremy@alum.mit.edu (Jeremy Hylton) (Jeremy Hylton) Date: Tue, 1 Apr 2003 23:30:32 -0500 Subject: [Compiler-sig] ast branch In-Reply-To: <20030401222546.GK16288@epoch.metaslash.com> References: <20030401123239.GH16288@epoch.metaslash.com> <1049222914.14149.36.camel@slothrop.zope.com> <20030401222546.GK16288@epoch.metaslash.com> Message-ID: <16010.26472.219301.756746@slothrop.zope.com> Thanks for the clue about the Raise statement in ast.c. It was completely broken, and fixing it provided a lot of progress. Running "python -v" got to an interpreter prompt: 'import site' failed; traceback: Traceback (most recent call last): File "/home/jeremy/src/python-ast/build-pydebug/../Lib/site.py", line 0, in File "/home/jeremy/src/python-ast/build-pydebug/../Lib/site.py", line 0, in File "/home/jeremy/src/python-ast/build-pydebug/../Lib/os.py", line 0, in File "/home/jeremy/src/python-ast/build-pydebug/../Lib/os.py", line 0, in NameError: name '_Environ' is not defined Python 2.3a0 (#71, Apr 1 2003, 23:20:02) [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2 Type "help", "copyright", "credits" or "license" for more information. filename= >>> It's not perfect, but it's nice to actually survive startup. As we've noted, it would sure be handy to have some line numbers. Jeremy From neal@metaslash.com Thu Apr 3 00:47:47 2003 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 02 Apr 2003 19:47:47 -0500 Subject: [Compiler-sig] ast branch In-Reply-To: <16010.26472.219301.756746@slothrop.zope.com> References: <20030401123239.GH16288@epoch.metaslash.com> <1049222914.14149.36.camel@slothrop.zope.com> <20030401222546.GK16288@epoch.metaslash.com> <16010.26472.219301.756746@slothrop.zope.com> Message-ID: <20030403004746.GB11263@epoch.metaslash.com> marshal_write_object() is empty in Python/asdl.c. The only use is in Python/Python-ast.c for writing a Number: case Num_kind: marshal_write_int(buf, off, 10); marshal_write_object(buf, off, o->v.Num.n); break; Not sure what needs to be done here. Do we need to handle arch sized ints? (ie 4 or 8 bytes). If not, couldn't we just use marshal_write_int? Neal From huey_jiang@yahoo.com Wed Apr 9 05:45:55 2003 From: huey_jiang@yahoo.com (Huey Jiang) Date: Tue, 8 Apr 2003 21:45:55 -0700 (PDT) Subject: [Compiler-sig] Chinese version Message-ID: <20030409044555.74727.qmail@web20001.mail.yahoo.com> Hi There, My name is Huey, a python lover. Recently I took my home visiting to China where IT industry is booming. However, I noticed that nobody even heard of python: people are now being trapped by Microsoft and Borland. Yet, while I was trying to show the people python, I fell into embaressment: the Unicode of python does not support Chinese language. What a shame! I wonder: can you guys make python to support Chinese language? In fact, I myself am interesting at working on it if somebody would help me. I guess python is now supporting non-English languages such as Spanish, Germny. So, I wish somebody worked on those would give me helps. Regards, Huey __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - File online, calculators, forms, and more http://tax.yahoo.com From jeremy@zope.com Mon Apr 28 19:52:36 2003 From: jeremy@zope.com (Jeremy Hylton) Date: 28 Apr 2003 14:52:36 -0400 Subject: [Compiler-sig] merged trunk to ast-branch -- line numbers working Message-ID: <1051555956.17289.122.camel@slothrop.zope.com> I did some work over the last couple of weeks to get line numbers working on the ast branch. If you get a traceback, it should not have meaningful line numbers. You've probably noticed that I updated the branch from the trunk in order to get there. This wasn't completely necessary, but was the first way I thought of to get the interpreter to use co_lnotab instead of SET_LINENO instructions. There's now even more code that gets run when the interpreter starts up. warnings.py was one that caused problems. There are some problems with the code generated. I haven't tracked down exactly what goes wrong, but I hope the line numbers help. Jeremy