[IronPython] IronPython 2.6 CodePlex Source Update

[merllab at microsoft.com]

This is an automated email letting you know that sources 
have recently been pushed out.  You can download these newer 
sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/54584.

MODIFIED SOURCES
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/Parameter.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/NameExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/MemberExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/IndexExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/Expression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/ErrorExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/DelStatement.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/ContinueStatement.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/ConstantExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/CallExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/BreakStatement.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/BinaryExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/AstGenerator.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/AssignmentStatement.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/YieldExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/TupleExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/SequenceExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/ReturnStatement.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Ast/ParenthesisExpression.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/TokenKind.Generated.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Tokenizer.cs
	$/IronPython/IronPython_Main/Src/IronPython/Compiler/Parser.cs
	$/IronPython/IronPython_Main/Src/Runtime/Microsoft.Scripting/Actions/NamespaceTracker.cs
	$/IronPython/IronPython_Main/Src/Runtime/Microsoft.Scripting/Actions/TopNamespaceTracker.cs
	$/IronPython/IronPython_Main/Src/Tests/hosting/editor_svcs/errorlistener.py
	$/IronPython/IronPython_Main/Src/IronPython/Runtime/Types/PythonAssemblyOps.cs
	$/IronPython/IronPython_Main/Src/Tests/hosting/editor_svcs/tokencategorizer.py
	$/IronPython/IronPython_Main/Src/Tests/test_syntax.py

CHECKIN COMMENTS
--------------------------------------------------------------------------------
Changeset Id: 941816
Date: 6/11/2009 9:22:19 AM

(dinov) Fixes these two issues:

22692 - compile() incompatibility with CPython
12907 - Missing PyCF_DONT_IMPLY_DEDENT compile Flag

Also moves all error reporting out of the AST generator and into the Parser.  This will enable total lazy transformation of the trees if we want to do it.  It also means that editors using our Parser for syntax highlighting will see all errors instead of missing things like break outside of a loop.  This is just removing the places in transform into explicit virtual calls (CheckAssign, CheckDelete, CheckAugmentedAssign) that we call and if we get a string back we report that string as the error.  

Most of the change is about tracking blank lines and reporting them from the tokenizer as NL tokens.  This isn’t what the documentation says we should be doing (all blank lines are to be ignored) but it matches what CPython’s tokenize module is doing and presumably what CPython’s actual tokenizer is doing as well.  Certainly it’s hard to imagine otherwise how code that only differs by new lines could report different exceptions.

So do this the tokenizer now needs to keep track whether it just gave out a newline and if so when it will give out another one it should instead give out a NL token.  Don’t imply dedent is also updated to look at this state and yield a real newline (if we don’t have one, for example consider “def f():    return 2”) and then yield the pending dedent(s).  

Also there’s a fun little hack to match CPython’s behavior where it differs on how it reports errors w/ “compile()” vs an imported file.  If we are doing a “compile” (we have interactive code or statements) then re-indenting at turns into an error ("def  Foo():\n\n    pass\n\n  ").  But if this is just some random source file that we’re bringing in via import then this is perfectly fine.

I’ve also updated most of the test_syntax tests to run on both CPython and IronPython – we still differ on some of the errors. 



(Shelveset: ReallySignificantWhitespaceFinal;REDMOND\dinov | SNAP CheckinId: m9439)