[Python-bugs-list] [Bug #115555] Parser crashes for deeply nested list displays

[noreply@sourceforge.net]

Bug #115555, was updated on 2000-Sep-28 03:42
Here is a current snapshot of the bug.

Project: Python
Category: Parser/Compiler
Status: Open
Resolution: None
Bug Group: Feature Request
Priority: 5
Summary: Parser crashes for deeply nested list displays

Details: Compiling

s=[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]

gives an error message

s_push: parser stack overflow

Python 1.5.2 then reports a MemoryError, 2.0b2 a SyntaxError.

Follow-Ups:

Date: 2000-Sep-28 12:35
By: jhylton

Comment:
What is the desired outcome here?  Python is reporting a SyntaxError; it's not crashing.  Would you like to increase the max stack size for the parser?  What should it's limit be?

The current parser stack limit allows eval("["*x+"]"*x) for values of x up to and including 35.  I'm not sure why the limit is set this low.  I bumped the value of MAXSTACK in parser.c from 500 to 5000 and it accepted the nest list expression for values of x up to 357.

-------------------------------------------------------

Date: 2000-Sep-29 09:08
By: loewis

Comment:
It was confusing that Python would produce a SyntaxError for an obviously-correct script, and that such a small limit was found in the parser.

Since the limit is not due to recursion on the C stack:  Would a patch removing this limitation be accepted (certainly not for 2.0). The two alternatives I see are to make the array completely dynamic, or to allocate a dynamic array in the stack if the static one overflows.
-------------------------------------------------------

Date: 2000-Sep-29 21:48
By: jhylton

Comment:
There is a limit that is based on the C stack, because the parser is recursive descent.  If I set the max stack size to 100000, I get a seg fault.
I'm not sure if a patch for this would be accepted post 2.0 or not; I'll talk to Guido and see what he thinks.

I think we could safely increase the static limit before 2.0, though I'm not 100% certain.  What nesting level did your application come up with?
I would guess that max stack == 10000 (700 nested lists) should be safe on all reasonable platforms and much more useful.

-------------------------------------------------------

Date: 2000-Sep-30 19:12
By: bwarsaw

Comment:
BTW, JPython gets to 133 nestings (on my RH6.1 system) before a java.lang.StackOverflow gets thrown.

Ever heard the old joke "Doctor, it hurts when I do this?" ... 
-------------------------------------------------------

Date: 2000-Oct-02 02:23
By: loewis

Comment:
The problem was found when printing an expression like parser.suite("3*4").tolist(),
then modifying the string, and feeding the outcome back to Python. It is not a serious problem that this works for every possible Python code - it is just confusing to get a SyntaxError when there is no syntax error in the input.

BTW, I believe the parser is *not* recursive on the C stack; I could not find any sign of recursion inside Parser/parser.c. Most likely, the crash you got when increasing the parser stack size comes from Python/compile.c; the com_* functions are recursive on the C stack. It would be possible to remove the recursion here as well (using an explicit stack that lives on the heap); that would require a larger rewrite of compile.c, though.
-------------------------------------------------------

Date: 2000-Oct-02 03:22
By: gvanrossum

Comment:
The SyntaxError is actually caused by a bug in the parser code! When s_push() reports a stack overflow it returns -1, which is passed through unchanged by push() -- but the code that calls it only checks for positive error codes! I've fixed this by returning E_NOMEM from s_push() instead.

The only downside I see of making MAXSTACK larger is that a stack of maximal size is allocated each time the parser is invoked. With MAXSTACK=500 on a 32-bit machine, that's 6K. Switch to 5000 and it's 60K. No big deal except perhaps for ports to PalmPilots and the like...

Note that with MAXSTACK=500, you can "only" have 35 nested lists.
-------------------------------------------------------

For detailed info, follow this link:
http://sourceforge.net/bugs/?func=detailbug&bug_id=115555&group_id=5470