Telling "incomplete input" from "invalid input" - FAQ out of date?

[Matt Setzer]

    Section 5.15 of the Python FAQ (How to tell "incomplete input" from
"invalid input") seems to be out of date.  Two code samples are given, but
neither appears to work with Python 2.0 under Win32.  The second code sample
reports the error "Indendation Error: expected an indented block" on the
string "for i in range(0,10 ):\n", rather than recognizing that the
statement is incomplete.  The first example uses a symbol _Py_grammar which
is not exported from the python library, but it seems to be looking for the
same condition that the second example is, suggesting that it would also
fail in the same way.

     I implemented the algorithm used in compile_command from codeop.py in
C++ (see below) and that seems to work, but I'm wondering if I missed
something in the FAQ to explain why the existing examples don't seem to
work, or if there's a better way of handling this problem.  If not, should I
submit the function below as an update to the FAQ entry?

Matt Setzer
Return address spam protected in a hopefully obvious fashion.

/*
 * Check to see if a string represents a complete python command, an
incomplete python command,
 * or an invalid python command.  Returns a PyCodeObject that can be passed
to PyEval_EvalCode
 * for complete commands, Py_None for incomplete commands, and NULL for
invalid commands.
 * Based on the algorithm for compile_command in codeop.py
 */
PyObject *CompileCommand( const string &s )
{
 PyObject *r = Py_CompileString( const_cast< char * >( s.c_str() ),
"<input>", Py_single_input );
 if ( r )
  return r;
 else if ( !PyErr_ExceptionMatches( PyExc_SyntaxError ) )
  return NULL;

 PyObject *typ, *val, *tb;
 PyErr_Fetch( &typ, &val, &tb );

 PyObject *r1 = Py_CompileString( const_cast< char * >( ( s + string(
"\n" ) ).c_str() ), "<input>", Py_single_input );
 if ( r1 )
 {
  Py_DECREF( r1 );
  Py_INCREF( Py_None );
  return Py_None;
 }
 else if ( !PyErr_ExceptionMatches( PyExc_SyntaxError ) )
  return NULL;

 PyObject *typ1, *val1, *tb1;
 PyErr_Fetch( &typ1, &val1, &tb1 );

 PyObject *r2 = Py_CompileString( const_cast< char * >( ( s + string(
"\n\n" ) ).c_str() ), "<input>", Py_single_input );
 assert( NULL == r2 ); // This should never compile if the previous two
attempts didn't
 if ( !PyErr_ExceptionMatches( PyExc_SyntaxError ) )
  return NULL;

 PyObject *typ2, *val2, *tb2;
 PyErr_Fetch( &typ2, &val2, &tb2 );

 PyObject *pystr1 = PyObject_Str( val1 );
 PyObject *pystr2 = PyObject_Str( val2 );

 const char *str1 = PyString_AsString( pystr1 );
 const char *str2 = PyString_AsString( pystr2 );

 bool match = false;
 if ( 0 == strcmp( str1, str2 ) )
  match = true;

 Py_XDECREF( typ1 );
 Py_XDECREF( val1 );
 Py_XDECREF( tb1 );
 Py_XDECREF( typ2 );
 Py_XDECREF( val2 );
 Py_XDECREF( tb2 );

 PyObject *result = NULL;
 if ( match )
 {
  PyErr_Restore( typ, val, tb );
 }
 else
 {
  Py_XDECREF( typ );
  Py_XDECREF( val );
  Py_XDECREF( tb );
  Py_INCREF( Py_None );
  result = Py_None;
 }

 return result;
}