"tuple index out of range" side effect of C extension??

[Tim Peters]

[Andrew MacIntyre]
> I'm seeing a bizarre situation where IndexErrors are being thrown with
> "tuple index out of range" error strings.
>
> The scenario is something like:
>
> l = [(v1a, v1b), (v2a, v2b), ..., (vna, vnb)]
>
> for a, b in l:
>     ...
>
>
>
> with the IndexError being thrown on the "for..." line after the last
> element of l has been through the for loop.
>
> I've also seen the same exception thrown on first access to an
> attribute of a simple class.
>
> The exception being thrown is not consistent,

It's probably the case then that C code is setting an exception in a
function F but neglecting to make F return its "there's an error" return
value.  The most common way this happens is for an extension module to call
a Python C API function and neglect to check *it* for an error return value.
In general, every Python C API function call must be checked for an error
return, and, when one occurs, the calling function must either return an
error code itself, or explicitly clear the error.

If an error return isn't made, ceval.c (the interpreter's main loop) doesn't
know an exception is pending, and not all bytecodes check for a pending
exception.  The exception doesn't get triggered then until the interpreter
happens to execute a bytecode that does check for a pending exception.  The
original C code that set the exception is long gone at that point, and the
reported exception doesn't appear to make sense.

> and reducing the number of tuples and lists in the code seems to
> reduce the likelihood of the exception being thrown.

You wish <wink>.

> The only thing unusual is the presence of a C extension

Bingo.

> which I am testing
> - an Ingres DB interface module, most of whose functionality seems to
> work when the IndexErrors don't get thrown.

It's probably doing something like PyList_GetItem with an out-of-bounds
index itself, then neglecting to check whether PyList_GetItem complained.

> Perusal of the source, while not exhaustive, hasn't turned up anything
> obvious; other than the embedded SQL stuff it seems to be in tune
> with the module layout in the Python docs.
>
> The complications are:-
> - I don't have MSVC, so can't build a standard Windows Python debug
>   setup to really look at the internals;

If you can do any sort of debug build of the interpreter, ceval in a debug
build inserts extra checks to try to catch "some routine set an exception
but didn't return an error value" mistakes sooner when they happen.  Look
for

#ifdef CHECKEXC

in ceval.c to find the extra checks made then.  You could #define CHECKEXC
alone to get that much.

> - I built the module with MinGW32 1.1 (gcc 2.95.3, it compiles with
>   only 1 warning which surprised me a bit) and Python can't yet be
>   built by MinGW (so that I could use gdb) AFAIK.

Beats me what happens then.  It *sounds* like a common mistake in extension
modules, though, so pursue that first.

> ...
> Replies cc'ed to the reply-to address in addition to the list
> appreciated.

Done.  Good luck!