SystemError: com_backpatch: offset too large

[Jeff Epler]

I wrote in a long-ago python-list thread:
> >Python supports a prefix opcode to give 32-bit operand values instead of
> >16-bit values, but the code generator isn't smart enough to use them.
> 
On Thu, Nov 04, 2004 at 11:21:46PM -0800, alban.minassian at wanadoo.fr wrote:
> There is a solution 'force' code generator 32-bit operand values instead of
> 16-bit values ?

The following patch (against CVS head, but it should apply manually to
any recent version of Python) makes all backwards branches use
EXTENDED_ARG unconditionally.  This means that the bytecode will be 3
bytes bigger for each such branch, and the bytecode optimizer will not
run (one of its assumptions is that there is no EXTENDED_ARG.  It should
detect this automatically, but I made it exit early unconditionally.  It
will now run only for functions without backpatched branches, which I
think excludes any function with 'if', 'for', 'and', 'or', chain
comparison, or anything else that makes for a useful function)

A better approach would probably be to add a variable to 'struct
compiling' to denote whether EXTENDED_ARG branches are being used.  It
would be initially set to zero, then if the com_backpatch error code is
hit, the compile is restarted with the flag set and all jumps become
EXTENDED_ARG jumps.  Now, the price is paid only by overly complex
code objects, but it's still paid by all branches.

The best approach would be to use either of the strategies above, but
then fix the optimizer to notice when an EXTENDED_ARG is zero and remove
it, shifting all affected branches accordingly, to get the best possible
code.

I did not run the test suite after this change, but it does fail to
error out on the two pieces of code in my older message. (it also
compiles this statement with the value increased from ~16k to ~300k:
>>> exec "if len:\n" + "\tNone\n" * 300000 + "else: pass"
}

You may use the code below under the terms of the Python license.

Jeff

Index: Python/compile.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Python/compile.c,v
retrieving revision 2.337
diff -u -r2.337 compile.c
--- Python/compile.c	7 Nov 2004 14:04:00 -0000	2.337
+++ Python/compile.c	8 Nov 2004 03:03:11 -0000
@@ -503,6 +503,7 @@
 	unsigned int *blocks = NULL;
 	char *name;
 
+	goto exitUnchanged;
 	/* Bail out if an exception is set */
 	if (PyErr_Occurred())
 		goto exitUnchanged;
@@ -1331,11 +1332,14 @@
 	/* Compile a forward reference for backpatching */
 	int here;
 	int anchor;
-	com_addbyte(c, op);
+	com_addbyte(c, EXTENDED_ARG);
 	here = c->c_nexti;
 	anchor = *p_anchor;
 	*p_anchor = here;
-	com_addint(c, anchor == 0 ? 0 : here - anchor);
+	anchor = anchor == 0 ? 0 : here - anchor;
+	com_addint(c, anchor >> 16);
+	com_addbyte(c, op);
+	com_addint(c, anchor & 0xffff);
 }
 
 static void
@@ -1347,17 +1351,16 @@
 	int prev;
 	for (;;) {
 		/* Make the JUMP instruction at anchor point to target */
-		prev = code[anchor] + (code[anchor+1] << 8);
-		dist = target - (anchor+2);
-		code[anchor] = dist & 0xff;
+		prev = (code[anchor] << 16)+ (code[anchor+1] << 24)
+		       + code[anchor+3] + (code[anchor+4] << 8);
+		dist = target - (anchor+5);
+		code[anchor+3] = dist & 0xff;
 		dist >>= 8;
-		code[anchor+1] = dist;
+		code[anchor+4] = dist & 0xff;
 		dist >>= 8;
-		if (dist) {
-			com_error(c, PyExc_SystemError,
-				  "com_backpatch: offset too large");
-			break;
-		}
+		code[anchor] = dist & 0xff;
+		dist >>= 8;
+		code[anchor+1] = dist & 0xff;
 		if (!prev)
 			break;
 		anchor -= prev;


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20041107/a74df0dc/attachment.sig>