[Python-Dev] binary incompatibility and tp_flags (was: Linker problems on Linux)

[Greg Stein]

On Sun, Jul 16, 2000 at 11:25:53PM +0200, Thomas Wouters wrote:
> On Sun, Jul 16, 2000 at 10:57:47PM +0200, Martin v. Loewis wrote:
> 
> > For example, if you have a 1.5.2-compiled type in an extension module,
> > and that type happens to implement PySequenceMethods, then applying 'a
> > in b' will likely crash: Python 2 will check for the sq_contains
> > field, but that is past the end of the function table in the old
> > module.
> 
> Nope. This is specifically fixed by the new tp_flags structmember (which was
> 'unused' before, so reading *that* will work ok) and the PyType_HasFeature
> macro. All code that wants to access one of the really new struct members
> (currently, bf_getcharbuffer, sq_contains and the cycle-gc stuff. And my
> augmented assignment patch adds another 13 ;) should check tp_flags first.
> 
> See the comment in Include/object.h regarding this.

Actually, those flags should only be used if you are attempting to maintain
binary compatibility.

At the point where you say "okay. binary compatibility is hereby broken.",
then we go and remove the flags for the new structure members.

The concept is pretty simple:

*) if an extension was compiled before the addition of the new member, then
   the core should not attempt to access it. to detect this case, the core
   looks at the flag bit, which will be zero.

*) when the extension is recompiled, the struct grows according to the new
   definition.
   
   -) The extension may still have zero in there, so Python still won't
      look, even though it could
   -) The extension added Py_TPFLAGS_DEFAULT and picks up the bit that says
      the new struct member is there. It is probably NULL, unless the author
      has filled in a value.

The last API bump was to 1009 for "Unicode API added". I'm not sure why
Guido bumped it since the *addition* of an API shouldn't create binary
incompatibility.

Regardless, I think we have introduced several semantic incompatibilities or
even signature changes in the APIs. We should do a bump for safety's sake
and we should toss the two Py_TPFLAGS_HAVE_* macros (and the corresponding
tests within the core interpreter).

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/