right list for SIGABRT python binary question ?

Wed Nov 1 07:40:56 EDT 2017

> my assumption would be that something clobbers 0x6aab7c,
> which seems to be in (?) _Py_ZeroStruct in this run. I'll
> re-run a few times to make sure the corruption "reliably"
> hits _Py_ZeroStruct.
> 
> If so, I'll set a memory write breakpoint on _Py_ZeroStruct.

Interestingly, on subsequent runs, it seems to hit the same
address, 0x6aab7c, belonging to the same symbol, _Py_ZeroStruct.

This is what happens:

	(gdb) watch *0x6aab7c
	Hardware watchpoint 1: *0x6aab7c
	(gdb) run
	Starting program: /usr/bin/python2.7-dbg ./bootstrap_gm_db_system.py --log-file=./bootstrap-latest.log --conf-file=bootstrap-latest.conf --
	[Thread debugging using libthread_db enabled]
	Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

	Hardware watchpoint 1: *0x6aab7c

	Old value = 0
	New value = -1208182272
	_Py_AddToAllObjects (op=False, force=0) at ../Objects/object.c:70
	70      ../Objects/object.c: Datei oder Verzeichnis nicht gefunden.
	(gdb)

which means I'll probably have to apply the delayed
breakpoint setting strategy, or else it is just some initial
relocation at startup. Let's see what "cont" brings. The next
hit after the Python script has run until just before it
usually aborts:

	Hardware watchpoint 1: *0x6aab7c

	Old value = -1208182272
	New value = 0
	_Py_ForgetReference (op=False) at ../Objects/object.c:2255
	2255    in ../Objects/object.c
	(gdb)

The backtrace at this point says:

	(gdb) bt
	#0  _Py_ForgetReference (op=False) at ../Objects/object.c:2255
	#1  0x00497be0 in _Py_Dealloc (op=False) at ../Objects/object.c:2261
	#2  0x004885d7 in insertdict_by_entry (mp=0xb7fc5674, key='dont_write_bytecode', hash=591857026, ep=0x7c5c08, value=None) at ../Objects/dictobject.c:519
	#3  0x00488857 in insertdict (mp=0xb7fc5674, key='dont_write_bytecode', hash=591857026, value=None) at ../Objects/dictobject.c:556
	#4  0x0048910f in dict_set_item_by_hash_or_entry (
	    op={'setrecursionlimit': None, 'dont_write_bytecode': None, 'getfilesystemencoding': <built-in function getfilesystemencoding>, 'long_info': <sys.long_info at remote
	0xb7f936e8>, 'path_importer_cache': None, 'stdout': <file at remote 0xb7fcd098>, 'getprofile': <built-in function getprofile>, '__stdin__': <file at remote 0xb7fcd028>, 'version_info': <sys.version_info at remote 0xb7fcfd80>, 'exc_clear': <built-in function exc_clear>, 'gettotalrefcount': <built-in function gettotalrefcount>, 'getrefcount': <built-in function getrefcount>, 'byteorder': 'little', '_clear_type_cache': None, 'excepthook': <built-in function excepthook>, 'subversion': ('CPython', '', ''), '_multiarch': None, 'exc_type': None, 'ps1': None, '__excepthook__': <built-in function excepthook>, 'executable': '/usr/bin/python2.7-dbg', 'float_info': None, 'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright (c) 1995-2001 Corporation for Nation...(truncated), key='dont_write_bytecode', hash=591857026, ep=0x0, value=None) at ../Objects/dictobject.c:795
	#5  0x00489285 in PyDict_SetItem (
	    op={'setrecursionlimit': None, 'dont_write_bytecode': None, 'getfilesystemencoding': <built-in function getfilesystemencoding>, 'long_info': <sys.long_info at remote
	    0xb7f936e8>, 'path_importer_cache': None, 'stdout': <file at remote 0xb7fcd098>, 'getprofile': <built-in function getprofile>, '__stdin__': <file at remote 0xb7fcd028>, 'version_info': <sys.version_info at remote 0xb7fcfd80>, 'exc_clear': <built-in function exc_clear>, 'gettotalrefcount': <built-in function gettotalrefcount>, 'getrefcount': <built-in function getrefcount>, 'byteorder': 'little', '_clear_type_cache': None, 'excepthook': <built-in function excepthook>, 'subversion': ('CPython', '', ''), '_multiarch': None, 'exc_type': None, 'ps1': None, '__excepthook__': <built-in function excepthook>, 'executable': '/usr/bin/python2.7-dbg', 'float_info': None, 'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright (c) 1995-2001 Corporation for Nation...(truncated), key='dont_write_bytecode', value=None) at ../Objects/dictobject.c:848
	#6  0x0049281f in _PyModule_Clear (m=<module at remote 0xb7f935d4>) at ../Objects/moduleobject.c:139
	#7  0x0054a3ec in PyImport_Cleanup () at ../Python/import.c:540
	#8  0x0055c53c in Py_Finalize () at ../Python/pythonrun.c:458
	#9  0x0055fe9c in Py_Exit (sts=1) at ../Python/pythonrun.c:1783
	#10 0x0055e0fc in handle_system_exit () at ../Python/pythonrun.c:1151
	#11 0x0055e152 in PyErr_PrintEx (set_sys_last_vars=1) at ../Python/pythonrun.c:1161
	#12 0x0055dd5b in PyErr_Print () at ../Python/pythonrun.c:1064
	#13 0x0055d61f in PyRun_SimpleFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:952
	#14 0x0055cc4e in PyRun_AnyFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:752
	#15 0x00577cb0 in Py_Main (argc=5, argv=0xbffff684) at ../Modules/main.c:645
	#16 0x004259c8 in main (argc=5, argv=0xbffff684) at ../Modules/python.c:20

And continuing hits the SIGABRT right away:

	(gdb) cont
	Continuing.
	Debug memory block at address p=0x6aab7c: API ''
	    0 bytes originally requested
	    The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
	        at p-3: 0x33 *** OUCH
	        at p-2: 0x47 *** OUCH
	        at p-1: 0x00 *** OUCH
	    Because memory is corrupted at the start, the count of bytes requested
	       may be bogus, and checking the trailing pad bytes may segfault.
	    The 4 pad bytes at tail=0x6aab7c are not all FORBIDDENBYTE (0xfb):
	        at tail+0: 0x00 *** OUCH
	        at tail+1: 0x00 *** OUCH
	        at tail+2: 0x00 *** OUCH
	        at tail+3: 0x00 *** OUCH
	    The block was made by call #0 to debug malloc/realloc.
	Fatal Python error: bad ID: Allocated using API '', verified using API 'o'

	Program received signal SIGABRT, Aborted.
	0xb7fd9ce9 in __kernel_vsyscall ()
	(gdb)

Does that help ?

Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346