[Python-Dev] Invalid memory read in PyObject_Free

[Tim Peters]

[Neal Norwitz]
> ...
> I never got around to submitting the attached patch which
> makes it easier and safer to use memory testing tools such
> as Purify and Valgrind.  The suppression then only needs to
> be applied to ADDRESS_IN_RANGE.

Please apply the patch, so that people can use this in 2.3 final.  Nits:

+ USING_MEMORY_DEBUGGER should not be defined by default.

+ USING_MEMORY_DEBUGGER should be documented in Misc/SpecialBuilds.txt.
  That's also a good place to put sample incantations to make
  Purify (etc) shut up about this.

+ USING_MEMORY_DEBUGGER should probably grow a Py_ prefix.

+ The declaration and definition of the function form of ADDRESS_IN_RANGE
  should be declared static.

+ C functions in Python source should be defined like

decorations
function_name(...

  instead of like

decorations function_name(...

> I haven't noticed a problem with pymalloc on Linux, Solaris, Tru64,
> AIX, HP-UX, FreeBSD.  So far there haven't seemed to be any problems
> with pymalloc.

Its non-standard assumptions are mild.  I don't know of any architecture now
where a pointer can't be faithfully cast to some unsigned integral type.
The biggest insecurity remaining then is addressed in a comment:

...

 * In theory, if SYSTEM_PAGE_SIZE is larger than the native page
 * size, then `POOL_ADDR(p)->arenaindex' could rarely cause a segmentation
 * violation fault.  4K is apparently OK for all the platforms that python
 * currently targets.

This comment in turn assumes that whatever memory protection gimmicks an OS
implements are at page granularity.  This assumption could be violated too
by, e.g., an OS that resolved memory read permission to word (or even byte)
boundaries, but I expect that's very rare due to lack of direct HW support.

> Somtimes, there are benefits to turning off pymalloc from time to time
> in order to diagnose memory in use

Really?  In a PYMALLOC_DEBUG build, and when the envar PYTHONMALLOCSTATS is
set, pymalloc delivers an exact accounting of every byte pymalloc knows
about.  Even bytes "lost" to various alignment requirements are accounted
for.

> and some other memory related issues.  Usually, pymalloc is a big win.

I agree <wink>.