[ python-Bugs-1721309 ] make testall shows many glibc detected malloc corruptions

Sun May 20 04:18:15 CEST 2007

Bugs item #1721309, was opened at 2007-05-18 06:42
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1721309&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Extension Modules
Group: Python 2.5
Status: Open
Resolution: Fixed
Priority: 5
Private: No
Submitted By: David Favor (dfavor)
Assigned to: Neal Norwitz (nnorwitz)
Summary: make testall shows many glibc detected malloc corruptions

Initial Comment:
export _MALLOC_CHECK_=1
make testall

test_bsddb3
*** glibc detected *** ./python: free(): invalid pointer: 0x40b92de4 ***
*** glibc detected *** ./python: free(): invalid pointer: 0x40b92de4 ***
*** glibc detected *** ./python: free(): invalid pointer: 0x40b92de4 ***
*** glibc detected *** ./python: free(): invalid pointer: 0x40b92de4 ***
*** glibc detected *** ./python: free(): invalid pointer: 0x40b92de4 ***
*** glibc detected *** ./python: free(): invalid pointer: 0x40b92de4 ***

followed by deadlock errors.

malloc: using debugging hooks
blattest test_tempfile failed -- Traceback (most recent call last):
  File "/build/work/Python-2.5.1/Lib/test/test_tempfile.py", line 310, in test_noinherit
    self.failIf(retval > 0, "child process reports failure %d"%retval)
AssertionError: child process reports failure 1

test_timeout
test test_timeout failed -- Traceback (most recent call last):
  File "/build/work/Python-2.5.1/Lib/test/test_timeout.py", line 128, in testConnectTimeout
    %(_delta, self.fuzz, _timeout))
AssertionError: timeout (8.20631) is more than 2 seconds more than expected (0.001)

----------------------------------------------------------------------

>Comment By: Neal Norwitz (nnorwitz)
Date: 2007-05-19 19:18

Message:
Logged In: YES 
user_id=33168
Originator: NO

Sigh, 4.4.20 and 4.5.20 behave differently.  I tested on both, but botched
it. :-(

It seemed like a good idea to have at least one of the buildbots to use
4.5.20.  I upgraded the g4 bot to 4.5.20, since that didn't have any
version of bsddb on it.  I also installed 4.4.20 on the amd64 buildbot. 
The x86 gentoo bot probably has 4.1, but it's down right now.  Martin's
machine has 4.2.

Both versions (4.4 and 4.5) require the flag DB_DBT_MALLOC (get() returns
an error 22 otherwise).  However only 4.4 actually allocates memory.  4.5
doesn't.  I checked in revs 55457 (2.5) which should handle this better. 
Could someone please test and verify I really fixed the problem this time. 
If it's really fixed, close this bug report.

The real change is to conditionally free data if it is a different pointer
than we pass in (orig_data).  Don't free if they are the same pointer.

Tested 4.5 memory leaks with the leaks program and 4.4 with valgrind.
File Added: db.c

----------------------------------------------------------------------

Comment By: David Favor (dfavor)
Date: 2007-05-19 08:05

Message:
Logged In: YES 
user_id=370230
Originator: YES

Just built with latest
http://svn.python.org/projects/python/branches/release25-maint
pull and problem still remains.

----------------------------------------------------------------------

Comment By: David Favor (dfavor)
Date: 2007-05-19 08:04

Message:
Logged In: YES 
user_id=370230
Originator: YES

My test was done with latest BerkeleyDB 4.5.20 plain (no patches).

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2007-05-19 03:31

Message:
Logged In: YES 
user_id=849994
Originator: NO

I can confirm that the checkin didn't change anything for me.

I use bsddb version 4.5.20_p2 from Gentoo.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2007-05-19 00:08

Message:
Logged In: YES 
user_id=21627
Originator: NO

(I'm not convinced that Neal's patch could actually fix anything)

Can people who were able to reproduce this please report their bdb version
also; I was using 4.4.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2007-05-19 00:04

Message:
Logged In: YES 
user_id=21627
Originator: NO

dfavor: I see. It is MALLOC_CHECK_, not _MALLOC_CHECK (as I used it), and
not _MALLOC_CHECK_ (as you wrote). I now see that debug hooks are used, but
I still cannot reproduce the problem...

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2007-05-18 21:38

Message:
Logged In: YES 
user_id=33168
Originator: NO

I screwed up the first checkin.  The CLEAR_DBT was in the wrong place. 
The fixes are in:

Committed revision 55452. 
Committed revision 55454. (2.5)

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2007-05-18 20:54

Message:
Logged In: YES 
user_id=33168
Originator: NO

Thanks for the report David.  It would be great if you could test the
change.  I couldn't find any other places that had a similar problem, but
you never know.  You will need to pull out of SVN.  It's a one line change
if you want to apply manually.

Committed revision 55450.
Committed revision 55451. (2.5)

----------------------------------------------------------------------

Comment By: David Favor (dfavor)
Date: 2007-05-18 14:26

Message:
Logged In: YES 
user_id=370230
Originator: YES

The os I'm running is Fedora 6 with latest patches.

gcc is 4.1.1

glibc is 2.5

I had to set MALLOC_CHECK_=1 just to get the suite to run as normal
MALLOC_CHECK_ defaults to die when something evil occurs.

I'll build gcc 4.2, rebuild and see what occurs.

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2007-05-18 14:02

Message:
Logged In: YES 
user_id=849994
Originator: NO

I don't even need to set an environment variable to get

~/devel/python> ./python Lib/test/regrtest.py -uall test_bsddb3
test_bsddb3
*** glibc detected *** ./python: free(): invalid pointer: 0x0827874c ***
[1]    17141 abort      ./python Lib/test/regrtest.py -uall test_bsddb3

(HEAD, linux x86, gcc 4.1.2, glibc 2.5)

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2007-05-18 10:43

Message:
Logged In: YES 
user_id=21627
Originator: NO

Unfortunately, I cannot reproduce this problem on Debian unstable (x86,
gcc 4.1.3, glibc 2.5) - 
the test passes just fine with _MALLOC_CHECK set.

Can you provide a few more details: glibc version, compiler version, Linux
distribution, processor architecture, precise Python version being test,
does the error happen on the first or a subsequent cycle of all tests?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1721309&group_id=5470