[ python-Bugs-868896 ] Multiple problems with GC in 2.3.3

[SourceForge.net]

Bugs item #868896, was opened at 2004-01-01 14:00
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=868896&group_id=5470

Category: Python Interpreter Core
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: washington irving (washirv)
Assigned to: Nobody/Anonymous (nobody)
Summary: Multiple problems with GC in 2.3.3

Initial Comment:
Hi. We're running a multithreaded application that
spiders some web pages, and parses them. We've had 2
types of problems: one where we have the python process
segfault. Another where python spins in an infinite
loop. We are running on FreeBSD 4.8-RELEASE. We have
not had this problem with 2.2. We have this problem
with both 2.3.2 and 2.3.3. This is repeatable, and
we're willing to help in every way to fix this. I've
attached the gdb stack trace for the process that
segfaulted, and the process that spins in an infinte
loop. We attached to it in gdb and ctrl-c'ed to check
the status. There are 2 separate gdb traces in the
attached file.

Thanks


----------------------------------------------------------------------

>Comment By: Neal Norwitz (nnorwitz)
Date: 2004-01-01 16:16

Message:
Logged In: YES 
user_id=33168

I also don't have enough time to help.  Another way to try
to find the problem is by using valgrind, dmalloc, electric
fence and/or purify or any other memory debugger.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-01-01 15:56

Message:
Logged In: YES 
user_id=31435

Sorry, I can't offer time to help debug this.  Maybe someone 
else can, but since the evidence so strongly points at pycurl 
(see below), it would be best to get someone from that 
project to volunteer.

FYI, *many* incorrect usages of the Python C API first show 
up when cyclic gc is running, simply because gc traverses 
every container object in existence.  gc is thus extremely 
sensitive to coding errors like uninitialized memory, wild 
stores, thread races, and incorrect usage of the C-level GC 
API.  By the time gc suffers the effects of such an error, it's 
*typical* that the code causing the error is long gone, having 
screwed up millions or billions of cycles before gc ran (gc 
doesn't run all that often).

It's also typical that such bugs are eventually traced to 
coding errors in extension modules -- the Python core is too 
heavily exercised by too many users on too many platforms 
for such fundamental bugs to survive long there.

That doesn't mean the Python core can't be at fault, but 
does mean it's unlikely to be in the core.  Add to that that 
the symptoms you report have been reported by, and only by, 
people using pycurl, and the evidence pointing to pycurl is 
simply overwhelming.

At least two earlier reports from pycurl users said Python 2.3 
died with a

    GC object already tracked

fatal error.  That's a new check in 2.3, added to try to catch 
one incorrect usage of the Python C API.  That symptom has 
also been reported only by pycurl users.

BTW, if pycurl also has some sort of debug-mode build option 
(don't know -- haven't used pycurl), it would be good to build 
with that too.

----------------------------------------------------------------------

Comment By: washington irving (washirv)
Date: 2004-01-01 15:30

Message:
Logged In: YES 
user_id=941550

We are using pycurl. And we have reported it, in case. The
reason I'm reporting it here is that the stack trace does
not involve pycurl in any way. it seems to be python all the
way. I will build with pydebug and report back.

The only test case I have is the program we're running now.
We haven't yet managed to reduce it to a simple test case.
We're working on it. We would be open to figuring out a way
to give you access to the code itself...

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-01-01 14:31

Message:
Logged In: YES 
user_id=31435

Are you using the pycurl extension module?  If so, you should 
report your problems to the pycurl project too.  I don't know 
whether the problem *is* in pycurl, but the only reports of 
this type ever seen before have come from people using 
pycurl.  If you're not using pycurl, it would be good to know 
that too.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2004-01-01 14:14

Message:
Logged In: YES 
user_id=33168

Have you tried building python --with-pydebug?  That may
lead to an assert or some other indication of the problem. 
Do you have a test case to reproduce this problem?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=868896&group_id=5470