import in threads: crashes & strange exceptions on dual core machines

robert no-spam at no-spam-no-spam.com
Mon Oct 30 08:49:24 EST 2006


I get python crashes and (in better cases) strange Python exceptions when (in most cases) importing and using cookielib lazy on demand in a thread. 
It is mainly with cookielib, but remember the problem also with other imports (e.g. urllib2 etc.). 
And again very often in all these cases where I get weired Python exceptions, the problem is around re-functions - usually during re.compile calls during import (see some of the exceptions below). But not only.

Very strange: The errors occur almost only on machines with dual core/multi processors - and very very rarely on very fast single core machines (>3GHz).

I'm using Python2.3.5 on Win with win32ui (build 210) - the cookielib taken from Python 2.5.

I took care that I'm not starting off thread things or main application loop etc. during an import (which would cause a simple & explainable deadlock freeze on the import lock)

With real OS-level crashes I know from user reports (packaged app), that these errors occur very likely early after app start - thus when lazy imports are likely to do real execution.

I researched this bug for some time. I think I can meanwhile exclude (ref-count, mem.leak) problems in win32ui (the only complex extension lib I use) as cause for this. All statistics point towards import problems.

Any ideas?
Are there problems known with the import lock (Python 2.3.5) ?

(I cannot easily change from Python 2.3 and it takes weeks to get significant feedback after random improvements)

-robert

PS:

The basic pattern of usage is:

==================
def f():
            ...
            opener = urlcookie_openers.get(user)
            if not opener:
                import cookielib                        #<----1
                cj=cookielib.CookieJar()                #<----2  
                build_opener = urllib2.build_opener
                httpCookieProcessor = urllib2.HTTPCookieProcessor(cj)
                if url2_proxy:
                    opener = build_opener(url2_proxy,httpCookieProcessor)
                else:
                    opener = build_opener(httpCookieProcessor)
                opener.addheaders   #$pycheck_no
                opener.addheaders= app_addheaders
                urlcookie_openers[user] = opener
            ufile = opener.open(urllib2.Request(url,data,dict(headers)))
            ...


thread.start_new(f,())
=========================

Symptoms:
__________

sometimes ufile is None and other weired invalid states.

typical Python exceptions when in better cases there is no OS-level crash:

---------

 # Attributes randomly missing like:
 #<----2

"AttributeError: \'module\' object has no attribute \'CookieJar\'\\n"]             


---------

# weired invalid states during computation like:
#<----1

...  File "cookielib.pyo", line 184, in ?\\n\', \'  File
"sre.pyo", line 179, in compile\\n\', \'  File "sre.pyo", line 228, in _compile\\n\', \'  File
"sre_compile.pyo", line 467, in compile\\n\', \'  File "sre_parse.pyo", line 624, in parse\\n\', \'
File "sre_parse.pyo", line 317, in _parse_sub\\n\', \'  File "sre_parse.pyo", line 588, in
_parse\\n\', \'  File "sre_parse.pyo", line 92, in closegroup\\n\', \'ValueError: list.remove(x): x
not in list\\n\'] 
...
'windows', "(5, 1, 2600, 2, 'Service Pack 2')/NP=2")


---------

#<----1


File "cookielib.pyo", line 116, in ?\\n\', \'  File "sre.pyo", line 179, in compile\\n\', \'  File "sre.pyo", line 228, in _compile\\n\', \'  File "sre_compile.pyo", line 467, in compile\\n\', \'  File "sre_parse.pyo", line 624, in parse\\n\', \'  File "sre_parse.pyo", line 317, in _parse_sub\\n\', \'  File "sre_parse.pyo", line 494, in _parse\\n\', \'  File "sre_parse.pyo", line 140, in __setitem__\\n\', \'IndexError: list assignment index out of range\\n\']

('windows', "(5, 1, 2600, 2, 'Service Pack 2')/NP=2"

---------

# weired errors in other threads:

# after dlg.DoModal() in main thread

File "wintools.pyo", line 115, in PreTranslateMessage\\n\', \'TypeError: an integer is required\\n\']

('windows', "(5, 1, 2600, 2, 'Service Pack 2')/NP=2")

---------

# after win32ui.PumpWaitingMessages(wc.WM_PAINT, wc.WM_MOUSELAST) in main thread

\'TypeError: argument list must be a tuple\\n\'


...



More information about the Python-list mailing list