[ python-Bugs-914148 ] xml.sax segfault on error
SourceForge.net
noreply at sourceforge.net
Tue Oct 4 06:50:59 CEST 2005
Bugs item #914148, was opened at 2004-03-11 06:14
Message generated for change (Settings changed) made by nnorwitz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=914148&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: XML
Group: Python 2.3
Status: Open
Resolution: None
>Priority: 6
Submitted By: Adam Sampson (adamsampson)
Assigned to: Neal Norwitz (nnorwitz)
Summary: xml.sax segfault on error
Initial Comment:
While (mistakenly) using Mark Pilgrim's feedparser
module to parse data from
<http://www.gothamist.com/archives/news_nyc/index.php>,
Python segfaults when it should invoke an error handler
for invalid XML. The attached code demonstrates the
problem; it occurs with Python 2.2.3 and 2.3.3 on my
system. I've tried to chop the example data down as far
as possible, but reducing it any further doesn't
exhibit the problem (it's currently just above 64k,
which might be a coincidence).
The gdb traceback I get from the example is as follows:
#0 normal_updatePosition (enc=0x404a4fc0,
ptr=0x40682000 <Address 0x40682000 out of bounds>,
end=0x81e87e0 "a></div>\n\n<div
id=\content\>\n\n<div class=\blog\>\n<!--\n<rdf:RDF
xmlns:rdf=\http://www.w3.org/1999/02/22-rdf-syntax-ns#\\n
xmlns:trackback=\http://madskills.com/public/xml/rss/module/trackback/\\n"...,
pos=0x81e7dac)
at
/120g/gar/python/python23/work/Python-2.3.3/Modules/expat/xmltok_impl.c:1745
#1 0x40484288 in XML_GetCurrentLineNumber
(parser=0x81e7c18)
at
/120g/gar/python/python23/work/Python-2.3.3/Modules/expat/xmlparse.c:1605
#2 0x40481fc5 in set_error (self=0x0,
code=XML_ERROR_TAG_MISMATCH)
at
/120g/gar/python/python23/work/Python-2.3.3/Modules/pyexpat.c:124
#3 0x40480ae7 in xmlparse_Parse (self=0x402fddac,
args=0x0)
at
/120g/gar/python/python23/work/Python-2.3.3/Modules/pyexpat.c:888
#4 0x080fc25a in PyCFunction_Call (func=0x402faa0c,
arg=0x402f338c,
kw=0xfffffffb) at Objects/methodobject.c:108
#5 0x080aa674 in call_function (pp_stack=0xbffff03c,
oparg=0)
at Python/ceval.c:3439
#6 0x080a8a2e in eval_frame (f=0x816e45c) at
Python/ceval.c:2116
#7 0x080a95bc in PyEval_EvalCodeEx (co=0x40303de0,
globals=0xfffffffb,
locals=0x0, args=0x816e5a8, argcount=2,
kws=0x816a9fc, kwcount=0,
defs=0x40321678, defcount=1, closure=0x0) at
Python/ceval.c:2663
#8 0x080aa729 in fast_function (func=0xfffffffb,
pp_stack=0xbffff1bc, n=2,
na=0, nk=135703028) at Python/ceval.c:3529
#9 0x080aa56c in call_function (pp_stack=0xbffff1bc,
oparg=0)
at Python/ceval.c:3458
#10 0x080a8a2e in eval_frame (f=0x816a894) at
Python/ceval.c:2116
#11 0x080a95bc in PyEval_EvalCodeEx (co=0x402fd2a0,
globals=0xfffffffb,
locals=0x0, args=0x402f3318, argcount=2, kws=0x0,
kwcount=0, defs=0x0,
defcount=0, closure=0x0) at Python/ceval.c:2663
#12 0x080fbda7 in function_call (func=0x4030617c,
arg=0x402f330c, kw=0x0)
at Objects/funcobject.c:504
#13 0x0805b899 in PyObject_Call (func=0x40682000,
arg=0x0, kw=0x0)
at Objects/abstract.c:1755
#14 0x08062288 in instancemethod_call (func=0x4030617c,
arg=0x402f330c, kw=0x0)
at Objects/classobject.c:2433
#15 0x0805b899 in PyObject_Call (func=0x40682000,
arg=0x0, kw=0x0)
at Objects/abstract.c:1755
#16 0x080aa892 in do_call (func=0x4032025c,
pp_stack=0x402f330c, na=0, nk=0)
at Python/ceval.c:3644
#17 0x080aa4f9 in call_function (pp_stack=0xbffff5fc,
oparg=0)
at Python/ceval.c:3460
#18 0x080a8a2e in eval_frame (f=0x818b414) at
Python/ceval.c:2116
#19 0x080aa7ad in fast_function (func=0xfffffffb,
pp_stack=0xbffff71c, n=2,
na=0, nk=1076865996) at Python/ceval.c:3518
#20 0x080aa56c in call_function (pp_stack=0xbffff71c,
oparg=0)
at Python/ceval.c:3458
#21 0x080a8a2e in eval_frame (f=0x8183814) at
Python/ceval.c:2116
#22 0x080a95bc in PyEval_EvalCodeEx (co=0x402ed2a0,
globals=0xfffffffb,
locals=0x0, args=0x0, argcount=0, kws=0x0,
kwcount=0, defs=0x0,
defcount=0, closure=0x0) at Python/ceval.c:2663
#23 0x080abdb9 in PyEval_EvalCode (co=0x0, globals=0x0,
locals=0x0)
at Python/ceval.c:537
#24 0x080d7d2b in run_node (n=0x402bb79c, filename=0x0,
globals=0x0,
locals=0x0, flags=0x0) at Python/pythonrun.c:1265
#25 0x080d74df in PyRun_SimpleFileExFlags (fp=0x8139050,
filename=0xbffffa4d "testexpat.py",
closeit=-1073743283, flags=0xbffff878)
at Python/pythonrun.c:862
#26 0x08054dd5 in Py_Main (argc=1, argv=0xbffff8f4) at
Modules/main.c:415
#27 0x0805492b in main (argc=0, argv=0x0) at
Modules/python.c:23
----------------------------------------------------------------------
Comment By: Neal Norwitz (nnorwitz)
Date: 2005-10-02 18:26
Message:
Logged In: YES
user_id=33168
This doesn't crash for me on 2.5 (CVS) (2.4 seems about the
same), but there is use of uninitialized memory in valgrind.
(It does crash when running under valgrind.) Either way,
there is still a problem.
Do you still see the crash?
==26881== Use of uninitialised value of size 8
==26881== at 0x12BA4518: normal_updatePosition
(xmltok_impl.c:1745)
==26881== by 0x12B91005: XML_GetCurrentLineNumber
(xmlparse.c:1782)
==26881== by 0x12B884EA: set_error (pyexpat.c:125)
==26881== by 0x12B8BCCD: get_parse_result (pyexpat.c:901)
==26881== by 0x12B8BD6A: xmlparse_Parse (pyexpat.c:923)
==26881== by 0x4E00B0: PyCFunction_Call (methodobject.c:73)
==26881== by 0x48B2F3: call_function (ceval.c:3580)
==26881== by 0x486F4C: PyEval_EvalFrameEx (ceval.c:2181)
==26881== by 0x48932B: PyEval_EvalCodeEx (ceval.c:2754)
==26881== by 0x48B76E: fast_function (ceval.c:3673)
==26881== by 0x48B436: call_function (ceval.c:3601)
==26881== by 0x486F4C: PyEval_EvalFrameEx (ceval.c:2181)
==26881== by 0x48932B: PyEval_EvalCodeEx (ceval.c:2754)
==26881== by 0x4DF7FA: function_call (funcobject.c:548)
==26881== by 0x418DE2: PyObject_Call (abstract.c:1777)
==26881==
==26881== Invalid read of size 1
==26881== at 0x12BA4514: normal_updatePosition
(xmltok_impl.c:1745)
==26881== by 0x12B91005: XML_GetCurrentLineNumber
(xmlparse.c:1782)
==26881== by 0x12B884EA: set_error (pyexpat.c:125)
==26881== by 0x12B8BCCD: get_parse_result (pyexpat.c:901)
==26881== by 0x12B8BD6A: xmlparse_Parse (pyexpat.c:923)
==26881== by 0x4E00B0: PyCFunction_Call (methodobject.c:73)
==26881== by 0x48B2F3: call_function (ceval.c:3580)
==26881== by 0x486F4C: PyEval_EvalFrameEx (ceval.c:2181)
==26881== by 0x48932B: PyEval_EvalCodeEx (ceval.c:2754)
==26881== by 0x48B76E: fast_function (ceval.c:3673)
==26881== by 0x48B436: call_function (ceval.c:3601)
==26881== by 0x486F4C: PyEval_EvalFrameEx (ceval.c:2181)
==26881== by 0x48932B: PyEval_EvalCodeEx (ceval.c:2754)
==26881== by 0x4DF7FA: function_call (funcobject.c:548)
==26881== by 0x418DE2: PyObject_Call (abstract.c:1777)
==26881== Address 0x12B720C0 is 0 bytes after a block of
size 131072 alloc'd
==26881== at 0x11B19F13: malloc (vg_replace_malloc.c:149)
==26881== by 0x12B90AB5: XML_GetBuffer (xmlparse.c:1634)
==26881== by 0x12B906B0: XML_Parse (xmlparse.c:1528)
==26881== by 0x12B8BD5F: xmlparse_Parse (pyexpat.c:923)
==26881== by 0x4E00B0: PyCFunction_Call (methodobject.c:73)
==26881== by 0x48B2F3: call_function (ceval.c:3580)
==26881== by 0x486F4C: PyEval_EvalFrameEx (ceval.c:2181)
==26881== by 0x48932B: PyEval_EvalCodeEx (ceval.c:2754)
==26881== by 0x48B76E: fast_function (ceval.c:3673)
==26881== by 0x48B436: call_function (ceval.c:3601)
==26881== by 0x486F4C: PyEval_EvalFrameEx (ceval.c:2181)
==26881== by 0x48932B: PyEval_EvalCodeEx (ceval.c:2754)
==26881== by 0x4DF7FA: function_call (funcobject.c:548)
==26881== by 0x418DE2: PyObject_Call (abstract.c:1777)
==26881== by 0x421034: instancemethod_call
(classobject.c:2447)
----------------------------------------------------------------------
Comment By: Jozef Behran (jojoworks)
Date: 2004-04-03 11:28
Message:
Logged In: YES
user_id=688090
Now when I thinked out a way how to put a breakpoint into
shared library and get GDB stopped on it I investigated on
this bug more and found following:
The bug is in XML_GetBuffer() located at xmlparse.c:1498.
When this call realizes that the buffer is too small, a
larger one is allocated and data are copied. The problem is
that m_eventPtr is not transformed to be pointing into the
new buffer during this transaction and so is still pointing
to the old (and invalid) buffer.
In the case described here the invalid pointer "m_eventPtr"
(invalidated after the buffer was moved by XML_GetBuffer) is
passed (xmlparse:1606) to XmlUpdatePosition(), which assumes
that it is valid. The XmlUpdatePosition() call touches
memory through the pointer, falls into "memory hole"
(because the memory to which the pointer is pointing was
freed) and segfaults.
----------------------------------------------------------------------
Comment By: Jozef Behran (jojoworks)
Date: 2004-03-19 14:45
Message:
Logged In: YES
user_id=688090
Result of a brief scan:
When the exceptional situation happens, the pyexpat.c module
trashes parser->m_positionPtr (aliased as positionPtr) (see
modules/expat/xmlparse.c, function
XML_GetCurrentLineNumber() and similar). When the
errorhandler forgets to raises an exception or exit, the
module tries to access memory through the garbage pointer
and segfaults.
It seems to be buffer overrun bug: the pointer gets thrashed
when some sort of input (the erroneous entity) is large
enough to reach it (or variable from which the pointer is
fetched).
It is impossible for me to do further investigations because
gdb/Mangrake GNU/Linux refuses to trace dlopen()ed shared
object and I don't understand it's code enough to "debug" it
by hand.
----------------------------------------------------------------------
Comment By: Mark Moraes (moraes)
Date: 2004-03-15 02:26
Message:
Logged In: YES
user_id=390363
#! /usr/bin/env python
dhead = """<?xml version="1.0" encoding="ISO-8859-1" ?>
<item><title>»</title></item>
<item><title>
"""
dtail = """</title></item>
"""
import xml.sax
from cStringIO import StringIO as _StringIO
class _StrictFeedParser:
def _err(self, errtype, exc):
print errtype, exc.getMessage(), 'line', exc.getLineNumber(), 'column', exc.getColumnNumber()
def fatalError(self, exc):
self._err('fatalError', exc)
# raise exc # avoids the problem
def error(self, exc):
self._err('error', exc)
def warning(self, exc):
self._err('warning', exc)
def parse(data):
feedparser = _StrictFeedParser()
saxparser = xml.sax.make_parser(["drv_libxml2"])
saxparser.setErrorHandler(feedparser)
source = xml.sax.xmlreader.InputSource()
source.setByteStream(_StringIO(data))
saxparser.parse(source)
if __name__ == '__main__':
for i in xrange(65427,66000,1):
print i
parse(dhead + 'x'*i + dtail)
----------------------------------------------------------------------
Comment By: Mark Moraes (moraes)
Date: 2004-03-15 02:22
Message:
Logged In: YES
user_id=390363
I ran into this as well -- turns out that 64k is relevant: I
have a simpler script that reproduces this problem -- create
an unterminated character ref such as "«" without the
trailing semi-colon and add roughly 64k of data after it.
The crash occurs if the sax parser has an ErrorHandler set
where the fatalError() method returns normally instead of
terminating/raising the exception.
As a defensive measure, I suggest that any call to the
fatalError method be followed by a raise of the exception if
fatalError returns.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=914148&group_id=5470
More information about the Python-bugs-list
mailing list