[ python-Bugs-736428 ] allow HTMLParser error recovery

[SourceForge.net]

Bugs item #736428, was opened at 2003-05-12 12:37
Message generated for change (Comment added) made by kingswood
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=736428&group_id=5470

Category: Python Library
Group: Feature Request
Status: Open
Resolution: None
Priority: 5
Submitted By: Steven Rosenthal (smroid)
Assigned to: Nobody/Anonymous (nobody)
Summary: allow HTMLParser error recovery

Initial Comment:
I'm using 2.3a2.

HTMLParser correctly raises a "malformed start tag"
error on:

<meta NAME=DESCRIPTION Content=Lands&#039; End quality...
outerwear and more.> 

because my application is imprecise by nature (web
scraping), I want to be able to continue after such errors.

I can override the error() method to not raise an
exception. To make this work, I also needed to alter
HTMLParser.py, near line 316, to read as:

            self.updatepos(i, j)
            self.error("malformed start tag")
            return j                    #  ADDED THIS LINE
        raise AssertionError("we should not get here!")

My enhancement request is for every place where
self.error() is called, to ensure that the "override
error() to not raise an exception" continuation
strategy works as well as can be hoped.

Thanks,

Steve


----------------------------------------------------------------------

Comment By: Frank Vorstenbosch (kingswood)
Date: 2004-04-03 19:04

Message:
Logged In: YES 
user_id=555155

This problem is actually more widespread than previously
indicated. Not only do all calls to self.error where that
function returns need to cope with that, and recover (the
HTMLParser defines that every character in the input will be
visited exactly once), but other modules are also affected.

In particular, feeding HTML (from spam) with a tag <!12345>
into HTMLParser causes markupbase._scan_name to emit an
error that now needs to recover.

The patch in #917188 may be better than the one suggested
here as it deals with all places where self.error() can return.
More is needed to fix the problem completely.
In markupbase.py, at least this is necessary

--- markupbase.py.orig  Sat Apr 03 17:43:48 2004
+++ markupbase.py       Sat Apr 03 18:02:48 2004
@@ -377,6 +377,8 @@
         else:
             self.updatepos(declstartpos, i)
             self.error("expected name token")
+        return None,rawdata.find(">",i)

     # To be overridden -- handlers for unknown objects
     def unknown_decl(self, data):

----------------------------------------------------------------------

Comment By: Frank Vorstenbosch (kingswood)
Date: 2004-03-16 09:53

Message:
Logged In: YES 
user_id=555155

Fixed by my patch against 2.3.3.
The patch adds recovery to ensure progress and tries to not
miss any data in the input.
The error() method is now commented as being overridable,
just def error(): pass to ignore any parsing errors.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=736428&group_id=5470