Simple allowing of HTML elements/attributes?

Fri Feb 13 10:01:22 EST 2004

[Alan Kennedy]
>> The optimal solution, IMHO, is to tidy the HTML into XML, and then use
>> SAX to filter out the stuff you don't want. Here is some code that
>> does the latter. This should be nice and fast, and use a lot less
>> memory than object-model based approaches.

[Robert Brewer]
> I rolled my own solution to this the other day,
> relying more on regexes. This might be more usable.

Not sure what you're saying here Robert. Certainly, a regex based
solution has the potential to be faster than the XML+SAX technique,
but is likely to be *much* harder to verify correct and secure
operation.

> One issue: the parser, as written, mangles well-formed xhtml tags like
> <br /> into <br></br>. Any recommendations besides brute-force (keeping
> a list of allowed empty tags) for dealing with this?

I'm assuming that when you say "the parser, as written" you mean my
code?

The issue of end-tags and empty elements is not a simple one.

If one wants to emit XML, then a start-tag + end-tag combination is
equivalent to an empty element tag, i.e. ' ' should be
interchangable with ' '.

http://www.w3.org/TR/2004/REC-xml-20040204/#dt-content

The statement "The representation of an empty element is either a
start-tag immediately followed by an end-tag, or an empty-element tag"
shows that, as far as XML is concerned, either must be acceptable to
XML processors.

Note the statement "For interoperability, the empty-element tag SHOULD
be used, and SHOULD only be used, for elements which are declared
EMPTY". The "SHOULD" implies that this is only recommended behaviour,
not required. Also, "declared empty" implies that this recommended
behaviour SHOULD only be enforced when a DTD is in use, implying the
use of a validating XML parser.

Since XHTML is simply an XML document type, all XHTML compliant
processors should accept both (i.e. ' ' and ' ') to
represent the same thing, an element with no children.

However, we're talking about HTML here, and there is a lot of
non-X[HT]ML aware software out there. Many of it (e.g. Netscape 4.x)
would not accept *either* option: they only render a line-break if
presented with ' ' (or maybe '<br' :-). So in order to generate
HTML that will work in these browsers, we should only output ' '
when generating for an older user-agent.

Which means that the filter I presented would have to different code
paths, depending on the user-agent it is rendering for.

The cleanest solution to this problem is to separate out the content
filtering from the output phase. So we do the output in a second
handler, which can be changed depending on the desired output, and our
content-filtering algorithm is then isolated from the output code.

Note that XSLT addresses this problem by allowing the user to declare
the "output method", so that they can indicate whether or not they
want well-formed XML in the output.

http://www.w3.org/TR/xslt#section-HTML-Output-Method

Here is an implementation of an XML output method.

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

class xml_output(xml.sax.handler.ContentHandler):

  def __init__(self):
    xml.sax.handler.ContentHandler.__init__(self)
    self.outbuf = StringIO.StringIO()

  def startElement(self, elemname, attrs):
    attrstr = ""
    for a in attrs.keys():
      attrstr = "%s%s" % (attrstr, " %s=%s" % (a,
quoteattr(attrs[a])))
    self.outbuf.write("<%s%s>" % (elemname, attrstr))

  def endElement(self, elemname):
    self.outbuf.write("</%s>" % (elemname,))

  def characters(self, s):
    self.outbuf.write("%s" % (escape(s),))

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

And we have to change the cleaning filter so that it now consumes and
generates SAX events, rather than simply outputting the data.

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

class cleaning_filter(xml.sax.handler.ContentHandler):

  def __init__(self, output_method):
    xml.sax.handler.ContentHandler.__init__(self)
    self.outmeth = output_method

  def startElement(self, elemname, attrs):
    if elemname in permittedElements:
      newattrs = {}
      for a in attrs.keys():
        if a in permittedAttrs:
          newattrs[a] = attrs[a]
      self.outmeth.startElement(elemname, newattrs)

  def endElement(self, elemname):
    if elemname in permittedElements:
      self.outmeth.endElement(elemname)

  def characters(self, s):
    self.outmeth.characters(s)

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Now we need an HTML output method. So we need to change the
startElement and endElement events of the SAX handler so that they
output empty elements in the old style. So we simply change the
startElement function to output the whole empty element. But in the
endElement function, we need some kind of flag which indicates whether
or not there have been any children since the corresponding
startElement call.

A simple way to store this kind of state information is in a stack. We
push a flag on the stack when we receive a start element call, which
is false, i.e. indicates that there are no children. We pop that flag
off the stack when in the endElement call, which tells us whether or
not to generate an end-tag. And whenever we receive any startElement
or characters call (or indeed any other type of child node), we modify
the flag on top of the stack, representing the child status of our
immediate parent, to indicate that children have been notified.

The code looks like this

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

HAS_NO_CHILDREN = 0
HAS_CHILDREN = 1

class html_output(xml.sax.handler.ContentHandler):

  def __init__(self):
    xml.sax.handler.ContentHandler.__init__(self)
    self.outbuf = StringIO.StringIO()
    self.child_flags = []

  def startElement(self, elemname, attrs):
    if len(self.child_flags):
      self.child_flags[-1] = HAS_CHILDREN
    self.child_flags.append(HAS_NO_CHILDREN)
    attrstr = ""
    for a in attrs.keys():
      attrstr = "%s%s" % (attrstr, " %s=%s" % (a,
quoteattr(attrs[a])))
    self.outbuf.write("<%s%s>" % (elemname, attrstr))

  def endElement(self, elemname):
    if self.child_flags[-1] == HAS_CHILDREN:
      self.outbuf.write("</%s>" % (elemname,))
    self.child_flags.pop()

  def characters(self, s):
    self.child_flags[-1] = HAS_CHILDREN
    self.outbuf.write("%s" % (escape(s),))

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

So combining all of the above, we end up with complete code like so

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
import xml.sax
from xml.sax.saxutils import escape, quoteattr
import cStringIO as StringIO
import random

permittedElements = ['html', 'body', 'b', 'i', 'p', 'hr', 'br',]
permittedAttrs = ['class', 'id', 'width',]

class xml_output(xml.sax.handler.ContentHandler):

  def __init__(self):
    xml.sax.handler.ContentHandler.__init__(self)
    self.outbuf = StringIO.StringIO()

  def startElement(self, elemname, attrs):
    attrstr = ""
    for a in attrs.keys():
      attrstr = "%s%s" % (attrstr, " %s=%s" % (a,
quoteattr(attrs[a])))
    self.outbuf.write("<%s%s>" % (elemname, attrstr))

  def endElement(self, elemname):
    self.outbuf.write("</%s>" % (elemname,))

  def characters(self, s):
    self.outbuf.write("%s" % (escape(s),))

HAS_NO_CHILDREN = 0
HAS_CHILDREN = 1

class html_output(xml.sax.handler.ContentHandler):

  def __init__(self):
    xml.sax.handler.ContentHandler.__init__(self)
    self.outbuf = StringIO.StringIO()
    self.child_flags = []

  def startElement(self, elemname, attrs):
    if len(self.child_flags):
      self.child_flags[-1] = HAS_CHILDREN
    self.child_flags.append(HAS_NO_CHILDREN)
    attrstr = ""
    for a in attrs.keys():
      attrstr = "%s%s" % (attrstr, " %s=%s" % (a,
quoteattr(attrs[a])))
    self.outbuf.write("<%s%s>" % (elemname, attrstr))

  def endElement(self, elemname):
    if self.child_flags.pop() == HAS_CHILDREN:
      self.outbuf.write("</%s>" % (elemname,))

  def characters(self, s):
    self.child_flags[-1] = HAS_CHILDREN
    self.outbuf.write("%s" % (escape(s),))

class cleaning_filter(xml.sax.handler.ContentHandler):

  def __init__(self, output_method):
    xml.sax.handler.ContentHandler.__init__(self)
    self.outmeth = output_method

  def startElement(self, elemname, attrs):
    if elemname in permittedElements:
      newattrs = {}
      for a in attrs.keys():
        if a in permittedAttrs:
          newattrs[a] = attrs[a]
      self.outmeth.startElement(elemname, newattrs)

  def endElement(self, elemname):
    if elemname in permittedElements:
      self.outmeth.endElement(elemname)

  def characters(self, s):
    self.outmeth.characters(s)

testdoc = """
<html>
  <body>
    <p class="1" id="2">This paragraph contains <b>only</b> permitted
elements.</p>
    <p>This paragraph contains <i 
    onclick="javascript:pop('porno.htm')">disallowed
attributes</i>.</p>
    <img src="http://www.blackhat.com/session_hijack.gif"/>
    <p>This paragraph contains
    <script src="blackhat.js"/>a potential script
    attack</p>
    <br/>
    <hr width="50%"/>
  </body>
</html>
"""

if __name__ == "__main__":
  is_old_user_agent = int(random.random() + 0.05)
  if is_old_user_agent:
    output_method = html_output()
  else:
    output_method = xml_output()
  mycleaner = cleaning_filter(output_method)
  parser = xml.sax.make_parser()
  parser.setContentHandler(mycleaner)
  parser.setFeature(xml.sax.handler.feature_namespaces, 0)
  parser.feed(testdoc)
  print output_method.outbuf.getvalue()
#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Note that it gets more complex if we want to output the empty-element
form from the xml_output handler: we can't decide in the startElement
function if we should output '/>' or '>' for the tag. But we can't
simply defer the decision until the endElement function, because if
there were intervening child elements, our output would be all messed
up. So we sould have to store "pending output" on the stack, or
"pending nodes", or something similar.

And now we get back to the statement in the XML spec: "the
empty-element tag SHOULD be used, and SHOULD only be used, for
elements which are declared EMPTY". What's basically stated here is
that it is recommended behaviour to output the empty element *when the
brute-force approach is possible*, i.e. when the processor *knows*
from the the content model, as declared in the DTD, that the element
is definitely empty. Which mostly only validating parsers do.
Requiring processors without a content model available to always
output the empty-element form would place too much of a burden of code
complexity on event-based-code writers, hence the "permissiveness" of
the XML spec in this regard.

It is also interesting to note that the original principle designer
for SAX, David Megginson, recognised the problem and created a
separate "emptyElement" handler in his "XMLWriter" java code, which
consumes SAX events and writes XML to output files.

http://www.megginson.com/Software/

would-love-to-have-time-to-write-an-xml-and-python-book-ly y'rs

-- 
alan kennedy
------------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan:              http://xhaus.com/contact/alan