Problem with minidom and special chars in HTML

Thu Feb 24 06:38:11 EST 2005

Horst Gutmann wrote:

> I currently have quite a big problem with minidom and special chars
> (for example ü)  in HTML.

Yes. Ignoring the issue of the wrong doctype, minidom is a pure XML
parser and knows nothing of XHTML and its doctype's entities 'uuml' and
the like. Only the built-in entities (& etc.) will work.

Unfortunately the parser minidom uses won't read external entities -
including the external subset of the DTD (which is where all the stuff
about what "ü" means is stored). And because minidom does not
support EntityReference nodes, the information that there was an entity
reference there at all gets thrown away as it is replaced with the
empty string. Which is kind of bad.

Possible workarounds:

1. pass minidom a different parser to use, one which supports external
entities and which will parse all the DTD stuff. I don't know if there
is anything suitable available, though...

2. use a DOM implementation with the option to support external
entities. For example, with pxdom, one can use DOM Level 3 LS methods,
or pxdom.parse(f, {'pxdom-external-entities': True}).

However note that reading and parsing an external entity will introduce
significant slowdown, especially in the case of the rather complex
multi-file XHTML DTD. Other possibilities:

3. hack the content on the way into the parser to replace the DOCTYPE
declaration with one including entity definitions in the internal
subset:

  <!DOCTYPE html PUBLIC "..." "..." [
    <!ENTITY uuml "ü">
    ...
  ]>
  <html>...

4. hack the content on the way into the parser to replace entity
references with character references, eg. ü -> ü. This is
'safe' for simple documents without an internal subset; charrefs and
entrefs can be used in the same places with the same meaning, except
for some issues in the internal subset.

5. use a DOM implementation that supports EntityReference nodes, such
as pxdom. Entity references with no replacement text (or all entity
references if the DOM Level 3 LS parameter 'entities' is set) will
exist as EntityReference DOM objects instead of being flattened to
text. They can safely be reserialized as ü without the
implementation having to know what text they represent.

Entities are a big source of complication and confusion, which I wish
had not made it into XML!

-- 
Andrew Clover
mailto:and at doxdesk.com
http://www.doxdesk.com/