[Python-Dev] XML codec?

[Walter Dörwald]

Martin v. Löwis wrote:

>> Then how about the suggested "xml-auto-detect"?
> 
> That is better.

OK.

>>> Then, I'd claim that the problem that the codec solves doesn't really
>>> exist. IOW, most XML parsers implement the auto-detection of encodings,
>>> anyway, and this is where architecturally this functionality belongs.
>> But not all XML parsers support all encodings. The XML codec makes it
>> trivial to add this support to an existing parser.
> 
> I would like to question this claim. Can you give an example of a parser
> that doesn't support a specific encoding

It seems that e.g. expat doesn't support UTF-32:

from xml.parsers import expat

p = expat.ParserCreate()
e = "utf-32"
s = (u"<?xml version='1.0' encoding=%r?><foo/>" % e).encode(e)
p.Parse(s, True)

This fails with:

Traceback (most recent call last):
   File "gurk.py", line 6, in <module>
     p.Parse(s, True)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, 
column 1

Replace "utf-32" with "utf-16" and the problem goes away.

> and where adding such a codec
> solves that problem?
> 
> In particular, why would that parser know how to process Python Unicode
> strings?

It doesn't have to. You can use an XML encoder to reencode the unicode 
string into bytes (forcing an encoding that the parser knows):

import codecs
from xml.parsers import expat

ci = codecs.lookup("xml-auto-detect")
p = expat.ParserCreate()
e = "utf-32"
s = (u"<?xml version='1.0' encoding=%r?><foo/>" % e).encode(e)
s = ci.encode(ci.decode(s)[0], encoding="utf-8")[0]
p.Parse(s, True)

>> Furthermore encoding-detection might be part of the responsibility of
>> the XML parser, but this decoding phase is totally distinct from the
>> parsing phase, so why not put the decoding into a common library?
> 
> I would not object to that - just to expose it as a codec. Adding it
> to the XML library is fine, IMO.

But it does make sense as a codec. The decoding phase of an XML parser 
has to turn a byte stream into a unicode stream. That's the job of a codec.

>> There's a (currently undocumented) codecs.detect_xml_encoding() in the
>> patch. We could document this function and make it public. But if
>> there's no codec that uses it, this function IMHO doesn't belong in the
>> codecs module. Should this function be available from xml/__init__.py or
>> should be put it into something like xml/utils.py?
> 
> Either - or.

OK, so should I put the C code into a _xml module?

>>> Finally, I think the codec is incorrect. When saving XML to a file
>>> (e.g. in a text editor), there should rarely be encoding errors, since
>>> one could use character references in many cases.
>> This requires some intelligent fiddling with the errors attribute of the
>> encoder.
> 
> Much more than that, I think - you cannot use a character reference
> in an XML Name. So the codec would have to parse the output stream
> to know whether or not a character reference could be used.

That's what I meant with "intelligent" fiddling. But I agree this is way 
beyond what a text editor should do. AFAIK it is way beyond what 
existing text editors do. However using the XML codec would at least 
guarantee that the encoding specified in the XML declaration and the 
encoding used for encoding the file stay consistent.

>> Correct, but as long as Python doesn't have an EBCDIC codec, that won't
>> help much. Adding *detection* of EBCDIC to detect_xml_encoding() is
>> rather simple though.
> 
> But it does! cp037 is EBCDIC, and supported by Python.

I didn't know that. I'm going to update the patch.

Servus,
    Walter