UTF_16 question

[Richard Damon]

> On Apr 29, 2024, at 12:23 PM, jak via Python-list <python-list at python.org> wrote:
> 
> Hi everyone,
> one thing that I do not understand is happening to me: I have some text
> files with different characteristics, among these there are that they
> have an UTF_32_le coding, utf_32be, utf_16_le, utf_16_be all of them
> without BOM. With those utf_32_xx I have no problem but with the
> UTF_16_xx I have. If I have an utf_16_le coded file and I read it with
> encoding='utf_16_le' I have no problem I read it, with
> encoding='utf_16_be' I can read it without any error even if the data I
> receive have the inverted bytes. The same thing happens with the
> utf_16_be codified file, I read it, both with encoding='utf_16_be' and
> with 'utf_16_le' without errors but in the last case the bytes are
> inverted. What did I not understand? What am I doing wrong?
> 
> thanks in advance
> 
> --
> https://mail.python.org/mailman/listinfo/python-list

That is why the BOM was created. A lot of files can be “correctly” read as either UTF-16-LE or UTF-1-BE encoded, as most of the 16 bit codes are valid, so unless the wrong encoding happens to hit something that is invalid (most likely something looking like a Surrogage Pair without a match), there isn’t an error in reading the file. The BOM character was specifically designed to be an invalid code if read by the wrong encoding (if you ignore the possibility of the file having a NUL right after the BOM)

If you know the files likely contains a lot of “ASCII” characters, then you might be able to detect that you got it wrong, due to seeing a lot of 0xXX00 characters and few 0x00XX characters, but that doesn’t create an “error” normally.