Python 2.6 StreamReader.readline()

[wxjmfauth at gmail.com]

On Wednesday, July 25, 2012 11:02:01 AM UTC+2, Walter Dörwald wrote:
> On 25.07.12 08:09, Ulrich Eckhardt wrote:
> 
> > Am 24.07.2012 17:01, schrieb cpppwner at gmail.com:
> >>      reader = codecs.getreader(encoding)
> >>      lines  =  []
> >>      with open(filename, 'rb') as f:
> >>          lines  = reader(f, 'strict').readlines(keepends=False)
> >>
> >> where encoding == 'utf-16-be'
> >> Everything works fine, except that lines[0] is equal to
> >> codecs.BOM_UTF16_BE
> >> Is this behaviour correct, that the BOM is still present?
> >
> > Yes, assuming the first line only contains that BOM. Technically it's a
> > space character, and why should those be removed?
> 
> If the first "character" in the file is a BOM the file encoding is 
> probably not utf-16-be but utf-16.
> 
> Servus,
>     Walter

The byte order mark, if present, is nothing else than
an encoded 

>>> ud.name('\ufeff')
'ZERO WIDTH NO-BREAK SPACE'

*code point*.

Five "BOM" are possible (Unicode consortium). utf-8-sig, utf-16-be,
utf-16-le, utf-32-be, utf-32-le. The codecs module provide many
aliases.

The fact that utf-16/32 does correspond to -le or to -be may
vary according to the platforms, the compilers, ...

>>> sys.version
'3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit
(Intel)]'
>>> codecs.BOM_UTF16_BE
b'\xfe\xff'
>>> codecs.BOM_UTF16_LE
b'\xff\xfe'
>>> codecs.BOM_UTF16
b'\xff\xfe'
>>>

---

As far as I know, Py 2.7 or Py 3.2 never return a "BOM" when
a file is read correctly.

>>> with open('a-utf-16-be.txt', 'r', encoding='utf-16-be') as f:
...     r = f.readlines()
...     for zeile in r:
...         print(zeile.rstrip())
...         
abc
élève
cœur
€uro
>>> 


jmf