MeCab UTF-8 Decoding Problem

fobos3 at gmail.com fobos3 at gmail.com
Sat Jun 29 07:29:23 EDT 2013


Hi,

I am trying to use a program called MeCab, which does syntax analysis on Japanese text. The problem I am having is that it returns a byte string and if I try to print it, it prints question marks for almost all characters. However, if I try to use .decide, it throws an error. Here is my code:

#!/usr/bin/python
# -*- coding:utf-8 -*-

import MeCab
tagger = MeCab.Tagger("-Owakati")
text = 'MeCabで遊んでみよう!'

result = tagger.parse(text)
print result

result = result.decode('utf-8')
print result

And here is the output:

MeCab �� �� ��んで�� �� ��う! 

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    result = result.decode('utf-8')
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 6-7: invalid continuation byte


------------------
(program exited with code: 1)
Press return to continue

Also my terminal is able to display Japanese characters properly. For example print '日本語' works perfectly fine.

Any ideas?



More information about the Python-list mailing list