[issue27240] 'UnstructuredTokenList' object has no attribute '_fold_as_ew'

[Zhang Wenbo]

Zhang Wenbo added the comment:

I think I find a case which can reproduct this bug: 
the UnstructuredTokenList has a ValueTerminal which is long enough(its encoded word is longer than policy.maxlen ) and not at the beginning.

For examples, the following subjects can cause this bug:

- 'Re: 我只是一个有那么一点点长的主题而已'
- '回复: 回复:【北京大学-2017年招生信息公告】'

So, the code can not fold a ValueTerminal correctly. 
Seems the method `_fold_as_ew` was designed to do this, but it is missing.

----------
Added file: http://bugs.python.org/file46941/test2.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue27240>
_______________________________________