Not found in the documentation

[Christian Gollwitzer]

Am 29.04.21 um 08:54 schrieb elas tica:
> Le mercredi 28 avril 2021 à 17:36:32 UTC+2, Chris Angelico a écrit :
> 
>> In what sense of the word "token" are you asking? The parser? You can
>> play around with the low-level tokenizer with the aptly-named
>> tokenizer module.
> 
> It was a good suggestion, and the PLR doesn't mention the tokeniser module. It should, this goes very well with the conversional style it has.
> 
> 
> 
> # --------------
> from tokenize import tokenize
> from io import BytesIO
> 
> s="""42 not\
>   in [42]"""
> g = tokenize(BytesIO(s.encode('utf-8')).readline)
> print(*(g), sep='\n')
> # --------------
> 
> the docs are wrong when they say:
> 
> ......................................
> using a backslash). A backslash is illegal elsewhere on a line outside a string literal.
> ......................................
> 

You're not passing a backslash. Try print(s).
It would be different with a raw string

s=r"""42 not\
    in [42]"""

	Christian