Problem with tokenize module and indents

[Fredrik Lundh]

Tim wrote:

> I ran into a problem with a script i was playing with to check code 
> indents and need some direction.  It seems to depend on if tabsize is 
> set to 4 in editor and spaces and tabs indents are mixed on consecutive 
> lines.  Works fine when editors tabsize was 8 regardless if indents are 
> mixed.
> 

> # nano -T4 tabspacing_4.py
> class Test:
>     """triple quote"""              #indent is 1 tab
>     def __init__(self, msg):    #indent is 4 spaces               << 
> this gets reported as a dedent when there is no change in indent level

http://pyref.infogami.com/lexical-analysis

   First, tabs are replaced (from left to right) by one to eight spaces
   such that the total number of characters up to and including the
   replacement is a multiple of eight (this is intended to be the same
   rule as used by Unix). The total number of spaces preceding the first
   non-blank character then determines the line's indentation.

</F>