[PEP draft 2] Adding new math operators

[Bernhard Herzog]

hzhu at localhost.localdomain (Huaiyu Zhu) writes:

> On 09 Aug 2000 23:59:11 +0200, Bernhard Herzog <herzog at online.de> wrote:
> 
> [ nice explanation about distinction between tokenizer and parser ]
> 
> >In the part you didn't quote:
> >
> >  You could probably work around it by also allowing ~+ and ~- to be
> >  used as unary operators, but that seems awkward.
> >
> >I was trying to say that to work around this, you have to extend the
> >grammar and compile the unary operators ~+ and ~- as if they mean ~ +
> >and ~ - respectively.
> 
> I don't quite follow here.  Isn't this the behavior of current Python? 

Well, currently Python doesn't have ~+ and ~-, so "~+" is actually two
tokens, ~ and +. If ~+ is added as an operator, "~+" becomes one token.
I'm using string literaly here to properly distibguish between the input
of the tokenizer and its output.

> The proposal does not introduce unary operators ~+ ~-. 

No, but it would have to, or at least the implementation would have to.

In current Python the input '~+1' is parsed as three tokens:

>>> import tokenize, StringIO
>>> tokenize.tokenize(StringIO.StringIO('~+1').readline)
1,0-1,1:        OP      '~'
1,1-1,2:        OP      '+'
1,2-1,3:        NUMBER  '1'
2,0-2,0:        ENDMARKER       ''
>>> 

Well, OK, actually four, but since this is not the Spanish Inquisition,
I'm not going to try again....

Now, if ~+ is an operator, you only get two tokens (Ok, three with the
ENDMARKER). Since ~+ is not implemented yet, I've hacked up a version of
tokenize that recognizes it:

>>> import mytokenize
>>> mytokenize.tokenize(StringIO.StringIO('~+1').readline)
1,0-1,2:        OP      '~+'
1,2-1,3:        NUMBER  '1'
2,0-2,0:        ENDMARKER       ''

If ~+ is only defined as a binary operator in the Python grammar, then
this sequence of tokens is a syntax error because ~+ can't be the first
token, there must be an operand preceding it.

> Maybe you could explain more the nature of work around needed here.

AFAICS, the only way to avoid the syntax error described above is to
also allow ~+ to be used as a unary operator. The compiler would have to
treat that unary operator as if it were the combination of ~ and + and
generate byte code that first applies the unary + operator and then
bitwise negation. 

That kind of workaround seems ugly to me.


-- 
Bernhard Herzog   | Sketch, a drawing program for Unix
herzog at online.de  | http://sketch.sourceforge.net/