[Python-Dev] Deprecation warning on integer shifts and such

[M.-A. Lemburg]

Guido van Rossum wrote:
>>Jack Jansen wrote:
>>
>>>As of recently I'm getting deprecation warnings on lots of constructs of 
>>>the form "0xff << 24", telling me that in Python 2.4 this will return a 
>>>long.
>>
>>Interesting. I wonder why the implementation warns about 0xff << 24...
>>0xff000000 fits nicely into a 32-bit integer. I don't see why the
>>"changing sign" is relevant here or even why it is mentioned in the
>>warning since the PEP doesn't say anything about it.
> 
> 
> PEP 237 *tries* mention it:
> 
>     - Currently, x<<n can lose bits for short ints.  This will be
>       changed to return a long int containing all the shifted-out
>       bits, if returning a short int would lose bits.
> 
> Maybe you don't see changing sign as "losing bits"?  I do!  Maybe I
> have to clarify this in the PEP.

I was talking about the sign bit which lies within the 32 bits for
32-bit integers, so no bits are lost. I am not talking about things
like 0xff << 28 where bits are actually moved beyond the 32 bits to the
left and lost that way (or preserved if you move to a 64-bit platform,
but that's another story ;-).

> PEP 237 is about erasing all differences between int and long.  When
> seen as a long, 0xff000000 has the value 4278190080.  But currently it
> is an int and has the value -16777216.  As a bit pattern that doesn't
> make much of a difference, but as a numeric value it makes a huge
> difference (2**32 to be exact :-).  So in Python 2.4, 0xff<<24, as
> well as the constant 0xff000000, will have the value 4278190080.
> 
> Note that larger constants are already longs in 2.2: e.g. 0x100000000
> equals 4294967296 (which happens to be representable only as a long).
> It's the oct and hex constants in range(2**31, 2**32) that currently
> behave anomalously, returning negative numbers despite looking like
> positive numbers (to everyone except people whose minds have been
> exposed to 32-bit bit-fiddling too long :-).
> 
> 
>>Changing these semantics would cause compatibility problems for
>>applications doing low-level bit manipulations or ones which use
>>the Python integer type to store unsigned integer values, e.g.
>>for use as bitmapped flags.
> 
> 
> That's why I'm adding the warnings to 2.3.  Note that the bit pattern
> in the lower 32 bits will remain the same; it's just the
> interpretation of the sign that will change.

That's exactly what I'd like too :-) With the only difference
that you seem to see the sign bit as not included in the 32 bits.

>>>As these things are bitpatterns (they're all generated from .h
>>>files for system call interfaces and such) that the user will pass
>>>to methods that wrap underlying API calls I don't want them to be
>>>longs. How do I force them to remain ints?
>>
> 
> Why do you want them to remain ints?  Does a long whose lower 32 bits
> have the right bit pattern not work?

No, because you usually pass these objects directly to some
Python C function (directly as parameter or indirectly as item
in a list or tuple) which often enough insists on getting a true
integer object.

> If you really want the int value, you have to do a little arithmetic.
> Here's something that's independent of the Python version and won't
> issue a warning:
> 
> def toint32(x):
>     x = x & 0xffffffffL # Force it to be a long in range(0, 2**32)
>     if x & 0x80000000L: # If sign bit set
>         x -= 0x100000000L # flip sign
>     return int(x)
> 
> You can also write it as a single expression:
> 
> def toint32(x):
>     return int((x & 0xffffffffL) - ((x & 0x80000000L) << 1))
> 
> In the long run you'll thank me for this.

No argument about this. It's just that I see a lot of programs
breaking because of the 0x1 << 31 returning a long. That needen't
be the case. People using this will know what they are doing and
use a long when possible anyway. However, tweaking C extensions to
also accept longs instead of integers requires hacking those
extensions which I'd like to avoid if possible. I already had
one of these instances with file.tell() returning a long and
that caused a lot of trouble then.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/