[issue12528] Implement configurable bitfield allocation strategy

[Vlad Riscutia]

New submission from Vlad Riscutia <riscutiavlad at gmail.com>:

Opened this issue to track configurable bitfield allocation strategy. This will address issues like http://bugs.python.org/issue6069, http://bugs.python.org/issue11920.

Summary: the way bitfields are allocated is up to the compiler not defined by standard. MSVC and GCC have different strategies to perform the allocation so the size of bitfield structures can be different depending on compiler. Currently we hardcode allocation strategy to be GCC-way on non-Windows and MSVC-way on Windows which raises issues when trying to interop on Windows with GCC binaries.

Short term this solution will enable interop between MSVC compiled Python with GCC compiled binaries under Windows. It will also enable addressing other possible compiler interop issues in the future, for compilers that don't use GCC strategy.

Following is copied from thread discussing this:


On 6/25/2011 12:33 PM, Vlad Riscutia wrote:
I recently started looking at some ctypes issues. I dug a bit into
http://bugs.python.org/issue6069 and then I found
http://bugs.python.org/issue11920. They both boil down to the fact that
bitfield allocation is up to the compiler, which is different in GCC and
MSVC. Currently we have hard-coded allocation strategy based on paltform
in cfields.c:

 if (bitsize /* this is a bitfield request */

       &&  *pfield_size /* we have a bitfield open */
 #ifdef MS_WIN32
       /* MSVC, GCC with -mms-bitfields */
       &&  dict->size * 8 == *pfield_size
 #else
       /* GCC */
       &&  dict->size * 8<= *pfield_size
 #endif
       &&  (*pbitofs + bitsize)<= *pfield_size) {
       /* continue bit field */
       fieldtype = CONT_BITFIELD;
 #ifndef MS_WIN32
   } else if (bitsize /* this is a bitfield request */
       &&  *pfield_size /* we have a bitfield open */
       &&  dict->size * 8>= *pfield_size
       &&  (*pbitofs + bitsize)<= dict->size * 8) {
       /* expand bit field */
       fieldtype = EXPAND_BITFIELD;
 #endif

So when creating a bitfield structure, it's size can be different on
Linux vs Windows.

class MyStructure(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_uint32, 32),
                   ("Data1",   ctypes.c_uint8, 3),
                   ("Data2",   ctypes.c_uint16, 12),
                  ]

sizeof for above structure is 6 on GCC build and 7 on MSVC build. This
leads to some confusion and issues, because we can't always interop
correctly with code compiled with different compiler than the one Python
is compiled with on the platform.

Just curious, are you saying that this is the 'cause' of the two bug reports, or 'just' something you discovered while investigating them?


> Short term solution is to add a warning in the documentation about this.

For 2.7/3.2, yes.


> Longer term though, I think it
would be better to add a property on the Structure class for
configurable allocation strategy, for example Native (default), GCC,
MSVC and when allocating the bitfield, use given strategy. Native would
behave similar to what happens now, but we would also allow GCC-style
allocation on Windows for example and the ability to extend this if we
ever run into similar issues with other compilers. This shouldn't be too
difficult to implement, will be backwards compatible and it would
improve interop. I would like to hear some opinions on this.

If this would allow the MSVC-compilied Python to better access dlls compiled with gcc (cygwin) on Windows, definitely -- in 3.3.
If the new feature is (currently) only useful on Windows, doc should say so.

-- 
Terry Jan Reedy

/copy

Attached is patch with initial refactoring of cfield.c to enable configurable allocation. Next step is to provide a way to configure this through Python library. I will also look at updating documentation to point out the known issue.

----------
components: ctypes
files: cfield_bitfield_refactoring.diff
keywords: patch
messages: 140088
nosy: terry.reedy, vladris
priority: normal
severity: normal
status: open
title: Implement configurable bitfield allocation strategy
type: feature request
versions: Python 3.3
Added file: http://bugs.python.org/file22617/cfield_bitfield_refactoring.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12528>
_______________________________________