[SciPy-dev] Notes from meeting with Guido regarding inclusion of array package in Python core

[Perry Greenfield]

On March 7th Travis Oliphant and Perry Greenfield met Guido and Paul 
Dubois to discuss some issues regarding the inclusion of an array 
package within core Python.

The following represents thoughts and conclusions regarding our meeting 
with Guido. They in no way represent the order of discussion with Guido 
and some of the points we raise weren't actually mentioned during the 
meeting, but instead were spurred by subsequent discussion after the 
meeting with Guido.

1) Including an array package in the Python core. To start, before the 
meeting we both agreed that we did not think that this itself was a 
high priority in itself. Rather we both felt that the most important 
issue was making arrays an acceptable and widely supported interchange 
format (it may not be apparent to some that this does not require 
arrays be in the core; more on that later). In discussing the 
desirability of including arrays in the core with Guido, we quickly 
came to the conclusion that not only was it not important, that in the 
near term (the next couple years and possibly much longer) it was a bad 
thing to do so. This is primarily because it would mean that updates to 
the array package would wait on Python releases potentially delaying 
important bug fixes, performance enhancements, or new capabilities 
greatly. Neither of us envisions any scenario regarding array packages, 
whether that be Numeric3 or numarray, where we would consider it to be 
something that would not *greatly* benefit from decoupling its release 
needs from that of Python (it's also true that it possibly introduces 
complications for Python releases if they need to synch with array 
schedules, but being inconsiderate louts, we don't care much about 
that). And when one considers that the move to multicore and 64-bit 
processors will introduce the need for significant changes in the 
internals to take advantage of these capabilities, it is unlike we will 
see a quiescent, maintenance-level state for an array package for some 
time. In short, this issue is a distraction at the moment and will only 
sap energy from what needs to be done to unify the array packages.

So what about supporting arrays as an interchange format? There are a 
number of possibilities to consider, none of which require inclusion of 
arrays into the core. It is possible for 3rd party extensions to 
optionally support arrays as an interchange format through one of the 
following mechanisms:

a) So long as the extension package has access to the necessary array 
include files, it can build the extension to use the arrays as a format 
without actually having the array package installed. The include files 
alone could be included into the core (Guido has previously been 
receptive to doing this though at this meeting he didn't seem quite as 
receptive instead suggesting the next option) or could be packaged with 
extension (we would prefer the former to reduce the possibilities of 
many copies of include files). The extension could then  be 
successfully compiled without actually having the array package 
present. The extension would, when requested to use arrays would see if 
it could import the array package, if not, then all use of arrays would 
result in exceptions. The advantage of this approach is that it does 
not require that arrays be installed before the extension is built for 
arrays to supported. It could be built, and then later the array 
package could be installed and no rebuilding would be necessary.

b) One could modify the extension build process to see if the package 
is installed and the include files are available, if so, it is built 
with the support, otherwise not. The advantage of this approach is that 
it doesn't require the include files be included with the core or be 
bundled with the extension, thus avoiding any potential version 
mismatches. The disadvantage is that later adding the array package 
require the extension to be rebuilt, and it results in more complex 
build process (more things to go wrong).

c) One could provide the support at the Python level by instead relying 
on the use of buffer objects by the extension at the C level, thus 
avoiding any dependence on the array C api. So long as the extension 
has the ability to return buffer objects containing the putative array 
data to the Python level and the necessary meta information (in this 
case, the shape, type, and other info, e.g., byteswapping, necessary to 
properly interpret the array) to Python, the extension can provide its 
own functions or methods to convert these buffer objects into arrays 
without copying of the data in the buffer object. The extension can try 
to import the array package, and if it is present, provide arrays as a 
data format using this scheme. In many respects this is the most 
attractive approach. It has no dependencies on include files, build 
order, etc. This approach led to the suggestion that Python develop a 
buffer object that could contain meta information, and a way of 
supporting community conventions (e.g., a name attribute indicating 
which conventions was being used) to facilitate the interchange of any 
sort of binary data, not just arrays. We also concluded that it would 
be nice to be able create buffer objects from Python with malloced 
memory (currently one can only create buffer objects from other objects 
that already have memory allocated; there is no way of creating newly 
allocated, writable memory from Python within a buffer object; one can 
create a buffer object from a string, but it is not writable). 
Nevertheless, if an extension is written in C, none of these changes 
are necessary to make use of this mechanism for interchange purposes 
now. This is the approach we recommend trying. The obvious case to 
apply it to is PIL as test case. We should do this ourselves and offer 
it as a patch to PIL. Other obvious cases are to support image 
interchange for GUIs (e.g., wxPython) and OpenGL.

2) Scalar support, rank-0 and related. Travis and I agreed (we 
certainly seek comments on this conclusion; we may have forgotten about 
key arguments arguing for one the different approaches) that the 
desirability of using rank-0 arrays as return values from single 
element indexing depends on other factors, most importantly Python's 
support for scalars in various aspects. This is a multifaceted issue 
that will need to be determined by considering all the facets 
simultaneously. The following tries to list the pro's and con's 
previously discussed for returning scalars (two cases previously 
discussed) or rank-0 arrays (input welcomed).

a) return only existing Python scalar types (cast upwards except for 
long long and long double based types)

Pros:
   - What users probably expect (except matlab users!)
   - No performance hit in subsequent scalar expressions
   - faster indexing performance (?)

Cons:
   - Doesn't support array attributes, numeric behaviors
   - What do you return for long long and long double?  No matter what 
is done, you will either lose precision or lose consistency. Or you 
create a few new Python scalar types for the unrepresentable    types? 
But, with subclassing in C the effort to create a few scalar types is 
very close to the effort to create many.

b) create new Python scalar types and return those (one for each basic 
array type)

Pros:
   - Exactly what numeric users expect in representation
   - No peformance hit in subsequent scalar expressions
   - faster indexing performance
   - Scalars have the same methods and attributes as arrays

Cons:
   - Might require great political energy to eventually get the 
arraytype with all of its scalartype-children into the Python core.  
This is really an unknown, though, since if the arrayobject is in the 
standard module and not in the types module, then people may not care 
(a new type is essentially a new-style class and there are many, many 
classes in the Python standard library). A good scientific-packaging 
solution that decreases the desireability of putting the arrayobject 
into the core would help alleviate this problem as well.
   - By itself it doesn't address different numeric behaviors for the 
"still-present" Python scalars throughout Python.

c) return rank-0 array

Pros:
   - supports all array behaviors, particularly with regard to numerical 
processing, particularly with regard to ieee exception handling (a 
matter of some controversy, some would like it also to be len()=1 and 
support [0] index, which strictly speaking rank-0 arrays should not 
support)

Cons:
   - Performance hit on all scalar operations (e.g., if one then does 
many loops over what appears to be a pure scalar expression, use of 
rank-0 will be much slower than Python scalars since use of arrays 
incurs significant overhead.
   - Doesn't eliminate the fact that one can still run into different 
numerical behavior involving operations between Python scalars.
   - Still necessary to write code that must deal with Python scalars 
"leaking" into code as inputs to functions.
   - Can't currently be used to index sequences (so not completely 
usable in place of scalars)

Out of this came two potential needs (The first isn't strictly 
necessary if approach a is taken, but could help smooth use of all 
integer types as indexes if approach b is taken):

If rank-0 arrays are returned, then Guido was very receptive to 
supporting a special method, __index__ which would allow any Python 
object to be used as an index to a sequence or mapping object. Calling 
this would return a value that would be suitable as index if the object 
was not itself suitable directly. Thus rank-0 arrays would have this 
method called to convert its internal integer value into a Python 
integer. There are some details about how this would work at the C 
level that need to be worked out. This would allow rank-0 integer 
arrays to be used as indices. To be useful, it would be necessary to 
get this into the core as quickly as possible (if there are C API 
issues that have lingering solutions that won't be solved right away, 
then a greatly delayed implementation in Python would make this less 
than useful).

We talked at some length about whether it was possible to change 
Python's numeric behavior for scalars, namely support for configurable 
handling of numeric exceptions in the way numarray does it (and 
Numeric3 as well). In short, not much was resolved. Guido didn't much 
like the stack approach to the exception handling mode. His argument (a 
reasonable one) was that even if the stack allowed pushing and popping 
modes, it was fragile for two reasons. If one called other functions in 
other modules that were previously written without knowledge that the 
mode could be changed, those functions presumed the previous behavior 
and thus could be broken with mode change (though we suppose that just 
puts the burden on the caller to guard all external calls with restores 
to default behavior; even so, many won't do that leading to spurious 
bug reports that may annoy maintainers to no end though no fault of 
their own). He also felt that some termination conditions may cause 
missed pops leading to incorrect modes. He suggested studying the use 
of the decimal's use of context to see if it could used as a model. 
Overall he seemed to think that setting mode on a module basis was a 
better approach. Travis and I wondered about how that could be 
implemented (it seems to imply that the exception handling needs to 
know what module or namespace is being executed in order to determine 
the mode.

So some more thought is needed regarding this. The difficulty of 
proposing such changes and getting them accepted is likely to be 
considerable. But Travis had a brilliant idea (some may see this as 
evil but I think it has great merit). Nothing prevents a C extension 
from hijacking the existing Python scalar objects behaviors. Once a 
reference is obtained to an integer, float or complex value, one can 
replace the table of operations on those objects with whatever code one 
wishes. In this way an array package could (optionally) change the 
behavior of Python scalars. In this way we could test the behavior of 
proposed changes quite easily, distribute that behavior quite easily in 
the community, and ultimately see if there are really any problems 
without expending any political energy to get it accepted. Once seeing 
if it really worked (without "forking" Python either), would place us 
in a much stronger position to have the new behaviors incorporated into 
the core. Even then, it  may never prove necessary if can be so 
customized by the array package. This holds out the potential of making 
scalar/array behavior much more consistent. Doing this may allow option 
a) as the ultimate solution, i.e., no changes needed to Python at all 
(as such), no rank-0 arrays. This will be studied further. One possible 
issue is that adding the necessary machinery to make numeric scalar 
processing consistent with that of the array package may introduce 
significant performance penalties (what is negligible overhead for 
arrays may not be for scalars).

One last comment is that it is unlikely that any choice in this area 
prevents the need for added helper functions to the array package to 
assist in writing code that works well with scalars and arrays. There 
are likely a number of such issues. A common approach is to wrap all 
unknown objects with "asarray". This works reasonably well but doesn't 
handle the following case: If you wish to write a function that will 
accept arrays or scalars, in principal it would be nice to return 
scalars if all that was supplied were scalars. So functions to help 
determine what the output type should be based on the inputs would be 
helpful, for example to distinguish from when someone provided a rank-0 
array as an input (or rank-1 len-1 array) and an actual scalar if 
asarray happens to map this to the same thing so that the return can 
properly return a scalar if that is what was originally input. Other 
such tools may help writing code that allows the main body to treat all 
objects as arrays without needing checks for scalars.

Other miscellaneous comments.

The old use of where() may be deprecated and only "nonzero" 
interpretation will be kept. A new function will be defined to replace 
the old usage of where (we deem that regular expression search and 
replaces should work pretty well to make changes in almost all old 
code).

With the use of buffer objects, tostring methods are likely to be 
deprecated.


Python PEPs needed
===================

 From the discussions it was clear that at least two Python PEPs need to 
be written and implemented, but that these needed to wait until the 
unification of the arrayobject takes place.

PEP 1:  Insertion of an __index__ special method and an as_index slot 
(perhaps in the as_sequence methods) in the C-level typeobject  into 
Python.

PEP 2:  Improvements on the buffer object and buffer builtin method so 
that buffer objects can be Python-tracked wrappers around allocated 
memory that extension packages can use and share.  Two extensions are 
considered so far.  1) The buffer objects have a meta attribute so that 
meta information can be passed around in a unified manner and 2) The 
buffer builtin should take an integer giving the size of writeable 
buffer object to create.