Assigning generator expressions to ctype arrays

[Terry Reedy]

On 10/28/2011 2:05 PM, Patrick Maupin wrote:

> On Oct 27, 10:23 pm, Terry Reedy<tjre... at udel.edu>  wrote:
>> I do not think everyone else should suffer substantial increase in space
>> and run time to avoid surprising you.
>
> What substantial increase?

of time and space, as I said, for the temporary array that I think would 
be needed and which I also described in the previous paragraph that you 
clipped

>  There's already a check that winds up
> raising an exception.  Just make it empty an iterator instead.

It? I have no idea what you intend that to refer to.


>>> It violates the principle of least surprise
>> for ctypes to do what is most efficient in 99.9% of uses?
>
> It doesn't work at all with an iterator, so it's most efficient 100%
> of the time now.  How do you know how many people would use iterators
> if it worked?

I doubt it would be very many because it is *impossible* to make it work 
in the way that I think people would want it to.

>> It could, but at some cost. Remember, people use ctypes for efficiency,

> yes, you just made my argument for me.  Thank you.  It is incredibly
> inefficient to have to create a temp array.

But necessary to work with blank box iterators. Now you are agreeing 
with my argument.

>> so the temp array path would have to be conditional.

> I don't understand this at all.  Right now, it just throws up its
> hands and says "I don't work with iterators."

If ctype_array slice assignment were to be augmented to work with 
iterators, that would, in my opinion (and see below), require use of 
temporary arrays. Since slice assignment does not use temporary arrays 
now (see below), that augmentation should be conditional on the source 
type being a non-sequence iterator.

> Why would it be a problem to change this?

CPython comes with immutable fixed-length arrays (tuples) that do not 
allow slice assignment and mutable variable-length arrays (lists) that 
do. The definition is 'replace the indicated slice with a new slice 
built from all values from an iterable'. Point 1: This works for any 
properly functioning iterable that produces any finite number of items. 
Iterators are always exhausted.

Replace can be thought of as delete follewed by add, but the 
implementation is not that naive. Point 2: If anything goes wrong and an 
exception is raised, the list is unchanged. This means that there must 
be temporary internal storage of either old or new references. An 
example that uses an improperly functioning generator.

 >>> a
[0, 1, 2, 3, 4, 5, 6, 7]
 >>> def g():
	yield None
	raise ValueError

 >>> a[3:6]=g()
Traceback (most recent call last):
   File "<pyshell#21>", line 1, in <module>
     a[3:6]=g()
   File "<pyshell#20>", line 3, in g
     raise ValueError
ValueError
 >>> a
[0, 1, 2, 3, 4, 5, 6, 7]

A c_uint array is a new kind of beast: a fixed-length mutable array. So 
it has to have a different definition of slice assignment than lists. 
Thomas Heller, the ctypes author, apparently chose 'replacement by a 
sequence with exactly the same number of items, else raise an 
exception'. though I do not know what the doc actually says.

An alternative definition would have been to replace as much of the 
slice as possible, from the beginning, while ignoring any items in 
excess of the slice length. This would work with any iterable. However, 
partial replacement of a slice would be a surprising innovation to most.

The current implementation assumes that the reported length of a 
sequence matches the valid indexes and dispenses with temporary storage. 
This is shown by the following:

from ctypes import c_uint
n = 20

class Liar:
     def __len__(self): return n
     def __getitem__(self, index):
         if index < 10:
             return 1
         else:
             raise ValueError()

x = (n * c_uint)()
print(list(x))
x[:] = range(n)
print(list(x))
try:
     x[:] = Liar()
except:
     pass
print(list(x))
 >>>
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

I consider such unintended partial replacement to be a glitch. An 
exception could be raised, but without adding temp storage, the array 
could not be restored. And making a change *and* raising an exception 
would be a different sort of glitch. (One possible with augmented 
assignment involving a mutable member of a tuple.) So I would leave this 
as undefined behavior for an input outside the proper domain of the 
function.

Anyway, as I said before, you are free to propose a specific change 
('work with iterators' is too vague) and provide a corresponding patch.

-- 
Terry Jan Reedy