Assigning generator expressions to ctype arrays

[Steven D'Aprano]

On Fri, 28 Oct 2011 16:27:37 -0700, Patrick Maupin wrote:

> And, BTW, the example you give of, e.g.
> 
> a,b,c = (some generator expression)
> 
> ALREADY LOSES DATA if the iterator isn't the right size and it raises an
> exception.

Yes. What's your point? This fact doesn't support your proposal in the 
slightest. You have argued against using a temporary array. I quote:

"It is incredibly inefficient to have to create a temp array."

[Aside: how do you know this is not just inefficient but *incredibly* so?]

But that's exactly what happens in tuple unpacking: the generator on the 
right hand side is unpacked into a temporary tuple, and then assigned to 
the targets on the left hand side. If the number of elements on both 
sides aren't the same, the assignment fails. That is, tuple unpacking 
behaves like this pseudo-code:

targets = left hand side
values = tuple(right hand side)
if len(targets) != len(values):
  fail
otherwise: 
  for each pair target, value:
    target = value

This has the important property that the assignment is atomic: it either 
succeeds in full, or it doesn't occur. The only side-effect is to exhaust 
the generator, which is unavoidable given that generators don't have a 
length.

Without temporary storage for the right hand side, you lose the property 
of atomicism. That would be unacceptable.

In the case of the ctypes array, the array slice assignment is also 
treated as atomic: it either succeeds in full, or it fails in full. This 
is an important property. Unlike tuple unpacking, the array is even more 
conservative about what is on the right hand size: it doesn't accept 
iterators at all, only sequences. This is a sensible default, because it 
is *easy* to work around: if you want to unpack the iterator, just make a 
temporary list:

array[:] = list(x+1 for x in range(32))

Assignment remains atomic, and the generator will be unpacked into a 
temporary list at full C speed.

If you don't care about assignment being atomic -- and it's your right to 
weaken the array contract in your own code -- feel free to write your own 
helper function based on your earlier suggestion:

"It merely needs to fill the slice and then ask for one more and check 
that StopIteration is raised."

def array_assign(array, values):
    try:
        if len(values) == len(array):
            array[:] = values  # runs at full C speed
    except TypeError:
        try:
            for i in xrange(len(array)):
                array[i] = next(values)  # or values.next in Python 2.5
        except StopIteration:
            raise TypeError('too few items')
        try:
            next(values)
        except StopIteration:
            pass
        else:
            raise TypeError('too many values')


But this non-atomic behaviour would be entirely inappropriate as the 
default behaviour for a ctypes array.



-- 
Steven