Assigning generator expressions to ctype arrays

[Patrick Maupin]

On Oct 28, 8:01 pm, Steven D'Aprano > > ALREADY LOSES DATA if the
iterator isn't the right size and it raises an
> > exception.
>
> Yes. What's your point? This fact doesn't support your proposal in the
> slightest.

You earlier made the argument that "If the slice has too few elements,
you've just blown away the entire iterator for no good reason.  If the
slice is the right length, but the iterator doesn't next raise
StopIteration, you've just thrown away one perfectly good value. Hope
it
wasn't something important."

In other words, you WERE arguing that it's very important not to lose
information.  Now that I show that information is ALREADY LOST in a
similar scenario, you are arguing that's irrelevant.  Whatever.

> You have argued against using a temporary array. I quote:
>
> [Aside: how do you know this is not just inefficient but *incredibly* so?]

By profiling my app, looking for places to save time without having to
resort to C.

> But that's exactly what happens in tuple unpacking:
> the generator on the right hand side is unpacked into
> a temporary tuple, and then assigned to the targets
> on the left hand side.

Agreed.  But Terry was making the (correct) argument that people using
ctypes are often looking for efficiency.

> If the number of elements on both sides aren't the same,
> the assignment fails. That is, tuple unpacking behaves
> like (snip)

Yes, and that loses information from the right hand side if it fails
and the right hand side happens to be an iterator.  For some reason
you're OK with that in this case, but it was the worst thing in the
world a few messages ago.

> This has the important property that the
> assignment is atomic: it either succeeds in full,
> or it doesn't occur.

Not right at all.

>The only side-effect is to exhaust the generator,
> which is unavoidable given that generators don't have a
> length.

Yes.  But earlier you were acting like it would be problematic for me
to lose information out of the generator if there were a failure, and
now the sanctity of the information on the LHS is so much more than on
the RHS.  Frankly, that's all a crock.  In your earlier email, you
argued that my proposal loses information, when in fact, in some cases
it PRESERVES information (the very information you are trying to
transfer) that ISN'T preserved when this temp array is tossed, and the
only information it LOSES is information the programmer declared his
clear intent to kill.  But that's an edge case anyway.

> Without temporary storage for the right hand side,
> you lose the property of atomicism. That would be
> unacceptable.

As if the temporary storage workaround exhibited the "necessary"
atomicity, or as if you have even showed a valid argument why the
atomicity is important for this case.

> In the case of the ctypes array, the array slice assignment is also
> treated as atomic: it either succeeds in full, or it fails in full.
> This is an important property. Unlike tuple unpacking, the array is even more
> conservative about what is on the right hand size: it doesn't accept
> iterators at all, only sequences. This is a sensible default,

How it works is not a bad start, but it's arguably unfinished.

> because it is *easy* to work around: if you want to unpack the iterator, just make a temporary list: (snip)

I already said I know the workaround.  I don't know why you can't
believe that.  But one of the purposes of the iterator protocol is to
reduce the number of cases where you have to create huge lists.

> Assignment remains atomic, and the generator will be unpacked into a
> temporary list at full C speed.

Which can be very slow if your list has several million items in it.

> If you don't care about assignment being atomic -- and it's your right to
> weaken the array contract in your own code

Currently, there IS no contract between ctype arrays and generators.

I'm suggesting that it would be useful to create one, and further
suggesting that if a programmer attempts to load a ctypes array from a
generator of the wrong size, it is certainly important to let the
programmer know he screwed up, but it is not at all important that
some elements of the ctypes array, that the programmer was in fact
trying to replace, were in fact correctly replaced before the size
error was noticed.

> -- feel free to write your own
> helper function based on your earlier suggestion: (snip)

Doing this in Python is also slow and painful, and the tradeoff of
this vs the temp list depends on the system, the caching, the amount
of memory available, and the size of the data to be transferred.  I
could do it in C, but that defeats my whole purpose of using ctypes to
avoid having to ship C code to customers.

> But this non-atomic behaviour would be entirely inappropriate as the
> default behaviour for a ctypes array.

That's obviously your opinion, but your supporting arguments are quite
weak.

Regards,
Pat