[Python.NET] Efficient copy of .NET Array to ctypes or numpy array

Wed Nov 5 20:28:04 CET 2014

Finally decided to generate the managed object on Python side and return it
to C# with no conversion necessary in C#. This way I can even wrap regular
function with Python with @decorator to handle the conversion.  I suppose
the dynamic version of pythonnet may have auto conversion for Python 3, but
I have not tried. I'm on Python 2.7.

On Wed, Nov 5, 2014 at 9:31 AM, Denis Akhiyarov <denis.akhiyarov at gmail.com>
wrote:

> And more important question - is it possible to generalize the copying of
> python array object to managed C# array object without knowing the data
> type/size/length?
>
> On Wed, Nov 5, 2014 at 8:58 AM, Denis Akhiyarov <denis.akhiyarov at gmail.com
> > wrote:
>
>> How to copy unmanaged array (python list/tuple or numpy array) into
>> managed C# array? I guess using Marshal.Copy, but can anyone point to
>> example?
>>
>> Thanks,
>> Denis
>>
>> On Thu, Oct 30, 2014 at 12:19 PM, Nikhil Garg <nikhilgarg.gju at gmail.com>
>> wrote:
>>
>>> Thanks Brad and Jeff for the detailed info. For now, fromiter is serving
>>> me well and has reduced my processing time considerably, so I am just going
>>> to stick with it.
>>>
>>>
>>> On 29 October 2014 11:04, Jeffrey Bush <jeff at coderforlife.com> wrote:
>>>
>>>> I finally have a chance to chime in, and Bradley is exactly right.
>>>> Marshall.Copy copies the raw data, and apparently your file library does
>>>> not store that data in a nice, contiguous, manner. While it is highly
>>>> likely that copying all the data to an array in C# will be faster than the
>>>> fromiter in Python, I am unsure if copying all the data to an array in C#
>>>> then copying all the data again to a numpy array will be faster than
>>>> fromiter (cause you have to copy it twice). The exception is if the file
>>>> library has a function like ToArray that is optimized to copy the data to a
>>>> linear chunk of data. So, what type is "Data"?
>>>>
>>>> Another factor is how long the chunk of data you are copying is. You
>>>> say the last axis is only 400 elements long. Check out my code and you will
>>>> see that at 400 elements long, fromiter is actually the fastest (at least
>>>> when I tried). An example run:
>>>>
>>>> Copy using for loop in 0.000884 sec
>>>> Copy using fromiter in 0.000144 sec # fastest
>>>> Copy using fromstring in 0.001460 sec # fairly slow, 10.3x slower than
>>>> fromiter
>>>> Copy using Marshal.Copy in 0.001680 sec # slowest, 11.7x slower than
>>>> fromiter
>>>>
>>>> I start to do better with Marshal.Copy then fromiter around 5000
>>>> elements copied. This is because the overhead of the mass copies is high
>>>> but adding each element doesn't take much time. fromstring has a lower
>>>> overhead but slightly longer per-element time (fromstring is better than
>>>> Marshal.Copy until ~200,000 elements).
>>>>
>>>> So you might be doing as good as you can possibly do. If I knew more
>>>> about your file format library I might be able to provide more insight.
>>>>
>>>> Jeff
>>>>
>>>> On Tue, Oct 28, 2014 at 2:45 PM, Bradley Friedman <brad at fie.us> wrote:
>>>>
>>>>> Well it makes sense to me that doing it via an iterator, and element
>>>>> at a time, would be slow.  There’s a lot of call overhead associated with
>>>>> each iteration step.  Whether it’s done in .net, or in python, or a call
>>>>> from one to the other, it will be slow.  It’s still a call where you’d be
>>>>> better off copying whole buffers.
>>>>>
>>>>> Ideally you’d pull the data into as simple and raw a data structure as
>>>>> you can on the dotnet side, in a buffered manner.  Then you’d execute a
>>>>> movement of the data across, a reasonably sized chunk of buffer at a time.
>>>>> This will reduce call overhead and also allow read-ahead caching to do its
>>>>> thing on the file-access side of things.
>>>>>
>>>>> Your suggestion of loading into a .net array and then moving that
>>>>> array over, makes sense.  But I think it comes down to what you can do with
>>>>> the third party file-format library. If its not going to provide you with
>>>>> the data as some kind of buffer with a cohesive and known format in memory,
>>>>> you’re not really going to be able to move it over without iterating over
>>>>> it and reformatting it at some point.
>>>>>
>>>>> Specifically, I’d point to Jeffery’s original caveat:
>>>>>
>>>>> "but does involve a number of assumptions (for example that the data
>>>>> in the two arrays are laid out in the same way)."
>>>>>
>>>>> The question is:  is there a way to get the data off of disk and in
>>>>> memory from dotnet library, where its layout in memory is known, and
>>>>> something you want exactly as it is, but in python?  If so, you should be
>>>>> able to use the methods from the afore linked thread.  If not, you’re
>>>>> probably stuck iterating somewhere to reformat it, no matter what.  Which
>>>>> is probably why you got garbage back.  I’m guessing the object returned
>>>>> from the dotnet file-format-library isn’t laid out right, as suggested in
>>>>> the afore referenced caveat.
>>>>>
>>>>>
>>>>> > On Oct 28, 2014, at 9:55 AM, Nikhil <nikhilgarg.gju at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hello,
>>>>> > Yeah, I read data from a file say at each node and each time step,
>>>>> but when i try to use Marshal approach i get gibberish but when i use
>>>>> simple iter i get correct values. i have been trying the approach used in
>>>>> example in the previous post and that example makes sense but it doesnt
>>>>> make sense when i use it in my case. I am right now assigning it to a
>>>>> variable, i am now thinking of exploring the possibility of saving data to
>>>>> a dot net array maybe using System.Array and saving data to it but not sure
>>>>> if that even make sense.
>>>>> >
>>>>> > Sent from my iPhone
>>>>>
>>>>> _________________________________________________
>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>
>>>>
>>>>
>>>> _________________________________________________
>>>> Python.NET mailing list - PythonDotNet at python.org
>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>
>>>
>>>
>>>
>>> --
>>> Regards
>>>
>>> Nikhil
>>>
>>> -------------------------------------------------------------------
>>> Big whirls have little whirls,
>>> Which feed on their velocity,
>>> And little whirls have lesser whirls,
>>> And so on to viscosity
>>> (Richardson, 1922)
>>>
>>> _________________________________________________
>>> Python.NET mailing list - PythonDotNet at python.org
>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141105/29aa5ff1/attachment.html>