[Python.NET] Efficient copy of .NET Array to ctypes or numpy array

[Bradley Friedman]

Well it makes sense to me that doing it via an iterator, and element at a time, would be slow.  There’s a lot of call overhead associated with each iteration step.  Whether it’s done in .net, or in python, or a call from one to the other, it will be slow.  It’s still a call where you’d be better off copying whole buffers.

Ideally you’d pull the data into as simple and raw a data structure as you can on the dotnet side, in a buffered manner.  Then you’d execute a movement of the data across, a reasonably sized chunk of buffer at a time.  This will reduce call overhead and also allow read-ahead caching to do its thing on the file-access side of things.

Your suggestion of loading into a .net array and then moving that array over, makes sense.  But I think it comes down to what you can do with the third party file-format library. If its not going to provide you with the data as some kind of buffer with a cohesive and known format in memory, you’re not really going to be able to move it over without iterating over it and reformatting it at some point.

Specifically, I’d point to Jeffery’s original caveat:

"but does involve a number of assumptions (for example that the data in the two arrays are laid out in the same way)."

The question is:  is there a way to get the data off of disk and in memory from dotnet library, where its layout in memory is known, and something you want exactly as it is, but in python?  If so, you should be able to use the methods from the afore linked thread.  If not, you’re probably stuck iterating somewhere to reformat it, no matter what.  Which is probably why you got garbage back.  I’m guessing the object returned from the dotnet file-format-library isn’t laid out right, as suggested in the afore referenced caveat.


> On Oct 28, 2014, at 9:55 AM, Nikhil <nikhilgarg.gju at gmail.com> wrote:
> 
> Hello,
> Yeah, I read data from a file say at each node and each time step, but when i try to use Marshal approach i get gibberish but when i use simple iter i get correct values. i have been trying the approach used in example in the previous post and that example makes sense but it doesnt make sense when i use it in my case. I am right now assigning it to a variable, i am now thinking of exploring the possibility of saving data to a dot net array maybe using System.Array and saving data to it but not sure if that even make sense. 
> 
> Sent from my iPhone