[Python.NET] Efficient copy of .NET Array to ctypes or numpy array

[Bradley Friedman]

You indicate that you are reading from a file.  The thread you reference was about copying data in memory.  I’d think matters of buffering and read-ahead caches would be far more relevant than anything else.  Am I missing something?

> On Oct 26, 2014, at 5:18 AM, Nikhil Garg <nikhilgarg.gju at gmail.com> wrote:
> 
> 
> Hi,
> 
> I have looked at the following thread to copy a c# array to numpy list using python dot net.
> 
> https://mail.python.org/pipermail/pythondotnet/2014-May/001525.html <https://mail.python.org/pipermail/pythondotnet/2014-May/001525.html>
> 
> When I try the methods mentioned in the complete thread I manage to get the array from simple iteration method or 
> np.fromiter, but I have not been able to make GCHandle or Marshal.Copy work for my case.
> 
> To explain about my issue, I use a computer code written in c# which produces output in a custom build format. I am able to access the data from the file system using python dot net, but it seems that it take almost half an hour to get the data from the file.
> 
> For example my file has an array of size 5018x73x400, if i use fromiter, i provide the last bit of array to copy to python due to the way file stores output.
> 
> for j in xrange(num_nodes-1):
> 	for i in xrange(nsteps):
> 		temp = file_in.ReadItemTimeStep(j+1,i).Data
> 		temp_1 = np.fromiter(temp, float)
> 		dest[j, i] = temp_1
> 
> Other way of doing same task would be to use a for loop but tht would be really slow.
> 
> for k in xrange(num_nodes-1):
>         for k in xrange(nsteps):
>                 for i in xrange(nvals):
>                        temp[k,j,i] = file_in.ReadItemTimeStep(k+1, j).Data[i]
> 
> I am trying to find a faster way to extract the data from the file. I was wondering if there is a possibility to use something like GChandle or Marshal.Copy. I have also attached same output from Marshal.Copy
> 
> Marshal.Copy(temp, 0, IntPtr.__overloads__[int](temp2.__array_interface__['data'][0]), len(temp))
> 
> 
>          0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
>          0.00000000e+000,   3.13551399e-242,   4.20993645e-161,
>          8.71245319e-119,   8.38370432e-089,   2.60762781e-075,
>          1.92616374e-072,   2.89006184e-072,   3.06007265e-082,
>          3.81879776e-315,   0.00000000e+000,   0.00000000e+000,
>          0.00000000e+000,   1.03959661e-297,   7.22038448e-196,
>          3.05237954e-130,   2.58469346e-093,   2.29495911e-070,
>          3.74354820e-062,   1.74288060e-061,   2.13172553e-067,
>          2.74305386e-077,   0.00000000e+000,   0.00000000e+000,
>          0.00000000e+000,   0.00000000e+000,   1.02484756e-240,
>          1.70193770e-154,   1.30625320e-108,   1.16793762e-075,
>          2.96889340e-061,   7.84687103e-058,   1.16879604e-057,
>          5.56256478e-068,   4.06437857e-315,   0.00000000e+000,
>          0.00000000e+000,   0.00000000e+000,   1.11705239e-296,
>          1.46978156e-194,   1.98034372e-128,   3.79615170e-091,
>          5.88936509e-068,   1.85305929e-059,   2.16186001e-058,
> 
> Whereas the actual values stored in the array are 
>          0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
>          0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
>          0.00000000e+00,   0.00000000e+00,   5.66641200e-37,
>          1.24961600e-30,   5.34846800e-25,   1.72519600e-20,
>          1.49402200e-17,   3.27685300e-15,   4.10687500e-13,
>          1.87978200e-11,   2.01397900e-10,   8.83561100e-10,
>          2.14397600e-09,   2.02600300e-09,   2.47599800e-09,
>          2.15987100e-09,   4.82694100e-10,   1.18759300e-10,
>          3.32024100e-11,   0.00000000e+00,   0.00000000e+00,
>          0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
>          0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
>          1.45288600e-37,   7.37245900e-31,   7.59282500e-25,
>          5.85463400e-20,   1.20634700e-16,   3.06414300e-14,
>          5.16041300e-12,   2.87562800e-10,   3.62841500e-09,
>          1.63540600e-08,   4.06988300e-08,   3.97764200e-08,
>          4.88768500e-08,   4.07385000e-08,   8.78868600e-09,
> 
> I would appreciate any help i can get regarding this matter.
> 
> Nikhil
> _________________________________________________
> Python.NET mailing list - PythonDotNet at python.org
> https://mail.python.org/mailman/listinfo/pythondotnet

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141027/decdabc6/attachment-0001.html>