From denis.akhiyarov at gmail.com  Wed Nov  5 15:58:08 2014
From: denis.akhiyarov at gmail.com (Denis Akhiyarov)
Date: Wed, 5 Nov 2014 08:58:08 -0600
Subject: [Python.NET] Efficient copy of .NET Array to ctypes or numpy
	array
In-Reply-To: <CAL3fuhmT9seWajW=mS_=uGBDc6o3y5mL6Wzxw2xiXs7oQ_ExRQ@mail.gmail.com>
References: <CAL3fuhnJb60pz5+H5+mJ8sORVuAZHU1B+J_YqK3b0dEyALLgfA@mail.gmail.com>
 <8BE6014E-0EB7-43B7-AF98-EB35C292E0D1@fie.us>
 <AC2FE74D-5440-4756-9FBB-E920835B1495@gmail.com>
 <B5C9A727-F367-4061-8EFE-EBEE11AF4385@fie.us>
 <CALOUSfnpQb7PUm0MrM1t1WuFiGXDxw4guz4tjz9JcyYxHUE3aw@mail.gmail.com>
 <CAL3fuhmT9seWajW=mS_=uGBDc6o3y5mL6Wzxw2xiXs7oQ_ExRQ@mail.gmail.com>
Message-ID: <CALxxJLTgveWW6UNxtoHKd54DfM26JLzmPC5hphH5jfBi2Y=yUg@mail.gmail.com>

How to copy unmanaged array (python list/tuple or numpy array) into managed
C# array? I guess using Marshal.Copy, but can anyone point to example?

Thanks,
Denis

On Thu, Oct 30, 2014 at 12:19 PM, Nikhil Garg <nikhilgarg.gju at gmail.com>
wrote:

> Thanks Brad and Jeff for the detailed info. For now, fromiter is serving
> me well and has reduced my processing time considerably, so I am just going
> to stick with it.
>
>
> On 29 October 2014 11:04, Jeffrey Bush <jeff at coderforlife.com> wrote:
>
>> I finally have a chance to chime in, and Bradley is exactly right.
>> Marshall.Copy copies the raw data, and apparently your file library does
>> not store that data in a nice, contiguous, manner. While it is highly
>> likely that copying all the data to an array in C# will be faster than the
>> fromiter in Python, I am unsure if copying all the data to an array in C#
>> then copying all the data again to a numpy array will be faster than
>> fromiter (cause you have to copy it twice). The exception is if the file
>> library has a function like ToArray that is optimized to copy the data to a
>> linear chunk of data. So, what type is "Data"?
>>
>> Another factor is how long the chunk of data you are copying is. You say
>> the last axis is only 400 elements long. Check out my code and you will see
>> that at 400 elements long, fromiter is actually the fastest (at least when
>> I tried). An example run:
>>
>> Copy using for loop in 0.000884 sec
>> Copy using fromiter in 0.000144 sec # fastest
>> Copy using fromstring in 0.001460 sec # fairly slow, 10.3x slower than
>> fromiter
>> Copy using Marshal.Copy in 0.001680 sec # slowest, 11.7x slower than
>> fromiter
>>
>> I start to do better with Marshal.Copy then fromiter around 5000 elements
>> copied. This is because the overhead of the mass copies is high but adding
>> each element doesn't take much time. fromstring has a lower overhead but
>> slightly longer per-element time (fromstring is better than Marshal.Copy
>> until ~200,000 elements).
>>
>> So you might be doing as good as you can possibly do. If I knew more
>> about your file format library I might be able to provide more insight.
>>
>> Jeff
>>
>> On Tue, Oct 28, 2014 at 2:45 PM, Bradley Friedman <brad at fie.us> wrote:
>>
>>> Well it makes sense to me that doing it via an iterator, and element at
>>> a time, would be slow.  There?s a lot of call overhead associated with each
>>> iteration step.  Whether it?s done in .net, or in python, or a call from
>>> one to the other, it will be slow.  It?s still a call where you?d be better
>>> off copying whole buffers.
>>>
>>> Ideally you?d pull the data into as simple and raw a data structure as
>>> you can on the dotnet side, in a buffered manner.  Then you?d execute a
>>> movement of the data across, a reasonably sized chunk of buffer at a time.
>>> This will reduce call overhead and also allow read-ahead caching to do its
>>> thing on the file-access side of things.
>>>
>>> Your suggestion of loading into a .net array and then moving that array
>>> over, makes sense.  But I think it comes down to what you can do with the
>>> third party file-format library. If its not going to provide you with the
>>> data as some kind of buffer with a cohesive and known format in memory,
>>> you?re not really going to be able to move it over without iterating over
>>> it and reformatting it at some point.
>>>
>>> Specifically, I?d point to Jeffery?s original caveat:
>>>
>>> "but does involve a number of assumptions (for example that the data in
>>> the two arrays are laid out in the same way)."
>>>
>>> The question is:  is there a way to get the data off of disk and in
>>> memory from dotnet library, where its layout in memory is known, and
>>> something you want exactly as it is, but in python?  If so, you should be
>>> able to use the methods from the afore linked thread.  If not, you?re
>>> probably stuck iterating somewhere to reformat it, no matter what.  Which
>>> is probably why you got garbage back.  I?m guessing the object returned
>>> from the dotnet file-format-library isn?t laid out right, as suggested in
>>> the afore referenced caveat.
>>>
>>>
>>> > On Oct 28, 2014, at 9:55 AM, Nikhil <nikhilgarg.gju at gmail.com> wrote:
>>> >
>>> > Hello,
>>> > Yeah, I read data from a file say at each node and each time step, but
>>> when i try to use Marshal approach i get gibberish but when i use simple
>>> iter i get correct values. i have been trying the approach used in example
>>> in the previous post and that example makes sense but it doesnt make sense
>>> when i use it in my case. I am right now assigning it to a variable, i am
>>> now thinking of exploring the possibility of saving data to a dot net array
>>> maybe using System.Array and saving data to it but not sure if that even
>>> make sense.
>>> >
>>> > Sent from my iPhone
>>>
>>> _________________________________________________
>>> Python.NET mailing list - PythonDotNet at python.org
>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>
>>
>>
>> _________________________________________________
>> Python.NET mailing list - PythonDotNet at python.org
>> https://mail.python.org/mailman/listinfo/pythondotnet
>>
>
>
>
> --
> Regards
>
> Nikhil
>
> -------------------------------------------------------------------
> Big whirls have little whirls,
> Which feed on their velocity,
> And little whirls have lesser whirls,
> And so on to viscosity
> (Richardson, 1922)
>
> _________________________________________________
> Python.NET mailing list - PythonDotNet at python.org
> https://mail.python.org/mailman/listinfo/pythondotnet
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141105/583e10d7/attachment-0001.html>

From denis.akhiyarov at gmail.com  Wed Nov  5 16:31:25 2014
From: denis.akhiyarov at gmail.com (Denis Akhiyarov)
Date: Wed, 5 Nov 2014 09:31:25 -0600
Subject: [Python.NET] Efficient copy of .NET Array to ctypes or numpy
	array
In-Reply-To: <CALxxJLTgveWW6UNxtoHKd54DfM26JLzmPC5hphH5jfBi2Y=yUg@mail.gmail.com>
References: <CAL3fuhnJb60pz5+H5+mJ8sORVuAZHU1B+J_YqK3b0dEyALLgfA@mail.gmail.com>
 <8BE6014E-0EB7-43B7-AF98-EB35C292E0D1@fie.us>
 <AC2FE74D-5440-4756-9FBB-E920835B1495@gmail.com>
 <B5C9A727-F367-4061-8EFE-EBEE11AF4385@fie.us>
 <CALOUSfnpQb7PUm0MrM1t1WuFiGXDxw4guz4tjz9JcyYxHUE3aw@mail.gmail.com>
 <CAL3fuhmT9seWajW=mS_=uGBDc6o3y5mL6Wzxw2xiXs7oQ_ExRQ@mail.gmail.com>
 <CALxxJLTgveWW6UNxtoHKd54DfM26JLzmPC5hphH5jfBi2Y=yUg@mail.gmail.com>
Message-ID: <CALxxJLTm6-0EegAfRk8hSxzcV_fcpnv4ENHSpHPdvVGHMeNxAQ@mail.gmail.com>

And more important question - is it possible to generalize the copying of
python array object to managed C# array object without knowing the data
type/size/length?

On Wed, Nov 5, 2014 at 8:58 AM, Denis Akhiyarov <denis.akhiyarov at gmail.com>
wrote:

> How to copy unmanaged array (python list/tuple or numpy array) into
> managed C# array? I guess using Marshal.Copy, but can anyone point to
> example?
>
> Thanks,
> Denis
>
> On Thu, Oct 30, 2014 at 12:19 PM, Nikhil Garg <nikhilgarg.gju at gmail.com>
> wrote:
>
>> Thanks Brad and Jeff for the detailed info. For now, fromiter is serving
>> me well and has reduced my processing time considerably, so I am just going
>> to stick with it.
>>
>>
>> On 29 October 2014 11:04, Jeffrey Bush <jeff at coderforlife.com> wrote:
>>
>>> I finally have a chance to chime in, and Bradley is exactly right.
>>> Marshall.Copy copies the raw data, and apparently your file library does
>>> not store that data in a nice, contiguous, manner. While it is highly
>>> likely that copying all the data to an array in C# will be faster than the
>>> fromiter in Python, I am unsure if copying all the data to an array in C#
>>> then copying all the data again to a numpy array will be faster than
>>> fromiter (cause you have to copy it twice). The exception is if the file
>>> library has a function like ToArray that is optimized to copy the data to a
>>> linear chunk of data. So, what type is "Data"?
>>>
>>> Another factor is how long the chunk of data you are copying is. You say
>>> the last axis is only 400 elements long. Check out my code and you will see
>>> that at 400 elements long, fromiter is actually the fastest (at least when
>>> I tried). An example run:
>>>
>>> Copy using for loop in 0.000884 sec
>>> Copy using fromiter in 0.000144 sec # fastest
>>> Copy using fromstring in 0.001460 sec # fairly slow, 10.3x slower than
>>> fromiter
>>> Copy using Marshal.Copy in 0.001680 sec # slowest, 11.7x slower than
>>> fromiter
>>>
>>> I start to do better with Marshal.Copy then fromiter around 5000
>>> elements copied. This is because the overhead of the mass copies is high
>>> but adding each element doesn't take much time. fromstring has a lower
>>> overhead but slightly longer per-element time (fromstring is better than
>>> Marshal.Copy until ~200,000 elements).
>>>
>>> So you might be doing as good as you can possibly do. If I knew more
>>> about your file format library I might be able to provide more insight.
>>>
>>> Jeff
>>>
>>> On Tue, Oct 28, 2014 at 2:45 PM, Bradley Friedman <brad at fie.us> wrote:
>>>
>>>> Well it makes sense to me that doing it via an iterator, and element at
>>>> a time, would be slow.  There?s a lot of call overhead associated with each
>>>> iteration step.  Whether it?s done in .net, or in python, or a call from
>>>> one to the other, it will be slow.  It?s still a call where you?d be better
>>>> off copying whole buffers.
>>>>
>>>> Ideally you?d pull the data into as simple and raw a data structure as
>>>> you can on the dotnet side, in a buffered manner.  Then you?d execute a
>>>> movement of the data across, a reasonably sized chunk of buffer at a time.
>>>> This will reduce call overhead and also allow read-ahead caching to do its
>>>> thing on the file-access side of things.
>>>>
>>>> Your suggestion of loading into a .net array and then moving that array
>>>> over, makes sense.  But I think it comes down to what you can do with the
>>>> third party file-format library. If its not going to provide you with the
>>>> data as some kind of buffer with a cohesive and known format in memory,
>>>> you?re not really going to be able to move it over without iterating over
>>>> it and reformatting it at some point.
>>>>
>>>> Specifically, I?d point to Jeffery?s original caveat:
>>>>
>>>> "but does involve a number of assumptions (for example that the data in
>>>> the two arrays are laid out in the same way)."
>>>>
>>>> The question is:  is there a way to get the data off of disk and in
>>>> memory from dotnet library, where its layout in memory is known, and
>>>> something you want exactly as it is, but in python?  If so, you should be
>>>> able to use the methods from the afore linked thread.  If not, you?re
>>>> probably stuck iterating somewhere to reformat it, no matter what.  Which
>>>> is probably why you got garbage back.  I?m guessing the object returned
>>>> from the dotnet file-format-library isn?t laid out right, as suggested in
>>>> the afore referenced caveat.
>>>>
>>>>
>>>> > On Oct 28, 2014, at 9:55 AM, Nikhil <nikhilgarg.gju at gmail.com> wrote:
>>>> >
>>>> > Hello,
>>>> > Yeah, I read data from a file say at each node and each time step,
>>>> but when i try to use Marshal approach i get gibberish but when i use
>>>> simple iter i get correct values. i have been trying the approach used in
>>>> example in the previous post and that example makes sense but it doesnt
>>>> make sense when i use it in my case. I am right now assigning it to a
>>>> variable, i am now thinking of exploring the possibility of saving data to
>>>> a dot net array maybe using System.Array and saving data to it but not sure
>>>> if that even make sense.
>>>> >
>>>> > Sent from my iPhone
>>>>
>>>> _________________________________________________
>>>> Python.NET mailing list - PythonDotNet at python.org
>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>
>>>
>>>
>>> _________________________________________________
>>> Python.NET mailing list - PythonDotNet at python.org
>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>
>>
>>
>>
>> --
>> Regards
>>
>> Nikhil
>>
>> -------------------------------------------------------------------
>> Big whirls have little whirls,
>> Which feed on their velocity,
>> And little whirls have lesser whirls,
>> And so on to viscosity
>> (Richardson, 1922)
>>
>> _________________________________________________
>> Python.NET mailing list - PythonDotNet at python.org
>> https://mail.python.org/mailman/listinfo/pythondotnet
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141105/c2ebfbf1/attachment-0001.html>

From denis.akhiyarov at gmail.com  Wed Nov  5 20:28:04 2014
From: denis.akhiyarov at gmail.com (Denis Akhiyarov)
Date: Wed, 5 Nov 2014 13:28:04 -0600
Subject: [Python.NET] Efficient copy of .NET Array to ctypes or numpy
	array
In-Reply-To: <CALxxJLTm6-0EegAfRk8hSxzcV_fcpnv4ENHSpHPdvVGHMeNxAQ@mail.gmail.com>
References: <CAL3fuhnJb60pz5+H5+mJ8sORVuAZHU1B+J_YqK3b0dEyALLgfA@mail.gmail.com>
 <8BE6014E-0EB7-43B7-AF98-EB35C292E0D1@fie.us>
 <AC2FE74D-5440-4756-9FBB-E920835B1495@gmail.com>
 <B5C9A727-F367-4061-8EFE-EBEE11AF4385@fie.us>
 <CALOUSfnpQb7PUm0MrM1t1WuFiGXDxw4guz4tjz9JcyYxHUE3aw@mail.gmail.com>
 <CAL3fuhmT9seWajW=mS_=uGBDc6o3y5mL6Wzxw2xiXs7oQ_ExRQ@mail.gmail.com>
 <CALxxJLTgveWW6UNxtoHKd54DfM26JLzmPC5hphH5jfBi2Y=yUg@mail.gmail.com>
 <CALxxJLTm6-0EegAfRk8hSxzcV_fcpnv4ENHSpHPdvVGHMeNxAQ@mail.gmail.com>
Message-ID: <CALxxJLS5VCzKoaGSpyf+30ZYfAvOfBxiNzK2c3yADx3_Pwxteg@mail.gmail.com>

Finally decided to generate the managed object on Python side and return it
to C# with no conversion necessary in C#. This way I can even wrap regular
function with Python with @decorator to handle the conversion.  I suppose
the dynamic version of pythonnet may have auto conversion for Python 3, but
I have not tried. I'm on Python 2.7.

On Wed, Nov 5, 2014 at 9:31 AM, Denis Akhiyarov <denis.akhiyarov at gmail.com>
wrote:

> And more important question - is it possible to generalize the copying of
> python array object to managed C# array object without knowing the data
> type/size/length?
>
> On Wed, Nov 5, 2014 at 8:58 AM, Denis Akhiyarov <denis.akhiyarov at gmail.com
> > wrote:
>
>> How to copy unmanaged array (python list/tuple or numpy array) into
>> managed C# array? I guess using Marshal.Copy, but can anyone point to
>> example?
>>
>> Thanks,
>> Denis
>>
>> On Thu, Oct 30, 2014 at 12:19 PM, Nikhil Garg <nikhilgarg.gju at gmail.com>
>> wrote:
>>
>>> Thanks Brad and Jeff for the detailed info. For now, fromiter is serving
>>> me well and has reduced my processing time considerably, so I am just going
>>> to stick with it.
>>>
>>>
>>> On 29 October 2014 11:04, Jeffrey Bush <jeff at coderforlife.com> wrote:
>>>
>>>> I finally have a chance to chime in, and Bradley is exactly right.
>>>> Marshall.Copy copies the raw data, and apparently your file library does
>>>> not store that data in a nice, contiguous, manner. While it is highly
>>>> likely that copying all the data to an array in C# will be faster than the
>>>> fromiter in Python, I am unsure if copying all the data to an array in C#
>>>> then copying all the data again to a numpy array will be faster than
>>>> fromiter (cause you have to copy it twice). The exception is if the file
>>>> library has a function like ToArray that is optimized to copy the data to a
>>>> linear chunk of data. So, what type is "Data"?
>>>>
>>>> Another factor is how long the chunk of data you are copying is. You
>>>> say the last axis is only 400 elements long. Check out my code and you will
>>>> see that at 400 elements long, fromiter is actually the fastest (at least
>>>> when I tried). An example run:
>>>>
>>>> Copy using for loop in 0.000884 sec
>>>> Copy using fromiter in 0.000144 sec # fastest
>>>> Copy using fromstring in 0.001460 sec # fairly slow, 10.3x slower than
>>>> fromiter
>>>> Copy using Marshal.Copy in 0.001680 sec # slowest, 11.7x slower than
>>>> fromiter
>>>>
>>>> I start to do better with Marshal.Copy then fromiter around 5000
>>>> elements copied. This is because the overhead of the mass copies is high
>>>> but adding each element doesn't take much time. fromstring has a lower
>>>> overhead but slightly longer per-element time (fromstring is better than
>>>> Marshal.Copy until ~200,000 elements).
>>>>
>>>> So you might be doing as good as you can possibly do. If I knew more
>>>> about your file format library I might be able to provide more insight.
>>>>
>>>> Jeff
>>>>
>>>> On Tue, Oct 28, 2014 at 2:45 PM, Bradley Friedman <brad at fie.us> wrote:
>>>>
>>>>> Well it makes sense to me that doing it via an iterator, and element
>>>>> at a time, would be slow.  There?s a lot of call overhead associated with
>>>>> each iteration step.  Whether it?s done in .net, or in python, or a call
>>>>> from one to the other, it will be slow.  It?s still a call where you?d be
>>>>> better off copying whole buffers.
>>>>>
>>>>> Ideally you?d pull the data into as simple and raw a data structure as
>>>>> you can on the dotnet side, in a buffered manner.  Then you?d execute a
>>>>> movement of the data across, a reasonably sized chunk of buffer at a time.
>>>>> This will reduce call overhead and also allow read-ahead caching to do its
>>>>> thing on the file-access side of things.
>>>>>
>>>>> Your suggestion of loading into a .net array and then moving that
>>>>> array over, makes sense.  But I think it comes down to what you can do with
>>>>> the third party file-format library. If its not going to provide you with
>>>>> the data as some kind of buffer with a cohesive and known format in memory,
>>>>> you?re not really going to be able to move it over without iterating over
>>>>> it and reformatting it at some point.
>>>>>
>>>>> Specifically, I?d point to Jeffery?s original caveat:
>>>>>
>>>>> "but does involve a number of assumptions (for example that the data
>>>>> in the two arrays are laid out in the same way)."
>>>>>
>>>>> The question is:  is there a way to get the data off of disk and in
>>>>> memory from dotnet library, where its layout in memory is known, and
>>>>> something you want exactly as it is, but in python?  If so, you should be
>>>>> able to use the methods from the afore linked thread.  If not, you?re
>>>>> probably stuck iterating somewhere to reformat it, no matter what.  Which
>>>>> is probably why you got garbage back.  I?m guessing the object returned
>>>>> from the dotnet file-format-library isn?t laid out right, as suggested in
>>>>> the afore referenced caveat.
>>>>>
>>>>>
>>>>> > On Oct 28, 2014, at 9:55 AM, Nikhil <nikhilgarg.gju at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hello,
>>>>> > Yeah, I read data from a file say at each node and each time step,
>>>>> but when i try to use Marshal approach i get gibberish but when i use
>>>>> simple iter i get correct values. i have been trying the approach used in
>>>>> example in the previous post and that example makes sense but it doesnt
>>>>> make sense when i use it in my case. I am right now assigning it to a
>>>>> variable, i am now thinking of exploring the possibility of saving data to
>>>>> a dot net array maybe using System.Array and saving data to it but not sure
>>>>> if that even make sense.
>>>>> >
>>>>> > Sent from my iPhone
>>>>>
>>>>> _________________________________________________
>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>
>>>>
>>>>
>>>> _________________________________________________
>>>> Python.NET mailing list - PythonDotNet at python.org
>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>
>>>
>>>
>>>
>>> --
>>> Regards
>>>
>>> Nikhil
>>>
>>> -------------------------------------------------------------------
>>> Big whirls have little whirls,
>>> Which feed on their velocity,
>>> And little whirls have lesser whirls,
>>> And so on to viscosity
>>> (Richardson, 1922)
>>>
>>> _________________________________________________
>>> Python.NET mailing list - PythonDotNet at python.org
>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141105/29aa5ff1/attachment.html>

From jeff at coderforlife.com  Wed Nov  5 23:32:45 2014
From: jeff at coderforlife.com (Jeffrey Bush)
Date: Wed, 5 Nov 2014 14:32:45 -0800
Subject: [Python.NET] Efficient copy of .NET Array to ctypes or numpy
	array
In-Reply-To: <CALxxJLS5VCzKoaGSpyf+30ZYfAvOfBxiNzK2c3yADx3_Pwxteg@mail.gmail.com>
References: <CAL3fuhnJb60pz5+H5+mJ8sORVuAZHU1B+J_YqK3b0dEyALLgfA@mail.gmail.com>
 <8BE6014E-0EB7-43B7-AF98-EB35C292E0D1@fie.us>
 <AC2FE74D-5440-4756-9FBB-E920835B1495@gmail.com>
 <B5C9A727-F367-4061-8EFE-EBEE11AF4385@fie.us>
 <CALOUSfnpQb7PUm0MrM1t1WuFiGXDxw4guz4tjz9JcyYxHUE3aw@mail.gmail.com>
 <CAL3fuhmT9seWajW=mS_=uGBDc6o3y5mL6Wzxw2xiXs7oQ_ExRQ@mail.gmail.com>
 <CALxxJLTgveWW6UNxtoHKd54DfM26JLzmPC5hphH5jfBi2Y=yUg@mail.gmail.com>
 <CALxxJLTm6-0EegAfRk8hSxzcV_fcpnv4ENHSpHPdvVGHMeNxAQ@mail.gmail.com>
 <CALxxJLS5VCzKoaGSpyf+30ZYfAvOfBxiNzK2c3yADx3_Pwxteg@mail.gmail.com>
Message-ID: <CALOUSfmP9AKHdJMwj5dpcknd5PD+T1FmsWMj0SOSs54X-EDVDA@mail.gmail.com>

To copy from a list or tuple you need to use the Python buffer() function.
You can use that on numpy arrays as well. In buffer form, the len() is the
byte length so then you don't need to know the data type or size. However,
some additional work would be required if you wanted to make sure the C#
array was of the proper type and length.

Jeff

On Wed, Nov 5, 2014 at 11:28 AM, Denis Akhiyarov <denis.akhiyarov at gmail.com>
wrote:

> Finally decided to generate the managed object on Python side and return
> it to C# with no conversion necessary in C#. This way I can even wrap
> regular function with Python with @decorator to handle the conversion.  I
> suppose the dynamic version of pythonnet may have auto conversion for
> Python 3, but I have not tried. I'm on Python 2.7.
>
> On Wed, Nov 5, 2014 at 9:31 AM, Denis Akhiyarov <denis.akhiyarov at gmail.com
> > wrote:
>
>> And more important question - is it possible to generalize the copying of
>> python array object to managed C# array object without knowing the data
>> type/size/length?
>>
>> On Wed, Nov 5, 2014 at 8:58 AM, Denis Akhiyarov <
>> denis.akhiyarov at gmail.com> wrote:
>>
>>> How to copy unmanaged array (python list/tuple or numpy array) into
>>> managed C# array? I guess using Marshal.Copy, but can anyone point to
>>> example?
>>>
>>> Thanks,
>>> Denis
>>>
>>> On Thu, Oct 30, 2014 at 12:19 PM, Nikhil Garg <nikhilgarg.gju at gmail.com>
>>> wrote:
>>>
>>>> Thanks Brad and Jeff for the detailed info. For now, fromiter is
>>>> serving me well and has reduced my processing time considerably, so I am
>>>> just going to stick with it.
>>>>
>>>>
>>>> On 29 October 2014 11:04, Jeffrey Bush <jeff at coderforlife.com> wrote:
>>>>
>>>>> I finally have a chance to chime in, and Bradley is exactly right.
>>>>> Marshall.Copy copies the raw data, and apparently your file library does
>>>>> not store that data in a nice, contiguous, manner. While it is highly
>>>>> likely that copying all the data to an array in C# will be faster than the
>>>>> fromiter in Python, I am unsure if copying all the data to an array in C#
>>>>> then copying all the data again to a numpy array will be faster than
>>>>> fromiter (cause you have to copy it twice). The exception is if the file
>>>>> library has a function like ToArray that is optimized to copy the data to a
>>>>> linear chunk of data. So, what type is "Data"?
>>>>>
>>>>> Another factor is how long the chunk of data you are copying is. You
>>>>> say the last axis is only 400 elements long. Check out my code and you will
>>>>> see that at 400 elements long, fromiter is actually the fastest (at least
>>>>> when I tried). An example run:
>>>>>
>>>>> Copy using for loop in 0.000884 sec
>>>>> Copy using fromiter in 0.000144 sec # fastest
>>>>> Copy using fromstring in 0.001460 sec # fairly slow, 10.3x slower than
>>>>> fromiter
>>>>> Copy using Marshal.Copy in 0.001680 sec # slowest, 11.7x slower than
>>>>> fromiter
>>>>>
>>>>> I start to do better with Marshal.Copy then fromiter around 5000
>>>>> elements copied. This is because the overhead of the mass copies is high
>>>>> but adding each element doesn't take much time. fromstring has a lower
>>>>> overhead but slightly longer per-element time (fromstring is better than
>>>>> Marshal.Copy until ~200,000 elements).
>>>>>
>>>>> So you might be doing as good as you can possibly do. If I knew more
>>>>> about your file format library I might be able to provide more insight.
>>>>>
>>>>> Jeff
>>>>>
>>>>> On Tue, Oct 28, 2014 at 2:45 PM, Bradley Friedman <brad at fie.us> wrote:
>>>>>
>>>>>> Well it makes sense to me that doing it via an iterator, and element
>>>>>> at a time, would be slow.  There?s a lot of call overhead associated with
>>>>>> each iteration step.  Whether it?s done in .net, or in python, or a call
>>>>>> from one to the other, it will be slow.  It?s still a call where you?d be
>>>>>> better off copying whole buffers.
>>>>>>
>>>>>> Ideally you?d pull the data into as simple and raw a data structure
>>>>>> as you can on the dotnet side, in a buffered manner.  Then you?d execute a
>>>>>> movement of the data across, a reasonably sized chunk of buffer at a time.
>>>>>> This will reduce call overhead and also allow read-ahead caching to do its
>>>>>> thing on the file-access side of things.
>>>>>>
>>>>>> Your suggestion of loading into a .net array and then moving that
>>>>>> array over, makes sense.  But I think it comes down to what you can do with
>>>>>> the third party file-format library. If its not going to provide you with
>>>>>> the data as some kind of buffer with a cohesive and known format in memory,
>>>>>> you?re not really going to be able to move it over without iterating over
>>>>>> it and reformatting it at some point.
>>>>>>
>>>>>> Specifically, I?d point to Jeffery?s original caveat:
>>>>>>
>>>>>> "but does involve a number of assumptions (for example that the data
>>>>>> in the two arrays are laid out in the same way)."
>>>>>>
>>>>>> The question is:  is there a way to get the data off of disk and in
>>>>>> memory from dotnet library, where its layout in memory is known, and
>>>>>> something you want exactly as it is, but in python?  If so, you should be
>>>>>> able to use the methods from the afore linked thread.  If not, you?re
>>>>>> probably stuck iterating somewhere to reformat it, no matter what.  Which
>>>>>> is probably why you got garbage back.  I?m guessing the object returned
>>>>>> from the dotnet file-format-library isn?t laid out right, as suggested in
>>>>>> the afore referenced caveat.
>>>>>>
>>>>>>
>>>>>> > On Oct 28, 2014, at 9:55 AM, Nikhil <nikhilgarg.gju at gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Hello,
>>>>>> > Yeah, I read data from a file say at each node and each time step,
>>>>>> but when i try to use Marshal approach i get gibberish but when i use
>>>>>> simple iter i get correct values. i have been trying the approach used in
>>>>>> example in the previous post and that example makes sense but it doesnt
>>>>>> make sense when i use it in my case. I am right now assigning it to a
>>>>>> variable, i am now thinking of exploring the possibility of saving data to
>>>>>> a dot net array maybe using System.Array and saving data to it but not sure
>>>>>> if that even make sense.
>>>>>> >
>>>>>> > Sent from my iPhone
>>>>>>
>>>>>> _________________________________________________
>>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________
>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards
>>>>
>>>> Nikhil
>>>>
>>>> -------------------------------------------------------------------
>>>> Big whirls have little whirls,
>>>> Which feed on their velocity,
>>>> And little whirls have lesser whirls,
>>>> And so on to viscosity
>>>> (Richardson, 1922)
>>>>
>>>> _________________________________________________
>>>> Python.NET mailing list - PythonDotNet at python.org
>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>
>>>
>>>
>>
>
> _________________________________________________
> Python.NET mailing list - PythonDotNet at python.org
> https://mail.python.org/mailman/listinfo/pythondotnet
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141105/9ddbbdd9/attachment-0001.html>

From nmcwilliam at quantifisolutions.com  Mon Nov 10 19:05:24 2014
From: nmcwilliam at quantifisolutions.com (Noel McWilliam)
Date: Mon, 10 Nov 2014 13:05:24 -0500
Subject: [Python.NET] debugging
Message-ID: <6a7514a8d84446a7a625fee3e806895d@QExchange.quantifisolutions.local>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141110/31d973a1/attachment.html>

From tony at pyxll.com  Tue Nov 11 09:29:20 2014
From: tony at pyxll.com (Tony Roberts)
Date: Tue, 11 Nov 2014 08:29:20 +0000
Subject: [Python.NET] debugging
In-Reply-To: <6a7514a8d84446a7a625fee3e806895d@QExchange.quantifisolutions.local>
References: <6a7514a8d84446a7a625fee3e806895d@QExchange.quantifisolutions.local>
Message-ID: <CA+XbcsZZPjpJ3yeiaP8nzBtH+ohrwcmQ6Wh3Nr2A5kGk=9HBCA@mail.gmail.com>

Hi Noel,

any python ide should work fine. If you're using windows you can just use
your regular python and import the clr module without using npython
(npython is useful for linux where some distros have python built a certain
way that means the clr module can't be imported). You could take a look at
Enthought's Canopy, PyCharm or Eclipse (with the PyDev plugin) - all should
work fine.

If you want to debug the .NET code you're probably best off using Visual
Studio.

Hope that helps,
Tony

On Mon, Nov 10, 2014 at 6:05 PM, Noel McWilliam <
nmcwilliam at quantifisolutions.com> wrote:

> Hi Tony,
>
>
>
> I?m new to Python and to Python for .NET. Can you recommend an IDE that
> can be configured to reference nPython?
>
>
>
> Thanks in advance
>
> Noel
>
>
>
>
> <http://gfidisc.SID=0105000000000005150000007d870efc375d6c3727be357eca050000>
>
> Disclaimer: This email contains proprietary information some or all of
> which may be legally privileged. It is for the intended recipient only. If
> an addressing or transmission error has misdirected this email, please
> notify the author by replying to this email. If you are not the intended
> recipient you must not use, disclose, distribute, copy, print or rely on
> this email.
> Quantifi Limited is a company registered in England and Wales with the
> number 5818192. Our registered office is Farringdon Place, 20, Farringdon
> Road, London EC1M 3AP
>
>
> <http://gfidisc.SID=0105000000000005150000007d870efc375d6c3727be357eca050000>
>
> _________________________________________________
> Python.NET mailing list - PythonDotNet at python.org
> https://mail.python.org/mailman/listinfo/pythondotnet
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141111/2856e3b9/attachment.html>

From aklein at bluemountaincapital.com  Fri Nov 14 20:16:45 2014
From: aklein at bluemountaincapital.com (Adam Klein)
Date: Fri, 14 Nov 2014 19:16:45 +0000
Subject: [Python.NET] pythonnet appdomain(s)
Message-ID: <60CDCA83B420BB48BF95B8AF66CE7FFF45A36D89@NYMBX02.bcna.corp>

Hello,

Is there a design reason that pythonnet loads C# assemblies into its own AppDomain rather than establishing a separate AppDomain? It would be useful for instance when developing in C# in parallel with writing python code (say, in the ipython REPL) to be able to reload C# assemblies dynamically.

Also, we've found it useful in Initialize() to enable shadow copying, to prevent file locking, ie

        internal static void Initialize() {
...

            AppDomain domain = AppDomain.CurrentDomain;

            // prevent file locking
            domain.SetShadowCopyFiles();
            domain.SetShadowCopyPath(null);

Although these are deprecated methods, and really it should be establishing a new AppDomain as above.

Regards,
  Adam

_________________________________
Adam D Klein
BlueMountain Capital Management LLC
280 Park Ave, 5th Floor East
New York, NY 10017
O:(212)905-2136
C:(917)331-8871

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141114/5eed006c/attachment.html>

From denis.akhiyarov at gmail.com  Thu Nov 20 01:25:44 2014
From: denis.akhiyarov at gmail.com (Denis Akhiyarov)
Date: Wed, 19 Nov 2014 18:25:44 -0600
Subject: [Python.NET] debugging
In-Reply-To: <6a7514a8d84446a7a625fee3e806895d@QExchange.quantifisolutions.local>
References: <6a7514a8d84446a7a625fee3e806895d@QExchange.quantifisolutions.local>
Message-ID: <CALxxJLSL=h0PnuMpk1o_8AeGTz-FMV6MENguZS_1BPsJ9Gt2xg@mail.gmail.com>

I would highly recommend PTVS:

1. supports mixed-mode debugging between Python, C#, C/C++, Fortran, etc.
2. integrates ipython nicely into workflow
3. intellisense is quite powerful

The only issue I found (reported as bug) is that remote debugging does not
work with pythonnet.

please note that my scenario is mainly embedding Python in C#.


On Mon, Nov 10, 2014 at 12:05 PM, Noel McWilliam <
nmcwilliam at quantifisolutions.com> wrote:

> Hi Tony,
>
>
>
> I?m new to Python and to Python for .NET. Can you recommend an IDE that
> can be configured to reference nPython?
>
>
>
> Thanks in advance
>
> Noel
>
>
>
>
> <http://gfidisc.SID=0105000000000005150000007d870efc375d6c3727be357eca050000>
>
> Disclaimer: This email contains proprietary information some or all of
> which may be legally privileged. It is for the intended recipient only. If
> an addressing or transmission error has misdirected this email, please
> notify the author by replying to this email. If you are not the intended
> recipient you must not use, disclose, distribute, copy, print or rely on
> this email.
> Quantifi Limited is a company registered in England and Wales with the
> number 5818192. Our registered office is Farringdon Place, 20, Farringdon
> Road, London EC1M 3AP
>
>
> <http://gfidisc.SID=0105000000000005150000007d870efc375d6c3727be357eca050000>
>
> _________________________________________________
> Python.NET mailing list - PythonDotNet at python.org
> https://mail.python.org/mailman/listinfo/pythondotnet
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141119/483a13ed/attachment-0001.html>

From denis.akhiyarov at gmail.com  Thu Nov 20 01:29:35 2014
From: denis.akhiyarov at gmail.com (Denis Akhiyarov)
Date: Wed, 19 Nov 2014 18:29:35 -0600
Subject: [Python.NET] Efficient copy of .NET Array to ctypes or numpy
	array
In-Reply-To: <CALOUSfmP9AKHdJMwj5dpcknd5PD+T1FmsWMj0SOSs54X-EDVDA@mail.gmail.com>
References: <CAL3fuhnJb60pz5+H5+mJ8sORVuAZHU1B+J_YqK3b0dEyALLgfA@mail.gmail.com>
 <8BE6014E-0EB7-43B7-AF98-EB35C292E0D1@fie.us>
 <AC2FE74D-5440-4756-9FBB-E920835B1495@gmail.com>
 <B5C9A727-F367-4061-8EFE-EBEE11AF4385@fie.us>
 <CALOUSfnpQb7PUm0MrM1t1WuFiGXDxw4guz4tjz9JcyYxHUE3aw@mail.gmail.com>
 <CAL3fuhmT9seWajW=mS_=uGBDc6o3y5mL6Wzxw2xiXs7oQ_ExRQ@mail.gmail.com>
 <CALxxJLTgveWW6UNxtoHKd54DfM26JLzmPC5hphH5jfBi2Y=yUg@mail.gmail.com>
 <CALxxJLTm6-0EegAfRk8hSxzcV_fcpnv4ENHSpHPdvVGHMeNxAQ@mail.gmail.com>
 <CALxxJLS5VCzKoaGSpyf+30ZYfAvOfBxiNzK2c3yADx3_Pwxteg@mail.gmail.com>
 <CALOUSfmP9AKHdJMwj5dpcknd5PD+T1FmsWMj0SOSs54X-EDVDA@mail.gmail.com>
Message-ID: <CALxxJLS-NVef-UG7zuxL6Dnfz-9+kSNSG9kQiRFGNenMVhbMEA@mail.gmail.com>

Jeff,

Thank you for quick reply. Can you give any example of how would buffer()
help converting  numpy/python arrays to managed?


For now I developed as decorator (on my personal time) that handles all I/O
conversion on python side (assumes one input and one output arbitrary
array):

def decornet(func,T=System.Object):
    def inner(*args,**kwargs):
        res = np.array(func(listit(args[0]),*args[1:],**kwargs))
        tnum = res.dtype.type
        if tnum is np.int32:
            tnet = System.Int32
        elif tnum is np.int64:
            tnet = System.Int64
        elif tnum is np.float:
            tnet = System.Single
        elif tnum is np.double:
            tnet = System.Double
        elif tnum is np.bool:
            tnet = System.Boolean
        else:
            tnet = T
        netarr = Array.CreateInstance(tnet,*res.shape)
        it = np.nditer(res,flags=['multi_index'])
        while not it.finished:
            ix = it.multi_index
            if len(ix)==1:
                netarr[ix[0]] = res[ix[0]]
            else:
                netarr[ix] = res[ix]
            it.iternext()
        return netarr
    return inner

Thanks,

Denis


On Wed, Nov 5, 2014 at 4:32 PM, Jeffrey Bush <jeff at coderforlife.com> wrote:

> To copy from a list or tuple you need to use the Python buffer() function.
> You can use that on numpy arrays as well. In buffer form, the len() is the
> byte length so then you don't need to know the data type or size. However,
> some additional work would be required if you wanted to make sure the C#
> array was of the proper type and length.
>
> Jeff
>
> On Wed, Nov 5, 2014 at 11:28 AM, Denis Akhiyarov <
> denis.akhiyarov at gmail.com> wrote:
>
>> Finally decided to generate the managed object on Python side and return
>> it to C# with no conversion necessary in C#. This way I can even wrap
>> regular function with Python with @decorator to handle the conversion.  I
>> suppose the dynamic version of pythonnet may have auto conversion for
>> Python 3, but I have not tried. I'm on Python 2.7.
>>
>> On Wed, Nov 5, 2014 at 9:31 AM, Denis Akhiyarov <
>> denis.akhiyarov at gmail.com> wrote:
>>
>>> And more important question - is it possible to generalize the copying
>>> of python array object to managed C# array object without knowing the data
>>> type/size/length?
>>>
>>> On Wed, Nov 5, 2014 at 8:58 AM, Denis Akhiyarov <
>>> denis.akhiyarov at gmail.com> wrote:
>>>
>>>> How to copy unmanaged array (python list/tuple or numpy array) into
>>>> managed C# array? I guess using Marshal.Copy, but can anyone point to
>>>> example?
>>>>
>>>> Thanks,
>>>> Denis
>>>>
>>>> On Thu, Oct 30, 2014 at 12:19 PM, Nikhil Garg <nikhilgarg.gju at gmail.com
>>>> > wrote:
>>>>
>>>>> Thanks Brad and Jeff for the detailed info. For now, fromiter is
>>>>> serving me well and has reduced my processing time considerably, so I am
>>>>> just going to stick with it.
>>>>>
>>>>>
>>>>> On 29 October 2014 11:04, Jeffrey Bush <jeff at coderforlife.com> wrote:
>>>>>
>>>>>> I finally have a chance to chime in, and Bradley is exactly right.
>>>>>> Marshall.Copy copies the raw data, and apparently your file library does
>>>>>> not store that data in a nice, contiguous, manner. While it is highly
>>>>>> likely that copying all the data to an array in C# will be faster than the
>>>>>> fromiter in Python, I am unsure if copying all the data to an array in C#
>>>>>> then copying all the data again to a numpy array will be faster than
>>>>>> fromiter (cause you have to copy it twice). The exception is if the file
>>>>>> library has a function like ToArray that is optimized to copy the data to a
>>>>>> linear chunk of data. So, what type is "Data"?
>>>>>>
>>>>>> Another factor is how long the chunk of data you are copying is. You
>>>>>> say the last axis is only 400 elements long. Check out my code and you will
>>>>>> see that at 400 elements long, fromiter is actually the fastest (at least
>>>>>> when I tried). An example run:
>>>>>>
>>>>>> Copy using for loop in 0.000884 sec
>>>>>> Copy using fromiter in 0.000144 sec # fastest
>>>>>> Copy using fromstring in 0.001460 sec # fairly slow, 10.3x slower
>>>>>> than fromiter
>>>>>> Copy using Marshal.Copy in 0.001680 sec # slowest, 11.7x slower than
>>>>>> fromiter
>>>>>>
>>>>>> I start to do better with Marshal.Copy then fromiter around 5000
>>>>>> elements copied. This is because the overhead of the mass copies is high
>>>>>> but adding each element doesn't take much time. fromstring has a lower
>>>>>> overhead but slightly longer per-element time (fromstring is better than
>>>>>> Marshal.Copy until ~200,000 elements).
>>>>>>
>>>>>> So you might be doing as good as you can possibly do. If I knew more
>>>>>> about your file format library I might be able to provide more insight.
>>>>>>
>>>>>> Jeff
>>>>>>
>>>>>> On Tue, Oct 28, 2014 at 2:45 PM, Bradley Friedman <brad at fie.us>
>>>>>> wrote:
>>>>>>
>>>>>>> Well it makes sense to me that doing it via an iterator, and element
>>>>>>> at a time, would be slow.  There?s a lot of call overhead associated with
>>>>>>> each iteration step.  Whether it?s done in .net, or in python, or a call
>>>>>>> from one to the other, it will be slow.  It?s still a call where you?d be
>>>>>>> better off copying whole buffers.
>>>>>>>
>>>>>>> Ideally you?d pull the data into as simple and raw a data structure
>>>>>>> as you can on the dotnet side, in a buffered manner.  Then you?d execute a
>>>>>>> movement of the data across, a reasonably sized chunk of buffer at a time.
>>>>>>> This will reduce call overhead and also allow read-ahead caching to do its
>>>>>>> thing on the file-access side of things.
>>>>>>>
>>>>>>> Your suggestion of loading into a .net array and then moving that
>>>>>>> array over, makes sense.  But I think it comes down to what you can do with
>>>>>>> the third party file-format library. If its not going to provide you with
>>>>>>> the data as some kind of buffer with a cohesive and known format in memory,
>>>>>>> you?re not really going to be able to move it over without iterating over
>>>>>>> it and reformatting it at some point.
>>>>>>>
>>>>>>> Specifically, I?d point to Jeffery?s original caveat:
>>>>>>>
>>>>>>> "but does involve a number of assumptions (for example that the data
>>>>>>> in the two arrays are laid out in the same way)."
>>>>>>>
>>>>>>> The question is:  is there a way to get the data off of disk and in
>>>>>>> memory from dotnet library, where its layout in memory is known, and
>>>>>>> something you want exactly as it is, but in python?  If so, you should be
>>>>>>> able to use the methods from the afore linked thread.  If not, you?re
>>>>>>> probably stuck iterating somewhere to reformat it, no matter what.  Which
>>>>>>> is probably why you got garbage back.  I?m guessing the object returned
>>>>>>> from the dotnet file-format-library isn?t laid out right, as suggested in
>>>>>>> the afore referenced caveat.
>>>>>>>
>>>>>>>
>>>>>>> > On Oct 28, 2014, at 9:55 AM, Nikhil <nikhilgarg.gju at gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Hello,
>>>>>>> > Yeah, I read data from a file say at each node and each time step,
>>>>>>> but when i try to use Marshal approach i get gibberish but when i use
>>>>>>> simple iter i get correct values. i have been trying the approach used in
>>>>>>> example in the previous post and that example makes sense but it doesnt
>>>>>>> make sense when i use it in my case. I am right now assigning it to a
>>>>>>> variable, i am now thinking of exploring the possibility of saving data to
>>>>>>> a dot net array maybe using System.Array and saving data to it but not sure
>>>>>>> if that even make sense.
>>>>>>> >
>>>>>>> > Sent from my iPhone
>>>>>>>
>>>>>>> _________________________________________________
>>>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _________________________________________________
>>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards
>>>>>
>>>>> Nikhil
>>>>>
>>>>> -------------------------------------------------------------------
>>>>> Big whirls have little whirls,
>>>>> Which feed on their velocity,
>>>>> And little whirls have lesser whirls,
>>>>> And so on to viscosity
>>>>> (Richardson, 1922)
>>>>>
>>>>> _________________________________________________
>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>
>>>>
>>>>
>>>
>>
>> _________________________________________________
>> Python.NET mailing list - PythonDotNet at python.org
>> https://mail.python.org/mailman/listinfo/pythondotnet
>>
>
>
> _________________________________________________
> Python.NET mailing list - PythonDotNet at python.org
> https://mail.python.org/mailman/listinfo/pythondotnet
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141119/ca4f0b35/attachment-0001.html>

From jeff at coderforlife.com  Fri Nov 21 05:08:01 2014
From: jeff at coderforlife.com (Jeffrey Bush)
Date: Thu, 20 Nov 2014 20:08:01 -0800
Subject: [Python.NET] Efficient copy of .NET Array to ctypes or numpy
	array
In-Reply-To: <CALxxJLS-NVef-UG7zuxL6Dnfz-9+kSNSG9kQiRFGNenMVhbMEA@mail.gmail.com>
References: <CAL3fuhnJb60pz5+H5+mJ8sORVuAZHU1B+J_YqK3b0dEyALLgfA@mail.gmail.com>
 <8BE6014E-0EB7-43B7-AF98-EB35C292E0D1@fie.us>
 <AC2FE74D-5440-4756-9FBB-E920835B1495@gmail.com>
 <B5C9A727-F367-4061-8EFE-EBEE11AF4385@fie.us>
 <CALOUSfnpQb7PUm0MrM1t1WuFiGXDxw4guz4tjz9JcyYxHUE3aw@mail.gmail.com>
 <CAL3fuhmT9seWajW=mS_=uGBDc6o3y5mL6Wzxw2xiXs7oQ_ExRQ@mail.gmail.com>
 <CALxxJLTgveWW6UNxtoHKd54DfM26JLzmPC5hphH5jfBi2Y=yUg@mail.gmail.com>
 <CALxxJLTm6-0EegAfRk8hSxzcV_fcpnv4ENHSpHPdvVGHMeNxAQ@mail.gmail.com>
 <CALxxJLS5VCzKoaGSpyf+30ZYfAvOfBxiNzK2c3yADx3_Pwxteg@mail.gmail.com>
 <CALOUSfmP9AKHdJMwj5dpcknd5PD+T1FmsWMj0SOSs54X-EDVDA@mail.gmail.com>
 <CALxxJLS-NVef-UG7zuxL6Dnfz-9+kSNSG9kQiRFGNenMVhbMEA@mail.gmail.com>
Message-ID: <CALOUSf=gOYxN4WMEnG_ZB8qXq6UDMUXPZCe=1Vnuo-DaB8tbLQ@mail.gmail.com>

If you know that they are always numpy arrays, then it doesn't really help.

However, if they could be numpy arrays, lists, strings, ... then using
buffer() on them gets you the memory buffer of the object, basically an
array of bytes you could copy. Basically it helps generalize the problem to
more types of objects.

Jeff

On Wed, Nov 19, 2014 at 4:29 PM, Denis Akhiyarov <denis.akhiyarov at gmail.com>
wrote:

> Jeff,
>
> Thank you for quick reply. Can you give any example of how would buffer()
> help converting  numpy/python arrays to managed?
>
>
> For now I developed as decorator (on my personal time) that handles all
> I/O conversion on python side (assumes one input and one output arbitrary
> array):
>
> def decornet(func,T=System.Object):
>     def inner(*args,**kwargs):
>         res = np.array(func(listit(args[0]),*args[1:],**kwargs))
>         tnum = res.dtype.type
>         if tnum is np.int32:
>             tnet = System.Int32
>         elif tnum is np.int64:
>             tnet = System.Int64
>         elif tnum is np.float:
>             tnet = System.Single
>         elif tnum is np.double:
>             tnet = System.Double
>         elif tnum is np.bool:
>             tnet = System.Boolean
>         else:
>             tnet = T
>         netarr = Array.CreateInstance(tnet,*res.shape)
>         it = np.nditer(res,flags=['multi_index'])
>         while not it.finished:
>             ix = it.multi_index
>             if len(ix)==1:
>                 netarr[ix[0]] = res[ix[0]]
>             else:
>                 netarr[ix] = res[ix]
>             it.iternext()
>         return netarr
>     return inner
>
> Thanks,
>
> Denis
>
>
> On Wed, Nov 5, 2014 at 4:32 PM, Jeffrey Bush <jeff at coderforlife.com>
> wrote:
>
>> To copy from a list or tuple you need to use the Python buffer()
>> function. You can use that on numpy arrays as well. In buffer form, the
>> len() is the byte length so then you don't need to know the data type or
>> size. However, some additional work would be required if you wanted to make
>> sure the C# array was of the proper type and length.
>>
>> Jeff
>>
>> On Wed, Nov 5, 2014 at 11:28 AM, Denis Akhiyarov <
>> denis.akhiyarov at gmail.com> wrote:
>>
>>> Finally decided to generate the managed object on Python side and return
>>> it to C# with no conversion necessary in C#. This way I can even wrap
>>> regular function with Python with @decorator to handle the conversion.  I
>>> suppose the dynamic version of pythonnet may have auto conversion for
>>> Python 3, but I have not tried. I'm on Python 2.7.
>>>
>>> On Wed, Nov 5, 2014 at 9:31 AM, Denis Akhiyarov <
>>> denis.akhiyarov at gmail.com> wrote:
>>>
>>>> And more important question - is it possible to generalize the copying
>>>> of python array object to managed C# array object without knowing the data
>>>> type/size/length?
>>>>
>>>> On Wed, Nov 5, 2014 at 8:58 AM, Denis Akhiyarov <
>>>> denis.akhiyarov at gmail.com> wrote:
>>>>
>>>>> How to copy unmanaged array (python list/tuple or numpy array) into
>>>>> managed C# array? I guess using Marshal.Copy, but can anyone point to
>>>>> example?
>>>>>
>>>>> Thanks,
>>>>> Denis
>>>>>
>>>>> On Thu, Oct 30, 2014 at 12:19 PM, Nikhil Garg <
>>>>> nikhilgarg.gju at gmail.com> wrote:
>>>>>
>>>>>> Thanks Brad and Jeff for the detailed info. For now, fromiter is
>>>>>> serving me well and has reduced my processing time considerably, so I am
>>>>>> just going to stick with it.
>>>>>>
>>>>>>
>>>>>> On 29 October 2014 11:04, Jeffrey Bush <jeff at coderforlife.com> wrote:
>>>>>>
>>>>>>> I finally have a chance to chime in, and Bradley is exactly right.
>>>>>>> Marshall.Copy copies the raw data, and apparently your file library does
>>>>>>> not store that data in a nice, contiguous, manner. While it is highly
>>>>>>> likely that copying all the data to an array in C# will be faster than the
>>>>>>> fromiter in Python, I am unsure if copying all the data to an array in C#
>>>>>>> then copying all the data again to a numpy array will be faster than
>>>>>>> fromiter (cause you have to copy it twice). The exception is if the file
>>>>>>> library has a function like ToArray that is optimized to copy the data to a
>>>>>>> linear chunk of data. So, what type is "Data"?
>>>>>>>
>>>>>>> Another factor is how long the chunk of data you are copying is. You
>>>>>>> say the last axis is only 400 elements long. Check out my code and you will
>>>>>>> see that at 400 elements long, fromiter is actually the fastest (at least
>>>>>>> when I tried). An example run:
>>>>>>>
>>>>>>> Copy using for loop in 0.000884 sec
>>>>>>> Copy using fromiter in 0.000144 sec # fastest
>>>>>>> Copy using fromstring in 0.001460 sec # fairly slow, 10.3x slower
>>>>>>> than fromiter
>>>>>>> Copy using Marshal.Copy in 0.001680 sec # slowest, 11.7x slower than
>>>>>>> fromiter
>>>>>>>
>>>>>>> I start to do better with Marshal.Copy then fromiter around 5000
>>>>>>> elements copied. This is because the overhead of the mass copies is high
>>>>>>> but adding each element doesn't take much time. fromstring has a lower
>>>>>>> overhead but slightly longer per-element time (fromstring is better than
>>>>>>> Marshal.Copy until ~200,000 elements).
>>>>>>>
>>>>>>> So you might be doing as good as you can possibly do. If I knew more
>>>>>>> about your file format library I might be able to provide more insight.
>>>>>>>
>>>>>>> Jeff
>>>>>>>
>>>>>>> On Tue, Oct 28, 2014 at 2:45 PM, Bradley Friedman <brad at fie.us>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Well it makes sense to me that doing it via an iterator, and
>>>>>>>> element at a time, would be slow.  There?s a lot of call overhead
>>>>>>>> associated with each iteration step.  Whether it?s done in .net, or in
>>>>>>>> python, or a call from one to the other, it will be slow.  It?s still a
>>>>>>>> call where you?d be better off copying whole buffers.
>>>>>>>>
>>>>>>>> Ideally you?d pull the data into as simple and raw a data structure
>>>>>>>> as you can on the dotnet side, in a buffered manner.  Then you?d execute a
>>>>>>>> movement of the data across, a reasonably sized chunk of buffer at a time.
>>>>>>>> This will reduce call overhead and also allow read-ahead caching to do its
>>>>>>>> thing on the file-access side of things.
>>>>>>>>
>>>>>>>> Your suggestion of loading into a .net array and then moving that
>>>>>>>> array over, makes sense.  But I think it comes down to what you can do with
>>>>>>>> the third party file-format library. If its not going to provide you with
>>>>>>>> the data as some kind of buffer with a cohesive and known format in memory,
>>>>>>>> you?re not really going to be able to move it over without iterating over
>>>>>>>> it and reformatting it at some point.
>>>>>>>>
>>>>>>>> Specifically, I?d point to Jeffery?s original caveat:
>>>>>>>>
>>>>>>>> "but does involve a number of assumptions (for example that the
>>>>>>>> data in the two arrays are laid out in the same way)."
>>>>>>>>
>>>>>>>> The question is:  is there a way to get the data off of disk and in
>>>>>>>> memory from dotnet library, where its layout in memory is known, and
>>>>>>>> something you want exactly as it is, but in python?  If so, you should be
>>>>>>>> able to use the methods from the afore linked thread.  If not, you?re
>>>>>>>> probably stuck iterating somewhere to reformat it, no matter what.  Which
>>>>>>>> is probably why you got garbage back.  I?m guessing the object returned
>>>>>>>> from the dotnet file-format-library isn?t laid out right, as suggested in
>>>>>>>> the afore referenced caveat.
>>>>>>>>
>>>>>>>>
>>>>>>>> > On Oct 28, 2014, at 9:55 AM, Nikhil <nikhilgarg.gju at gmail.com>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > Hello,
>>>>>>>> > Yeah, I read data from a file say at each node and each time
>>>>>>>> step, but when i try to use Marshal approach i get gibberish but when i use
>>>>>>>> simple iter i get correct values. i have been trying the approach used in
>>>>>>>> example in the previous post and that example makes sense but it doesnt
>>>>>>>> make sense when i use it in my case. I am right now assigning it to a
>>>>>>>> variable, i am now thinking of exploring the possibility of saving data to
>>>>>>>> a dot net array maybe using System.Array and saving data to it but not sure
>>>>>>>> if that even make sense.
>>>>>>>> >
>>>>>>>> > Sent from my iPhone
>>>>>>>>
>>>>>>>> _________________________________________________
>>>>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________
>>>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards
>>>>>>
>>>>>> Nikhil
>>>>>>
>>>>>> -------------------------------------------------------------------
>>>>>> Big whirls have little whirls,
>>>>>> Which feed on their velocity,
>>>>>> And little whirls have lesser whirls,
>>>>>> And so on to viscosity
>>>>>> (Richardson, 1922)
>>>>>>
>>>>>> _________________________________________________
>>>>>> Python.NET mailing list - PythonDotNet at python.org
>>>>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> _________________________________________________
>>> Python.NET mailing list - PythonDotNet at python.org
>>> https://mail.python.org/mailman/listinfo/pythondotnet
>>>
>>
>>
>> _________________________________________________
>> Python.NET mailing list - PythonDotNet at python.org
>> https://mail.python.org/mailman/listinfo/pythondotnet
>>
>
>
> _________________________________________________
> Python.NET mailing list - PythonDotNet at python.org
> https://mail.python.org/mailman/listinfo/pythondotnet
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythondotnet/attachments/20141120/711ddf2f/attachment-0001.html>