From vsakharuk at tower-research.com  Tue Oct  4 09:05:16 2016
From: vsakharuk at tower-research.com (Vladimir Sakharuk)
Date: Tue, 4 Oct 2016 13:05:16 +0000
Subject: [C++-sig] Boost Python. access pandas columns in the same order as
 in python
Message-ID: <1475586316331.55938@tower-research.com>

Hello All,


Trying to figure out how to c++ access pandas dataframe columns in the same order as they exist in the python.

in python:

    import pandas


    df = pandas.DataFrame.from_items([('CCC', [5]), ('BBB', [1]), ('AAA', [4])])


    df.columns


    # outputs Index([u'CCC', u'BBB', u'AAA'], dtype='object')

in C++ when passing df to foo.

    foo(const boost::python::object in_obj)


    {


      boost::python::stl_input_iterator<boost::python::object> it (in_obj), end;


      int index=0;


      for(; it != end; it++)     {    auto extracted = boost::python::extract<char const *> (*it);   char const * colname = extracted;     std::cout<<"index="<<index<<", colname="<<colname<<std::endl;   index++;     }     }     //outputs regardless of original order of column names.     //index=0, colname=AAA     //index=1, colname=BBB     //index=2, colname=CCC

As you can see pythons' order 'CCC','BBB','AAA' is not same as c++ 'AAA', 'BBB', 'CCC'. Looks like those order depend on object hasing.


How could I get those values in the original order of pandas dataframe from C++?


Thank you.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20161004/f535992f/attachment.html>

From stefan at seefeld.name  Wed Oct  5 12:31:49 2016
From: stefan at seefeld.name (Stefan Seefeld)
Date: Wed, 5 Oct 2016 12:31:49 -0400
Subject: [C++-sig] Boost Python. access pandas columns in the same order
 as in python
In-Reply-To: <1475586316331.55938@tower-research.com>
References: <1475586316331.55938@tower-research.com>
Message-ID: <5cfc5fca-bb74-3830-de18-58378dcb6804@seefeld.name>

On 04.10.2016 09:05, Vladimir Sakharuk wrote:
>
> Hello All,
>
>
> Trying to figure out how to c++ access pandas dataframe columns in the
> same order as they exist in the python.
>
> in python:
>

[...]

> ||
> |    //outputs regardless of original order of column names.   
> //index=0, colname=AAA    //index=1, colname=BBB    //index=2,
> colname=CCC|
>
> As you can see pythons' order 'CCC','BBB','AAA' is not same as c++
> 'AAA', 'BBB', 'CCC'. Looks like those order depend on object hasing.
>
>
> How could I get those values in the original order of pandas dataframe
> from C++?
>

I can't reproduce that; I see the same order (['CCC', 'BBB', 'AAA'])
both in Python and in C++ with your code above.

FWIW,
        Stefan


-- 

      ...ich hab' noch einen Koffer in Berlin...


From jbosch at astro.princeton.edu  Wed Oct  5 12:36:42 2016
From: jbosch at astro.princeton.edu (Jim Bosch)
Date: Wed, 5 Oct 2016 12:36:42 -0400
Subject: [C++-sig] Boost Python. access pandas columns in the same order
 as in python
In-Reply-To: <5cfc5fca-bb74-3830-de18-58378dcb6804@seefeld.name>
References: <1475586316331.55938@tower-research.com>
 <5cfc5fca-bb74-3830-de18-58378dcb6804@seefeld.name>
Message-ID: <CAN+0q26_4Vy-E4UBN_Eds2TX9W0JjcamojchOkxDHtaEgx7Fhw@mail.gmail.com>

Note that in Python 3.3+, dict order is nondeterministic, not just
arbitrary:

http://stackoverflow.com/questions/14956313/dictionary-ordering-non-deterministic-in-python3

So if these are invoking dict ordering under the hood, anything is possible.

Jim




On Wed, Oct 5, 2016 at 12:31 PM, Stefan Seefeld <stefan at seefeld.name> wrote:

> On 04.10.2016 09:05, Vladimir Sakharuk wrote:
> >
> > Hello All,
> >
> >
> > Trying to figure out how to c++ access pandas dataframe columns in the
> > same order as they exist in the python.
> >
> > in python:
> >
>
> [...]
>
> > ||
> > |    //outputs regardless of original order of column names.
> > //index=0, colname=AAA    //index=1, colname=BBB    //index=2,
> > colname=CCC|
> >
> > As you can see pythons' order 'CCC','BBB','AAA' is not same as c++
> > 'AAA', 'BBB', 'CCC'. Looks like those order depend on object hasing.
> >
> >
> > How could I get those values in the original order of pandas dataframe
> > from C++?
> >
>
> I can't reproduce that; I see the same order (['CCC', 'BBB', 'AAA'])
> both in Python and in C++ with your code above.
>
> FWIW,
>         Stefan
>
>
> --
>
>       ...ich hab' noch einen Koffer in Berlin...
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> https://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20161005/adb5149a/attachment.html>

From vsakharuk at tower-research.com  Wed Oct  5 13:06:16 2016
From: vsakharuk at tower-research.com (Vladimir Sakharuk)
Date: Wed, 5 Oct 2016 17:06:16 +0000
Subject: [C++-sig] Boost Python. access pandas columns in the same order
 as in python
In-Reply-To: <CAN+0q26_4Vy-E4UBN_Eds2TX9W0JjcamojchOkxDHtaEgx7Fhw@mail.gmail.com>
References: <1475586316331.55938@tower-research.com>
 <5cfc5fca-bb74-3830-de18-58378dcb6804@seefeld.name>,
 <CAN+0q26_4Vy-E4UBN_Eds2TX9W0JjcamojchOkxDHtaEgx7Fhw@mail.gmail.com>
Message-ID: <1475687176432.64218@tower-research.com>

It looks like I had other "bug" aka replacing with OrderedDict solved the problem.

________________________________
From: Cplusplus-sig <cplusplus-sig-bounces+vsakharuk=tower-research.com at python.org> on behalf of Jim Bosch <jbosch at astro.princeton.edu>
Sent: Wednesday, October 5, 2016 12:36 PM
To: Development of Python/C++ integration
Subject: Re: [C++-sig] Boost Python. access pandas columns in the same order as in python

Note that in Python 3.3+, dict order is nondeterministic, not just arbitrary:

http://stackoverflow.com/questions/14956313/dictionary-ordering-non-deterministic-in-python3

So if these are invoking dict ordering under the hood, anything is possible.

Jim




On Wed, Oct 5, 2016 at 12:31 PM, Stefan Seefeld <stefan at seefeld.name<mailto:stefan at seefeld.name>> wrote:
On 04.10.2016 09:05, Vladimir Sakharuk wrote:
>
> Hello All,
>
>
> Trying to figure out how to c++ access pandas dataframe columns in the
> same order as they exist in the python.
>
> in python:
>

[...]

> ||
> |    //outputs regardless of original order of column names.
> //index=0, colname=AAA    //index=1, colname=BBB    //index=2,
> colname=CCC|
>
> As you can see pythons' order 'CCC','BBB','AAA' is not same as c++
> 'AAA', 'BBB', 'CCC'. Looks like those order depend on object hasing.
>
>
> How could I get those values in the original order of pandas dataframe
> from C++?
>

I can't reproduce that; I see the same order (['CCC', 'BBB', 'AAA'])
both in Python and in C++ with your code above.

FWIW,
        Stefan


--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig at python.org<mailto:Cplusplus-sig at python.org>
https://mail.python.org/mailman/listinfo/cplusplus-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20161005/6d802d42/attachment.html>

From stefan at seefeld.name  Sun Oct  9 00:21:11 2016
From: stefan at seefeld.name (Stefan Seefeld)
Date: Sun, 9 Oct 2016 00:21:11 -0400
Subject: [C++-sig] Boost.Python NumPy extension
Message-ID: <340b13f2-2996-5f8d-8d12-d68578cb3385@seefeld.name>

Hello,

I'm happy to announce that I just merged the NumPy C++ wrappers from
https://github.com/ndarray/Boost.NumPy into Boost.Python's development
branch. (Online documentation for it can be browsed here:
http://boostorg.github.io/python/develop/doc/html/numpy)

I'd very much like to be able to merge this into the master branch for
the next Boost release (version 1.63).
At present I'm using SCons to build Boost.Python (stand-alone, against a
separately installed Boost), which is also the infrastructure I use for
CI testing (https://travis-ci.org/boostorg/python).

I would like to ask for help with integrating the new NumPy extension
into the Boost.Build-based build system, such that this will also be
picked up by the regular Boost infrastructure. Please refer to
https://github.com/boostorg/python/issues/89 if you'd like to help with
this.

Many thanks !

        Stefan

-- 

      ...ich hab' noch einen Koffer in Berlin...


From amohr at pixar.com  Sun Oct  9 00:40:33 2016
From: amohr at pixar.com (Alex Mohr)
Date: Sat, 8 Oct 2016 21:40:33 -0700
Subject: [C++-sig] Boost.Python NumPy extension
In-Reply-To: <340b13f2-2996-5f8d-8d12-d68578cb3385@seefeld.name>
References: <340b13f2-2996-5f8d-8d12-d68578cb3385@seefeld.name>
Message-ID: <ec29616e-d674-2c59-4c22-67ee0cafcaed@pixar.com>

Cool!  Sorry, I don't have the expertise to help with the boost.build 
aspect, but I did have some questions and comments regarding the feature.

Does this add a dependency of boost.python on numpy?  We'd love to be 
able to continue use boost.python without a numpy dependency.  We did a 
nontrivial amount of work to remove this dependency in USD (openusd.org, 
https://github.com/PixarAnimationStudios/USD), a C++ library that 
provides python bindings via boost.python, so adding it back would be a 
problem for us.

Instead of depending on numpy, we made our array structures implement 
the python buffer protocol.  This lets clients that wish to use numpy do 
so easily, since numpy supports the buffer protocol, but it does not 
burden those that don't with the numpy dependency.  We did this manually 
of course but supporting the buffer protocol would be a cool feature for 
boost.python.

Alex

On 10/8/16 9:21 PM, Stefan Seefeld wrote:
> Hello,
>
> I'm happy to announce that I just merged the NumPy C++ wrappers from
> https://github.com/ndarray/Boost.NumPy into Boost.Python's development
> branch. (Online documentation for it can be browsed here:
> http://boostorg.github.io/python/develop/doc/html/numpy)
>
> I'd very much like to be able to merge this into the master branch for
> the next Boost release (version 1.63).
> At present I'm using SCons to build Boost.Python (stand-alone, against a
> separately installed Boost), which is also the infrastructure I use for
> CI testing (https://travis-ci.org/boostorg/python).
>
> I would like to ask for help with integrating the new NumPy extension
> into the Boost.Build-based build system, such that this will also be
> picked up by the regular Boost infrastructure. Please refer to
> https://github.com/boostorg/python/issues/89 if you'd like to help with
> this.
>
> Many thanks !
>
>         Stefan
>

From stefan at seefeld.name  Sun Oct  9 08:40:59 2016
From: stefan at seefeld.name (Stefan Seefeld)
Date: Sun, 9 Oct 2016 08:40:59 -0400
Subject: [C++-sig] Boost.Python NumPy extension
In-Reply-To: <ec29616e-d674-2c59-4c22-67ee0cafcaed@pixar.com>
References: <340b13f2-2996-5f8d-8d12-d68578cb3385@seefeld.name>
 <ec29616e-d674-2c59-4c22-67ee0cafcaed@pixar.com>
Message-ID: <749af4dd-821c-9e80-fa5c-a2aa961f2aa1@seefeld.name>

Hi Alex,

On 09.10.2016 00:40, Alex Mohr wrote:
> Cool!  Sorry, I don't have the expertise to help with the boost.build
> aspect, but I did have some questions and comments regarding the feature.
>
> Does this add a dependency of boost.python on numpy?  We'd love to be
> able to continue use boost.python without a numpy dependency.  We did
> a nontrivial amount of work to remove this dependency in USD
> (openusd.org, https://github.com/PixarAnimationStudios/USD), a C++
> library that provides python bindings via boost.python, so adding it
> back would be a problem for us.
>
> Instead of depending on numpy, we made our array structures implement
> the python buffer protocol.  This lets clients that wish to use numpy
> do so easily, since numpy supports the buffer protocol, but it does
> not burden those that don't with the numpy dependency.  We did this
> manually of course but supporting the buffer protocol would be a cool
> feature for boost.python.

Short answer: The dependency on NumPy is only added to the extension
(compiled into a separate library); Boost.Python without that extension
remains exactly the same.

Somewhat longer answer: it all depends how Boost.Python is installed.
There is a configure flag to decide whether to build with or without
NumPy support. Building with NumPy support will yield two libraries:
libboost_python.so and libboost_numpy.so, so I expect downstream
packagers to wrap both into separate packages. (The new headers are
likewise kept separate for the same reason.)

            Stefan


-- 

      ...ich hab' noch einen Koffer in Berlin...


From jbosch at astro.princeton.edu  Sun Oct  9 11:17:23 2016
From: jbosch at astro.princeton.edu (Jim Bosch)
Date: Sun, 9 Oct 2016 11:17:23 -0400
Subject: [C++-sig] Boost.Python NumPy extension
In-Reply-To: <ec29616e-d674-2c59-4c22-67ee0cafcaed@pixar.com>
References: <340b13f2-2996-5f8d-8d12-d68578cb3385@seefeld.name>
 <ec29616e-d674-2c59-4c22-67ee0cafcaed@pixar.com>
Message-ID: <CAN+0q24_3QiQ+Z6J0OW4H5BDbXTQ4+vO=Y_SvzFvVZeTXESHCQ@mail.gmail.com>

On Sun, Oct 9, 2016 at 12:40 AM, Alex Mohr <amohr at pixar.com> wrote:

>
> Instead of depending on numpy, we made our array structures implement the
> python buffer protocol.  This lets clients that wish to use numpy do so
> easily, since numpy supports the buffer protocol, but it does not burden
> those that don't with the numpy dependency.  We did this manually of course
> but supporting the buffer protocol would be a cool feature for boost.python.


I think this is a very good idea - it wasn't available as an option back
when these NumPy bindings were written, but it's certainly the way I'd
recommend writing them now if we were starting over.  That would also
eliminate the need to call an initialization function in any Python modules
that use the NumPy bindings - failure to do that is by far the most common
problem people have in using the NumPy bindings.

That said, I think it'd be a fair amount of work to rewrite most of the
library's functionality with no NumPy dependency, and I think some parts of
it - like constructing numpy.dtype objects that correspond to C++ types -
would probably be impossible to do without access to the NumPy C API, so
we'd have to remove some functionality as well.


Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20161009/f6fa0bb5/attachment.html>