From fghoussen at gmail.com Fri Feb 7 16:06:27 2020 From: fghoussen at gmail.com (HOUSSEN Franck) Date: Fri, 7 Feb 2020 22:06:27 +0100 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ Message-ID: With boost python, how to pass a tuple of lists from python to C++ ? Code attached : line 12 crashes ?! Googling but no fix for now. Expected results should look like this: >> make; python dummyTest.py g++ -I/usr/include/python2.7 -o dummy.so -shared -fPIC dummy.cpp -lboost_python -lboost_numpy doStuffs nbInt 3 ptrInt ...... ptrInt[0] = 1 ptrInt[1] = 2 ptrInt[2] = 3 nbfloat 3 ptrFloat .... ptrFloat[0] = 1.2 ptrFloat[1] = 2.3 ptrFloat[2] = 3.4 -- Bonne journ?e, Franck HOUSSEN -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummyTest.py Type: text/x-python Size: 191 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 100 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummy.cpp Type: text/x-c++src Size: 1041 bytes Desc: not available URL: From stefan at seefeld.name Fri Feb 7 16:43:10 2020 From: stefan at seefeld.name (stefan) Date: Fri, 7 Feb 2020 16:43:10 -0500 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ In-Reply-To: References: Message-ID: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> On 2020-02-07 4:06 p.m., HOUSSEN Franck wrote: > With boost python, how to pass a tuple of lists from python to C++ ? > Code attached : line 12 crashes ?! Googling but no fix for now. You are referring to this line: ? np::ndarray lsInt = boost::python::extract(t[0]); Note that this line is actually doing two things: 1) it creates a boost::python::extract object 2) it calls conversion operator on it to convert to np::ndarray Note that the conversion may fail (for example if the object doesn't contain an object of that type). In case of doubt, you may want to call extract::check() to see whether it's actually valid. See https://www.boost.org/doc/libs/1_72_0/libs/python/doc/html/reference/to_from_python_type_conversion.html#to_from_python_type_conversion.boost_python_extract_hpp.class_template_extract for details. Stefan -- ...ich hab' noch einen Koffer in Berlin... -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.png Type: image/png Size: 1478 bytes Desc: not available URL: From fghoussen at gmail.com Sat Feb 8 12:07:27 2020 From: fghoussen at gmail.com (HOUSSEN Franck) Date: Sat, 8 Feb 2020 18:07:27 +0100 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ In-Reply-To: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> References: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> Message-ID: I tried to play with extract::check() without much success : line 13 crashes !?... This is a very simple exemple : would like to get it to work. Could somebody help ? Still googling / searching for a solution Le ven. 7 f?vr. 2020 ? 22:43, stefan a ?crit : > > On 2020-02-07 4:06 p.m., HOUSSEN Franck wrote: > > With boost python, how to pass a tuple of lists from python to C++ ? Code > attached : line 12 crashes ?! Googling but no fix for now. > > You are referring to this line: > > np::ndarray lsInt = boost::python::extract(t[0]); > > Note that this line is actually doing two things: > > 1) it creates a boost::python::extract object > > 2) it calls conversion operator on it to convert to np::ndarray > > Note that the conversion may fail (for example if the object doesn't > contain an object of that type). In case of doubt, you may want to call > extract::check() to see whether it's actually valid. > > See > https://www.boost.org/doc/libs/1_72_0/libs/python/doc/html/reference/to_from_python_type_conversion.html#to_from_python_type_conversion.boost_python_extract_hpp.class_template_extract > for details. > > > > [image: Stefan] > > -- > > ...ich hab' noch einen Koffer in Berlin... > > > _______________________________________________ > Cplusplus-sig mailing list > Cplusplus-sig at python.org > https://mail.python.org/mailman/listinfo/cplusplus-sig > -- Bonne journ?e, Franck HOUSSEN -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.png Type: image/png Size: 1478 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummy.cpp Type: text/x-c++src Size: 1537 bytes Desc: not available URL: From stefan at seefeld.name Sat Feb 8 23:04:08 2020 From: stefan at seefeld.name (Stefan Seefeld) Date: Sat, 8 Feb 2020 23:04:08 -0500 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ In-Reply-To: References: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> Message-ID: On 2020-02-08 12:07 p.m., HOUSSEN Franck wrote: > I tried to play with extract::check() without much success : line 13 > crashes !?... > This is a very simple exemple : would like to get it to work. Could > somebody help ? Still googling / searching for a solution You need to call np::initialize() in your module, i.e. insert ? `np::initialize();` at the top of your `dummy` module definition. Stefan -- ...ich hab' noch einen Koffer in Berlin... -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: .signature.png Type: image/png Size: 2754 bytes Desc: not available URL: From fghoussen at gmail.com Sun Feb 9 04:58:06 2020 From: fghoussen at gmail.com (HOUSSEN Franck) Date: Sun, 9 Feb 2020 10:58:06 +0100 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ In-Reply-To: References: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> Message-ID: Oh men ! Totally missed / forget that... Thanks Sefan : the dummy simple example works now :D Unfortunately, this dummy example was extracted from a much more complex one which seems to do things like dummy but still does not work... so I am still debugging the real case... Anyway, thanks for the help on the dummy case ! Hope this could help some other people Franck Le dim. 9 f?vr. 2020 ? 05:04, Stefan Seefeld a ?crit : > > On 2020-02-08 12:07 p.m., HOUSSEN Franck wrote: > > I tried to play with extract::check() without much success : line 13 > crashes !?... > This is a very simple exemple : would like to get it to work. Could > somebody help ? Still googling / searching for a solution > > You need to call np::initialize() in your module, i.e. insert > > `np::initialize();` > > at the top of your `dummy` module definition. > > [image: Stefan] > -- > > ...ich hab' noch einen Koffer in Berlin... > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: .signature.png Type: image/png Size: 2754 bytes Desc: not available URL: From fghoussen at gmail.com Tue Feb 11 07:24:11 2020 From: fghoussen at gmail.com (HOUSSEN Franck) Date: Tue, 11 Feb 2020 13:24:11 +0100 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ In-Reply-To: References: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> Message-ID: Finally able to reproduce the "real" problem with a "dummy" example : seems that, at python side, when you use "np.append" C++ get screwed data ?!... (note that with or without "np.append" all is OK at python side). Can somebody help here ? Known problem ? Possible workaround or fix ? Should I open a bug for that ? (if yes where) Franck >> make; python dummyTest.py ; python dummyTest2.py g++ -I/usr/include/python2.7 -o dummy.so -shared -fPIC dummy.cpp -lboost_python -lboost_numpy [1 2 3] [1.2 2.3 3.4] ptrInt[0] = 1, 0x5643649d5640 ptrInt[1] = 2, 0x5643649d5644 ptrInt[2] = 3, 0x5643649d5648 ptrFloat[0] = 1.2, 0x5643649d5660 ptrFloat[1] = 2.3, 0x5643649d5664 ptrFloat[2] = 3.4, 0x5643649d5668 [1 2 3] [1.2 2.3 3.4] ptrInt[0] = 1, 0x559e0710e550 ptrInt[1] = 0, 0x559e0710e554 ptrInt[2] = 2, 0x559e0710e558 ptrFloat[0] = 4.17233e-08, 0x559e0710e570 ptrFloat[1] = 1.9, 0x559e0710e574 ptrFloat[2] = 2.72008e+23, 0x559e0710e578 Le dim. 9 f?vr. 2020 ? 10:58, HOUSSEN Franck a ?crit : > Oh men ! Totally missed / forget that... Thanks Sefan : the dummy simple > example works now :D > Unfortunately, this dummy example was extracted from a much more complex > one which seems to do things like dummy but still does not work... so I am > still debugging the real case... > Anyway, thanks for the help on the dummy case ! Hope this could help some > other people > > Franck > > Le dim. 9 f?vr. 2020 ? 05:04, Stefan Seefeld a > ?crit : > >> >> On 2020-02-08 12:07 p.m., HOUSSEN Franck wrote: >> >> I tried to play with extract::check() without much success : line 13 >> crashes !?... >> This is a very simple exemple : would like to get it to work. Could >> somebody help ? Still googling / searching for a solution >> >> You need to call np::initialize() in your module, i.e. insert >> >> `np::initialize();` >> >> at the top of your `dummy` module definition. >> >> [image: Stefan] >> -- >> >> ...ich hab' noch einen Koffer in Berlin... >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: .signature.png Type: image/png Size: 2754 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 100 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummy.cpp Type: text/x-c++src Size: 1050 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummyTest2.py Type: text/x-python Size: 386 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummyTest.py Type: text/x-python Size: 220 bytes Desc: not available URL: From jvansanten at gmail.com Tue Feb 11 08:56:58 2020 From: jvansanten at gmail.com (Jakob van Santen) Date: Tue, 11 Feb 2020 14:56:58 +0100 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ In-Reply-To: References: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> Message-ID: > On Feb 11, 2020, at 13:24, HOUSSEN Franck wrote: > > Finally able to reproduce the "real" problem with a "dummy" example : seems that, at python side, when you use "np.append" C++ get screwed data ?!... (note that with or without "np.append" all is OK at python side). > Can somebody help here ? Known problem ? Possible workaround or fix ? Should I open a bug for that ? (if yes where) This is a side-effect of how np.append() broadcasts its arguments to find a common type. You've asked it to append a Python float (which is secretly a double) to an array of np.float32. The only safe way to do that is to cast both to numpy.float64, which np.append() happily does for you. Before you reinterpret_cast the data pointer of an ndarray to a given type, you need to check whether the dtype is compatible (and also, strictly speaking, whether the flags are CARRAY or CARRAY_RO). There's a whole lot more background information in the Numpy C API docs: https://docs.scipy.org/doc/numpy/reference/c-api.html Cheers, Jakob From fghoussen at gmail.com Tue Feb 11 14:05:08 2020 From: fghoussen at gmail.com (HOUSSEN Franck) Date: Tue, 11 Feb 2020 20:05:08 +0100 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ In-Reply-To: References: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> Message-ID: OK, I understand this is a type related problem. I found a workaround (dummyTest3.py - init numpy arrays with list seems to work). But, I unfortunately do not get how to change the code to get dummyTest2.py to work ?!... Should I read reinterpret_cast'ed data sizeof(double) by sizeof(double) : seems cryptic (?!), what's the natural way to do that ? Also a question raises : if the size of the data type you want to pass from python to C++ is smaller than sizeof(double), python will always allocate MORE memory than needed ? Why is that ? In case the size is large, the extra memory allocated may be large and trigger problems like slowdown (swapping) or out-of-memory, no ?... Why is this like so ? Can somebody help to fix the dummy code : this would be the best explanation !... :D Franck Le mar. 11 f?vr. 2020 ? 14:57, Jakob van Santen a ?crit : > > > On Feb 11, 2020, at 13:24, HOUSSEN Franck wrote: > > > > Finally able to reproduce the "real" problem with a "dummy" example : > seems that, at python side, when you use "np.append" C++ get screwed data > ?!... (note that with or without "np.append" all is OK at python side). > > Can somebody help here ? Known problem ? Possible workaround or fix ? > Should I open a bug for that ? (if yes where) > > This is a side-effect of how np.append() broadcasts its arguments to find > a common type. You've asked it to append a Python float (which is secretly > a double) to an array of np.float32. The only safe way to do that is to > cast both to numpy.float64, which np.append() happily does for you. Before > you reinterpret_cast the data pointer of an ndarray to a given type, you > need to check whether the dtype is compatible (and also, strictly speaking, > whether the flags are CARRAY or CARRAY_RO). > > There's a whole lot more background information in the Numpy C API docs: > https://docs.scipy.org/doc/numpy/reference/c-api.html > > Cheers, > Jakob > _______________________________________________ > Cplusplus-sig mailing list > Cplusplus-sig at python.org > https://mail.python.org/mailman/listinfo/cplusplus-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummy.cpp Type: text/x-c++src Size: 1460 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummyTest3.py Type: text/x-python Size: 324 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummyTest1.py Type: text/x-python Size: 300 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 100 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummyTest2.py Type: text/x-python Size: 561 bytes Desc: not available URL: From stefan at seefeld.name Tue Feb 11 14:12:20 2020 From: stefan at seefeld.name (stefan) Date: Tue, 11 Feb 2020 14:12:20 -0500 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ In-Reply-To: References: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> Message-ID: <93edac45-5978-f044-ad8c-842c92d38011@seefeld.name> On 2020-02-11 2:05 p.m., HOUSSEN Franck wrote: > OK, I understand this is a type related problem. I found a workaround > (dummyTest3.py - init numpy arrays with list seems to work). > > But, I unfortunately do not get how to change the code to get > dummyTest2.py to work ?!... Should I read reinterpret_cast'ed data > sizeof(double) by sizeof(double) : seems cryptic (?!), what's the > natural way to do that ? Can't you change your Python code so instead of ? np.append(lsFloat, 1.2) you would call ? np.append(lsFloat, np.float32(1.2)) to make sure the casting happens before the broadcasting. (Note: I haven't actually tried that; it just seems the natural fix to the issue ;-) ) Stefan -- ...ich hab' noch einen Koffer in Berlin... -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.png Type: image/png Size: 1478 bytes Desc: not available URL: From fghoussen at gmail.com Tue Feb 11 16:13:02 2020 From: fghoussen at gmail.com (HOUSSEN Franck) Date: Tue, 11 Feb 2020 22:13:02 +0100 Subject: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++ In-Reply-To: <93edac45-5978-f044-ad8c-842c92d38011@seefeld.name> References: <3e37c9be-163e-4468-75fb-00e7aa7583a0@seefeld.name> <93edac45-5978-f044-ad8c-842c92d38011@seefeld.name> Message-ID: Works great ! Thanks ! Code attached. Franck >> make; python dummyTest1.py; python dummyTest2.py; python dummyTest3.py g++ -I/usr/include/python2.7 -o dummy.so -shared -fPIC dummy.cpp -lboost_python -lboost_numpy [1 2 3] [1.2 2.3 3.4] [4.5 5.6 6.7] ptrInt[0] = 1, 0x555a42aede80 ptrInt[1] = 2, 0x555a42aede84 ptrInt[2] = 3, 0x555a42aede88 ptrFloat[0] = 1.2, 0x555a429c4f60 ptrFloat[1] = 2.3, 0x555a429c4f64 ptrFloat[2] = 3.4, 0x555a429c4f68 ptrDouble[0] = 4.5, 0x555a429c4f80 ptrDouble[1] = 5.6, 0x555a429c4f88 ptrDouble[2] = 6.7, 0x555a429c4f90 [1 2 3] [1.2 2.3 3.4] [4.5 5.6 6.7] ptrInt[0] = 1, 0x55dbaa536470 ptrInt[1] = 2, 0x55dbaa536474 ptrInt[2] = 3, 0x55dbaa536478 ptrFloat[0] = 1.2, 0x55dbaa4b9bb0 ptrFloat[1] = 2.3, 0x55dbaa4b9bb4 ptrFloat[2] = 3.4, 0x55dbaa4b9bb8 ptrDouble[0] = 4.5, 0x55dbaa4b9bd0 ptrDouble[1] = 5.6, 0x55dbaa4b9bd8 ptrDouble[2] = 6.7, 0x55dbaa4b9be0 [1 2 3] [1.2 2.3 3.4] [4.5 5.6 6.7] ptrInt[0] = 1, 0x563725f958f0 ptrInt[1] = 2, 0x563725f958f4 ptrInt[2] = 3, 0x563725f958f8 ptrFloat[0] = 1.2, 0x563725e87a10 ptrFloat[1] = 2.3, 0x563725e87a14 ptrFloat[2] = 3.4, 0x563725e87a18 ptrDouble[0] = 4.5, 0x563725fc0de0 ptrDouble[1] = 5.6, 0x563725fc0de8 ptrDouble[2] = 6.7, 0x563725fc0df0 Le mar. 11 f?vr. 2020 ? 20:12, stefan a ?crit : > > On 2020-02-11 2:05 p.m., HOUSSEN Franck wrote: > > OK, I understand this is a type related problem. I found a workaround > (dummyTest3.py - init numpy arrays with list seems to work). > > But, I unfortunately do not get how to change the code to get > dummyTest2.py to work ?!... Should I read reinterpret_cast'ed data > sizeof(double) by sizeof(double) : seems cryptic (?!), what's the natural > way to do that ? > > Can't you change your Python code so instead of > > np.append(lsFloat, 1.2) > > you would call > > np.append(lsFloat, np.float32(1.2)) > > to make sure the casting happens before the broadcasting. > > (Note: I haven't actually tried that; it just seems the natural fix to the > issue ;-) ) > [image: Stefan] > > -- > > ...ich hab' noch einen Koffer in Berlin... > > > _______________________________________________ > Cplusplus-sig mailing list > Cplusplus-sig at python.org > https://mail.python.org/mailman/listinfo/cplusplus-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.png Type: image/png Size: 1478 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummyTest3.py Type: text/x-python Size: 324 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummyTest1.py Type: text/x-python Size: 300 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 100 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummyTest2.py Type: text/x-python Size: 663 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dummy.cpp Type: text/x-c++src Size: 1460 bytes Desc: not available URL: