From jdemeyer at cage.ugent.be  Mon Aug  1 02:38:14 2016
From: jdemeyer at cage.ugent.be (Jeroen Demeyer)
Date: Mon, 1 Aug 2016 08:38:14 +0200
Subject: [Cython] Cython docs are for version 0.15pre
Message-ID: <579EEE56.9080003@cage.ugent.be>

The page
https://cython.readthedocs.io/en/latest/
has the documentation for Cython version 0.15pre.

How did that happen???

From robertwb at gmail.com  Mon Aug  1 13:40:49 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 1 Aug 2016 10:40:49 -0700
Subject: [Cython] Cython docs are for version 0.15pre
In-Reply-To: <579EEE56.9080003@cage.ugent.be>
References: <579EEE56.9080003@cage.ugent.be>
Message-ID: <CADiQ+QBzjtvJXJ4qy1_=MMqKZ7qHnPhzpuwPtPFNXtFGHCXnFA@mail.gmail.com>

Looks like it was trying to pull from the old docs repo. Thanks for
pointing this out. Fixed.

On Sun, Jul 31, 2016 at 11:38 PM, Jeroen Demeyer <jdemeyer at cage.ugent.be> wrote:
> The page
> https://cython.readthedocs.io/en/latest/
> has the documentation for Cython version 0.15pre.
>
> How did that happen???
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From lists at onerussian.com  Wed Aug 10 09:39:01 2016
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Wed, 10 Aug 2016 09:39:01 -0400
Subject: [Cython] long_double_to_float_int failure on s390x (BE) (0.24.1)
Message-ID: <20160810133901.GF8170@onerussian.com>

Dear Cython ppl,

Didn't spot any obvious commit/log since 0.24.1 so decided just to report:
full build log:
https://buildd.debian.org/status/fetch.php?pkg=cython&arch=s390x&ver=0.24.1-1&stamp=1470118175

1 test has failed on s390x (only).  Quick resolution would help to
assure cython's healthy status in debian coming to a freeze point.

======================================================================
FAIL: long_double_to_float_int (line 195) (int_float_builtins_as_casts_T400.__test__)
Doctest: int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
  File "/?PKGBUILDDIR?/build/work-dir/run/c/int_float_builtins_as_casts_T400/int_float_builtins_as_casts_T400.so", line unknown line number, in long_double_to_float_int (line 195)

----------------------------------------------------------------------
File "/?PKGBUILDDIR?/build/work-dir/run/c/int_float_builtins_as_casts_T400/int_float_builtins_as_casts_T400.so", line ?, in int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
Failed example:
    long_double_to_float_int(4.1)
Expected:
    4.0
Got:
    0.0
----------------------------------------------------------------------
File "/?PKGBUILDDIR?/build/work-dir/run/c/int_float_builtins_as_casts_T400/int_float_builtins_as_casts_T400.so", line ?, in int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
Failed example:
    long_double_to_float_int(-4.1)
Expected:
    -4.0
Got:
    0.0
----------------------------------------------------------------------
File "/?PKGBUILDDIR?/build/work-dir/run/c/int_float_builtins_as_casts_T400/int_float_builtins_as_casts_T400.so", line ?, in int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
Failed example:
    long_double_to_float_int(4)
Expected:
    4.0
Got:
    0.0


----------------------------------------------------------------------
Ran 9957 tests in 4237.557s

FAILED (failures=1, skipped=2)


-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From robertwb at math.washington.edu  Wed Aug 10 23:55:10 2016
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 10 Aug 2016 20:55:10 -0700
Subject: [Cython] long_double_to_float_int failure on s390x (BE) (0.24.1)
In-Reply-To: <20160810133901.GF8170@onerussian.com>
References: <20160810133901.GF8170@onerussian.com>
Message-ID: <CADiQ+QCghq6W+p947yv_xEvhdukMFpVLTnXOY3R2b7kXwF-hpA@mail.gmail.com>

Is this a change from 0.24? Do you have any s390x hardware that we
could test on?

On Wed, Aug 10, 2016 at 6:39 AM, Yaroslav Halchenko
<lists at onerussian.com> wrote:
> Dear Cython ppl,
>
> Didn't spot any obvious commit/log since 0.24.1 so decided just to report:
> full build log:
> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=s390x&ver=0.24.1-1&stamp=1470118175
>
> 1 test has failed on s390x (only).  Quick resolution would help to
> assure cython's healthy status in debian coming to a freeze point.
>
> ======================================================================
> FAIL: long_double_to_float_int (line 195) (int_float_builtins_as_casts_T400.__test__)
> Doctest: int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/doctest.py", line 2226, in runTest
>     raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
>   File "/?PKGBUILDDIR?/build/work-dir/run/c/int_float_builtins_as_casts_T400/int_float_builtins_as_casts_T400.so", line unknown line number, in long_double_to_float_int (line 195)
>
> ----------------------------------------------------------------------
> File "/?PKGBUILDDIR?/build/work-dir/run/c/int_float_builtins_as_casts_T400/int_float_builtins_as_casts_T400.so", line ?, in int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
> Failed example:
>     long_double_to_float_int(4.1)
> Expected:
>     4.0
> Got:
>     0.0
> ----------------------------------------------------------------------
> File "/?PKGBUILDDIR?/build/work-dir/run/c/int_float_builtins_as_casts_T400/int_float_builtins_as_casts_T400.so", line ?, in int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
> Failed example:
>     long_double_to_float_int(-4.1)
> Expected:
>     -4.0
> Got:
>     0.0
> ----------------------------------------------------------------------
> File "/?PKGBUILDDIR?/build/work-dir/run/c/int_float_builtins_as_casts_T400/int_float_builtins_as_casts_T400.so", line ?, in int_float_builtins_as_casts_T400.__test__.long_double_to_float_int (line 195)
> Failed example:
>     long_double_to_float_int(4)
> Expected:
>     4.0
> Got:
>     0.0
>
>
> ----------------------------------------------------------------------
> Ran 9957 tests in 4237.557s
>
> FAILED (failures=1, skipped=2)
>
>
> --
> Yaroslav O. Halchenko
> Center for Open Neuroscience     http://centerforopenneuroscience.org
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From lists at onerussian.com  Thu Aug 11 00:52:28 2016
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Thu, 11 Aug 2016 00:52:28 -0400
Subject: [Cython] long_double_to_float_int failure on s390x (BE) (0.24.1)
In-Reply-To: <CADiQ+QCghq6W+p947yv_xEvhdukMFpVLTnXOY3R2b7kXwF-hpA@mail.gmail.com>
References: <20160810133901.GF8170@onerussian.com>
 <CADiQ+QCghq6W+p947yv_xEvhdukMFpVLTnXOY3R2b7kXwF-hpA@mail.gmail.com>
Message-ID: <20160811045228.GG8170@onerussian.com>


On Wed, 10 Aug 2016, Robert Bradshaw wrote:

> Is this a change from 0.24? Do you have any s390x hardware that we
> could test on?

if anything it is a "change" from 0.23.4+git4-g7eed8d8-2 ... but the
thing is that since my email Jakub Wilk reported that he hasn't
observed this failure while building on s390x only few days ago, so it
might have been some toolchain fluke.  I have asked for a rebuild, so
let's wait for it and if reproduces, I will see what we could do about
access or at least more info to troubleshoot.

Cheers
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From lists at onerussian.com  Thu Aug 11 12:02:01 2016
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Thu, 11 Aug 2016 12:02:01 -0400
Subject: [Cython] long_double_to_float_int failure on s390x (BE) (0.24.1)
In-Reply-To: <20160811045228.GG8170@onerussian.com>
References: <20160810133901.GF8170@onerussian.com>
 <CADiQ+QCghq6W+p947yv_xEvhdukMFpVLTnXOY3R2b7kXwF-hpA@mail.gmail.com>
 <20160811045228.GG8170@onerussian.com>
Message-ID: <20160811160201.GH8170@onerussian.com>


On Thu, 11 Aug 2016, Yaroslav Halchenko wrote:


> On Wed, 10 Aug 2016, Robert Bradshaw wrote:

> > Is this a change from 0.24? Do you have any s390x hardware that we
> > could test on?

> if anything it is a "change" from 0.23.4+git4-g7eed8d8-2 ... but the
> thing is that since my email Jakub Wilk reported that he hasn't
> observed this failure while building on s390x only few days ago, so it
> might have been some toolchain fluke.  I have asked for a rebuild, so
> let's wait for it and if reproduces, I will see what we could do about
> access or at least more info to troubleshoot.

confirming that issue was gone in rebuild:
https://buildd.debian.org/status/logs.php?pkg=cython&ver=0.24.1-1&arch=s390x

Thank you and Cheers
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From nevion at gmail.com  Sat Aug 13 21:28:35 2016
From: nevion at gmail.com (Jason Newton)
Date: Sat, 13 Aug 2016 21:28:35 -0400
Subject: [Cython] Any way to embed C/C++ in a single source compilation of a
 pyx file?
Message-ID: <CAGou9MiR=VYH96zoGhwXH3xRmkXJPJ6c6uqHN8tX7Dp2zZGirg@mail.gmail.com>

I just added inline_module's and submitted a PR today here:
https://github.com/cython/cython/pull/555

My intent was something like OpenCL's JIT/AOC compilation (PyCUDA/PyOpenCL
also work this way) which is simple and high performance, and I saw cython
had this capability and just needed a little bit of a wrapper to expose it
better.

But after trying to put it into use for my own problems, where I would
regularly want to use Eigen or other integration problems I realized I'd
still have to write and save [many] code fragments to other files which
cythonize-ing would read in through includes, when you need to work with
something that doesn't have a cython cimport module.  This is pretty ugly
and doesn't achieve the easier flow that I had wanted.

However if there was a way to embed C/C++ inside a pyx file (that is inline
to strings), there would be no such issue.

I submitted an issue here https://github.com/cython/cython/issues/556 but I
was hoping to get the ball rolling if the ML is where the development
conversations happen - it's not clear to me right now the best place for
the Cython project to request this from, so sorry to cross post.

Regards,
-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160813/9c777042/attachment.html>

From robertwb at gmail.com  Wed Aug 17 17:49:19 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 17 Aug 2016 14:49:19 -0700
Subject: [Cython] Migration of tickets from trac to github
In-Reply-To: <579B4E23.6040503@cage.ugent.be>
References: <CADiQ+QBM8maiieSpuW54Vpy5Nyt=iwDcXdFTsDvhTN_SJH1cYA@mail.gmail.com>
 <579B4E23.6040503@cage.ugent.be>
Message-ID: <CADiQ+QBarYk3SpwDK343rPbDroaafBiJQg5j70aMP8nH-vQE1g@mail.gmail.com>

On Fri, Jul 29, 2016 at 5:37 AM, Jeroen Demeyer <jdemeyer at cage.ugent.be> wrote:
> We should also about a convention about naming testsuite files.
>
> For example, there is
> tests/compile/distutils_libraries_T845.srctree
>
> Should this be changed to
> tests/compile/distutils_libraries_GH1234.srctree
>
> if this becomes GitHub issue #1234?

Sounds like a fine convention to me. I don't think we need to go
renaming all the Txxx ones though.

From robertwb at gmail.com  Thu Aug 18 01:49:50 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 17 Aug 2016 22:49:50 -0700
Subject: [Cython] Cython on GitHub
Message-ID: <CADiQ+QD0nxnzEB2FsoAtQ2_FHuwpgc+ZgOyY2VZVFMZwgzmeWA@mail.gmail.com>

All of of Cython serving infrastructure, save the jenkins buildbot,
has been moved to GitHub. (Documentation is served via
readthedocs.org, and tarballs served from pypi, built from GitHub
head). Each trac ticket has been migrated to a github issue. I have
updated/redirected what links I can.

As for backups, I created
https://github.com/cython/cython-issues/tree/master to store the
non-repository data, which will be periodically updated. Note that the
original trac data (minus session/user info) is there as well.

I will also be locally cloning all the repositories in the cython
organization, and would recommend at least one other person do the
same.

- Robert

From stefan_ml at behnel.de  Thu Aug 18 02:49:12 2016
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Aug 2016 08:49:12 +0200
Subject: [Cython] Fwd: Re: [Speed] New benchmark suite for Python
In-Reply-To: <CAKJDb-M3AyRiqxtd26JA50fS5r+rZKRt1hUU4eRHcKV26Ac44Q@mail.gmail.com>
References: <CAKJDb-M3AyRiqxtd26JA50fS5r+rZKRt1hUU4eRHcKV26Ac44Q@mail.gmail.com>
Message-ID: <1db3e44d-52b2-25f4-deb7-21e2808a7785@behnel.de>

This could replace the benchmark runs that we had on Jenkins.

Stefan


-------- Forwarded Message --------
Subject: Re: [Speed] New benchmark suite for Python
Date: Wed, 17 Aug 2016 23:47:29 -0500
From: Zachary Ware <zachary.ware+pydev(AT)gmail.com>
To: speed(AT)python.org

On Wed, Aug 17, 2016 at 7:37 PM, Victor Stinner wrote:
> PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to
> take a look at my project and test it ;-) Tell me what is broken, what
> is missing, and I will try to help you to move your project to this
> new benchmark suite!

Also, if you're interested in having your interpreter benchmarked on
speed.python.org, contact me with clear instructions (preferably in
the form of a shell or Python script) on how to build, test, install,
and invoke your interpreter from a fresh Ubuntu 16.04 installation.
As an example, here's an untested version for CPython 3.x:

#!/bin/sh
# set up dependencies
sudo apt-get build-dep -y python3
sudo apt-get install -y --no-install-recommends mercurial
# get the code
hg clone https://hg.python.org/cpython
cd cpython
# build
./configure --prefix=/opt/python/default
make -j12
# test
make buildbottest TESTOPTS=-j12
# install
make install
# invoke
/opt/python/default/bin/python3


I don't know when I'll have a chance to work on it, but I'd like to
get as many projects as possible benchmarked on speed.python.org.

Victor: Thanks for getting the new repository set up and for all your
work on the new runner!  I'm looking forward to trying it out.

-- 
Zach

_______________________________________________
Speed mailing list
https://mail.python.org/mailman/listinfo/speed


From nevion at gmail.com  Thu Aug 18 15:05:53 2016
From: nevion at gmail.com (Jason Newton)
Date: Thu, 18 Aug 2016 15:05:53 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the code
 emitter
Message-ID: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>

Accidentally posted to an already-opened tab for the cython-users ML
yesterday, moving to here. Following up from a github opened issue here:

https://github.com/cython/cython/issues/1440

I was hoping we could give a way to drop straight into C/C++ inside of
Cython pyx files.

Why?

-It [helps] avoid needing to declare every class/function in cython, a
somewhat daunting/impossible task that I think everyone hates.  Have you
libraries like Eigen or others that use complex template based techniques?
How about those with tons of [member] functions to boot or getting C++
inheritance involved.
-It works around having the Cython compiler know about all of C++'s nuances
- as an advanced C++ developer these are painful and it is a 2nd class
citizen to Cython's simpler C-support - that's no good.  Just right now I
was bitten by yet another template argument bug and it's clear C++ template
arguments have been kind of dicy since support appeared.
-It would allow single source files - I think this is important for runtime
compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide

The idea is that Cython glue makes the playing field for extracting data
easy, but that once it's extracted to a cdef variable for instance, cython
doesn't need to know what happens.  Maybe in a way sort of like the GCC asm
extension.  Hopefully simpler variable passing though.

The alternative to not having this construct is a concrete wall.  I'll have
to segment up files for C++ for the rest of time until I get function forms
that Cython can handle. At that point I just say screw it and use boost
python.  Of course cython does the definitions, data extraction, and
compilation so much easier than boost.python.  It would be a shame to not
consider this plight C++ developers have been cornered into and you can't
say the C++ libraries are broken, it is the strategy Cython is taking that
cannot work.

I did some examination of how this could be implemented - my idea was to
base on the print/exec statements handling and simply emit their arguments
from the Nodes to the code generator.  Proposing inline_c or inline_ as the
statement.  Such a statement could be used to do macros, pull in includes,
and modify/declare c-variables.

-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160818/9012491a/attachment.html>

From robertwb at gmail.com  Fri Aug 19 05:36:59 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 19 Aug 2016 02:36:59 -0700
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
Message-ID: <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>

On Thu, Aug 18, 2016 at 12:05 PM, Jason Newton <nevion at gmail.com> wrote:
> Accidentally posted to an already-opened tab for the cython-users ML
> yesterday, moving to here. Following up from a github opened issue here:
>
> https://github.com/cython/cython/issues/1440
>
> I was hoping we could give a way to drop straight into C/C++ inside of
> Cython pyx files.
>
> Why?
>
> -It [helps] avoid needing to declare every class/function in cython, a
> somewhat daunting/impossible task that I think everyone hates.  Have you
> libraries like Eigen or others that use complex template based techniques?
> How about those with tons of [member] functions to boot or getting C++
> inheritance involved.

I agree that this is a pain, and better tooling should be developed to
(mostly?) eliminate this (e.g. see the recent thread at
https://groups.google.com/forum/#!topic/cython-users/c8ChI6jERzY ).

Of course having symbols magically appear (e.g. due some #Include, who
knows which) has its downsides too, which is why import * is often
discouraged in Python too.

> -It works around having the Cython compiler know about all of C++'s nuances
> - as an advanced C++ developer these are painful and it is a 2nd class
> citizen to Cython's simpler C-support - that's no good.  Just right now I
> was bitten by yet another template argument bug and it's clear C++ template
> arguments have been kind of dicy since support appeared.

Yes, C++ is an extraordinarily complicated language, and exposing all
of that would add significant amounts of complexity to Cython itself,
and perhaps more importantly increase the barrier of entry to reading
Cython code. One of the guiding principles is that we try to stay as
close to Python as possible (if you know Python, you can probably read
Cython, and with a minimal amount of C knowledge start writing it) and
much of C++ simply isn't Pythonic.

Though, as you discovered, there are some basic things like non-type
template arguments that we would like to have. If there are other
specific C++ constructs that are commonly used but impossible to
express in Cython it'd be useful to know.

> -It would allow single source files - I think this is important for runtime
> compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide

Not quite following what you're saying here.

> The idea is that Cython glue makes the playing field for extracting data
> easy, but that once it's extracted to a cdef variable for instance, cython
> doesn't need to know what happens.  Maybe in a way sort of like the GCC asm
> extension.  Hopefully simpler variable passing though.

Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any
possible conflicts. Specifying what/how to mangle could get as ugly as
GCC's asm variable passing. And embedded variable declarations, let
alone control flow statements (especially return, break, ...) could
get really messy. It obscures analysis Cython can do on the code, such
as whether variables are used or what values they may take. Little
code snippets are not always local either, e.g. do they often need to
refer to variables (or headers) referenced elsewhere. And they must
all be mutually compatible.

That's aside from the jarring nature of interleaving Python and C++
code. Would semicolons be required? How would parsers (including IDEs)
handle the C++ snippets? Or would they be placed in opaque
strings/comments (which I'd rather avoid)?

Feels like you want PyInline or weave.inline, but at compile time.

> The alternative to not having this construct is a concrete wall.  I'll have
> to segment up files for C++ for the rest of time until I get function forms
> that Cython can handle. At that point I just say screw it and use boost
> python.  Of course cython does the definitions, data extraction, and
> compilation so much easier than boost.python.  It would be a shame to not
> consider this plight C++ developers have been cornered into and you can't
> say the C++ libraries are broken, it is the strategy Cython is taking that
> cannot work.

Some C++ libraries are not very amenable to being used outside a C++
context. They don't expose an "FFI-friendly" interface and calling
them from other languages is more painful. That doesn't mean they're
broken, just C++ centric.

The strategy that Cython uses is that you use (as close to possible)
Python structures and syntax to write your bindings. This is great for
people who know and understand Python, but for those who would rather
write their code (including bindings) in C++ there's Boost.Python and
others (including writing C extensions by hand).

If your library is full of C++isms, another option is to create a C
wrapper, and expose that to Python (which is what people did before
there was any C++ support). If exposing these as Python functions,
with automatic conversion to/from Python types is the primary value
you're getting from Cython, simply declare these as cpdef functions in
your extern blocks and wrappers will automatically be created. If your
C wrappers are simple enough, you might even be able to auto-generate
the .pxd and .pyx files with some of the tools listed at
https://github.com/cython/cython/wiki/AutoPxd . Yes, you'd have your
C++ code in a C++ file and "Python" code in a .pyx file rather than
interleaving them, but the amount of code you write would be the same.

> I did some examination of how this could be implemented - my idea was to
> base on the print/exec statements handling and simply emit their arguments
> from the Nodes to the code generator.  Proposing inline_c or inline_ as the
> statement.  Such a statement could be used to do macros, pull in includes,
> and modify/declare c-variables.
>
> -Jason
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From nevion at gmail.com  Fri Aug 19 14:19:54 2016
From: nevion at gmail.com (Jason Newton)
Date: Fri, 19 Aug 2016 14:19:54 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
Message-ID: <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>

On Fri, Aug 19, 2016 at 5:36 AM, Robert Bradshaw <robertwb at gmail.com> wrote:

> On Thu, Aug 18, 2016 at 12:05 PM, Jason Newton <nevion at gmail.com> wrote:
> > Accidentally posted to an already-opened tab for the cython-users ML
> > yesterday, moving to here. Following up from a github opened issue here:
> >
> > https://github.com/cython/cython/issues/1440
> >
> > I was hoping we could give a way to drop straight into C/C++ inside of
> > Cython pyx files.
> >
> > Why?
> >
> > -It [helps] avoid needing to declare every class/function in cython, a
> > somewhat daunting/impossible task that I think everyone hates.  Have you
> > libraries like Eigen or others that use complex template based
> techniques?
> > How about those with tons of [member] functions to boot or getting C++
> > inheritance involved.
>
> I agree that this is a pain, and better tooling should be developed to
> (mostly?) eliminate this (e.g. see the recent thread at
> https://groups.google.com/forum/#!topic/cython-users/c8ChI6jERzY ).
>
> Of course having symbols magically appear (e.g. due some #Include, who
> knows which) has its downsides too, which is why import * is often
> discouraged in Python too.
>
> > -It works around having the Cython compiler know about all of C++'s
> nuances
> > - as an advanced C++ developer these are painful and it is a 2nd class
> > citizen to Cython's simpler C-support - that's no good.  Just right now I
> > was bitten by yet another template argument bug and it's clear C++
> template
> > arguments have been kind of dicy since support appeared.
>
> Yes, C++ is an extraordinarily complicated language, and exposing all
> of that would add significant amounts of complexity to Cython itself,
> and perhaps more importantly increase the barrier of entry to reading
> Cython code. One of the guiding principles is that we try to stay as
> close to Python as possible (if you know Python, you can probably read
> Cython, and with a minimal amount of C knowledge start writing it) and
> much of C++ simply isn't Pythonic.
>

Maybe it's off topic but I debate the guiding principle of Cython -  I was
not able to comprehend Cython before reading tutorials and this is not my
first time looking at it, I had a couple runins over the last 5 years with
it on projects such as h5py easily got lost on what was going on between
all the wrapper-wrappers (pure py code wrapping py-invokable code) and
re-declarations.  These projects complied with Cython's current philosophy
to the degradation of clarity, context, and overall idea of how code was
hooked up.  Perhaps Cython should take the lessons learned from it's
inception, time, and the results of the state of the c-python userbase to
guide us into a new philosophy.



>
> Though, as you discovered, there are some basic things like non-type
> template arguments that we would like to have. If there are other
> specific C++ constructs that are commonly used but impossible to
> express in Cython it'd be useful to know.
>

I haven't had the capability to use Cython sufficiently to learn more of
them because it currently can't solve my problems.  From prior SWIG et al
experiences, my outlook is that it is treacherous path to walk and unless
you brought in llvm/clang into the project for parsing/AST, I'd hold onto
that outlook.


>
> > -It would allow single source files - I think this is important for
> runtime
> > compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide
>
> Not quite following what you're saying here.
>

Maybe PyInline was a better example off the bat for you, but something a
little more flexible but also with less work is needed.  Compare with
PyOpenCL:
https://documen.tician.de/pyopencl/ - check out some examples.   There is a
c runtime api between the contexts hooking things up (this is the OpenCL
runtime part) - it's a pretty similar story to PyCuda (and by the same
author, execpt for that project has to jump out to nvcc and cache kernel
compilation like the inline function implementation does).  There's no
limit to the number of functions you can declare though and the OpenCL side
is kept simple - things are generally pretty typesafe/do what you would
expect on dispatch.  PyInline for comparison looks like it might lean on
the Python c-api for it's work more and maybe limited in the number of
functions per snippet it can declare.  I don't expect to be able to work
with Numpy ndarray data easily with it.


>
> > The idea is that Cython glue makes the playing field for extracting data
> > easy, but that once it's extracted to a cdef variable for instance,
> cython
> > doesn't need to know what happens.  Maybe in a way sort of like the GCC
> asm
> > extension.  Hopefully simpler variable passing though.
>
> Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any
> possible conflicts. Specifying what/how to mangle could get as ugly as
> GCC's asm variable passing. And embedded variable declarations, let
> alone control flow statements (especially return, break, ...) could
> get really messy. It obscures analysis Cython can do on the code, such
> as whether variables are used or what values they may take. Little
> code snippets are not always local either, e.g. do they often need to
> refer to variables (or headers) referenced elsewhere. And they must
> all be mutually compatible.
>

Like gcc's asm, let's let adults do what they want and let them worry about
the consequences of flow control/stray includes. I'm not even sure how most
of this would be an issue (switch/break/if) if you are properly nesting
pyxd output.  The only thing I think is an issue here is mangled names.  I
haven't yet figured out why (cdef) variable names must be mangled.  Can you
explain?  Maybe we add an option to allow it to be unmangled in their
declaration? C++ has extern "C" for example.


>
> That's aside from the jarring nature of interleaving Python and C++
> code. Would semicolons be required? How would parsers (including IDEs)
> handle the C++ snippets? Or would they be placed in opaque
> strings/comments (which I'd rather avoid)?
>

Opaque strings.  It's a good and time tested solution to the issue.  I'm
very happy with it in the contexts I use it in.


>
> Feels like you want PyInline or weave.inline, but at compile time.
>

You must realize that almost any other python driven way to compile c-code
in the spirit these projects do is deprecated/dead.  Cython has absorbed
all the reputation and users that didn't go to pure-c/boost.python -
pybind11 is the new kid on the block there so I'm not including it (I'm of
the opinion that SWIG users stayed unchanged).  Community
belief/QA/designers/google all think of Cython first.  Weave has
effectively closed up it's doors and I'm not even sure it had the power to
do what I wanted anyway because Cython provides a language that eases the
data-extraction/typecasting part of inlining C/C++.


>
> > The alternative to not having this construct is a concrete wall.  I'll
> have
> > to segment up files for C++ for the rest of time until I get function
> forms
> > that Cython can handle. At that point I just say screw it and use boost
> > python.  Of course cython does the definitions, data extraction, and
> > compilation so much easier than boost.python.  It would be a shame to not
> > consider this plight C++ developers have been cornered into and you can't
> > say the C++ libraries are broken, it is the strategy Cython is taking
> that
> > cannot work.
>
> Some C++ libraries are not very amenable to being used outside a C++
> context. They don't expose an "FFI-friendly" interface and calling
> them from other languages is more painful. That doesn't mean they're
> broken, just C++ centric.
>
> The strategy that Cython uses is that you use (as close to possible)
> Python structures and syntax to write your bindings. This is great for
> people who know and understand Python, but for those who would rather
> write their code (including bindings) in C++ there's Boost.Python and
> others (including writing C extensions by hand).
>

As a multi-year user and contributor to the project of Boost.Python, it's
not all it's cracked up to be - Cython with inline C/C++ is often a better
approach (and better for scientific code acceleration/glue).
Boost.Python's autoconversion magic is very unpredictable/broken - and this
is required.  I'm interested in Pybind for large projects but not as much
as in Cython for numeric processing.  Writing C extensions by hand is
usually a waste of time, which is why so many people tried to automate it.

Cython's audience *I believe* is not people not knowing C/C++ and only
familiar with python but is C/C++ users trying to wrap stuff with Cython.
Further, I don't see how this is detrimental - it's segmented.  For small
stuff the cognitive burden of finding and going to an external file is far
higher which is the way I've seen it end up so far.


> If your library is full of C++isms, another option is to create a C
> wrapper, and expose that to Python (which is what people did before
> there was any C++ support). If exposing these as Python functions,
> with automatic conversion to/from Python types is the primary value
> you're getting from Cython, simply declare these as cpdef functions in
> your extern blocks and wrappers will automatically be created. If your
> C wrappers are simple enough, you might even be able to auto-generate
> the .pxd and .pyx files with some of the tools listed at
> https://github.com/cython/cython/wiki/AutoPxd . Yes, you'd have your
> C++ code in a C++ file and "Python" code in a .pyx file rather than
> interleaving them, but the amount of code you write would be the same.
>

So many nopes to this approach.  I listed it myself effectively above, just
not the autopxd part.  No I don't want to make C bindings for all of C++
libraries, I don't buy this saved anything and only creates a ridiculous
amount of work again where the cost (far) exceeds the gains.

-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160819/d573f7db/attachment.html>

From wstein at gmail.com  Fri Aug 19 15:19:47 2016
From: wstein at gmail.com (William Stein)
Date: Fri, 19 Aug 2016 09:19:47 -1000
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
Message-ID: <CACLE5GB=SwGuZ7vnfAzNSTNrBJ_ZE+MOS5hw570cMQ17C+_Mhw@mail.gmail.com>

On Fri, Aug 19, 2016 at 8:19 AM, Jason Newton <nevion at gmail.com> wrote:
> You must realize that almost any other python driven way to compile c-code
> in the spirit these projects do is deprecated/dead.  Cython has absorbed all
> the reputation and users that didn't go to pure-c/boost.python - pybind11 is
> the new kid on the block there so I'm not including it (I'm of the opinion
> that SWIG users stayed unchanged).  Community belief/QA/designers/google all
> think of Cython first.  Weave has effectively closed up it's doors [...]

May I ask why "any other python driven way to compile c-code in the
spirit these projects do is deprecated/dead?"    I'm curious since
when I started Sage (and Cython based on forking Pyrex), it was
because none of the other approaches seemed like they would work for
the developer base I envisioned growing for Sage.   That was a long
time ago, and I'm always pleasantly surprised that Cython has become
very popular.  However, I didn't realize the other approaches were
deprecated/dead.

William



-- 
William (http://wstein.org)

From nevion at gmail.com  Fri Aug 19 15:32:37 2016
From: nevion at gmail.com (Jason Newton)
Date: Fri, 19 Aug 2016 15:32:37 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CACLE5GB=SwGuZ7vnfAzNSTNrBJ_ZE+MOS5hw570cMQ17C+_Mhw@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CACLE5GB=SwGuZ7vnfAzNSTNrBJ_ZE+MOS5hw570cMQ17C+_Mhw@mail.gmail.com>
Message-ID: <CAGou9MgBs1TJ1BFdhvYRdhyrXn4JctCDyMASY544pKFxRbCijA@mail.gmail.com>

On Fri, Aug 19, 2016 at 3:19 PM, William Stein <wstein at gmail.com> wrote:

> On Fri, Aug 19, 2016 at 8:19 AM, Jason Newton <nevion at gmail.com> wrote:
> > You must realize that almost any other python driven way to compile
> c-code
> > in the spirit these projects do is deprecated/dead.  Cython has absorbed
> all
> > the reputation and users that didn't go to pure-c/boost.python -
> pybind11 is
> > the new kid on the block there so I'm not including it (I'm of the
> opinion
> > that SWIG users stayed unchanged).  Community belief/QA/designers/google
> all
> > think of Cython first.  Weave has effectively closed up it's doors [...]
>
> May I ask why "any other python driven way to compile c-code in the
> spirit these projects do is deprecated/dead?"    I'm curious since
> when I started Sage (and Cython based on forking Pyrex), it was
> because none of the other approaches seemed like they would work for
> the developer base I envisioned growing for Sage.   That was a long
> time ago, and I'm always pleasantly surprised that Cython has become
> very popular.  However, I didn't realize the other approaches were
> deprecated/dead.
>

PyInline's last news update was in 2004 where the author gives a "Hats
of[f] to PyRex", prior to that only a few blog entries in 2001/2002, I've
not come across any project using it but maybe that is not sufficient to
call it deprecated/dead?  Does it work with Python 3?
http://pyinline.sourceforge.net/

Pyrex no longer has a userbase.  Last ML post was 2014.

Weave only supports python2, got ripped out of Scipy and also directs to
check out Cython and strongly implies the project is all but
dead/maintenance mode: https://github.com/scipy/weave

Did I miss any of the python driven ways?

-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160819/d19f2687/attachment.html>

From krthornt at uci.edu  Fri Aug 19 12:34:57 2016
From: krthornt at uci.edu (Kevin Thornton)
Date: Fri, 19 Aug 2016 16:34:57 +0000
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
Message-ID: <CALciEZeL=m6DvSORmH1EcSGOBUQf+XCFip77CyG3p-zkZ0v0YQ@mail.gmail.com>

Hi Robert,

Related to this point, I think there are three important features that
Cython would benefit from, as far as C++11 and newer are concerned.  In
order of what I'm guessing to be increased complexity are:

1. Non-type template parameters.
2. "rvalue" references for standalone functions and class member functions:

    void foo[T]( T && t )

3. Variadic templates (member and non-member).  More generally, "parameter
packs":

    http://en.cppreference.com/w/cpp/language/parameter_pack

#2 is important for efficiency reasons--it isn't just a convenience
feature.  Without it, Cython cannot support all the member function of
std::vector, etc.  The implication is that some expensive operations, like
moving a big object into a vector, require extra copies in Cython while a
C++ program would just move the object into the new location.

#3 seems like the hardest, but would enable std::tuple, std::bind, and
other handy things to be used in a .pyx file.

Support for "auto" would be nice, but perhaps unrealistic.

I know that PRs have been submitted, and since stalled out, on a couple of
these features.  I've tried to look at the Cython parser myself, but I'm
not able to follow it, sadly.

Thanks for all your hard work,

Kevin

On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb at gmail.com> wrote:

>
> Though, as you discovered, there are some basic things like non-type
> template arguments that we would like to have. If there are other
> specific C++ constructs that are commonly used but impossible to
> express in Cython it'd be useful to know.
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160819/b47d745b/attachment.html>

From stefan_ml at behnel.de  Sat Aug 20 01:19:26 2016
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 20 Aug 2016 07:19:26 +0200
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
Message-ID: <37d971fb-067e-5bed-54ff-c4c3d003aa15@behnel.de>

Robert Bradshaw schrieb am 19.08.2016 um 11:36:
> On Thu, Aug 18, 2016 at 12:05 PM, Jason Newton wrote:
>> I was hoping we could give a way to drop straight into C/C++ inside of
>> Cython pyx files.
>>
>> Why?
>> [...]
>> -It works around having the Cython compiler know about all of C++'s nuances
>> - as an advanced C++ developer these are painful and it is a 2nd class
>> citizen to Cython's simpler C-support - that's no good.  Just right now I
>> was bitten by yet another template argument bug and it's clear C++ template
>> arguments have been kind of dicy since support appeared.
> 
> Yes, C++ is an extraordinarily complicated language, and exposing all
> of that would add significant amounts of complexity to Cython itself,
> and perhaps more importantly increase the barrier of entry to reading
> Cython code. One of the guiding principles is that we try to stay as
> close to Python as possible (if you know Python, you can probably read
> Cython, and with a minimal amount of C knowledge start writing it) and
> much of C++ simply isn't Pythonic.
> 
> Though, as you discovered, there are some basic things like non-type
> template arguments that we would like to have. If there are other
> specific C++ constructs that are commonly used but impossible to
> express in Cython it'd be useful to know.

Especially with respect to C++, we've always tried to make things safe and
helpful that Cython can support directly, and to make things possible that
are too complex to handle safely. But I agree with Robert that allowing
arbitrary C++ code snippets inside of Cython code is not a good idea.

If there are code constructs that we can add to improve the C++ integration
then we should do that, even if it means that Cython would (initially?) not
be able to validate that code. But we should otherwise keep the language
such that it allows us to get back to a state where Cython can validate and
understand what it sees.

Stefan


From nevion at gmail.com  Sat Aug 20 17:46:19 2016
From: nevion at gmail.com (Jason Newton)
Date: Sat, 20 Aug 2016 17:46:19 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <37d971fb-067e-5bed-54ff-c4c3d003aa15@behnel.de>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <37d971fb-067e-5bed-54ff-c4c3d003aa15@behnel.de>
Message-ID: <CAGou9Mi_yqEemA9rniH64kyF96g4RtiKcfwA9LMUfedjO8unFg@mail.gmail.com>

On Sat, Aug 20, 2016 at 1:19 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

>
> Especially with respect to C++, we've always tried to make things safe and
> helpful that Cython can support directly, and to make things possible that
> are too complex to handle safely. But I agree with Robert that allowing
> arbitrary C++ code snippets inside of Cython code is not a good idea.


Why is allowing arbitrary code inside not a good idea?  We're not talking
something necessarily like eval here and the reputation it got, we're
talking C/C++ glue and for the scale of Cython's development effort, proper
C++ support is insurmountable simply because the language is that hard to
work with on the compiler side.  You can't just go and take something that
is readily in the process of evolving (as newer standards actively come
out), that very slowly achieves full compiler support in it's own domain,
and pretend/aspire you can work like that though high level constructs in
your language - this project and none before it have not accomplished
anywhere near that and simply don't have the man-years/decades to
contribute.  I truly think it is irresponsible to think a) we should do
nothing or b) delude  ourselfs that we can make high level language level
features to get around this.



>


> If there are code constructs that we can add to improve the C++ integration
> then we should do that, even if it means that Cython would (initially?) not
> be able to validate that code. But we should otherwise keep the language
> such that it allows us to get back to a state where Cython can validate and
> understand what it sees.
>

I'm not going to argue against you making C++ developer life easier but I
have no idea how you could wrangle the beast in another way than letting
the C++ compiler handle things.   I don't think it's important that Cython
compiler can validate / understand everything it sees - I'm not saying that
by default, but just like the various flags you can use with cpdef,
strategic sacrifices are acceptable for consenting adults. Otherwise you
are still saying you will limit capability, which many can't agree with and
avoid like the plague.

Remember - cython is "the way" that comes up for how to glue C/C++ into
python these days, not just the program in cish python frontend for speed.



>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160820/990d9c07/attachment.html>

From insertinterestingnamehere at gmail.com  Sun Aug 21 01:36:37 2016
From: insertinterestingnamehere at gmail.com (Ian Henriksen)
Date: Sun, 21 Aug 2016 05:36:37 +0000
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CAGou9Mi_yqEemA9rniH64kyF96g4RtiKcfwA9LMUfedjO8unFg@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <37d971fb-067e-5bed-54ff-c4c3d003aa15@behnel.de>
 <CAGou9Mi_yqEemA9rniH64kyF96g4RtiKcfwA9LMUfedjO8unFg@mail.gmail.com>
Message-ID: <CAPWbfpQqk-14_38LdOqPdzhpjcuDoWx9GY1fyXTVFS5800rL0A@mail.gmail.com>

>
>
> Remember - cython is "the way" that comes up for how to glue C/C++ into
> python these days, not just the program in cish python frontend for speed.
>
>
There's some truth to the issues you raise, but there are a lot of nasty
details
behind the scenes. Do you have some more concrete details for what you're
proposing? In particular, Cython mangles all of its variable and function
names.
How do you want to work around that? How should exception forwarding and
throwing work?

You're right that it's unrealistic to ever expect Cython to support a full
set of C++
like features, but, at the same time, often C++ libraries use many of the
fancier
features to create higher level interfaces that, conceptually, have a much
simpler
interface than we give them credit for. In many cases, the metaprogramming,
though a part of the "interface" in C++ really is more of an implementation
detail
and is hidden from Cython. I wouldn't expect Cython to ever properly
implement
things like SFINAE, expression SFINAE, or even perfect forwarding, but it
often still makes sense for it to interact with interfaces that are written
using
these features. Given that not all features actually need to be implemented
in
Cython, the question becomes more about what is good enough to support most
interfaces nicely.

Regardless of exactly which C++ features need to be supported right now,
here are
a few ways you can do things like this right now.
- You can define inline functions, macros, or templates in separate headers.
  Though this does introduce additional files, it routes things through
fairly standard
  code paths and makes things pretty reasonable. This is what I prefer to
do when
  need more C++ features that Cython provides.
- Tell Cython how the interface you're using is used, not how it is
constructed. C
  style variadic function calls are especially good for this since they
essentially
  bypass all type checking on a given function's arguments.
- You can currently exploit user-specified C names of functions and
variables to
  inject expressions into compiled code. This is an awful hack that happens
to
  combine supported features in a useful way. You can inject a wide variety
of
  expressions into the generated C++ code, but it's painful enough that
doing
  something else is almost always preferable. It totally kills readability.

All that said, I really think that there are more useful and concrete
things to work on
at the moment. There's plenty of work to do to make Cython up to scratch for
wrapping arbitrary (reasonably well designed) C++ interfaces. Inlining C++
code
into Cython seems like a nice idea up front, but in practice it's less
clear what that
would mean and how exactly all the accompanying semantics would work.

Thanks for raising a lot of these issues though. Interfacing between Python
and
modern C++ is definitely worth the discussion.

Best,

Ian Henriksen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160821/c6461dc7/attachment.html>

From robertwb at gmail.com  Sun Aug 21 02:13:49 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sat, 20 Aug 2016 23:13:49 -0700
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CALciEZeL=m6DvSORmH1EcSGOBUQf+XCFip77CyG3p-zkZ0v0YQ@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CALciEZeL=m6DvSORmH1EcSGOBUQf+XCFip77CyG3p-zkZ0v0YQ@mail.gmail.com>
Message-ID: <CADiQ+QDHcKr98o9MtvxtZv+LNkNuzgRUcQmU+jKJWpFqxCNyyA@mail.gmail.com>

On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt at uci.edu> wrote:
> Hi Robert,
>
> Related to this point, I think there are three important features that
> Cython would benefit from, as far as C++11 and newer are concerned.  In
> order of what I'm guessing to be increased complexity are:
>
> 1. Non-type template parameters.

This is actually probably the hardest :). There is a PR that's stalled
out, probably needs some attention by both the author and us; it'd be
great to get this in.

> 2. "rvalue" references for standalone functions and class member functions:
>
>     void foo[T]( T && t )

>From a callers perspective, one can just declare an extern foo as
foo(T) in Cython, it doesn't care if the actual call is made by value
or by reference--the calling code is exactly the same.

Actually implementing it for Cython-defined functions (as the callee)
would require additional support, but would probably be pretty easy
(implement a new type, with parsing, which generally behaves the same
as a reference or value from Python's perspective--the actual heavy
lifting happens in the C compiler).

Want to file an issue on github for this?

> 3. Variadic templates (member and non-member).  More generally, "parameter
> packs":
>
>     http://en.cppreference.com/w/cpp/language/parameter_pack

This'd be interesting (especially as to how it relates to changing the
arity of functions for concrete instantiations). There's also a
question of syntax. Maybe

    void foo[A, int N, ... Types](A a, Types t):
        ...

It'd likely require #1, e.g. to be able to declare things like
tuple_element::type such that std::tuple.get<N> would be correctly
typed. Unfortunately seems declaring the (somewhat intuitive) API of
std::tuple requires being able to declare a fair amount of its
implementation.

Still likely feasible though, and I think the notion of variadic
templates is natural enough in a Python setting (which has had
heterogeneously typed tuples since its inception, though of course not
statically typed).

> #2 is important for efficiency reasons--it isn't just a convenience feature.
> Without it, Cython cannot support all the member function of std::vector,
> etc.  The implication is that some expensive operations, like moving a big
> object into a vector, require extra copies in Cython while a C++ program
> would just move the object into the new location.

It's interesting to call this an essential feature, given that C++
itself didn't have it until recently :). Not that it can't be a big
win, and libraries will more and more probably be designed assuming
it.

> #3 seems like the hardest, but would enable std::tuple, std::bind, and other
> handy things to be used in a .pyx file.
>
> Support for "auto" would be nice, but perhaps unrealistic.

Just leave the cdef out and you basically get auto, i.e.

    my_var = expr

gives my_var the type of expr. If multiple assignments are made to
my_var, it attempts to find a suitable union type that can hold all
assignments. The one exception is for integers, where it infers the
type to be object iff my_var is used in arithmetic expressions (that
it can't deduce would not overflow with the finite-sized C integer
type).

> I know that PRs have been submitted, and since stalled out, on a couple of
> these features.  I've tried to look at the Cython parser myself, but I'm not
> able to follow it, sadly.
>
> Thanks for all your hard work,
>
> Kevin
>
> On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb at gmail.com> wrote:
>>
>>
>> Though, as you discovered, there are some basic things like non-type
>> template arguments that we would like to have. If there are other
>> specific C++ constructs that are commonly used but impossible to
>> express in Cython it'd be useful to know.
>>
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at gmail.com  Sun Aug 21 05:11:43 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sun, 21 Aug 2016 02:11:43 -0700
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CAGou9MgBs1TJ1BFdhvYRdhyrXn4JctCDyMASY544pKFxRbCijA@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CACLE5GB=SwGuZ7vnfAzNSTNrBJ_ZE+MOS5hw570cMQ17C+_Mhw@mail.gmail.com>
 <CAGou9MgBs1TJ1BFdhvYRdhyrXn4JctCDyMASY544pKFxRbCijA@mail.gmail.com>
Message-ID: <CADiQ+QBNm6x47ysEf52MMK2jA00k5wzHdyf9NRKuyuaDE2ZRmA@mail.gmail.com>

On Fri, Aug 19, 2016 at 12:32 PM, Jason Newton <nevion at gmail.com> wrote:
>
>
> On Fri, Aug 19, 2016 at 3:19 PM, William Stein <wstein at gmail.com> wrote:
>>
>> On Fri, Aug 19, 2016 at 8:19 AM, Jason Newton <nevion at gmail.com> wrote:
>> > You must realize that almost any other python driven way to compile
>> > c-code
>> > in the spirit these projects do is deprecated/dead.  Cython has absorbed
>> > all
>> > the reputation and users that didn't go to pure-c/boost.python -
>> > pybind11 is
>> > the new kid on the block there so I'm not including it (I'm of the
>> > opinion
>> > that SWIG users stayed unchanged).  Community belief/QA/designers/google
>> > all
>> > think of Cython first.  Weave has effectively closed up it's doors [...]
>>
>> May I ask why "any other python driven way to compile c-code in the
>> spirit these projects do is deprecated/dead?"    I'm curious since
>> when I started Sage (and Cython based on forking Pyrex), it was
>> because none of the other approaches seemed like they would work for
>> the developer base I envisioned growing for Sage.   That was a long
>> time ago, and I'm always pleasantly surprised that Cython has become
>> very popular.  However, I didn't realize the other approaches were
>> deprecated/dead.
>
>
> PyInline's last news update was in 2004 where the author gives a "Hats of[f]
> to PyRex", prior to that only a few blog entries in 2001/2002, I've not come
> across any project using it but maybe that is not sufficient to call it
> deprecated/dead?  Does it work with Python 3?
> http://pyinline.sourceforge.net/
>
> Pyrex no longer has a userbase.  Last ML post was 2014.
>
> Weave only supports python2, got ripped out of Scipy and also directs to
> check out Cython and strongly implies the project is all but
> dead/maintenance mode: https://github.com/scipy/weave
>
> Did I miss any of the python driven ways?

Pybindgen? And of course there's ctypes/cffi. I could have sworn I saw
another project pop up several years ago that was a lot like what
you're suggesting, e.g. a listing of

    def foo(a, b):
       [c code using a and b]

I don't recall how the types for a and b were declared/converted, and
I don't think it did much non-trivial stuff yet, but I can't find a
trace of it now (but it's hard to search for laking a name).

From robertwb at gmail.com  Sun Aug 21 05:30:52 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sun, 21 Aug 2016 02:30:52 -0700
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
Message-ID: <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>

On Fri, Aug 19, 2016 at 11:19 AM, Jason Newton <nevion at gmail.com> wrote:
>
> On Fri, Aug 19, 2016 at 5:36 AM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>
>> On Thu, Aug 18, 2016 at 12:05 PM, Jason Newton <nevion at gmail.com> wrote:
>> > Accidentally posted to an already-opened tab for the cython-users ML
>> > yesterday, moving to here. Following up from a github opened issue here:
>> >
>> > https://github.com/cython/cython/issues/1440
>> >
>> > I was hoping we could give a way to drop straight into C/C++ inside of
>> > Cython pyx files.
>> >
>> > Why?
>> >
>> > -It [helps] avoid needing to declare every class/function in cython, a
>> > somewhat daunting/impossible task that I think everyone hates.  Have you
>> > libraries like Eigen or others that use complex template based
>> > techniques?
>> > How about those with tons of [member] functions to boot or getting C++
>> > inheritance involved.
>>
>> I agree that this is a pain, and better tooling should be developed to
>> (mostly?) eliminate this (e.g. see the recent thread at
>> https://groups.google.com/forum/#!topic/cython-users/c8ChI6jERzY ).
>>
>> Of course having symbols magically appear (e.g. due some #Include, who
>> knows which) has its downsides too, which is why import * is often
>> discouraged in Python too.
>>
>> > -It works around having the Cython compiler know about all of C++'s
>> > nuances
>> > - as an advanced C++ developer these are painful and it is a 2nd class
>> > citizen to Cython's simpler C-support - that's no good.  Just right now
>> > I
>> > was bitten by yet another template argument bug and it's clear C++
>> > template
>> > arguments have been kind of dicy since support appeared.
>>
>> Yes, C++ is an extraordinarily complicated language, and exposing all
>> of that would add significant amounts of complexity to Cython itself,
>> and perhaps more importantly increase the barrier of entry to reading
>> Cython code. One of the guiding principles is that we try to stay as
>> close to Python as possible (if you know Python, you can probably read
>> Cython, and with a minimal amount of C knowledge start writing it) and
>> much of C++ simply isn't Pythonic.
>
> Maybe it's off topic but I debate the guiding principle of Cython -  I was
> not able to comprehend Cython before reading tutorials and this is not my
> first time looking at it, I had a couple runins over the last 5 years with
> it on projects such as h5py easily got lost on what was going on between all
> the wrapper-wrappers (pure py code wrapping py-invokable code) and
> re-declarations.

In my experience Cython has generally been fairly easy to pick up for
people who already know Python. And Python often easy to pick up for
people who already know C/C++. Of course for many wrappings it often
takes non-trivial knowledge of the wrapped library itself too, but
typically at the same level as would be required to grok code written
against that same library directly from C/C++.

Yes, there's bad code out there in any language (no offense meant
towards h5py--I haven't looked at that project myself). Much of it due
to cargo-cult perpetuations or archaic (or simply flat-out-wrong)
contortions due to historical limitations (e.g. creating a Python
module to wrap an _extension module, avoiding all C++ features with
extensive C wrappers, ...). (You're familiar with C++, so likely no
stranger to this effect.)

> These projects complied with Cython's current philosophy
> to the degradation of clarity, context, and overall idea of how code was
> hooked up.  Perhaps Cython should take the lessons learned from it's
> inception, time, and the results of the state of the c-python userbase to
> guide us into a new philosophy.

I fail to see how "staying close to Python" caused "degradation of
clarity, context, etc." If anything, the lessons learned over time
have validated this philosophy. More on this later.

>> Though, as you discovered, there are some basic things like non-type
>> template arguments that we would like to have. If there are other
>> specific C++ constructs that are commonly used but impossible to
>> express in Cython it'd be useful to know.
>
> I haven't had the capability to use Cython sufficiently to learn more of
> them because it currently can't solve my problems.  From prior SWIG et al
> experiences, my outlook is that it is treacherous path to walk and unless
> you brought in llvm/clang into the project for parsing/AST, I'd hold onto
> that outlook.

I agree that any efforts to trying to parsing C++ without building on
an actual compiler are fraught with danger. That's not the case with
generating C++ code, which is the direction we're going. In
particular, our goal is to understand C++ enough to invoke it, which
allows us to be much less pedantic.

A that automatically extract definitions (or even wrappings, even
partially) from C++ headers is another topic, and should almost
certainly lean on an existing compiler.

>> > -It would allow single source files - I think this is important for
>> > runtime
>> > compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide
>>
>> Not quite following what you're saying here.
>
> Maybe PyInline was a better example off the bat for you, but something a
> little more flexible but also with less work is needed.  Compare with
> PyOpenCL:
> https://documen.tician.de/pyopencl/ - check out some examples.   There is a
> c runtime api between the contexts hooking things up (this is the OpenCL
> runtime part) - it's a pretty similar story to PyCuda (and by the same
> author, execpt for that project has to jump out to nvcc and cache kernel
> compilation like the inline function implementation does).  There's no limit
> to the number of functions you can declare though and the OpenCL side is
> kept simple - things are generally pretty typesafe/do what you would expect
> on dispatch.  PyInline for comparison looks like it might lean on the Python
> c-api for it's work more and maybe limited in the number of functions per
> snippet it can declare.  I don't expect to be able to work with Numpy
> ndarray data easily with it.

The fact that you're essentially defining a kernel/ufunc with a well
defined API in another language makes this a somewhat more natural and
tractable case than freely sliding back and forth between C and Python
data structures, function calls, etc. multiple times in a function's
body. (Not to say that these aren't quite sophisticated projects--they
tackle more interesting difficulties elsewhere.)

>> > The idea is that Cython glue makes the playing field for extracting data
>> > easy, but that once it's extracted to a cdef variable for instance,
>> > cython
>> > doesn't need to know what happens.  Maybe in a way sort of like the GCC
>> > asm
>> > extension.  Hopefully simpler variable passing though.
>>
>> Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any
>> possible conflicts. Specifying what/how to mangle could get as ugly as
>> GCC's asm variable passing. And embedded variable declarations, let
>> alone control flow statements (especially return, break, ...) could
>> get really messy. It obscures analysis Cython can do on the code, such
>> as whether variables are used or what values they may take. Little
>> code snippets are not always local either, e.g. do they often need to
>> refer to variables (or headers) referenced elsewhere. And they must
>> all be mutually compatible.
>
> Like gcc's asm, let's let adults do what they want and let them worry about
> the consequences of flow control/stray includes. I'm not even sure how most
> of this would be an issue (switch/break/if) if you are properly nesting pyxd
> output.  The only thing I think is an issue here is mangled names.  I
> haven't yet figured out why (cdef) variable names must be mangled.  Can you
> explain?  Maybe we add an option to allow it to be unmangled in their
> declaration? C++ has extern "C" for example.

Name mangling is done for the standard reasons--to avoid possible
conflicts with all other symbols that may be defined. E.g. We don't
want things to suddenly break if I happen to create a variable called
"PyNone." Or "__pyx_something_we_defined_implicitly." And of course we
want to mangle globals, function names, etc. lest they conflict with
some otherwise irrelevant symbol defined in some (possibly
recursively) included header somewhere.

Again, you could just say "Don't name things like that." This exposes
some more guiding principles. (1) If it's valid Python, it should be
valid Cython and (2) we always try to produce valid C code--if you
haven't lied to us (too much) about your external declarations, a
successful Cython compilation results in a valid C/C++ output. Also
(3) you shouldn't have to read or understand the generated C and the
Python/C API to use, let alone debug, Cython (though you're happy to
do so if you want, like Java developers sometimes read bytecodes, but
not usually, though understanding implementation can sometimes be
helpful when chasing performance (for all languages)).

There's an obvious tension between giving users all the rope they want
vs. providing an API that is possibly more restrictive, but inherently
correct by construction. I'll concede that Cython necessarily has
pointers, so I'll give that there's plenty of room for foot-shooting
(and better interfacing with modern C++ would be good help there), but
the kind of errors one runs into by injecting arbitrary code snippets
take things to a whole new level (and specifically violate (3) when
developing and debugging).

The escape hatch is to wrap the C++ in an actual C++ file and invoke
the wrapping. Typically this is the minority of one's code--if it's
the "whole library" then you probably have an API that's only
understandable to someone well versed in C++ anyways. You've given a
single example (non-type template arguments) that we would like to
support that's blocking you.

> Why is allowing arbitrary code inside not a good idea?  We're not talking
> something necessarily like eval here and the reputation it got,

Actually, I think it's a whole lot like eval. It's taking an opaque
(string) chunk of data and executing it as code. But potentially worse
as it's in a different language and evaluated in a transformed (even
if the names were unmangled) context.

If we were to go this direction, I might go with a function call (like
weave, maybe even follow it) rather than a new statement as the latter
is difficult to extend with the myriad of optional configuration
parameters, etc. that would beg to follow.

> we're
> talking C/C++ glue and for the scale of Cython's development effort, proper
> C++ support is insurmountable simply because the language is that hard to
> work with on the compiler side.  You can't just go and take something that
> is readily in the process of evolving (as newer standards actively come
> out), that very slowly achieves full compiler support in it's own domain,
> and pretend/aspire you can work like that though high level constructs in
> your language - this project and none before it have not accomplished
> anywhere near that and simply don't have the man-years/decades to
> contribute.  I truly think it is irresponsible to think a) we should do
> nothing or b) delude  ourselfs that we can make high level language level
> features to get around this.

As Ian said, we can declare the interface, not necessarily the
implementation, and when things get so bad that you need to express
them in C++, express them in direct C++ in a separate file and invoke
the nicer interface from Cython.

C++ is evolving, but most (not all) of that is happening at the level
of implementation details and much of the API changes are in terms of
syntactic sugar/new features that look a lot like old features. Cython
only needs concern itself with that which commonly affects a library's
API.

As I said before, if there are specific missing features that block
wrapping much of C++ code, it's worth looking at how they could fit
into the language. We aim for a clean API, but also strive to be
pragmatic. There is much room for improvement.

>> That's aside from the jarring nature of interleaving Python and C++
>> code. Would semicolons be required? How would parsers (including IDEs)
>> handle the C++ snippets? Or would they be placed in opaque
>> strings/comments (which I'd rather avoid)?
>
> Opaque strings.  It's a good and time tested solution to the issue.  I'm
> very happy with it in the contexts I use it in.
>
>> Feels like you want PyInline or weave.inline, but at compile time.
>
> You must realize that almost any other python driven way to compile c-code
> in the spirit these projects do is deprecated/dead.  Cython has absorbed all
> the reputation and users that didn't go to pure-c/boost.python - pybind11 is
> the new kid on the block there so I'm not including it (I'm of the opinion
> that SWIG users stayed unchanged).  Community belief/QA/designers/google all
> think of Cython first.  Weave has effectively closed up it's doors and I'm
> not even sure it had the power to do what I wanted anyway because Cython
> provides a language that eases the data-extraction/typecasting part of
> inlining C/C++.

You seem to be repeatedly bringing up the points

* Many (most?) of these string-based approaches are essentially dead,
often pointing people to Cython instead, but
* Cython should adopt the string-embedding approach of these earlier projects.

You ask at the beginning of the email whether time has vindicated our
philosophy. I think, based on the mindshare vs. these other attempts
at integrating with C, in large part it has. It has served us and our
users well; we will strive to stay close to Python.

Tight interleaving of multiple languages in is cute for making a
polyglot script, but I do not think it leads to legible code. An
"eval_cpp" operator would be a lot like the builtin eval--it'd be
really tempting to do the "quick and easy" hack of dropping in some
executable string instead of thinking how to structure things such
that that could be avoided, but putting in this effort leads to more
comprehensible code.

It's hard to say "no" to features, but I think such an introduction
would fundamentally change Cython and how it's written for the worse.

- Robert

From stefan_ml at behnel.de  Sun Aug 21 05:38:00 2016
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 21 Aug 2016 11:38:00 +0200
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QBNm6x47ysEf52MMK2jA00k5wzHdyf9NRKuyuaDE2ZRmA@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CACLE5GB=SwGuZ7vnfAzNSTNrBJ_ZE+MOS5hw570cMQ17C+_Mhw@mail.gmail.com>
 <CAGou9MgBs1TJ1BFdhvYRdhyrXn4JctCDyMASY544pKFxRbCijA@mail.gmail.com>
 <CADiQ+QBNm6x47ysEf52MMK2jA00k5wzHdyf9NRKuyuaDE2ZRmA@mail.gmail.com>
Message-ID: <f2429bc3-ee3c-9d14-0adb-1e8e3b5c202d@behnel.de>

Robert Bradshaw schrieb am 21.08.2016 um 11:11:
> On Fri, Aug 19, 2016 at 12:32 PM, Jason Newton wrote:
>> PyInline's last news update was in 2004 where the author gives a "Hats of[f]
>> to PyRex", prior to that only a few blog entries in 2001/2002, I've not come
>> across any project using it but maybe that is not sufficient to call it
>> deprecated/dead?  Does it work with Python 3?
>> http://pyinline.sourceforge.net/
>>
>> Pyrex no longer has a userbase.  Last ML post was 2014.
>>
>> Weave only supports python2, got ripped out of Scipy and also directs to
>> check out Cython and strongly implies the project is all but
>> dead/maintenance mode: https://github.com/scipy/weave
>>
>> Did I miss any of the python driven ways?
> 
> Pybindgen? And of course there's ctypes/cffi. I could have sworn I saw
> another project pop up several years ago that was a lot like what
> you're suggesting, e.g. a listing of
> 
>     def foo(a, b):
>        [c code using a and b]
> 
> I don't recall how the types for a and b were declared/converted, and
> I don't think it did much non-trivial stuff yet, but I can't find a
> trace of it now (but it's hard to search for laking a name).

This page lists some, although I can't see any real candidates for what you
describe:

https://wiki.python.org/moin/IntegratingPythonWithOtherLanguages

And most of those are dead links and/or projects.

Stefan


From stefan_ml at behnel.de  Sun Aug 21 06:14:10 2016
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 21 Aug 2016 12:14:10 +0200
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
Message-ID: <cac83600-584d-0dda-0bc7-dddd65d137ad@behnel.de>

Robert Bradshaw schrieb am 21.08.2016 um 11:30:
> You ask at the beginning of the email whether time has vindicated our
> philosophy. I think, based on the mindshare vs. these other attempts
> at integrating with C, in large part it has. It has served us and our
> users well; we will strive to stay close to Python.
> 
> Tight interleaving of multiple languages in is cute for making a
> polyglot script, but I do not think it leads to legible code. An
> "eval_cpp" operator would be a lot like the builtin eval--it'd be
> really tempting to do the "quick and easy" hack of dropping in some
> executable string instead of thinking how to structure things such
> that that could be avoided, but putting in this effort leads to more
> comprehensible code.

+1


> It's hard to say "no" to features, but I think such an introduction
> would fundamentally change Cython and how it's written for the worse.

... and restrict the further development of Cython to anything that doesn't
break what users have relied on in their interwoven code.

I agree with all of your arguments. But I'd like to note that it should
still be quite possible for the OP to write a preprocessor for pyx-ish
files that extracts any contained C/C++ code snippets (e.g. from specially
marked Python strings with a bit of a metadata DSL), drops them into an
external C/C++ header file as functions etc., and then replaces them with
external function calls in the original file, before passing it through Cython.

That's essentially how the described feature would work anyway, but it
would keep Cython itself free from any eval-my-string special handling and
user code integration overhead, while allowing users to play around with
the possibilities of keeping everything in one source file. If that turns
out to become the Next Great Thing, we will at least have learned something
from real usage by the time we get there, rather than building something
now that eventually everyone (but a few) will hate and not use.

I've seen some people turn their back on Cython because they didn't
understand it and instead started their own projects their own way. It's
obviously everyone's good right to do with their free time whatever they
want, but most of those attempts have visibly failed by now. They usually
didn't even survive the first couple of months or weeks. It often seems so
much easier (and certainly more fun, right?) to start something out in the
open countryside that never gets anywhere, than to build on something
existing which not quite but almost does what you need.

I'd be happy to help making it easier for people to integrate their own
tooling with Cython. Not everything needs to be an integrated part of the
compiler itself, and certainly not everything should be maintained or
developed by us.

Stefan


From nevion at gmail.com  Sun Aug 21 06:46:07 2016
From: nevion at gmail.com (Jason Newton)
Date: Sun, 21 Aug 2016 06:46:07 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <f2429bc3-ee3c-9d14-0adb-1e8e3b5c202d@behnel.de>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CACLE5GB=SwGuZ7vnfAzNSTNrBJ_ZE+MOS5hw570cMQ17C+_Mhw@mail.gmail.com>
 <CAGou9MgBs1TJ1BFdhvYRdhyrXn4JctCDyMASY544pKFxRbCijA@mail.gmail.com>
 <CADiQ+QBNm6x47ysEf52MMK2jA00k5wzHdyf9NRKuyuaDE2ZRmA@mail.gmail.com>
 <f2429bc3-ee3c-9d14-0adb-1e8e3b5c202d@behnel.de>
Message-ID: <CAGou9MiadgydrUsEK_tegsHDSLUxmWSwJ6HE2u9yunhwHsMgHg@mail.gmail.com>

ctypes/cffi aren't in the same class for comparison.  Just at the start of
the list is basically a restriction from C++ name mangling fouling things
up.

Pybindgen uses a programmable model to generate bindings for existing
functions/classes only.  It's a fair bit different from anything I had in
mind for the python side and even in the C++ binding side (boost.python
class) or pyinline class - no additional code of any kind allowed.  Last
activity 2014 - I don't really have a handle on how alive it is, but I
think it's inactive.

On Sun, Aug 21, 2016 at 5:38 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Robert Bradshaw schrieb am 21.08.2016 um 11:11:
> > On Fri, Aug 19, 2016 at 12:32 PM, Jason Newton wrote:
> >> PyInline's last news update was in 2004 where the author gives a "Hats
> of[f]
> >> to PyRex", prior to that only a few blog entries in 2001/2002, I've not
> come
> >> across any project using it but maybe that is not sufficient to call it
> >> deprecated/dead?  Does it work with Python 3?
> >> http://pyinline.sourceforge.net/
> >>
> >> Pyrex no longer has a userbase.  Last ML post was 2014.
> >>
> >> Weave only supports python2, got ripped out of Scipy and also directs to
> >> check out Cython and strongly implies the project is all but
> >> dead/maintenance mode: https://github.com/scipy/weave
> >>
> >> Did I miss any of the python driven ways?
> >
> > Pybindgen? And of course there's ctypes/cffi. I could have sworn I saw
> > another project pop up several years ago that was a lot like what
> > you're suggesting, e.g. a listing of
> >
> >     def foo(a, b):
> >        [c code using a and b]
> >
> > I don't recall how the types for a and b were declared/converted, and
> > I don't think it did much non-trivial stuff yet, but I can't find a
> > trace of it now (but it's hard to search for laking a name).
>
> This page lists some, although I can't see any real candidates for what you
> describe:
>
> https://wiki.python.org/moin/IntegratingPythonWithOtherLanguages
>
> And most of those are dead links and/or projects.
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160821/33c71c2c/attachment.html>

From nevion at gmail.com  Sun Aug 21 06:59:24 2016
From: nevion at gmail.com (Jason Newton)
Date: Sun, 21 Aug 2016 06:59:24 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CAPWbfpQqk-14_38LdOqPdzhpjcuDoWx9GY1fyXTVFS5800rL0A@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <37d971fb-067e-5bed-54ff-c4c3d003aa15@behnel.de>
 <CAGou9Mi_yqEemA9rniH64kyF96g4RtiKcfwA9LMUfedjO8unFg@mail.gmail.com>
 <CAPWbfpQqk-14_38LdOqPdzhpjcuDoWx9GY1fyXTVFS5800rL0A@mail.gmail.com>
Message-ID: <CAGou9MiPsbUGoZcfS_sm79y_8EDvmq0Cty7rmqh3aD6-mbvdyA@mail.gmail.com>

On Sun, Aug 21, 2016 at 1:36 AM, Ian Henriksen <
insertinterestingnamehere at gmail.com> wrote:

>
>> Remember - cython is "the way" that comes up for how to glue C/C++ into
>> python these days, not just the program in cish python frontend for speed.
>>
>>
> There's some truth to the issues you raise, but there are a lot of nasty
> details
> behind the scenes. Do you have some more concrete details for what you're
> proposing? In particular, Cython mangles all of its variable and function
> names.
> How do you want to work around that? How should exception forwarding and
> throwing work?
>

Recall I'm not one of the Cython devs :-)  I think it is reasonable without
knowing detail to the internals of Cython that if mangling must be done,
there can be an opt-out specifier to the mangling done on cdef'd functions
and variables.  Variables are where the meat is at for me but there could
be some benefit to functions as well if there's no reason they couldn't
come along for the ride if they don't add significant complication.

I think for exception forwarding and throwing, the responsibility would lay
to the wrapped code to guarantee either no thrown exceptions (you see this
in multithreaded software all the time) or they must code the exception
support in themselves.  However easier user-handled translations patterns
apply to make this a bit easier - set a cdef'd flag and a message and check
that flag after the eval'd portion - then decode and possibly raise an
exception.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160821/31028c60/attachment.html>

From nevion at gmail.com  Sun Aug 21 10:26:24 2016
From: nevion at gmail.com (Jason Newton)
Date: Sun, 21 Aug 2016 10:26:24 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
Message-ID: <CAGou9MjDw9M4EmBn4jTe-B=kh5xYjUq=rBYjCOpHDip7Bf6h-g@mail.gmail.com>

On Sun, Aug 21, 2016 at 5:30 AM, Robert Bradshaw <robertwb at gmail.com> wrote:

>
> In my experience Cython has generally been fairly easy to pick up for
> people who already know Python. And Python often easy to pick up for
> people who already know C/C++. Of course for many wrappings it often
> takes non-trivial knowledge of the wrapped library itself too, but
> typically at the same level as would be required to grok code written
> against that same library directly from C/C++.
>

Is your experience drawn from binding moderately complex libraries or play
code (of complexity like from a tutorial).  Bottom up or top down?  Sorry
if this sounds asinine to you.  To clarify, I'm coming from the case where
I didn't read the whole tutorial/docs before being faced with pyx in the
projects I previously mentioned, while on tight turn around time - I was
not able to grok in that context.

For myself and any other ML user who comes across this thread - can you
list a few libraries that do things the right way?


>
> Yes, there's bad code out there in any language (no offense meant
> towards h5py--I haven't looked at that project myself). Much of it due
> to cargo-cult perpetuations or archaic (or simply flat-out-wrong)
> contortions due to historical limitations (e.g. creating a Python
> module to wrap an _extension module, avoiding all C++ features with
> extensive C wrappers, ...). (You're familiar with C++, so likely no
> stranger to this effect.)
>
> > These projects complied with Cython's current philosophy
> > to the degradation of clarity, context, and overall idea of how code was
> > hooked up.  Perhaps Cython should take the lessons learned from it's
> > inception, time, and the results of the state of the c-python userbase to
> > guide us into a new philosophy.
>
> I fail to see how "staying close to Python" caused "degradation of
> clarity, context, etc." If anything, the lessons learned over time
> have validated this philosophy. More on this later.
>

My point was that multifile multi-level wrapper that I mentioned earlier -
if you're saying that those projects did Cython extensions wrong, then I'm
incorrect at faulting Cython and should fault the libraries using it.  I
didn't say staying close to python caused $blurb.

I don't know in a situation as confusing as to all the binding projects if
this should be taken as validation of philosophy either - I think it is
reasonable to consider the attrition of these projects as a function of
manpower, number of early on project supporters/authors, and if a project
(like sage) indirectly, through dependency,  kept the project alive.  And
good old fashioned luck.  I noted most of them don't use distutils and
something custom but less capable instead which maybe plays a roll in how
mature/usable/smalltime they where/are.


> I agree that any efforts to trying to parsing C++ without building on
> an actual compiler are fraught with danger. That's not the case with
> generating C++ code, which is the direction we're going. In
> particular, our goal is to understand C++ enough to invoke it, which
> allows us to be much less pedantic.
>

I understand and agree with the logic in stating it's a less complicated
goal but what comparable success stories exist?  I strongly think "devils
in the details" in correctly making that work and that they will be tough
solvable problems.  And then you're going and promising on unfamiliar
territory.  But what the ultimate takeaway for me is that you won't have it
ready in any near term.  Do you have the skills and resources to implement
this in under 2 years?  And then the other question is are you and the team
reasonably confident you will have it working and usable by then.
Otherwise you are not being pragmatic.

On the other hand, if it was reasonably simple as many of your other points
in future emails point out, I'd really like to know why you hadn't
addressed them earlier.


>
> A that automatically extract definitions (or even wrappings, even
> partially) from C++ headers is another topic, and should almost
> certainly lean on an existing compiler.
>
> >> > -It would allow single source files - I think this is important for
> >> > runtime
> >> > compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide
> >>
> >> Not quite following what you're saying here.
> >
> > Maybe PyInline was a better example off the bat for you, but something a
> > little more flexible but also with less work is needed.  Compare with
> > PyOpenCL:
> > https://documen.tician.de/pyopencl/ - check out some examples.   There
> is a
> > c runtime api between the contexts hooking things up (this is the OpenCL
> > runtime part) - it's a pretty similar story to PyCuda (and by the same
> > author, execpt for that project has to jump out to nvcc and cache kernel
> > compilation like the inline function implementation does).  There's no
> limit
> > to the number of functions you can declare though and the OpenCL side is
> > kept simple - things are generally pretty typesafe/do what you would
> expect
> > on dispatch.
>
> The fact that you're essentially defining a kernel/ufunc with a well
> defined API in another language makes this a somewhat more natural and
> tractable case than freely sliding back and forth between C and Python
> data structures, function calls, etc. multiple times in a function's
> body. (Not to say that these aren't quite sophisticated projects--they
> tackle more interesting difficulties elsewhere.)
>

After you get used to the mental gymnastics you do there, in that kind of
development (CUDA/OpenCL), this is pretty simple by comparison. It does
feels like some of the same areas of my brain would be activated though in
this case. :-)  Minor quibble - no you're not defining a ufunc... even a
kernel is misnomer, it's a full program on the other end with no main.


>
> >> > The idea is that Cython glue makes the playing field for extracting
> data
> >> > easy, but that once it's extracted to a cdef variable for instance,
> >> > cython
> >> > doesn't need to know what happens.  Maybe in a way sort of like the
> GCC
> >> > asm
> >> > extension.  Hopefully simpler variable passing though.
> >>
> >> Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any
> >> possible conflicts. Specifying what/how to mangle could get as ugly as
> >> GCC's asm variable passing. And embedded variable declarations, let
> >> alone control flow statements (especially return, break, ...) could
> >> get really messy. It obscures analysis Cython can do on the code, such
> >> as whether variables are used or what values they may take. Little
> >> code snippets are not always local either, e.g. do they often need to
> >> refer to variables (or headers) referenced elsewhere. And they must
> >> all be mutually compatible.
> >
> > Like gcc's asm, let's let adults do what they want and let them worry
> about
> > the consequences of flow control/stray includes. I'm not even sure how
> most
> > of this would be an issue (switch/break/if) if you are properly nesting
> pyxd
> > output.  The only thing I think is an issue here is mangled names.  I
> > haven't yet figured out why (cdef) variable names must be mangled.  Can
> you
> > explain?  Maybe we add an option to allow it to be unmangled in their
> > declaration? C++ has extern "C" for example.
>
> Name mangling is done for the standard reasons--to avoid possible
> conflicts with all other symbols that may be defined. E.g. We don't
> want things to suddenly break if I happen to create a variable called
> "PyNone." Or "__pyx_something_we_defined_implicitly." And of course we
> want to mangle globals, function names, etc. lest they conflict with
> some otherwise irrelevant symbol defined in some (possibly
> recursively) included header somewhere.
>
> Again, you could just say "Don't name things like that." This exposes
> some more guiding principles. (1) If it's valid Python, it should be
> valid Cython and (2) we always try to produce valid C code--if you
> haven't lied to us (too much) about your external declarations, a
> successful Cython compilation results in a valid C/C++ output. Also
> (3) you shouldn't have to read or understand the generated C and the
> Python/C API to use, let alone debug, Cython (though you're happy to
> do so if you want, like Java developers sometimes read bytecodes, but
> not usually, though understanding implementation can sometimes be
> helpful when chasing performance (for all languages)).
>
> There's an obvious tension between giving users all the rope they want
> vs. providing an API that is possibly more restrictive, but inherently
> correct by construction. I'll concede that Cython necessarily has
> pointers, so I'll give that there's plenty of room for foot-shooting
> (and better interfacing with modern C++ would be good help there), but
> the kind of errors one runs into by injecting arbitrary code snippets
> take things to a whole new level (and specifically violate (3) when
> developing and debugging).
>

I think injecting arbitrary code snippets has a reasonably good probably of
not breaking 3 in your above, provided we have a way to get at unmangled
identifiers (*or* document and stick to the mangling strategy, assuming
it's easy) - that or we scan the snippet code and replace identifiers
(significantly more complex, instincts make me think fragile for a while
until it's gotten right - esp without LLVM).  Perhaps syntax errors would
be an issue if you're just coding things up... but again there's the opt-in
to this construct and we could make life easier by annotating the snippet
in the output - to help localize the user.


>
> The escape hatch is to wrap the C++ in an actual C++ file and invoke
> the wrapping. Typically this is the minority of one's code--if it's
> the "whole library" then you probably have an API that's only
> understandable to someone well versed in C++ anyways. You've given a
> single example (non-type template arguments) that we would like to
> support that's blocking you.
>

My lack of examples is due to insufficient time playing with Cython - I hit
nonstarters so I stop and abandon; as I said, to date, Cython has never
been able to solve my C++ problems and none of them seem extraordinary.  I
think you've got alot of more unknown-unknowns here than you give credit to
but we can't discover that until you at least fix that template bug
(properly).  I'm still not looking forward to forward declaring every
identifier/function/whatever from C++ land in Cython though and I still
strongly dislike that there's no single source way of doing Cython with
something like a kernel/ufunc that needs to escape to C/C++.  This makes
doing something like the mako based templates I mentioned in the OP email
much more cumbersome/hard and Cython would provide no built in mechanism
(like inline/inline_module) for making that work.


>
> > Why is allowing arbitrary code inside not a good idea?  We're not talking
> > something necessarily like eval here and the reputation it got,
>
> Actually, I think it's a whole lot like eval. It's taking an opaque
> (string) chunk of data and executing it as code. But potentially worse
> as it's in a different language and evaluated in a transformed (even
> if the names were unmangled) context.
>
> If we were to go this direction, I might go with a function call (like
> weave, maybe even follow it) rather than a new statement as the latter
> is difficult to extend with the myriad of optional configuration
> parameters, etc. that would beg to follow.
>

My point with eval is it's bad reputation was mostly due to security
vulnerabilities and it joined the league of evil like goto and other tools
that are great in the right hands and times.  The string is known and fixed
at pyx body cython compile time which is one of the things that made me
think it had to be a statement rather than a function.  But again, I'm not
a cython developer.  If you think an inlining function that takes in all
the args it needs to inline successfully is the mechanism that achieves the
effect on pyx compile, I don't think I'd mind.   It sure is a privileged
and weird function though, to be able to emit code and not be a runtime
statement.

2 Examples I wanted to pull out real quick are:

https://github.com/scipy/weave/blob/master/examples/wx_example.py
https://github.com/scipy/weave/blob/master/examples/binary_search.py

But I think I detect magic going on here for adding includes in the wx
example and that is not usable/reliable approach.  The cython.inline
implementation also did include magic for numpy variables... Just a
warning.  I don't think weave failed because of Python 3 support, I think
it was because it was too limited to be useful because of that magic and
the walls around getting something say like Eigen in, so nobody used it.


> > You must realize that almost any other python driven way to compile
> c-code
> > in the spirit these projects do is deprecated/dead.  Cython has absorbed
> all
> > the reputation and users that didn't go to pure-c/boost.python -
> pybind11 is
> > the new kid on the block there so I'm not including it (I'm of the
> opinion
> > that SWIG users stayed unchanged).  Community belief/QA/designers/google
> all
> > think of Cython first.  Weave has effectively closed up it's doors and
> I'm
> > not even sure it had the power to do what I wanted anyway because Cython
> > provides a language that eases the data-extraction/typecasting part of
> > inlining C/C++.
>
> You seem to be repeatedly bringing up the points[:]
>
> * Many (most?) of these string-based approaches are essentially dead,
> often pointing people to Cython instead, but
> * Cython should adopt the string-embedding approach of these earlier
> projects.
>

Hoho - zing!  No that is not a conclusion you should be drawing.   Your
faults here are to imply those projects failed because they used
string-embedding approaches and to imply string-embedding based approaches
are the approaches that failed - *most* have failed over a variety of
implementations both Python driven and not.  I restricted to python-driven
for the sake of brevity and mentioned the selection. As I tried to hint
earlier, the several other projects failure happened because of any number
of unrelated reasons to it being a string based approach.  I believe
additionally that there were too many options (confusion) and too small a
potential userbase (at least in those years) to bolster and attract blood
to each of the projects and make them thrive.  Probably something akin to
the ton of orphaned projects on pipi, it doesn't mean it happened because
the approach was wrong.  One thing I did want you to take away is that
Cython needs to absorb the responsibility of it's reputation and status -
the last survivor of a somewhat diverse class with different capabilities,
if you will, that went outside of your original usage.


> You ask at the beginning of the email whether time has vindicated our
> philosophy. I think, based on the mindshare vs. these other attempts
> at integrating with C, in large part it has. It has served us and our
> users well; we will strive to stay close to Python.
>
> Tight interleaving of multiple languages in is cute for making a
> polyglot script, but I do not think it leads to legible code. An
> "eval_cpp" operator would be a lot like the builtin eval--it'd be
> really tempting to do the "quick and easy" hack of dropping in some
> executable string instead of thinking how to structure things such
> that that could be avoided, but putting in this effort leads to more
> comprehensible code.
>

It's served your *C* and faster-python users well.  If you had proper
constructs, I'm sure people wouldn't choose to do it with inline_c unless
there was a compelling solid reasoning.

I'm not saying you can't make those constructs and that users wouldn't use
them when they appear, but you are not being pragmatic again.  You
currently don't have all capabilities and are on risky turf for supporting
all c++ standards for the rest of time.  I'm betting against that you will
produce in a useful timeframe (maybe this a 5 year scale?) the usable
constructs needed - which I equate as turning your head away and giving the
middle finger to C++ developers.  inline_c would allow a forwards
compatible way to use anything the target c++ compiler allows with some
very minimal guarantees on Cython's side.  It is a very elegant and capable
solution to a hard problem.


>
> It's hard to say "no" to features, but I think such an introduction
> would fundamentally change Cython and how it's written for the worse.
>

I agree with the statement but I don't think you've classified the feature
correctly.

I watched one of your old talks for Sage Days 29 and at the bottom of a
slide you have "Cython is a very pragmatic project, driven by user needs".
I'm calling foul.  Go watch that video again and tell me what's changed
since 2011.

-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160821/c7dbb197/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Mon Aug 22 02:59:02 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 22 Aug 2016 18:59:02 +1200
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
Message-ID: <57BAA2B6.2070404@canterbury.ac.nz>

Robert Bradshaw wrote:
> Name mangling is done for the standard reasons--to avoid possible
> conflicts with all other symbols that may be defined.

The main reason for mangling names is that top-level declarations
in Cython belong to a module namespace, whereas C just has a
single namespace for all globals. So some way is needed to
distinguish things with the same name in different modules.

Also, a single Cython declaration often leads to several
different related things being declared in C, e.g. a class
has a PyType instance, a struct declaration, and possibly
a vtable, all of which need distinct names in the C code,
so prefixes are used to distinguish them.

It might be possible to get away without mangling a few
things, such as local variables, but for simplicity they're
treated the same as everything else.

-- 
Greg

From krthornt at uci.edu  Mon Aug 22 09:54:41 2016
From: krthornt at uci.edu (Kevin Thornton)
Date: Mon, 22 Aug 2016 13:54:41 +0000
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QDHcKr98o9MtvxtZv+LNkNuzgRUcQmU+jKJWpFqxCNyyA@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CALciEZeL=m6DvSORmH1EcSGOBUQf+XCFip77CyG3p-zkZ0v0YQ@mail.gmail.com>
 <CADiQ+QDHcKr98o9MtvxtZv+LNkNuzgRUcQmU+jKJWpFqxCNyyA@mail.gmail.com>
Message-ID: <CALciEZe6QamKmEr2ck_ii75PLXQ38PrUpAiPo1UvU=gMD4rV+g@mail.gmail.com>

On Sat, Aug 20, 2016 at 11:15 PM Robert Bradshaw <robertwb at gmail.com> wrote:

> On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt at uci.edu> wrote:
> > Hi Robert,
> >
> > Related to this point, I think there are three important features that
> > Cython would benefit from, as far as C++11 and newer are concerned.  In
> > order of what I'm guessing to be increased complexity are:
> >
> > 1. Non-type template parameters.
>
> This is actually probably the hardest :). There is a PR that's stalled
> out, probably needs some attention by both the author and us; it'd be
> great to get this in.
>
> > 2. "rvalue" references for standalone functions and class member
> functions:
> >
> >     void foo[T]( T && t )
>
> From a callers perspective, one can just declare an extern foo as
> foo(T) in Cython, it doesn't care if the actual call is made by value
> or by reference--the calling code is exactly the same.
>
> Actually implementing it for Cython-defined functions (as the callee)
> would require additional support, but would probably be pretty easy
> (implement a new type, with parsing, which generally behaves the same
> as a reference or value from Python's perspective--the actual heavy
> lifting happens in the C compiler).
>
> Want to file an issue on github for this?
>

Sure--will do.


>
> > 3. Variadic templates (member and non-member).  More generally,
> "parameter
> > packs":
> >
> >     http://en.cppreference.com/w/cpp/language/parameter_pack
>
> This'd be interesting (especially as to how it relates to changing the
> arity of functions for concrete instantiations). There's also a
> question of syntax. Maybe
>
>     void foo[A, int N, ... Types](A a, Types t):
>         ...
>
> It'd likely require #1, e.g. to be able to declare things like
> tuple_element::type such that std::tuple.get<N> would be correctly
> typed. Unfortunately seems declaring the (somewhat intuitive) API of
> std::tuple requires being able to declare a fair amount of its
> implementation.
>
> Still likely feasible though, and I think the notion of variadic
> templates is natural enough in a Python setting (which has had
> heterogeneously typed tuples since its inception, though of course not
> statically typed).
>
> > #2 is important for efficiency reasons--it isn't just a convenience
> feature.
> > Without it, Cython cannot support all the member function of std::vector,
> > etc.  The implication is that some expensive operations, like moving a
> big
> > object into a vector, require extra copies in Cython while a C++ program
> > would just move the object into the new location.
>
> It's interesting to call this an essential feature, given that C++
> itself didn't have it until recently :). Not that it can't be a big
> win, and libraries will more and more probably be designed assuming
> it.
>

Also, libraries not designed assuming it can still take advantage of it in
many  contexts.  For lots of objects, the move constructor can be
implicitly created by the constructor.

Also, this would solve some quirks for Cython.  Right now, std::unique_ptr
is supported, but the object is non-copyable.  One needs std::move in order
to have this type be fully-functional.



> > #3 seems like the hardest, but would enable std::tuple, std::bind, and
> other
> > handy things to be used in a .pyx file.
> >
> > Support for "auto" would be nice, but perhaps unrealistic.
>
> Just leave the cdef out and you basically get auto, i.e.
>
>     my_var = expr
>
> gives my_var the type of expr. If multiple assignments are made to
> my_var, it attempts to find a suitable union type that can hold all
> assignments. The one exception is for integers, where it infers the
> type to be object iff my_var is used in arithmetic expressions (that
> it can't deduce would not overflow with the finite-sized C integer
> type).
>

This is surprising to me.  Would my_var be passable to a "cdef nogil"
function?  I'll have to give this a try--thanks for the tip!


>
> > I know that PRs have been submitted, and since stalled out, on a couple
> of
> > these features.  I've tried to look at the Cython parser myself, but I'm
> not
> > able to follow it, sadly.
> >
> > Thanks for all your hard work,
> >
> > Kevin
> >
> > On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb at gmail.com>
> wrote:
> >>
> >>
> >> Though, as you discovered, there are some basic things like non-type
> >> template arguments that we would like to have. If there are other
> >> specific C++ constructs that are commonly used but impossible to
> >> express in Cython it'd be useful to know.
> >>
> >>
> >> _______________________________________________
> >> cython-devel mailing list
> >> cython-devel at python.org
> >> https://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160822/bb3f1b17/attachment.html>

From krthornt at uci.edu  Mon Aug 22 10:23:32 2016
From: krthornt at uci.edu (Kevin Thornton)
Date: Mon, 22 Aug 2016 14:23:32 +0000
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QDHcKr98o9MtvxtZv+LNkNuzgRUcQmU+jKJWpFqxCNyyA@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CALciEZeL=m6DvSORmH1EcSGOBUQf+XCFip77CyG3p-zkZ0v0YQ@mail.gmail.com>
 <CADiQ+QDHcKr98o9MtvxtZv+LNkNuzgRUcQmU+jKJWpFqxCNyyA@mail.gmail.com>
Message-ID: <CALciEZcWOK15VuZ-8vsgpjB6EoK2O4OcSvj4_imYVTBpNKLS1Q@mail.gmail.com>

On Sat, Aug 20, 2016 at 11:15 PM Robert Bradshaw <robertwb at gmail.com> wrote:

> On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt at uci.edu> wrote:
> > Hi Robert,
> >
> > Related to this point, I think there are three important features that
> > Cython would benefit from, as far as C++11 and newer are concerned.  In
> > order of what I'm guessing to be increased complexity are:
> >
> > 1. Non-type template parameters.
>
> This is actually probably the hardest :). There is a PR that's stalled
> out, probably needs some attention by both the author and us; it'd be
> great to get this in.
>
> > 2. "rvalue" references for standalone functions and class member
> functions:
> >
> >     void foo[T]( T && t )
>
> From a callers perspective, one can just declare an extern foo as
> foo(T) in Cython, it doesn't care if the actual call is made by value
> or by reference--the calling code is exactly the same.
>
> Actually implementing it for Cython-defined functions (as the callee)
> would require additional support, but would probably be pretty easy
> (implement a new type, with parsing, which generally behaves the same
> as a reference or value from Python's perspective--the actual heavy
> lifting happens in the C compiler).
>
> Want to file an issue on github for this?
>
> > 3. Variadic templates (member and non-member).  More generally,
> "parameter
> > packs":
> >
> >     http://en.cppreference.com/w/cpp/language/parameter_pack
>
> This'd be interesting (especially as to how it relates to changing the
> arity of functions for concrete instantiations). There's also a
> question of syntax. Maybe
>
>     void foo[A, int N, ... Types](A a, Types t):
>         ...
>
> It'd likely require #1, e.g. to be able to declare things like
> tuple_element::type such that std::tuple.get<N> would be correctly
> typed. Unfortunately seems declaring the (somewhat intuitive) API of
> std::tuple requires being able to declare a fair amount of its
> implementation.
>
> Still likely feasible though, and I think the notion of variadic
> templates is natural enough in a Python setting (which has had
> heterogeneously typed tuples since its inception, though of course not
> statically typed).
>
> > #2 is important for efficiency reasons--it isn't just a convenience
> feature.
> > Without it, Cython cannot support all the member function of std::vector,
> > etc.  The implication is that some expensive operations, like moving a
> big
> > object into a vector, require extra copies in Cython while a C++ program
> > would just move the object into the new location.
>
> It's interesting to call this an essential feature, given that C++
> itself didn't have it until recently :). Not that it can't be a big
> win, and libraries will more and more probably be designed assuming
> it.
>
> > #3 seems like the hardest, but would enable std::tuple, std::bind, and
> other
> > handy things to be used in a .pyx file.
> >
> > Support for "auto" would be nice, but perhaps unrealistic.
>
> Just leave the cdef out and you basically get auto, i.e.
>
>     my_var = expr
>
> gives my_var the type of expr. If multiple assignments are made to
> my_var, it attempts to find a suitable union type that can hold all
> assignments. The one exception is for integers, where it infers the
> type to be object iff my_var is used in arithmetic expressions (that
> it can't deduce would not overflow with the finite-sized C integer
> type).
>

Wow--that does work (at least for simple cases).  It may be nice to have a
paragraph about this in the dox?  Something like "Emulating auto in C++"?
We've been doing a lot of copy/pasting of type names b/c we weren't aware
of this.


>
> > I know that PRs have been submitted, and since stalled out, on a couple
> of
> > these features.  I've tried to look at the Cython parser myself, but I'm
> not
> > able to follow it, sadly.
> >
> > Thanks for all your hard work,
> >
> > Kevin
> >
> > On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb at gmail.com>
> wrote:
> >>
> >>
> >> Though, as you discovered, there are some basic things like non-type
> >> template arguments that we would like to have. If there are other
> >> specific C++ constructs that are commonly used but impossible to
> >> express in Cython it'd be useful to know.
> >>
> >>
> >> _______________________________________________
> >> cython-devel mailing list
> >> cython-devel at python.org
> >> https://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160822/fc5c2cf5/attachment-0001.html>

From krthornt at uci.edu  Mon Aug 22 10:44:35 2016
From: krthornt at uci.edu (Kevin Thornton)
Date: Mon, 22 Aug 2016 14:44:35 +0000
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QDHcKr98o9MtvxtZv+LNkNuzgRUcQmU+jKJWpFqxCNyyA@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CALciEZeL=m6DvSORmH1EcSGOBUQf+XCFip77CyG3p-zkZ0v0YQ@mail.gmail.com>
 <CADiQ+QDHcKr98o9MtvxtZv+LNkNuzgRUcQmU+jKJWpFqxCNyyA@mail.gmail.com>
Message-ID: <CALciEZdMkq5zD5B2+O5hrJGvJYH_G2yam3hbQ_eW0kGVmwj_HQ@mail.gmail.com>

I actually just realized another nagging issue with C++/Cython: member
typedefs are not supported:

cdef vector[T]:
      ctypedef size_t size_type

These "scoped" typedefs are very widely-used.  We can live without it, and
we have been, but while we're making a list.. :)

--Kevin

On Sat, Aug 20, 2016 at 11:15 PM Robert Bradshaw <robertwb at gmail.com> wrote:

> On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt at uci.edu> wrote:
> > Hi Robert,
> >
> > Related to this point, I think there are three important features that
> > Cython would benefit from, as far as C++11 and newer are concerned.  In
> > order of what I'm guessing to be increased complexity are:
> >
> > 1. Non-type template parameters.
>
> This is actually probably the hardest :). There is a PR that's stalled
> out, probably needs some attention by both the author and us; it'd be
> great to get this in.
>
> > 2. "rvalue" references for standalone functions and class member
> functions:
> >
> >     void foo[T]( T && t )
>
> From a callers perspective, one can just declare an extern foo as
> foo(T) in Cython, it doesn't care if the actual call is made by value
> or by reference--the calling code is exactly the same.
>
> Actually implementing it for Cython-defined functions (as the callee)
> would require additional support, but would probably be pretty easy
> (implement a new type, with parsing, which generally behaves the same
> as a reference or value from Python's perspective--the actual heavy
> lifting happens in the C compiler).
>
> Want to file an issue on github for this?
>
> > 3. Variadic templates (member and non-member).  More generally,
> "parameter
> > packs":
> >
> >     http://en.cppreference.com/w/cpp/language/parameter_pack
>
> This'd be interesting (especially as to how it relates to changing the
> arity of functions for concrete instantiations). There's also a
> question of syntax. Maybe
>
>     void foo[A, int N, ... Types](A a, Types t):
>         ...
>
> It'd likely require #1, e.g. to be able to declare things like
> tuple_element::type such that std::tuple.get<N> would be correctly
> typed. Unfortunately seems declaring the (somewhat intuitive) API of
> std::tuple requires being able to declare a fair amount of its
> implementation.
>
> Still likely feasible though, and I think the notion of variadic
> templates is natural enough in a Python setting (which has had
> heterogeneously typed tuples since its inception, though of course not
> statically typed).
>
> > #2 is important for efficiency reasons--it isn't just a convenience
> feature.
> > Without it, Cython cannot support all the member function of std::vector,
> > etc.  The implication is that some expensive operations, like moving a
> big
> > object into a vector, require extra copies in Cython while a C++ program
> > would just move the object into the new location.
>
> It's interesting to call this an essential feature, given that C++
> itself didn't have it until recently :). Not that it can't be a big
> win, and libraries will more and more probably be designed assuming
> it.
>
> > #3 seems like the hardest, but would enable std::tuple, std::bind, and
> other
> > handy things to be used in a .pyx file.
> >
> > Support for "auto" would be nice, but perhaps unrealistic.
>
> Just leave the cdef out and you basically get auto, i.e.
>
>     my_var = expr
>
> gives my_var the type of expr. If multiple assignments are made to
> my_var, it attempts to find a suitable union type that can hold all
> assignments. The one exception is for integers, where it infers the
> type to be object iff my_var is used in arithmetic expressions (that
> it can't deduce would not overflow with the finite-sized C integer
> type).
>
> > I know that PRs have been submitted, and since stalled out, on a couple
> of
> > these features.  I've tried to look at the Cython parser myself, but I'm
> not
> > able to follow it, sadly.
> >
> > Thanks for all your hard work,
> >
> > Kevin
> >
> > On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb at gmail.com>
> wrote:
> >>
> >>
> >> Though, as you discovered, there are some basic things like non-type
> >> template arguments that we would like to have. If there are other
> >> specific C++ constructs that are commonly used but impossible to
> >> express in Cython it'd be useful to know.
> >>
> >>
> >> _______________________________________________
> >> cython-devel mailing list
> >> cython-devel at python.org
> >> https://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160822/942481d7/attachment.html>

From robertwb at gmail.com  Mon Aug 22 15:26:35 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 22 Aug 2016 12:26:35 -0700
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CAGou9MjDw9M4EmBn4jTe-B=kh5xYjUq=rBYjCOpHDip7Bf6h-g@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
 <CAGou9MjDw9M4EmBn4jTe-B=kh5xYjUq=rBYjCOpHDip7Bf6h-g@mail.gmail.com>
Message-ID: <CADiQ+QDid2KjCpDDmnkVAVy1dRgi6AKyqgYX2OiE+Ke86uiubw@mail.gmail.com>

On Sun, Aug 21, 2016 at 7:26 AM, Jason Newton <nevion at gmail.com> wrote:
>
> On Sun, Aug 21, 2016 at 5:30 AM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>
>> In my experience Cython has generally been fairly easy to pick up for
>> people who already know Python. And Python often easy to pick up for
>> people who already know C/C++. Of course for many wrappings it often
>> takes non-trivial knowledge of the wrapped library itself too, but
>> typically at the same level as would be required to grok code written
>> against that same library directly from C/C++.
>
> Is your experience drawn from binding moderately complex libraries or play
> code (of complexity like from a tutorial).  Bottom up or top down?  Sorry if
> this sounds asinine to you.

A whole variety: from simple string manipulation libraries to complex
ML and big data processing frameworks. And second hand with complete
amateurs and students to senior software engineers.

> To clarify, I'm coming from the case where I
> didn't read the whole tutorial/docs before being faced with pyx in the
> projects I previously mentioned, while on tight turn around time - I was not
> able to grok in that context.

Understood--I am short on time too. Though your self-admitted lack of
familiarity doesn't bolster your argument that Cython's doing it
wrong, we need this new feature.

> For myself and any other ML user who comes across this thread - can you list
> a few libraries that do things the right way?
>
>> Yes, there's bad code out there in any language (no offense meant
>> towards h5py--I haven't looked at that project myself). Much of it due
>> to cargo-cult perpetuations or archaic (or simply flat-out-wrong)
>> contortions due to historical limitations (e.g. creating a Python
>> module to wrap an _extension module, avoiding all C++ features with
>> extensive C wrappers, ...). (You're familiar with C++, so likely no
>> stranger to this effect.)
>>
>> > These projects complied with Cython's current philosophy
>> > to the degradation of clarity, context, and overall idea of how code was
>> > hooked up.  Perhaps Cython should take the lessons learned from it's
>> > inception, time, and the results of the state of the c-python userbase
>> > to
>> > guide us into a new philosophy.
>>
>> I fail to see how "staying close to Python" caused "degradation of
>> clarity, context, etc." If anything, the lessons learned over time
>> have validated this philosophy. More on this later.
>
>
> My point was that multifile multi-level wrapper that I mentioned earlier -
> if you're saying that those projects did Cython extensions wrong, then I'm
> incorrect at faulting Cython and should fault the libraries using it.  I
> didn't say staying close to python caused $blurb.
>
> I don't know in a situation as confusing as to all the binding projects if
> this should be taken as validation of philosophy either - I think it is
> reasonable to consider the attrition of these projects as a function of
> manpower, number of early on project supporters/authors, and if a project
> (like sage) indirectly, through dependency,  kept the project alive.  And
> good old fashioned luck.  I noted most of them don't use distutils and
> something custom but less capable instead which maybe plays a roll in how
> mature/usable/smalltime they where/are.

Certainly the success of a project depends on many external factors,
and even raw luck plays a part. But the approach and philosophy taken
to attack a problem and guard its API (and the users/contributors that
such decisions attract or repel) can't be discounted either,
especially when taken over a long timeframe.

But if anyone wants to believe that Cython's become popular because of
pure luck despite wrongheaded guiding principles or philosophies,
it'll be difficult to persuade them otherwise.

>> I agree that any efforts to trying to parsing C++ without building on
>> an actual compiler are fraught with danger. That's not the case with
>> generating C++ code, which is the direction we're going. In
>> particular, our goal is to understand C++ enough to invoke it, which
>> allows us to be much less pedantic.
>
> I understand and agree with the logic in stating it's a less complicated
> goal but what comparable success stories exist?  I strongly think "devils in
> the details" in correctly making that work and that they will be tough
> solvable problems.  And then you're going and promising on unfamiliar
> territory.  But what the ultimate takeaway for me is that you won't have it
> ready in any near term.  Do you have the skills and resources to implement
> this in under 2 years?  And then the other question is are you and the team
> reasonably confident you will have it working and usable by then.  Otherwise
> you are not being pragmatic.
>
> On the other hand, if it was reasonably simple as many of your other points
> in future emails point out, I'd really like to know why you hadn't addressed
> them earlier.

Other higher priority items for limited resources. And non-type
template args are not necessarily that simple given the way things are
structured now.

>> >> > The idea is that Cython glue makes the playing field for extracting
>> >> > data
>> >> > easy, but that once it's extracted to a cdef variable for instance,
>> >> > cython
>> >> > doesn't need to know what happens.  Maybe in a way sort of like the
>> >> > GCC
>> >> > asm
>> >> > extension.  Hopefully simpler variable passing though.
>> >>
>> >> Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any
>> >> possible conflicts. Specifying what/how to mangle could get as ugly as
>> >> GCC's asm variable passing. And embedded variable declarations, let
>> >> alone control flow statements (especially return, break, ...) could
>> >> get really messy. It obscures analysis Cython can do on the code, such
>> >> as whether variables are used or what values they may take. Little
>> >> code snippets are not always local either, e.g. do they often need to
>> >> refer to variables (or headers) referenced elsewhere. And they must
>> >> all be mutually compatible.
>> >
>> > Like gcc's asm, let's let adults do what they want and let them worry
>> > about
>> > the consequences of flow control/stray includes. I'm not even sure how
>> > most
>> > of this would be an issue (switch/break/if) if you are properly nesting
>> > pyxd
>> > output.  The only thing I think is an issue here is mangled names.  I
>> > haven't yet figured out why (cdef) variable names must be mangled.  Can
>> > you
>> > explain?  Maybe we add an option to allow it to be unmangled in their
>> > declaration? C++ has extern "C" for example.
>>
>> Name mangling is done for the standard reasons--to avoid possible
>> conflicts with all other symbols that may be defined. E.g. We don't
>> want things to suddenly break if I happen to create a variable called
>> "PyNone." Or "__pyx_something_we_defined_implicitly." And of course we
>> want to mangle globals, function names, etc. lest they conflict with
>> some otherwise irrelevant symbol defined in some (possibly
>> recursively) included header somewhere.
>>
>> Again, you could just say "Don't name things like that." This exposes
>> some more guiding principles. (1) If it's valid Python, it should be
>> valid Cython and (2) we always try to produce valid C code--if you
>> haven't lied to us (too much) about your external declarations, a
>> successful Cython compilation results in a valid C/C++ output. Also
>> (3) you shouldn't have to read or understand the generated C and the
>> Python/C API to use, let alone debug, Cython (though you're happy to
>> do so if you want, like Java developers sometimes read bytecodes, but
>> not usually, though understanding implementation can sometimes be
>> helpful when chasing performance (for all languages)).
>>
>> There's an obvious tension between giving users all the rope they want
>> vs. providing an API that is possibly more restrictive, but inherently
>> correct by construction. I'll concede that Cython necessarily has
>> pointers, so I'll give that there's plenty of room for foot-shooting
>> (and better interfacing with modern C++ would be good help there), but
>> the kind of errors one runs into by injecting arbitrary code snippets
>> take things to a whole new level (and specifically violate (3) when
>> developing and debugging).
>
> I think injecting arbitrary code snippets has a reasonably good probably of
> not breaking 3 in your above, provided we have a way to get at unmangled
> identifiers (*or* document and stick to the mangling strategy, assuming it's
> easy) - that or we scan the snippet code and replace identifiers
> (significantly more complex, instincts make me think fragile for a while
> until it's gotten right - esp without LLVM).  Perhaps syntax errors would be
> an issue if you're just coding things up... but again there's the opt-in to
> this construct and we could make life easier by annotating the snippet in
> the output - to help localize the user.

You're missing the point of (3). The fact that we're generating C
code, and not fortran or directly assembly, should mostly be an
implementation detail. It's not realistic to embed snippets without
caring about the surrounding context in all but the simplest of cases.
And there'd be feature creep here--you're in the middle of a C snippet
and want to report an error, or access a Python object, or ...

>> The escape hatch is to wrap the C++ in an actual C++ file and invoke
>> the wrapping. Typically this is the minority of one's code--if it's
>> the "whole library" then you probably have an API that's only
>> understandable to someone well versed in C++ anyways. You've given a
>> single example (non-type template arguments) that we would like to
>> support that's blocking you.
>
> My lack of examples is due to insufficient time playing with Cython - I hit
> nonstarters so I stop and abandon; as I said, to date, Cython has never been
> able to solve my C++ problems and none of them seem extraordinary.  I think
> you've got alot of more unknown-unknowns here than you give credit to but we
> can't discover that until you at least fix that template bug (properly).

It would be helpful for you to enumerate the (implied many)
nonstarters and problems you've had, to at least get the
known-unknowns out on the table.

> I'm still not looking forward to forward declaring every
> identifier/function/whatever from C++ land in Cython though

This is largely a separate issue, and I agree a big pain point.

> and I still
> strongly dislike that there's no single source way of doing Cython with
> something like a kernel/ufunc that needs to escape to C/C++.  This makes
> doing something like the mako based templates I mentioned in the OP email
> much more cumbersome/hard and Cython would provide no built in mechanism
> (like inline/inline_module) for making that work.

The issue here that you want to inline C code snippets into your
inlined Cython code snippet?

>> > Why is allowing arbitrary code inside not a good idea?  We're not
>> > talking
>> > something necessarily like eval here and the reputation it got,
>>
>> Actually, I think it's a whole lot like eval. It's taking an opaque
>> (string) chunk of data and executing it as code. But potentially worse
>> as it's in a different language and evaluated in a transformed (even
>> if the names were unmangled) context.
>>
>> If we were to go this direction, I might go with a function call (like
>> weave, maybe even follow it) rather than a new statement as the latter
>> is difficult to extend with the myriad of optional configuration
>> parameters, etc. that would beg to follow.
>
> My point with eval is it's bad reputation was mostly due to security
> vulnerabilities and it joined the league of evil like goto and other tools
> that are great in the right hands and times.  The string is known and fixed
> at pyx body cython compile time which is one of the things that made me
> think it had to be a statement rather than a function.  But again, I'm not a
> cython developer.  If you think an inlining function that takes in all the
> args it needs to inline successfully is the mechanism that achieves the
> effect on pyx compile, I don't think I'd mind.   It sure is a privileged and
> weird function though, to be able to emit code and not be a runtime
> statement.

My intuition is that the "interface" would need more complicated
specification which will grow over time--weave.inline itself takes 27
parameters.

> 2 Examples I wanted to pull out real quick are:
>
> https://github.com/scipy/weave/blob/master/examples/wx_example.py
> https://github.com/scipy/weave/blob/master/examples/binary_search.py
>
> But I think I detect magic going on here for adding includes in the wx
> example and that is not usable/reliable approach.  The cython.inline
> implementation also did include magic for numpy variables... Just a warning.
> I don't think weave failed because of Python 3 support, I think it was
> because it was too limited to be useful because of that magic and the walls
> around getting something say like Eigen in, so nobody used it.
>
>>
>> > You must realize that almost any other python driven way to compile
>> > c-code
>> > in the spirit these projects do is deprecated/dead.  Cython has absorbed
>> > all
>> > the reputation and users that didn't go to pure-c/boost.python -
>> > pybind11 is
>> > the new kid on the block there so I'm not including it (I'm of the
>> > opinion
>> > that SWIG users stayed unchanged).  Community belief/QA/designers/google
>> > all
>> > think of Cython first.  Weave has effectively closed up it's doors and
>> > I'm
>> > not even sure it had the power to do what I wanted anyway because Cython
>> > provides a language that eases the data-extraction/typecasting part of
>> > inlining C/C++.
>>
>> You seem to be repeatedly bringing up the points[:]
>>
>> * Many (most?) of these string-based approaches are essentially dead,
>> often pointing people to Cython instead, but
>> * Cython should adopt the string-embedding approach of these earlier
>> projects.
>
> Hoho - zing!  No that is not a conclusion you should be drawing.   Your
> faults here are to imply those projects failed because they used
> string-embedding approaches and to imply string-embedding based approaches
> are the approaches that failed - *most* have failed over a variety of
> implementations both Python driven and not.  I restricted to python-driven
> for the sake of brevity and mentioned the selection. As I tried to hint
> earlier, the several other projects failure happened because of any number
> of unrelated reasons to it being a string based approach.  I believe
> additionally that there were too many options (confusion) and too small a
> potential userbase (at least in those years) to bolster and attract blood to
> each of the projects and make them thrive.  Probably something akin to the
> ton of orphaned projects on pipi, it doesn't mean it happened because the
> approach was wrong.  One thing I did want you to take away is that Cython
> needs to absorb the responsibility of it's reputation and status - the last
> survivor of a somewhat diverse class with different capabilities, if you
> will, that went outside of your original usage.

I'm not saying that these projects died because of the string-based
approach they took, rather that if this "embed strings" approach were
so critical, so superior, it should at least kept one of them alive.
Or a new project could have formed around this approach (e.g. letting
all the executable code be C++, with Python syntax for the structure,
could be an interesting point in the design space). It has its pros
and cons. I think for Cython it's the wrong direction. But I'm in
favor of letting many flowers bloom--and we're in luck that these are
all open source to boot.

>> You ask at the beginning of the email whether time has vindicated our
>> philosophy. I think, based on the mindshare vs. these other attempts
>> at integrating with C, in large part it has. It has served us and our
>> users well; we will strive to stay close to Python.
>>
>> Tight interleaving of multiple languages in is cute for making a
>> polyglot script, but I do not think it leads to legible code. An
>> "eval_cpp" operator would be a lot like the builtin eval--it'd be
>> really tempting to do the "quick and easy" hack of dropping in some
>> executable string instead of thinking how to structure things such
>> that that could be avoided, but putting in this effort leads to more
>> comprehensible code.
>
> It's served your *C* and faster-python users well.  If you had proper
> constructs, I'm sure people wouldn't choose to do it with inline_c unless
> there was a compelling solid reasoning.
>
> I'm not saying you can't make those constructs and that users wouldn't use
> them when they appear, but you are not being pragmatic again.  You currently
> don't have all capabilities and are on risky turf for supporting all c++
> standards for the rest of time.  I'm betting against that you will produce
> in a useful timeframe (maybe this a 5 year scale?) the usable constructs
> needed - which I equate as turning your head away and giving the middle
> finger to C++ developers.  inline_c would allow a forwards compatible way to
> use anything the target c++ compiler allows with some very minimal
> guarantees on Cython's side.  It is a very elegant and capable solution to a
> hard problem.
>
>> It's hard to say "no" to features, but I think such an introduction
>> would fundamentally change Cython and how it's written for the worse.
>
> I agree with the statement but I don't think you've classified the feature
> correctly.

My conclusion is based on thinking a lot about where this feature
would lead us, not just the immediate possibilities it would open up.

> I watched one of your old talks for Sage Days 29 and at the bottom of a
> slide you have "Cython is a very pragmatic project, driven by user needs".
> I'm calling foul.  Go watch that video again and tell me what's changed
> since 2011.

Embedding code is not a need, it's a means to an end. This is why I've
been asking for concrete features that are the most important to try
to support.

- Robert

From robertwb at gmail.com  Mon Aug 22 15:27:29 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 22 Aug 2016 12:27:29 -0700
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CALciEZdMkq5zD5B2+O5hrJGvJYH_G2yam3hbQ_eW0kGVmwj_HQ@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CALciEZeL=m6DvSORmH1EcSGOBUQf+XCFip77CyG3p-zkZ0v0YQ@mail.gmail.com>
 <CADiQ+QDHcKr98o9MtvxtZv+LNkNuzgRUcQmU+jKJWpFqxCNyyA@mail.gmail.com>
 <CALciEZdMkq5zD5B2+O5hrJGvJYH_G2yam3hbQ_eW0kGVmwj_HQ@mail.gmail.com>
Message-ID: <CADiQ+QB7Mv2=Ahohq6zh9_mQHH_EcrE1OYYH=Geebe4ppSNjPA@mail.gmail.com>

On Mon, Aug 22, 2016 at 7:44 AM, Kevin Thornton <krthornt at uci.edu> wrote:
> I actually just realized another nagging issue with C++/Cython: member
> typedefs are not supported:
>
> cdef vector[T]:
>       ctypedef size_t size_type
>
> These "scoped" typedefs are very widely-used.  We can live without it, and
> we have been, but while we're making a list.. :)

Yes, that's a good one. Worth an issue :).

> On Sat, Aug 20, 2016 at 11:15 PM Robert Bradshaw <robertwb at gmail.com> wrote:
>>
>> On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt at uci.edu> wrote:
>> > Hi Robert,
>> >
>> > Related to this point, I think there are three important features that
>> > Cython would benefit from, as far as C++11 and newer are concerned.  In
>> > order of what I'm guessing to be increased complexity are:
>> >
>> > 1. Non-type template parameters.
>>
>> This is actually probably the hardest :). There is a PR that's stalled
>> out, probably needs some attention by both the author and us; it'd be
>> great to get this in.
>>
>> > 2. "rvalue" references for standalone functions and class member
>> > functions:
>> >
>> >     void foo[T]( T && t )
>>
>> From a callers perspective, one can just declare an extern foo as
>> foo(T) in Cython, it doesn't care if the actual call is made by value
>> or by reference--the calling code is exactly the same.
>>
>> Actually implementing it for Cython-defined functions (as the callee)
>> would require additional support, but would probably be pretty easy
>> (implement a new type, with parsing, which generally behaves the same
>> as a reference or value from Python's perspective--the actual heavy
>> lifting happens in the C compiler).
>>
>> Want to file an issue on github for this?
>>
>> > 3. Variadic templates (member and non-member).  More generally,
>> > "parameter
>> > packs":
>> >
>> >     http://en.cppreference.com/w/cpp/language/parameter_pack
>>
>> This'd be interesting (especially as to how it relates to changing the
>> arity of functions for concrete instantiations). There's also a
>> question of syntax. Maybe
>>
>>     void foo[A, int N, ... Types](A a, Types t):
>>         ...
>>
>> It'd likely require #1, e.g. to be able to declare things like
>> tuple_element::type such that std::tuple.get<N> would be correctly
>> typed. Unfortunately seems declaring the (somewhat intuitive) API of
>> std::tuple requires being able to declare a fair amount of its
>> implementation.
>>
>> Still likely feasible though, and I think the notion of variadic
>> templates is natural enough in a Python setting (which has had
>> heterogeneously typed tuples since its inception, though of course not
>> statically typed).
>>
>> > #2 is important for efficiency reasons--it isn't just a convenience
>> > feature.
>> > Without it, Cython cannot support all the member function of
>> > std::vector,
>> > etc.  The implication is that some expensive operations, like moving a
>> > big
>> > object into a vector, require extra copies in Cython while a C++ program
>> > would just move the object into the new location.
>>
>> It's interesting to call this an essential feature, given that C++
>> itself didn't have it until recently :). Not that it can't be a big
>> win, and libraries will more and more probably be designed assuming
>> it.
>>
>> > #3 seems like the hardest, but would enable std::tuple, std::bind, and
>> > other
>> > handy things to be used in a .pyx file.
>> >
>> > Support for "auto" would be nice, but perhaps unrealistic.
>>
>> Just leave the cdef out and you basically get auto, i.e.
>>
>>     my_var = expr
>>
>> gives my_var the type of expr. If multiple assignments are made to
>> my_var, it attempts to find a suitable union type that can hold all
>> assignments. The one exception is for integers, where it infers the
>> type to be object iff my_var is used in arithmetic expressions (that
>> it can't deduce would not overflow with the finite-sized C integer
>> type).
>>
>> > I know that PRs have been submitted, and since stalled out, on a couple
>> > of
>> > these features.  I've tried to look at the Cython parser myself, but I'm
>> > not
>> > able to follow it, sadly.
>> >
>> > Thanks for all your hard work,
>> >
>> > Kevin
>> >
>> > On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb at gmail.com>
>> > wrote:
>> >>
>> >>
>> >> Though, as you discovered, there are some basic things like non-type
>> >> template arguments that we would like to have. If there are other
>> >> specific C++ constructs that are commonly used but impossible to
>> >> express in Cython it'd be useful to know.
>> >>
>> >>
>> >> _______________________________________________
>> >> cython-devel mailing list
>> >> cython-devel at python.org
>> >> https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From nevion at gmail.com  Mon Aug 22 19:48:35 2016
From: nevion at gmail.com (Jason Newton)
Date: Mon, 22 Aug 2016 19:48:35 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QDid2KjCpDDmnkVAVy1dRgi6AKyqgYX2OiE+Ke86uiubw@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
 <CAGou9MjDw9M4EmBn4jTe-B=kh5xYjUq=rBYjCOpHDip7Bf6h-g@mail.gmail.com>
 <CADiQ+QDid2KjCpDDmnkVAVy1dRgi6AKyqgYX2OiE+Ke86uiubw@mail.gmail.com>
Message-ID: <CAGou9MiHVoahDC=MQK9wFbjk0fxHpaAttJBzdoJc3iPRc0YVpQ@mail.gmail.com>

On Mon, Aug 22, 2016 at 3:26 PM, Robert Bradshaw <robertwb at gmail.com> wrote:

> > To clarify, I'm coming from the case where I
> > didn't read the whole tutorial/docs before being faced with pyx in the
> > projects I previously mentioned, while on tight turn around time - I was
> not
> > able to grok in that context.
>
> Understood--I am short on time too. Though your self-admitted lack of
> familiarity doesn't bolster your argument that Cython's doing it
> wrong, we need this new feature.
>

Bare in mind that was then, and while I'm not Cython developer level I do
have a better idea of how Cython works now, possibly better than your
average user.  I've thought about this problem and the state of the overall
C++ python binding capability alot as I've had to reevaluate it possibly
more than most, and I've used them more deeply than playcode.



> > For myself and any other ML user who comes across this thread - can you
> list
> > a few libraries that do things the right way?
>
>



> > My point was that multifile multi-level wrapper that I mentioned earlier
> -
> > if you're saying that those projects did Cython extensions wrong, then
> I'm
> > incorrect at faulting Cython and should fault the libraries using it.  I
> > didn't say staying close to python caused $blurb.
> >
> > I don't know in a situation as confusing as to all the binding projects
> if
> > this should be taken as validation of philosophy either - I think it is
> > reasonable to consider the attrition of these projects as a function of
> > manpower, number of early on project supporters/authors, and if a project
> > (like sage) indirectly, through dependency,  kept the project alive.  And
> > good old fashioned luck.  I noted most of them don't use distutils and
> > something custom but less capable instead which maybe plays a roll in how
> > mature/usable/smalltime they where/are.
>
> Certainly the success of a project depends on many external factors,
> and even raw luck plays a part. But the approach and philosophy taken
> to attack a problem and guard its API (and the users/contributors that
> such decisions attract or repel) can't be discounted either,
> especially when taken over a long timeframe.
>
> But if anyone wants to believe that Cython's become popular because of
> pure luck despite wrongheaded guiding principles or philosophies,
> it'll be difficult to persuade them otherwise.
>

You snidely play with extremes, I never said pure luck was the main factor
- I listed several factors of very high significance.  Further, Cython
branched off of pyrex - why do you think that project failed (you might
actually know)?  Did your philosophy carry separate the two or was it
because you needed things to work (for sage), had a larger set of smart
developers and improve it faster than pyrex would allow?  I think it's the
bustling of life that got Cython where it is, I don't believe it's
philosophy in this case - I think it was lack of feature competitiveness
and Cython probably worked better (more bugs/cases fixed).


>
> >> I agree that any efforts to trying to parsing C++ without building on
> >> an actual compiler are fraught with danger. That's not the case with
> >> generating C++ code, which is the direction we're going. In
> >> particular, our goal is to understand C++ enough to invoke it, which
> >> allows us to be much less pedantic.
> >
> > I understand and agree with the logic in stating it's a less complicated
> > goal but what comparable success stories exist?  I strongly think
> "devils in
> > the details" in correctly making that work and that they will be tough
> > solvable problems.  And then you're going and promising on unfamiliar
> > territory.  But what the ultimate takeaway for me is that you won't have
> it
> > ready in any near term.  Do you have the skills and resources to
> implement
> > this in under 2 years?  And then the other question is are you and the
> team
> > reasonably confident you will have it working and usable by then.
> Otherwise
> > you are not being pragmatic.
> >
> > On the other hand, if it was reasonably simple as many of your other
> points
> > in future emails point out, I'd really like to know why you hadn't
> addressed
> > them earlier.
>
> Other higher priority items for limited resources. And non-type
> template args are not necessarily that simple given the way things are
> structured now.


There's always limited resources and priorities but hen it sounds like it's
going to be a very (ever)? long time until everything is working great
unless the need is becoming a clear driver or this email thread has
reassigned priorities.  Do you think you can sketch out a time table for
the known issues?  Maybe not in this thread but I'd appreciate a back of
the envelope calculation - realistic time of usefulness is part of being
pragmatic.


>
> You're missing the point of (3). The fact that we're generating C
> code, and not fortran or directly assembly, should mostly be an
> implementation detail. It's not realistic to embed snippets without
> caring about the surrounding context in all but the simplest of cases.
> And there'd be feature creep here--you're in the middle of a C snippet
> and want to report an error, or access a Python object, or ...
>

I know you took a different philosophy than pyrex, but I don't understand
why.  In that sage days presentation, you mentioned pyrex chose to
distinguish itself as a way to write c extensions, emphasizing it (I don't
know to the degree if it offered what wish cython offered, doubted).  It's
pretty clear to me you wanted mostly accelerated python with some
c-trampolining capability through externally defined functions.  I *really*
think you should remember your root's goals though.



>
> >> The escape hatch is to wrap the C++ in an actual C++ file and invoke
> >> the wrapping. Typically this is the minority of one's code--if it's
> >> the "whole library" then you probably have an API that's only
> >> understandable to someone well versed in C++ anyways. You've given a
> >> single example (non-type template arguments) that we would like to
> >> support that's blocking you.
> >
> > My lack of examples is due to insufficient time playing with Cython - I
> hit
> > nonstarters so I stop and abandon; as I said, to date, Cython has never
> been
> > able to solve my C++ problems and none of them seem extraordinary.  I
> think
> > you've got alot of more unknown-unknowns here than you give credit to
> but we
> > can't discover that until you at least fix that template bug (properly).
>
> It would be helpful for you to enumerate the (implied many)
> nonstarters and problems you've had, to at least get the
> known-unknowns out on the table.
>

Honestly I can't remember the other 2 times I reevaluated its applicability
(we're talking a few days each over the last ~10 years), it was too long
ago and capability of Cython has grown right?  My answer in each of those
times, originally coming from SWIG, was boost.python... and in the mean
time it's looking like it might be pybind11 (I don't have the luxury of
waiting for what I'm working on, but I'm not forgetting about Cython
either).


>
> > I'm still not looking forward to forward declaring every
> > identifier/function/whatever from C++ land in Cython though
>
> This is largely a separate issue, and I agree a big pain point.
>
> > and I still
> > strongly dislike that there's no single source way of doing Cython with
> > something like a kernel/ufunc that needs to escape to C/C++.  This makes
> > doing something like the mako based templates I mentioned in the OP email
> > much more cumbersome/hard and Cython would provide no built in mechanism
> > (like inline/inline_module) for making that work.
>
> The issue here that you want to inline C code snippets into your
> inlined Cython code snippet?
>

Multiple files is more a PITA to making the mako-templated strategy work -
I didn't say impossible, just more a PITA. I learned the JIT-module
compilation technique from OpenCL/PyCUDA and found it makes a metric ton of
sense to do, the performance boost can be massive from using -DDEFINES and
embedding runtime variables/constants right into C (recall the point of
Cython seems to be for speed :-). This is one of the reasons why I'd like
to stick with a python driven way of doing C++ bindings (which hopefully
can use C++ directly).   It's alot easier for a single file to work and
simply pass a string to something like inline which does all the file
writing, and module placement.


> I'm not saying that these projects died because of the string-based
> approach they took, rather that if this "embed strings" approach were
> so critical, so superior, it should at least kept one of them alive.
> Or a new project could have formed around this approach (e.g. letting
> all the executable code be C++, with Python syntax for the structure,
> could be an interesting point in the design space). It has its pros
> and cons. I think for Cython it's the wrong direction. But I'm in
> favor of letting many flowers bloom--and we're in luck that these are
> all open source to boot.
>

I think it's because of their as-is-limitations, not of the technique they
implemented but as a whole system.  I looked at the source for pyinline the
other day and it looked like it only supports primitive types and would
choke on C++ for instance and doesn't really offer convenience to extract
data from python objects like Cython can so easily (the whole system,
including it's custom C (no ++) lexer is ~700 lines).  I tend to think of a
few/most of them are just passed the working proof of concept stage and
were significantly lacking in overall capability.  Most python projects (on
PyPi) are single contributors with short lifespans too which seems to be
the case for those projects.  I'm not sure when/how Pyrex started out (1
guy at Uni who set up a full website?) but how did it happen with Cython?


> > I watched one of your old talks for Sage Days 29 and at the bottom of a
> > slide you have "Cython is a very pragmatic project, driven by user
> needs".
> > I'm calling foul.  Go watch that video again and tell me what's changed
> > since 2011.
>
> Embedding code is not a need, it's a means to an end. This is why I've
> been asking for concrete features that are the most important to try
> to support.
>

I have another idea/iteration to run by you then.  One of your chief
quibbles, although I don't think it's your underlying one, is Cython must
understand what's going on.  So how about we support a block of C/C++ code
as a proper construct.  Same name but now, I guess braces maybe get
involved (since semicolons do too).  Then to transform (mangle) the nested
identifiers in accordance with scoping/shadowing rules we pull in the
python bindings to llvm/Clang to parse that complicated code, work with the
AST (or further down the rabbit-hole) appropriately and spit it back out.
Obviously you could detect and analyze embedded statements to your hearts
content to decide to do something smart about that (you mentioned exception
handling and return statement as concerning items).

The potential hazards here are that LLVM/Clang is brought in (LLVM is
already on ~ everybody's linux boxes as a graphics shader dependency) and
it's [python] api changes somewhat often.  I've used it before with success
for python driven C++ POD boiler plate (IO, size checks, visitors methods
etc) autocoding.

If you can understand the underlying C/C++ and get over that Cython will
always run on a C/C++ compiler in our universe,  and make that a feature to
embrace, not a detail to hide (who actually benefits in the later case?) -
are you still against the significant convenience - and potentially only
way to make things work when Cython is not supporting a C++ way of doing
things (say things aren't finished/working yet)?


-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160822/bc733a76/attachment-0001.html>

From robertwb at gmail.com  Tue Aug 23 00:44:41 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 22 Aug 2016 21:44:41 -0700
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CAGou9MiHVoahDC=MQK9wFbjk0fxHpaAttJBzdoJc3iPRc0YVpQ@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
 <CAGou9MjDw9M4EmBn4jTe-B=kh5xYjUq=rBYjCOpHDip7Bf6h-g@mail.gmail.com>
 <CADiQ+QDid2KjCpDDmnkVAVy1dRgi6AKyqgYX2OiE+Ke86uiubw@mail.gmail.com>
 <CAGou9MiHVoahDC=MQK9wFbjk0fxHpaAttJBzdoJc3iPRc0YVpQ@mail.gmail.com>
Message-ID: <CADiQ+QCa9jY7dfSNpXzgOHVd5+W721F7+nzscUB62vQaYEZUUg@mail.gmail.com>

On Mon, Aug 22, 2016 at 4:48 PM, Jason Newton <nevion at gmail.com> wrote:
>
> Bare in mind that was then, and while I'm not Cython developer level I do
> have a better idea of how Cython works now, possibly better than your
> average user.  I've thought about this problem and the state of the overall
> C++ python binding capability alot as I've had to reevaluate it possibly
> more than most, and I've used them more deeply than playcode.

Good to know.

>> > My point was that multifile multi-level wrapper that I mentioned earlier
>> > -
>> > if you're saying that those projects did Cython extensions wrong, then
>> > I'm
>> > incorrect at faulting Cython and should fault the libraries using it.  I
>> > didn't say staying close to python caused $blurb.
>> >
>> > I don't know in a situation as confusing as to all the binding projects
>> > if
>> > this should be taken as validation of philosophy either - I think it is
>> > reasonable to consider the attrition of these projects as a function of
>> > manpower, number of early on project supporters/authors, and if a
>> > project
>> > (like sage) indirectly, through dependency,  kept the project alive.
>> > And
>> > good old fashioned luck.  I noted most of them don't use distutils and
>> > something custom but less capable instead which maybe plays a roll in
>> > how
>> > mature/usable/smalltime they where/are.
>>
>> Certainly the success of a project depends on many external factors,
>> and even raw luck plays a part. But the approach and philosophy taken
>> to attack a problem and guard its API (and the users/contributors that
>> such decisions attract or repel) can't be discounted either,
>> especially when taken over a long timeframe.
>>
>> But if anyone wants to believe that Cython's become popular because of
>> pure luck despite wrongheaded guiding principles or philosophies,
>> it'll be difficult to persuade them otherwise.
>
> You snidely play with extremes, I never said pure luck was the main factor -
> I listed several factors of very high significance.  Further, Cython
> branched off of pyrex - why do you think that project failed (you might
> actually know)?  Did your philosophy carry separate the two or was it
> because you needed things to work (for sage), had a larger set of smart
> developers and improve it faster than pyrex would allow?  I think it's the
> bustling of life that got Cython where it is, I don't believe it's
> philosophy in this case - I think it was lack of feature competitiveness and
> Cython probably worked better (more bugs/cases fixed).

I wouldn't say that Pyrex failed--it gave birth to Cython. At a high
level where I'd say they diverged is that Pyrex was viewed as a tool
to write C extensions, whereas Cython was viewed as a Python compiler.
The latter meant prioritizing things like being an optimizing
compiler, full coverage of the Python API (e.g. generators),
numpy/buffer integration, and Python 3 support were higher priority,
as opposed to being non-goals for Pyrex.

But the key insight of using Python, or at least Python-inspired,
syntax to declare and invoke C constructs lives on. Which is quite
relevant to this thread.

The fact that Greg didn't seem to have the time/interest in corralling
an open source project factored in here as well. If you have anything
to add, Greg, always happy to have you chime in.

>> >> I agree that any efforts to trying to parsing C++ without building on
>> >> an actual compiler are fraught with danger. That's not the case with
>> >> generating C++ code, which is the direction we're going. In
>> >> particular, our goal is to understand C++ enough to invoke it, which
>> >> allows us to be much less pedantic.
>> >
>> > I understand and agree with the logic in stating it's a less complicated
>> > goal but what comparable success stories exist?  I strongly think
>> > "devils in
>> > the details" in correctly making that work and that they will be tough
>> > solvable problems.  And then you're going and promising on unfamiliar
>> > territory.  But what the ultimate takeaway for me is that you won't have
>> > it
>> > ready in any near term.  Do you have the skills and resources to
>> > implement
>> > this in under 2 years?  And then the other question is are you and the
>> > team
>> > reasonably confident you will have it working and usable by then.
>> > Otherwise
>> > you are not being pragmatic.
>> >
>> > On the other hand, if it was reasonably simple as many of your other
>> > points
>> > in future emails point out, I'd really like to know why you hadn't
>> > addressed
>> > them earlier.
>>
>> Other higher priority items for limited resources. And non-type
>> template args are not necessarily that simple given the way things are
>> structured now.
>
> There's always limited resources and priorities but hen it sounds like it's
> going to be a very (ever)? long time until everything is working great
> unless the need is becoming a clear driver or this email thread has
> reassigned priorities.  Do you think you can sketch out a time table for the
> known issues?  Maybe not in this thread but I'd appreciate a back of the
> envelope calculation - realistic time of usefulness is part of being
> pragmatic.

For a small project like this run completely by volunteers, the only
way to set a timetable is to to the work yourself :). However, this
discussion has re-motivated me to iron some of these issues out, and I
might (surprisingly) have some time on my hands to do so.

>> You're missing the point of (3). The fact that we're generating C
>> code, and not fortran or directly assembly, should mostly be an
>> implementation detail. It's not realistic to embed snippets without
>> caring about the surrounding context in all but the simplest of cases.
>> And there'd be feature creep here--you're in the middle of a C snippet
>> and want to report an error, or access a Python object, or ...
>
> I know you took a different philosophy than pyrex, but I don't understand
> why.  In that sage days presentation, you mentioned pyrex chose to
> distinguish itself as a way to write c extensions, emphasizing it (I don't
> know to the degree if it offered what wish cython offered, doubted).  It's
> pretty clear to me you wanted mostly accelerated python with some
> c-trampolining capability through externally defined functions.  I *really*
> think you should remember your root's goals though.

Yep. "C extension" is an implementation detail. (One can't always get
away from implementation details leaking though, but I rather like to
avoid prominently baking them in.)

>> The issue here that you want to inline C code snippets into your
>> inlined Cython code snippet?
>
> Multiple files is more a PITA to making the mako-templated strategy work - I
> didn't say impossible, just more a PITA. I learned the JIT-module
> compilation technique from OpenCL/PyCUDA and found it makes a metric ton of
> sense to do, the performance boost can be massive from using -DDEFINES and
> embedding runtime variables/constants right into C (recall the point of
> Cython seems to be for speed :-). This is one of the reasons why I'd like to
> stick with a python driven way of doing C++ bindings (which hopefully can
> use C++ directly).   It's alot easier for a single file to work and simply
> pass a string to something like inline which does all the file writing, and
> module placement.

I just had one idea to ease the pain--cython.inline[_module] could
take a "extra C source" parameter that would be #included in the file.
I'm much more comfortable with this over making changes to the
language itself.

>> I'm not saying that these projects died because of the string-based
>> approach they took, rather that if this "embed strings" approach were
>> so critical, so superior, it should at least kept one of them alive.
>> Or a new project could have formed around this approach (e.g. letting
>> all the executable code be C++, with Python syntax for the structure,
>> could be an interesting point in the design space). It has its pros
>> and cons. I think for Cython it's the wrong direction. But I'm in
>> favor of letting many flowers bloom--and we're in luck that these are
>> all open source to boot.
>
>
> I think it's because of their as-is-limitations, not of the technique they
> implemented but as a whole system.  I looked at the source for pyinline the
> other day and it looked like it only supports primitive types and would
> choke on C++ for instance and doesn't really offer convenience to extract
> data from python objects like Cython can so easily (the whole system,
> including it's custom C (no ++) lexer is ~700 lines).  I tend to think of a
> few/most of them are just passed the working proof of concept stage and were
> significantly lacking in overall capability.  Most python projects (on PyPi)
> are single contributors with short lifespans too which seems to be the case
> for those projects.  I'm not sure when/how Pyrex started out (1 guy at Uni
> who set up a full website?) but how did it happen with Cython?
>
>>
>> > I watched one of your old talks for Sage Days 29 and at the bottom of a
>> > slide you have "Cython is a very pragmatic project, driven by user
>> > needs".
>> > I'm calling foul.  Go watch that video again and tell me what's changed
>> > since 2011.
>>
>> Embedding code is not a need, it's a means to an end. This is why I've
>> been asking for concrete features that are the most important to try
>> to support.
>
> I have another idea/iteration to run by you then.  One of your chief
> quibbles, although I don't think it's your underlying one, is Cython must
> understand what's going on.  So how about we support a block of C/C++ code
> as a proper construct.  Same name but now, I guess braces maybe get involved
> (since semicolons do too).  Then to transform (mangle) the nested
> identifiers in accordance with scoping/shadowing rules we pull in the python
> bindings to llvm/Clang to parse that complicated code, work with the AST (or
> further down the rabbit-hole) appropriately and spit it back out.  Obviously
> you could detect and analyze embedded statements to your hearts content to
> decide to do something smart about that (you mentioned exception handling
> and return statement as concerning items).
>
> The potential hazards here are that LLVM/Clang is brought in (LLVM is
> already on ~ everybody's linux boxes as a graphics shader dependency) and
> it's [python] api changes somewhat often.  I've used it before with success
> for python driven C++ POD boiler plate (IO, size checks, visitors methods
> etc) autocoding.
>
> If you can understand the underlying C/C++ and get over that Cython will
> always run on a C/C++ compiler in our universe,  and make that a feature to
> embrace, not a detail to hide (who actually benefits in the later case?) -
> are you still against the significant convenience - and potentially only way
> to make things work when Cython is not supporting a C++ way of doing things
> (say things aren't finished/working yet)?

That would help solve the analysis, though enormous cost in complexity
(one now has two ASTs one must understand and reconcile, and a huge
amount of C++ knowledge must be baked in to understand the AST (or
even lower levels) and it'd be neigh impossible to do "partial
analysis" for features we don't yet understand, probably harder than
putting them into Cython itself modulo parsing) and making LLVM/Clang
a dependency for Cython is probably a non-starter as well.

But you're right, it's not my primary quibble, which is that a tight
alternating of C++ and Python statements seems, well, not very
Pythonic and would IMHO lead to hard to read and debug code. That and
I think the interface (including expectations and dependencies,
possibly non-local) between the two would grow a lot messier than a
question of how to resolve name mangling. I don't think snippets would
solve the fundamental differences like about scoping discrepancies
between Python and C++ (e.g. for RAII). And it would be a step back
for the nice auto-conversion facilities you care about, e.g.

    py_func(c_func(py_value))

would become

    c_value = py_value
    cdef some_type c_ret
    inline_cpp "c_ret = c_func(c_value)"
    py_func(c_ret)

It's just choosing a smaller unit (line vs. file) to swap between languages.

- Robert

From robertwb at gmail.com  Tue Aug 23 01:03:14 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 22 Aug 2016 22:03:14 -0700
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QCa9jY7dfSNpXzgOHVd5+W721F7+nzscUB62vQaYEZUUg@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
 <CAGou9MjDw9M4EmBn4jTe-B=kh5xYjUq=rBYjCOpHDip7Bf6h-g@mail.gmail.com>
 <CADiQ+QDid2KjCpDDmnkVAVy1dRgi6AKyqgYX2OiE+Ke86uiubw@mail.gmail.com>
 <CAGou9MiHVoahDC=MQK9wFbjk0fxHpaAttJBzdoJc3iPRc0YVpQ@mail.gmail.com>
 <CADiQ+QCa9jY7dfSNpXzgOHVd5+W721F7+nzscUB62vQaYEZUUg@mail.gmail.com>
Message-ID: <CADiQ+QALmgKsAJCT-e787PmMj+Y20x-1oWdKfy2Hwq8NL3nXog@mail.gmail.com>

On Mon, Aug 22, 2016 at 9:44 PM, Robert Bradshaw <robertwb at gmail.com> wrote:

> And it would be a step back
> for the nice auto-conversion facilities you care about, e.g.
>
>     py_func(c_func(py_value))
>
> would become
>
>     c_value = py_value
>     cdef some_type c_ret
>     inline_cpp "c_ret = c_func(c_value)"
>     py_func(c_ret)
>
> It's just choosing a smaller unit (line vs. file) to swap between languages.

And on this note it would also encourage use of the Python/C API (over
swapping "back" to Python) which is often a dangerous step backwards.

From alex0player at gmail.com  Tue Aug 23 05:15:27 2016
From: alex0player at gmail.com (Alex S.)
Date: Tue, 23 Aug 2016 12:15:27 +0300
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QALmgKsAJCT-e787PmMj+Y20x-1oWdKfy2Hwq8NL3nXog@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
 <CAGou9MjDw9M4EmBn4jTe-B=kh5xYjUq=rBYjCOpHDip7Bf6h-g@mail.gmail.com>
 <CADiQ+QDid2KjCpDDmnkVAVy1dRgi6AKyqgYX2OiE+Ke86uiubw@mail.gmail.com>
 <CAGou9MiHVoahDC=MQK9wFbjk0fxHpaAttJBzdoJc3iPRc0YVpQ@mail.gmail.com>
 <CADiQ+QCa9jY7dfSNpXzgOHVd5+W721F7+nzscUB62vQaYEZUUg@mail.gmail.com>
 <CADiQ+QALmgKsAJCT-e787PmMj+Y20x-1oWdKfy2Hwq8NL3nXog@mail.gmail.com>
Message-ID: <84110049-66EB-4242-8060-DF79613117A8@gmail.com>

Watching this thread for a long time now, and I must say I absolutely love how someone just barges in ?oh just add parsing of C++ AST?.
C++ is a language with an extremely complex context-dependent grammar, much more so than C. Not to mention they took to updating the standard with new awesome features every 3 or 4  years, which all seem to absolutely need their own syntactic support.
> 23 ???. 2016 ?., ? 8:03, Robert Bradshaw <robertwb at gmail.com> ???????(?):
> 
> On Mon, Aug 22, 2016 at 9:44 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
> 
>> And it would be a step back
>> for the nice auto-conversion facilities you care about, e.g.
>> 
>>    py_func(c_func(py_value))
>> 
>> would become
>> 
>>    c_value = py_value
>>    cdef some_type c_ret
>>    inline_cpp "c_ret = c_func(c_value)"
>>    py_func(c_ret)
>> 
>> It's just choosing a smaller unit (line vs. file) to swap between languages.
> 
> And on this note it would also encourage use of the Python/C API (over
> swapping "back" to Python) which is often a dangerous step backwards.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel


From nevion at gmail.com  Tue Aug 23 08:08:47 2016
From: nevion at gmail.com (Jason Newton)
Date: Tue, 23 Aug 2016 08:08:47 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <CADiQ+QCa9jY7dfSNpXzgOHVd5+W721F7+nzscUB62vQaYEZUUg@mail.gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
 <CAGou9MjDw9M4EmBn4jTe-B=kh5xYjUq=rBYjCOpHDip7Bf6h-g@mail.gmail.com>
 <CADiQ+QDid2KjCpDDmnkVAVy1dRgi6AKyqgYX2OiE+Ke86uiubw@mail.gmail.com>
 <CAGou9MiHVoahDC=MQK9wFbjk0fxHpaAttJBzdoJc3iPRc0YVpQ@mail.gmail.com>
 <CADiQ+QCa9jY7dfSNpXzgOHVd5+W721F7+nzscUB62vQaYEZUUg@mail.gmail.com>
Message-ID: <CAGou9Mgk0TGPj7aO6Gz9=ssrZkOJcxEh+d_X5Nuv4St3ZwPYGw@mail.gmail.com>

>
> > I have another idea/iteration to run by you then.  One of your chief
> > quibbles, although I don't think it's your underlying one, is Cython must
> > understand what's going on.  So how about we support a block of C/C++
> code
> > as a proper construct.  Same name but now, I guess braces maybe get
> involved
> > (since semicolons do too).  Then to transform (mangle) the nested
> > identifiers in accordance with scoping/shadowing rules we pull in the
> python
> > bindings to llvm/Clang to parse that complicated code, work with the AST
> (or
> > further down the rabbit-hole) appropriately and spit it back out.
> Obviously
> > you could detect and analyze embedded statements to your hearts content
> to
> > decide to do something smart about that (you mentioned exception handling
> > and return statement as concerning items).
> >
> > The potential hazards here are that LLVM/Clang is brought in (LLVM is
> > already on ~ everybody's linux boxes as a graphics shader dependency) and
> > it's [python] api changes somewhat often.  I've used it before with
> success
> > for python driven C++ POD boiler plate (IO, size checks, visitors methods
> > etc) autocoding.
> >
> > If you can understand the underlying C/C++ and get over that Cython will
> > always run on a C/C++ compiler in our universe,  and make that a feature
> to
> > embrace, not a detail to hide (who actually benefits in the later case?)
> -
> > are you still against the significant convenience - and potentially only
> way
> > to make things work when Cython is not supporting a C++ way of doing
> things
> > (say things aren't finished/working yet)?
>
> That would help solve the analysis, though enormous cost in complexity
> (one now has two ASTs one must understand and reconcile, and a huge
> amount of C++ knowledge must be baked in to understand the AST (or
> even lower levels) and it'd be neigh impossible to do "partial
> analysis" for features we don't yet understand, probably harder than
> putting them into Cython itself modulo parsing) and making LLVM/Clang
> a dependency for Cython is probably a non-starter as well.
>

Well, one thing occurred to me is that leaning on Clang may be the only way
you'll be able to remove/improve the DRY violating Cython side declaration
of C/C++ land stuff.

-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160823/d6e10f87/attachment.html>

From nevion at gmail.com  Tue Aug 23 08:22:51 2016
From: nevion at gmail.com (Jason Newton)
Date: Tue, 23 Aug 2016 08:22:51 -0400
Subject: [Cython] RFC: an inline_ function that dumps c/c++ code to the
 code emitter
In-Reply-To: <84110049-66EB-4242-8060-DF79613117A8@gmail.com>
References: <CAGou9Mj7N6YDWvjno41sAAzxAMWGm436T_Kfey_zpDWFuCL4ig@mail.gmail.com>
 <CADiQ+QB3PDVke5DdAPOY98_nLyiK+Qw3WJ9Tfih5GCTuFsEnXw@mail.gmail.com>
 <CAGou9MhiQEjnHcL=pQOvajMU=xArYAjJDrhou594XfZtTGpjxg@mail.gmail.com>
 <CADiQ+QAuOMqi+c2ksV0sbqxHh_OwGCMRjb6f0d2YzWJD0gHAZA@mail.gmail.com>
 <CAGou9MjDw9M4EmBn4jTe-B=kh5xYjUq=rBYjCOpHDip7Bf6h-g@mail.gmail.com>
 <CADiQ+QDid2KjCpDDmnkVAVy1dRgi6AKyqgYX2OiE+Ke86uiubw@mail.gmail.com>
 <CAGou9MiHVoahDC=MQK9wFbjk0fxHpaAttJBzdoJc3iPRc0YVpQ@mail.gmail.com>
 <CADiQ+QCa9jY7dfSNpXzgOHVd5+W721F7+nzscUB62vQaYEZUUg@mail.gmail.com>
 <CADiQ+QALmgKsAJCT-e787PmMj+Y20x-1oWdKfy2Hwq8NL3nXog@mail.gmail.com>
 <84110049-66EB-4242-8060-DF79613117A8@gmail.com>
Message-ID: <CAGou9Mj8XdFGNZ+9U993-tSY_ZGR6VKTC9=6iDKcNX37VZZbPA@mail.gmail.com>

Just trying and improve the state of the art.  Yea maybe we have to add
some tricks to separate the two grammar wise, so they can play together.
Besides, I didn't start with that complexity, it evolved into seeing if
that was a viable path forward.

On Tue, Aug 23, 2016 at 5:15 AM, Alex S. <alex0player at gmail.com> wrote:

> Watching this thread for a long time now, and I must say I absolutely love
> how someone just barges in ?oh just add parsing of C++ AST?.
> C++ is a language with an extremely complex context-dependent grammar,
> much more so than C. Not to mention they took to updating the standard with
> new awesome features every 3 or 4  years, which all seem to absolutely need
> their own syntactic support.
> > 23 ???. 2016 ?., ? 8:03, Robert Bradshaw <robertwb at gmail.com>
> ???????(?):
> >
> > On Mon, Aug 22, 2016 at 9:44 PM, Robert Bradshaw <robertwb at gmail.com>
> wrote:
> >
> >> And it would be a step back
> >> for the nice auto-conversion facilities you care about, e.g.
> >>
> >>    py_func(c_func(py_value))
> >>
> >> would become
> >>
> >>    c_value = py_value
> >>    cdef some_type c_ret
> >>    inline_cpp "c_ret = c_func(c_value)"
> >>    py_func(c_ret)
> >>
> >> It's just choosing a smaller unit (line vs. file) to swap between
> languages.
> >
> > And on this note it would also encourage use of the Python/C API (over
> > swapping "back" to Python) which is often a dangerous step backwards.
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160823/d300d60c/attachment.html>