From njs at pobox.com Sat Jul 9 05:21:33 2016 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 9 Jul 2016 02:21:33 -0700 Subject: [Python-compilers] Workshop updates (for both local and remote attendees) Message-ID: Hi all, I've updated the web page with a set of useful links at the top: https://python-compilers-workshop.github.io/ Note in particular: - Since a number of folks have asked about attending remotely, I've arranged for what should hopefully be a decent camera/microphone setup, and there's a Google Hangouts link there that we'll use for that. - There's a shared document to use for capturing notes and discussion. It also has a more detailed preliminary schedule. Currently, I'm aware of the following projects planning to present short talks: PyPy Pyjion HOPE Numba NumPy DyND GT-Py If you're not on this list and think you should be (or vice-versa), please speak up! Looking forward to meeting you all, -n -- Nathaniel J. Smith -- https://vorpus.org From markdewing at gmail.com Sun Jul 10 01:23:44 2016 From: markdewing at gmail.com (Mark Dewing) Date: Sun, 10 Jul 2016 00:23:44 -0500 Subject: [Python-compilers] Wrapping JIT-compiled functions Message-ID: Hello, The first bullet point for the Python Compilers Workshop talks about passing JIT'ed code to wrapped C-functions. Let me describe some work on a special case, which may be useful for the discussion. The starting point was the scipy integration routines (quad, etc) that accept a native function for the integrand callback. I wrote a Sympy to LLVM JIT that would compile expressions into callbacks for scipy quad, and then extended that to several other libraries. There is a longer description here: https://markdewing.github.io/blog/posts/integration-callbacks/ The python wrappers around the Cubature and Cuba libraries required modifications to accept native code callbacks. It seems any wrapper around a native code library that uses callbacks will need code to handle a native code callback vs. a Python callback. For details, see these pull requests: https://github.com/saullocastro/cubature/pull/16 https://github.com/JohannesBuchner/PyMultiNest/pull/73/files One of the issues in generalizing this is how to scale to different libraries and callbacks without needing to program knowledge of each of them into Sympy. There needs to be some way to describe the parameters expected in the callback, but it should be at a higher level than just C types - which ones are inputs, outputs, array lengths, etc. Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sun Jul 10 12:09:40 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 10 Jul 2016 18:09:40 +0200 Subject: [Python-compilers] Wrapping JIT-compiled functions In-Reply-To: References: Message-ID: <57827344.7020909@behnel.de> Mark Dewing schrieb am 10.07.2016 um 07:23: > It seems any wrapper around > a native code library that uses > callbacks will need code to handle a native code callback vs. a Python > callback. > For details, see these pull requests: > https://github.com/saullocastro/cubature/pull/16 > https://github.com/JohannesBuchner/PyMultiNest/pull/73/files > > One of the issues in generalizing this is how to scale to different > libraries and callbacks without needing to program knowledge of each of > them into Sympy. > There needs to be some way to describe the parameters expected in the > callback, but it should be at a higher level than just C types - which ones > are inputs, outputs, array lengths, etc. A couple of years ago, we did some thinking about this in the Cython project. Here's the write-up from back then, don't remember how complete it is. It never lead anywhere since no-one stepped up to implement that protocol. https://github.com/cython/cython/wiki/enhancements-cep1000 Stefan From siu at continuum.io Sun Jul 10 16:20:24 2016 From: siu at continuum.io (Siu Kwan Lam) Date: Sun, 10 Jul 2016 20:20:24 +0000 Subject: [Python-compilers] Wrapping JIT-compiled functions In-Reply-To: <57827344.7020909@behnel.de> References: <57827344.7020909@behnel.de> Message-ID: We recently added a similar functionality to Numba. Via a @cfunc decorator, we can compile functions at runtime and expose a ctypes function object for use in other libraries. We also have an example on using scipy quad at http://numba.pydata.org/numba-doc/latest/user/cfunc.html. On Sun, Jul 10, 2016 at 11:22 AM Stefan Behnel wrote: > Mark Dewing schrieb am 10.07.2016 um 07:23: > > It seems any wrapper around > > a native code library that uses > > callbacks will need code to handle a native code callback vs. a Python > > callback. > > For details, see these pull requests: > > https://github.com/saullocastro/cubature/pull/16 > > https://github.com/JohannesBuchner/PyMultiNest/pull/73/files > > > > One of the issues in generalizing this is how to scale to different > > libraries and callbacks without needing to program knowledge of each of > > them into Sympy. > > There needs to be some way to describe the parameters expected in the > > callback, but it should be at a higher level than just C types - which > ones > > are inputs, outputs, array lengths, etc. > > A couple of years ago, we did some thinking about this in the Cython > project. Here's the write-up from back then, don't remember how complete it > is. It never lead anywhere since no-one stepped up to implement that > protocol. > > https://github.com/cython/cython/wiki/enhancements-cep1000 > > Stefan > > _______________________________________________ > Python-compilers mailing list > Python-compilers at python.org > https://mail.python.org/mailman/listinfo/python-compilers > -- Siu Kwan Lam Software Engineer Continuum Analytics -------------- next part -------------- An HTML attachment was scrubbed... URL: From yarkot1 at gmail.com Mon Jul 11 10:54:27 2016 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Mon, 11 Jul 2016 09:54:27 -0500 Subject: [Python-compilers] Workshop updates (for both local and remote attendees) In-Reply-To: References: Message-ID: Sadly the google hangouts is full. On Saturday, July 9, 2016, Nathaniel Smith wrote: > Hi all, > > I've updated the web page with a set of useful links at the top: > > https://python-compilers-workshop.github.io/ > > Note in particular: > > - Since a number of folks have asked about attending remotely, I've > arranged for what should hopefully be a decent camera/microphone > setup, and there's a Google Hangouts link there that we'll use for > that. > > - There's a shared document to use for capturing notes and discussion. > It also has a more detailed preliminary schedule. Currently, I'm aware > of the following projects planning to present short talks: > > PyPy > Pyjion > HOPE > Numba > NumPy > DyND > GT-Py > > If you're not on this list and think you should be (or vice-versa), > please speak up! > > Looking forward to meeting you all, > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Python-compilers mailing list > Python-compilers at python.org > https://mail.python.org/mailman/listinfo/python-compilers > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markdewing at gmail.com Mon Jul 11 11:03:08 2016 From: markdewing at gmail.com (Mark Dewing) Date: Mon, 11 Jul 2016 10:03:08 -0500 Subject: [Python-compilers] Workshop updates (for both local and remote attendees) In-Reply-To: References: Message-ID: Gitter chat https://www.google.com/url?q=https%3A%2F%2Fgitter.im%2Fpython-compilers-workshop%2Fchat&sa=D&sntz=1&usg=AFQjCNF3ojD4k_mQf8AEju3hOmMGGEW-hw Overflow google hangout https://hangouts.google.com/hangouts/_/berkeley.edu/compilersworkshop2 On Mon, Jul 11, 2016 at 9:54 AM, Yarko Tymciurak wrote: > Sadly the google hangouts is full. > > > > On Saturday, July 9, 2016, Nathaniel Smith wrote: > >> Hi all, >> >> I've updated the web page with a set of useful links at the top: >> >> https://python-compilers-workshop.github.io/ >> >> Note in particular: >> >> - Since a number of folks have asked about attending remotely, I've >> arranged for what should hopefully be a decent camera/microphone >> setup, and there's a Google Hangouts link there that we'll use for >> that. >> >> - There's a shared document to use for capturing notes and discussion. >> It also has a more detailed preliminary schedule. Currently, I'm aware >> of the following projects planning to present short talks: >> >> PyPy >> Pyjion >> HOPE >> Numba >> NumPy >> DyND >> GT-Py >> >> If you're not on this list and think you should be (or vice-versa), >> please speak up! >> >> Looking forward to meeting you all, >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> _______________________________________________ >> Python-compilers mailing list >> Python-compilers at python.org >> https://mail.python.org/mailman/listinfo/python-compilers >> > > _______________________________________________ > Python-compilers mailing list > Python-compilers at python.org > https://mail.python.org/mailman/listinfo/python-compilers > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yarkot1 at gmail.com Mon Jul 11 11:04:10 2016 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Mon, 11 Jul 2016 10:04:10 -0500 Subject: [Python-compilers] Workshop updates (for both local and remote attendees) In-Reply-To: References: Message-ID: Thanks; have retried the overflow hangout twice now (to timeout); will keep trying On Monday, July 11, 2016, Mark Dewing wrote: > Gitter chat > > > https://www.google.com/url?q=https%3A%2F%2Fgitter.im%2Fpython-compilers-workshop%2Fchat&sa=D&sntz=1&usg=AFQjCNF3ojD4k_mQf8AEju3hOmMGGEW-hw > > > Overflow google hangout > https://hangouts.google.com/hangouts/_/berkeley.edu/compilersworkshop2 > > On Mon, Jul 11, 2016 at 9:54 AM, Yarko Tymciurak > wrote: > >> Sadly the google hangouts is full. >> >> >> >> On Saturday, July 9, 2016, Nathaniel Smith > > wrote: >> >>> Hi all, >>> >>> I've updated the web page with a set of useful links at the top: >>> >>> https://python-compilers-workshop.github.io/ >>> >>> Note in particular: >>> >>> - Since a number of folks have asked about attending remotely, I've >>> arranged for what should hopefully be a decent camera/microphone >>> setup, and there's a Google Hangouts link there that we'll use for >>> that. >>> >>> - There's a shared document to use for capturing notes and discussion. >>> It also has a more detailed preliminary schedule. Currently, I'm aware >>> of the following projects planning to present short talks: >>> >>> PyPy >>> Pyjion >>> HOPE >>> Numba >>> NumPy >>> DyND >>> GT-Py >>> >>> If you're not on this list and think you should be (or vice-versa), >>> please speak up! >>> >>> Looking forward to meeting you all, >>> -n >>> >>> -- >>> Nathaniel J. Smith -- https://vorpus.org >>> _______________________________________________ >>> Python-compilers mailing list >>> Python-compilers at python.org >>> https://mail.python.org/mailman/listinfo/python-compilers >>> >> >> _______________________________________________ >> Python-compilers mailing list >> Python-compilers at python.org >> >> https://mail.python.org/mailman/listinfo/python-compilers >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.s.cohn at intel.com Mon Jul 11 11:13:23 2016 From: robert.s.cohn at intel.com (Cohn, Robert S) Date: Mon, 11 Jul 2016 15:13:23 +0000 Subject: [Python-compilers] Workshop updates (for both local and remote attendees) In-Reply-To: References: Message-ID: <017981773BBEDE44A0F34FBEE7EAF2DA323796FC@ORSMSX113.amr.corp.intel.com> Yarko, I am in the overflow, keep trying. From: Python-compilers [mailto:python-compilers-bounces+robert.s.cohn=intel.com at python.org] On Behalf Of Yarko Tymciurak Sent: Monday, July 11, 2016 11:04 AM To: Mark Dewing Cc: python-compilers at python.org Subject: Re: [Python-compilers] Workshop updates (for both local and remote attendees) Thanks; have retried the overflow hangout twice now (to timeout); will keep trying On Monday, July 11, 2016, Mark Dewing > wrote: Gitter chat https://www.google.com/url?q=https%3A%2F%2Fgitter.im%2Fpython-compilers-workshop%2Fchat&sa=D&sntz=1&usg=AFQjCNF3ojD4k_mQf8AEju3hOmMGGEW-hw Overflow google hangout https://hangouts.google.com/hangouts/_/berkeley.edu/compilersworkshop2 On Mon, Jul 11, 2016 at 9:54 AM, Yarko Tymciurak > wrote: Sadly the google hangouts is full. On Saturday, July 9, 2016, Nathaniel Smith > wrote: Hi all, I've updated the web page with a set of useful links at the top: https://python-compilers-workshop.github.io/ Note in particular: - Since a number of folks have asked about attending remotely, I've arranged for what should hopefully be a decent camera/microphone setup, and there's a Google Hangouts link there that we'll use for that. - There's a shared document to use for capturing notes and discussion. It also has a more detailed preliminary schedule. Currently, I'm aware of the following projects planning to present short talks: PyPy Pyjion HOPE Numba NumPy DyND GT-Py If you're not on this list and think you should be (or vice-versa), please speak up! Looking forward to meeting you all, -n -- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Python-compilers mailing list Python-compilers at python.org https://mail.python.org/mailman/listinfo/python-compilers _______________________________________________ Python-compilers mailing list Python-compilers at python.org https://mail.python.org/mailman/listinfo/python-compilers -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Jul 11 11:46:41 2016 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 11 Jul 2016 10:46:41 -0500 Subject: [Python-compilers] Workshop updates (for both local and remote attendees) In-Reply-To: References: Message-ID: The overflow is for some reason set to require me to notice and manually approve everyone who attempts to join. So I think what happens is you have to try, and then after I've noticed and clicked accept (which might take a little while), you have to try again. Sorry about this -- we'll try to figure out something a little more streamlined after lunch I think. On Jul 11, 2016 10:04 AM, "Yarko Tymciurak" wrote: > Thanks; have retried the overflow hangout twice now (to timeout); will > keep trying > > On Monday, July 11, 2016, Mark Dewing wrote: > >> Gitter chat >> >> >> https://www.google.com/url?q=https%3A%2F%2Fgitter.im%2Fpython-compilers-workshop%2Fchat&sa=D&sntz=1&usg=AFQjCNF3ojD4k_mQf8AEju3hOmMGGEW-hw >> >> >> Overflow google hangout >> https://hangouts.google.com/hangouts/_/berkeley.edu/compilersworkshop2 >> >> On Mon, Jul 11, 2016 at 9:54 AM, Yarko Tymciurak >> wrote: >> >>> Sadly the google hangouts is full. >>> >>> >>> >>> On Saturday, July 9, 2016, Nathaniel Smith wrote: >>> >>>> Hi all, >>>> >>>> I've updated the web page with a set of useful links at the top: >>>> >>>> https://python-compilers-workshop.github.io/ >>>> >>>> Note in particular: >>>> >>>> - Since a number of folks have asked about attending remotely, I've >>>> arranged for what should hopefully be a decent camera/microphone >>>> setup, and there's a Google Hangouts link there that we'll use for >>>> that. >>>> >>>> - There's a shared document to use for capturing notes and discussion. >>>> It also has a more detailed preliminary schedule. Currently, I'm aware >>>> of the following projects planning to present short talks: >>>> >>>> PyPy >>>> Pyjion >>>> HOPE >>>> Numba >>>> NumPy >>>> DyND >>>> GT-Py >>>> >>>> If you're not on this list and think you should be (or vice-versa), >>>> please speak up! >>>> >>>> Looking forward to meeting you all, >>>> -n >>>> >>>> -- >>>> Nathaniel J. Smith -- https://vorpus.org >>>> _______________________________________________ >>>> Python-compilers mailing list >>>> Python-compilers at python.org >>>> https://mail.python.org/mailman/listinfo/python-compilers >>>> >>> >>> _______________________________________________ >>> Python-compilers mailing list >>> Python-compilers at python.org >>> https://mail.python.org/mailman/listinfo/python-compilers >>> >>> >> > _______________________________________________ > Python-compilers mailing list > Python-compilers at python.org > https://mail.python.org/mailman/listinfo/python-compilers > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Mon Jul 11 13:04:25 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 11 Jul 2016 10:04:25 -0700 Subject: [Python-compilers] Standard for C Functions backing Python Callables Message-ID: I started a doc for the first discussion bullet point at in https://python-compilers-workshop.github.io/ . - Robert From antoine at python.org Tue Jul 12 05:38:35 2016 From: antoine at python.org (Antoine Pitrou) Date: Tue, 12 Jul 2016 11:38:35 +0200 Subject: [Python-compilers] Standard for C Functions backing Python Callables In-Reply-To: References: Message-ID: <5784BA9B.50503@python.org> Hi, Le 11/07/2016 19:04, Robert Bradshaw a ?crit : > I started a doc for the first discussion bullet point at in > https://python-compilers-workshop.github.io/ . Where is the doc you are alluding to? One possible step would be to add the carray() and farray() functions described below to Numpy: http://numba.pydata.org/numba-doc/latest/user/cfunc.html#dealing-with-pointers-and-array-memory http://numba.pydata.org/numba-doc/latest/reference/utils.html#numba.carray The current pure Python implementation is here: https://github.com/numba/numba/blob/master/numba/numpy_support.py#L371-L430 Regards Antoine. From njs at pobox.com Tue Jul 12 10:54:18 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 12 Jul 2016 09:54:18 -0500 Subject: [Python-compilers] Standard for C Functions backing Python Callables In-Reply-To: References: Message-ID: On Mon, Jul 11, 2016 at 12:04 PM, Robert Bradshaw wrote: > I started a doc for the first discussion bullet point at in > https://python-compilers-workshop.github.io/ . Here's the direct link: https://docs.google.com/document/d/1NQJTnZCGmB9Hc7KT6HRGsW_GzM18aP4SSpJZjFAo8wY/edit I wrote some notes last night: Use cases: - {Numba,PyPy,Pyston,Pyjion} calling into {Numba,PyPy,Pyston,Pyjion} or {Cython,SWIG,CFFI,...} - scipy.integrate calling into any of the above - Cython calling into ...? - type-safe access to Cython capsules CEP 1000: https://github.com/cython/cython/wiki/enhancements-cep1000 huge focus on speed, because the goal is for cdef func(cdef void(*)(void) callback): callback() to execute quickly and safely, where void(*)(void) is checked at function entry. OTOH all of the other use cases above don't care about speed that much... Need some data structure to represent C types CFFI... doesn't really give us this! Numba exporting cffi: casts to "void*", because they don't want to deal with CFFI's pickiness about int != int32_t -- for a "fast call" mechanism you care about actual runtime ABI compatibility, not theoretical compatibility across machines CFFI also doesn't seem to provide any public API for introspecting types, and to the extent it does I'm not sure it's shared between CPython and PyPy implementations? And when Numba calls into CFFI, it has some weird thing where it needs you to "register" cffi modules (but only "out-of-line" modules): http://numba.pydata.org/numba-doc/dev/reference/pysupported.html#cffi I have no idea what this is or why it exists Simplest API: - Someone (probably CFFI?) should define a standard API for representing C types, and support basic operations like "are these API compatible?" and "are these ABI compatible?" - take this data structure for representing C types, attach it to callables, and require that people use interned versions for fast equality checking Fancier add-ons: - have a table of multiple type options - have a callable where you show it a C type and ask if the JIT can give you one of those - have a callable where you show it the Python types and ask if the JIT can give you one of those -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Tue Jul 12 12:43:17 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 12 Jul 2016 11:43:17 -0500 Subject: [Python-compilers] Standard for C Functions backing Python Callables In-Reply-To: References: Message-ID: On Tue, Jul 12, 2016 at 9:54 AM, Nathaniel Smith wrote: > On Mon, Jul 11, 2016 at 12:04 PM, Robert Bradshaw wrote: >> I started a doc for the first discussion bullet point at in >> https://python-compilers-workshop.github.io/ . > > Here's the direct link: > https://docs.google.com/document/d/1NQJTnZCGmB9Hc7KT6HRGsW_GzM18aP4SSpJZjFAo8wY/edit > > I wrote some notes last night: > > Use cases: > - {Numba,PyPy,Pyston,Pyjion} calling into {Numba,PyPy,Pyston,Pyjion} > or {Cython,SWIG,CFFI,...} > - scipy.integrate calling into any of the above > - Cython calling into ...? > - type-safe access to Cython capsules > > CEP 1000: https://github.com/cython/cython/wiki/enhancements-cep1000 > huge focus on speed, because the goal is for > > cdef func(cdef void(*)(void) callback): > callback() > > to execute quickly and safely, where void(*)(void) is checked at function entry. > > OTOH all of the other use cases above don't care about speed that much... > > Need some data structure to represent C types > CFFI... doesn't really give us this! Ah-hah, it turns out that I am wrong! The magic is: ffi.typeof(function_object) So I think the main things we might be missing are: - a fast way to get at this type object pointer - a way to do "abi level" comparison and interning of these objects > Numba exporting cffi: casts to "void*", because they don't want to > deal with CFFI's pickiness about int != int32_t -- for a "fast call" > mechanism you care about actual runtime ABI compatibility, not > theoretical compatibility across machines > > CFFI also doesn't seem to provide any public API for introspecting > types, and to the extent it does I'm not sure it's shared between > CPython and PyPy implementations? > And when Numba calls into CFFI, it has some weird thing where it needs > you to "register" cffi modules (but only "out-of-line" modules): > http://numba.pydata.org/numba-doc/dev/reference/pysupported.html#cffi > I have no idea what this is or why it exists > > Simplest API: > - Someone (probably CFFI?) should define a standard API for > representing C types, and support basic operations like "are these API > compatible?" and "are these ABI compatible?" > - take this data structure for representing C types, attach it to > callables, and require that people use interned versions for fast > equality checking > > Fancier add-ons: > - have a table of multiple type options > - have a callable where you show it a C type and ask if the JIT can > give you one of those > - have a callable where you show it the Python types and ask if the > JIT can give you one of those > > > -- > Nathaniel J. Smith -- https://vorpus.org -- Nathaniel J. Smith -- https://vorpus.org From siu at continuum.io Sat Jul 16 16:26:51 2016 From: siu at continuum.io (Siu Kwan Lam) Date: Sat, 16 Jul 2016 20:26:51 +0000 Subject: [Python-compilers] a minimal pyir interpreter Message-ID: Hi all, I am hacking together a minimal PyIR interpreter. It contains just enough code to run the "array sum" example we made up during our meeting. It's currently at https://github.com/sklam/pyir_interpreter. It is pure-python + numpy. Run test.py to try it out. Best, Siu -- Siu Kwan Lam Software Engineer Continuum Analytics -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge.guelton at telecom-bretagne.eu Mon Jul 18 12:02:49 2016 From: serge.guelton at telecom-bretagne.eu (serge guelton) Date: Mon, 18 Jul 2016 18:02:49 +0200 Subject: [Python-compilers] Python2/3 ast compatibility Message-ID: <20160718160249.GA3969@lakota> Hi all, As pythran struggled to have a python2/3 ast abstraction, I packed this minimal package: https://github.com/serge-sans-paille/gast That provides an AST that can represent 2.7, 3.0, 3.1, 3.2, 3.3, 3.4 and 3.5 AST with a limited set of change. It can be useful for those of us that use the Python AST as input. Feedbacks appreciated ;-)