From jdemeyer at cage.ugent.be  Wed Jun  1 16:56:24 2016
From: jdemeyer at cage.ugent.be (Jeroen Demeyer)
Date: Wed, 1 Jun 2016 22:56:24 +0200
Subject: [Cython] Supporting cython.operator in C
In-Reply-To: <CADiQ+QC5tNYTAb7BFWL60Z1bOwX43MtOaF2+uTgjMrLMFjXNmw@mail.gmail.com>
References: <574D675B.1040207@cage.ugent.be>
 <CADiQ+QC5tNYTAb7BFWL60Z1bOwX43MtOaF2+uTgjMrLMFjXNmw@mail.gmail.com>
Message-ID: <574F4BF8.6030607@cage.ugent.be>

On 2016-05-31 20:35, Robert Bradshaw wrote:
> I can't think of any fundamental reason these couldn't be implemented
> in C, but there's really no need to do so as they can't be overridden
> in C.

Well, for the comma operator there might be no reason. However, it would 
be nice if one could generate arbitrary code of the form

(x) (operator) (y)

for any C/C++ values x and y and any operator. Two particular cases I 
encountered where this would be useful:

1. "x = y" where x is something that Cython does not consider an lvalue.

2. "x || y" because Cython's "x or y" generates complicated code which 
isn't optimized as well as "x || y".


Jeroen.

From robertwb at gmail.com  Wed Jun  1 23:43:21 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 1 Jun 2016 20:43:21 -0700
Subject: [Cython] Supporting cython.operator in C
In-Reply-To: <574F4BF8.6030607@cage.ugent.be>
References: <574D675B.1040207@cage.ugent.be>
 <CADiQ+QC5tNYTAb7BFWL60Z1bOwX43MtOaF2+uTgjMrLMFjXNmw@mail.gmail.com>
 <574F4BF8.6030607@cage.ugent.be>
Message-ID: <CADiQ+QAqNtGUtEKitG5u0ZBOA0ruHXuGhTifEkVnHqFghuZg3w@mail.gmail.com>

On Wed, Jun 1, 2016 at 1:56 PM, Jeroen Demeyer <jdemeyer at cage.ugent.be> wrote:
> On 2016-05-31 20:35, Robert Bradshaw wrote:
>>
>> I can't think of any fundamental reason these couldn't be implemented
>> in C, but there's really no need to do so as they can't be overridden
>> in C.
>
>
> Well, for the comma operator there might be no reason. However, it would be
> nice if one could generate arbitrary code of the form
>
> (x) (operator) (y)
>
> for any C/C++ values x and y and any operator. Two particular cases I
> encountered where this would be useful:
>
> 1. "x = y" where x is something that Cython does not consider an lvalue.

Cython doesn't support operator= even for C++, nor provide an
operator.X definition for it. (These are only provided for operators
that don't have a Python equivalent.) Are there cases in C where one
has an lvalue but Cython can't tell? (I suppose foo(x) if foo is a
macro--is that the only case?)

You can always define your own macro #define ASSIGN(a, b) (a = b).
Redefining what a valid lvalue is in the language (and syntax) is a
whole can of worms I'd rather avoid getting into...

> 2. "x || y" because Cython's "x or y" generates complicated code which isn't
> optimized as well as "x || y".

I just tried using gcc -O3 and the resulting assembly is identical.
Granted, "x or y" does have a different meaning in Python (and
Cython), returning the first of x or y that is True rather than a
boolean. To get the C behavior, write <bint>(x or y). I think this is
automatically done when it's used as a condition. The "more
complicated" code is necessary to get the correct short-circuiting
behavior, in particular when the right hand side is not expressible as
a pure C expression (e.g. requires intermediates).

- Robert

From jdemeyer at cage.ugent.be  Thu Jun  2 03:41:08 2016
From: jdemeyer at cage.ugent.be (Jeroen Demeyer)
Date: Thu, 2 Jun 2016 09:41:08 +0200
Subject: [Cython] Supporting cython.operator in C
In-Reply-To: <CADiQ+QAqNtGUtEKitG5u0ZBOA0ruHXuGhTifEkVnHqFghuZg3w@mail.gmail.com>
References: <574D675B.1040207@cage.ugent.be>
 <CADiQ+QC5tNYTAb7BFWL60Z1bOwX43MtOaF2+uTgjMrLMFjXNmw@mail.gmail.com>
 <574F4BF8.6030607@cage.ugent.be>
 <CADiQ+QAqNtGUtEKitG5u0ZBOA0ruHXuGhTifEkVnHqFghuZg3w@mail.gmail.com>
Message-ID: <574FE314.6030007@cage.ugent.be>

On 2016-06-02 05:43, Robert Bradshaw wrote:
> You can always define your own macro #define ASSIGN(a, b) (a = b).

That's basically my whole point. This ASSIGN should be called 
cython.operator.assign.

From jdemeyer at cage.ugent.be  Thu Jun  2 03:57:46 2016
From: jdemeyer at cage.ugent.be (Jeroen Demeyer)
Date: Thu, 2 Jun 2016 09:57:46 +0200
Subject: [Cython] Supporting cython.operator in C
In-Reply-To: <CADiQ+QAqNtGUtEKitG5u0ZBOA0ruHXuGhTifEkVnHqFghuZg3w@mail.gmail.com>
References: <574D675B.1040207@cage.ugent.be>
 <CADiQ+QC5tNYTAb7BFWL60Z1bOwX43MtOaF2+uTgjMrLMFjXNmw@mail.gmail.com>
 <574F4BF8.6030607@cage.ugent.be>
 <CADiQ+QAqNtGUtEKitG5u0ZBOA0ruHXuGhTifEkVnHqFghuZg3w@mail.gmail.com>
Message-ID: <574FE6FA.3060708@cage.ugent.be>

On 2016-06-02 05:43, Robert Bradshaw wrote:
>> 2. "x || y" because Cython's "x or y" generates complicated code which isn't
>> optimized as well as "x || y".
>
> I just tried using gcc -O3 and the resulting assembly is identical.

Well, that would depend on what x and y are.

A very concrete example:

"if isinstance(x, (int,long))", which is parsed as "if PyInt_Check(x) or 
PyLong_Check(x)" is slightly slower than "if PyInt_Check(x) || 
PyLong_Check(x)".

From robertwb at gmail.com  Thu Jun  2 15:12:14 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 2 Jun 2016 12:12:14 -0700
Subject: [Cython] Supporting cython.operator in C
In-Reply-To: <574FE6FA.3060708@cage.ugent.be>
References: <574D675B.1040207@cage.ugent.be>
 <CADiQ+QC5tNYTAb7BFWL60Z1bOwX43MtOaF2+uTgjMrLMFjXNmw@mail.gmail.com>
 <574F4BF8.6030607@cage.ugent.be>
 <CADiQ+QAqNtGUtEKitG5u0ZBOA0ruHXuGhTifEkVnHqFghuZg3w@mail.gmail.com>
 <574FE6FA.3060708@cage.ugent.be>
Message-ID: <CADiQ+QA=UPMFdwpCkw=2TDeBZoYCZ+jF21ZteXt+nOT1d0zg3w@mail.gmail.com>

On Thu, Jun 2, 2016 at 12:57 AM, Jeroen Demeyer <jdemeyer at cage.ugent.be> wrote:
> On 2016-06-02 05:43, Robert Bradshaw wrote:
>>>
>>> 2. "x || y" because Cython's "x or y" generates complicated code which
>>> isn't
>>> optimized as well as "x || y".
>>
>> I just tried using gcc -O3 and the resulting assembly is identical.
>
> Well, that would depend on what x and y are.
>
> A very concrete example:
>
> "if isinstance(x, (int,long))", which is parsed as "if PyInt_Check(x) or
> PyLong_Check(x)" is slightly slower than "if PyInt_Check(x) ||
> PyLong_Check(x)".

That is very surprising. How much slower? If this is the case, we
should be trying to fix Cython's code generation rather than adding
extra operators like || that one needs to know to use. I wonder if
this is due to gcc's expectations about the likelyhood of truth values
of the subexpressions in a logical or vs. an if...

For both assign and or, the problem boils down to the fact that
subexpressions in Python do not necessarily map to subexpressions in
C. If operator.assign(a, b) is simply syntactic sugar for "a = b" and
operator.logical_or syntactic sugar for "a or b" then there is no
advantage to providing them as attributes of operator. On the other
hand, if one wants to just "drop in" the C operators "=" and "||" then
consider

   a[i].x = foo()
   if b or a[i].x:
       ...

This would translate to

  [statements extracting a[i].x into tmp1]
  tmp1 = foo();
  [statements extracting a[i].x into tmp2 with side effects]
  if (b || tmp2) {
    ...
  }

The trouble is, in both cases one needs to come up with a *C
expression* representing a[i].x (and for assignment it must be an
lvalue) and add extra logic forbidding this to be assigned to a
temporary. This is not always (or even often?) possible, especially
when a is an arbitrary Python object.

Saying these operators are only allowed when it's possible, and having
extra logic to not store any intermediates in temporaries in that
case, makes the language inconsistent.

- Robert

From jdemeyer at cage.ugent.be  Thu Jun  2 16:06:33 2016
From: jdemeyer at cage.ugent.be (Jeroen Demeyer)
Date: Thu, 2 Jun 2016 22:06:33 +0200
Subject: [Cython] Supporting cython.operator in C
In-Reply-To: <CADiQ+QA=UPMFdwpCkw=2TDeBZoYCZ+jF21ZteXt+nOT1d0zg3w@mail.gmail.com>
References: <574D675B.1040207@cage.ugent.be>
 <CADiQ+QC5tNYTAb7BFWL60Z1bOwX43MtOaF2+uTgjMrLMFjXNmw@mail.gmail.com>
 <574F4BF8.6030607@cage.ugent.be>
 <CADiQ+QAqNtGUtEKitG5u0ZBOA0ruHXuGhTifEkVnHqFghuZg3w@mail.gmail.com>
 <574FE6FA.3060708@cage.ugent.be>
 <CADiQ+QA=UPMFdwpCkw=2TDeBZoYCZ+jF21ZteXt+nOT1d0zg3w@mail.gmail.com>
Message-ID: <575091C9.5030107@cage.ugent.be>

On 2016-06-02 21:12, Robert Bradshaw wrote:
>> "if isinstance(x, (int,long))", which is parsed as "if PyInt_Check(x) or
>> PyLong_Check(x)" is slightly slower than "if PyInt_Check(x) ||
>> PyLong_Check(x)".
>
> That is very surprising. How much slower?

With GCC 4.9.3, roughly 4% for a simple loop like

for x in L:
     if isinstance(x, (int, long)):
         return True

> I wonder if
> this is due to gcc's expectations about the likelyhood of truth values
> of the subexpressions in a logical or vs. an if...

I doubt it. It's just the optimizer which optimizes arithmetic better 
than if/goto.

From robertwb at gmail.com  Thu Jun  2 19:08:06 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 2 Jun 2016 16:08:06 -0700
Subject: [Cython] Supporting cython.operator in C
In-Reply-To: <575091C9.5030107@cage.ugent.be>
References: <574D675B.1040207@cage.ugent.be>
 <CADiQ+QC5tNYTAb7BFWL60Z1bOwX43MtOaF2+uTgjMrLMFjXNmw@mail.gmail.com>
 <574F4BF8.6030607@cage.ugent.be>
 <CADiQ+QAqNtGUtEKitG5u0ZBOA0ruHXuGhTifEkVnHqFghuZg3w@mail.gmail.com>
 <574FE6FA.3060708@cage.ugent.be>
 <CADiQ+QA=UPMFdwpCkw=2TDeBZoYCZ+jF21ZteXt+nOT1d0zg3w@mail.gmail.com>
 <575091C9.5030107@cage.ugent.be>
Message-ID: <CADiQ+QAO6qa7rU8=-CyYJ5feL1McgDQjrpJYpf99f1E5LXckMg@mail.gmail.com>

On Thu, Jun 2, 2016 at 1:06 PM, Jeroen Demeyer <jdemeyer at cage.ugent.be> wrote:
> On 2016-06-02 21:12, Robert Bradshaw wrote:
>>>
>>> "if isinstance(x, (int,long))", which is parsed as "if PyInt_Check(x) or
>>> PyLong_Check(x)" is slightly slower than "if PyInt_Check(x) ||
>>> PyLong_Check(x)".
>>
>> That is very surprising. How much slower?
>
> With GCC 4.9.3, roughly 4% for a simple loop like
>
> for x in L:
>     if isinstance(x, (int, long)):
>         return True
>
>> I wonder if
>> this is due to gcc's expectations about the likelyhood of truth values
>> of the subexpressions in a logical or vs. an if...
>
> I doubt it. It's just the optimizer which optimizes arithmetic better than
> if/goto.

I'm able to replicate the 4% slowdown in that example. However, if I
replace L with a list of ints (rather than a list of non-ints) and
change the return to an integer increment the isinstance version goes
to 4% faster than the || version, so I'd call this microbenchmark a
wash.

From wstein at gmail.com  Sun Jun  5 23:27:46 2016
From: wstein at gmail.com (William Stein)
Date: Sun, 5 Jun 2016 23:27:46 -0400
Subject: [Cython] cython.org
Message-ID: <CACLE5GBeJmehj_4-YnOQ1SbHLVob1pgsHSCEBo=WUOuqK7XX_w@mail.gmail.com>

Hi Cython devs,

Unfortunately, cython.org is down, since it was hosted at UW, and
pretty much all UW math computing infrastructure I have is dead.
I don't know why, though I suspect an attack of some sort.   I don't
know whether there are any backups of cython.org as it was hosted on
hardware that I didn't manage at all.    Sorry.  There is no ETA for
recovery.

 -- William

-- 
William (http://wstein.org)

From robertwb at gmail.com  Mon Jun  6 01:43:51 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sun, 5 Jun 2016 22:43:51 -0700
Subject: [Cython] cython.org
In-Reply-To: <CACLE5GBeJmehj_4-YnOQ1SbHLVob1pgsHSCEBo=WUOuqK7XX_w@mail.gmail.com>
References: <CACLE5GBeJmehj_4-YnOQ1SbHLVob1pgsHSCEBo=WUOuqK7XX_w@mail.gmail.com>
Message-ID: <CADiQ+QBNRd+qbJXY=OS6u_7r9ziy1CiHw=L3Pr4ZDcJVncC4NQ@mail.gmail.com>

Thanks for letting us know.

cython.org site - UW - sure I can find a copy
code - github
wiki - github
docs - UW - easy to generate
trac - UW - the big loss (not sure how old any backup is)
buildnot - UW - ephemeral except some configs (which'd be nice, but
not essential, to find)

I'll look more into this tomorrow.


On Sun, Jun 5, 2016 at 8:27 PM, William Stein <wstein at gmail.com> wrote:
> Hi Cython devs,
>
> Unfortunately, cython.org is down, since it was hosted at UW, and
> pretty much all UW math computing infrastructure I have is dead.
> I don't know why, though I suspect an attack of some sort.   I don't
> know whether there are any backups of cython.org as it was hosted on
> hardware that I didn't manage at all.    Sorry.  There is no ETA for
> recovery.
>
>  -- William
>
> --
> William (http://wstein.org)
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From robertwb at gmail.com  Mon Jun  6 12:11:31 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 6 Jun 2016 09:11:31 -0700
Subject: [Cython] cython.org
In-Reply-To: <CADiQ+QBNRd+qbJXY=OS6u_7r9ziy1CiHw=L3Pr4ZDcJVncC4NQ@mail.gmail.com>
References: <CACLE5GBeJmehj_4-YnOQ1SbHLVob1pgsHSCEBo=WUOuqK7XX_w@mail.gmail.com>
 <CADiQ+QBNRd+qbJXY=OS6u_7r9ziy1CiHw=L3Pr4ZDcJVncC4NQ@mail.gmail.com>
Message-ID: <CADiQ+QDHnv9iP2r8-94WzFMyZUbeCDSb0b3w-2-s_ofD2mxKdg@mail.gmail.com>

FYI, this was due to AC failure in the server room.

On Sun, Jun 5, 2016 at 10:43 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
> Thanks for letting us know.
>
> cython.org site - UW - sure I can find a copy
> code - github
> wiki - github
> docs - UW - easy to generate
> trac - UW - the big loss (not sure how old any backup is)
> buildnot - UW - ephemeral except some configs (which'd be nice, but
> not essential, to find)
>
> I'll look more into this tomorrow.
>
>
> On Sun, Jun 5, 2016 at 8:27 PM, William Stein <wstein at gmail.com> wrote:
>> Hi Cython devs,
>>
>> Unfortunately, cython.org is down, since it was hosted at UW, and
>> pretty much all UW math computing infrastructure I have is dead.
>> I don't know why, though I suspect an attack of some sort.   I don't
>> know whether there are any backups of cython.org as it was hosted on
>> hardware that I didn't manage at all.    Sorry.  There is no ETA for
>> recovery.
>>
>>  -- William
>>
>> --
>> William (http://wstein.org)
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel

From erik.m.bray at gmail.com  Tue Jun  7 02:30:20 2016
From: erik.m.bray at gmail.com (Erik Bray)
Date: Tue, 7 Jun 2016 08:30:20 +0200
Subject: [Cython] Cygwin: Handling missing C99 long double functions
In-Reply-To: <CADiQ+QBMRHdTWXEQOoskv7+nvGroYYwqnJew59=aFDSDCmcHzQ@mail.gmail.com>
References: <CAOTD34Z_dDwtr=HZpWEY6qYJ62Uz+tt6vU06cMuvm36xGNA-wQ@mail.gmail.com>
 <CAAWYfq1H_76_oK08aAQTpWcY5Ucdt_2HQw6BGNKYdM038RLKYw@mail.gmail.com>
 <CAOTD34bs5jw5KF-jJcGL3aPQfm6nep44wDV3SWZXjZ=ff2gMDg@mail.gmail.com>
 <CADiQ+QC-fGwPCocN6V9GBE9GKUYh_QUOGkByaK=dFku4dfoW9g@mail.gmail.com>
 <CAOTD34a=sBSoFHKcmqEhdBrzqAA5Vn8KZCvu8ef86-5timXC7w@mail.gmail.com>
 <CADiQ+QBjeGufq1FU3EpUYvGFfCdNQXmLvMKbDxnH7ztS_UqHuw@mail.gmail.com>
 <CAOTD34at75DpD1OdZVQsh6UvkyfJt+a5472e4vpBP3+2V14hkg@mail.gmail.com>
 <CADiQ+QDxu5t6N-JSsRm4+vn50zemU=GAmz9Lzk-WD4o=Ay1tgw@mail.gmail.com>
 <CAOTD34ZHfy7ydrW4DtecR+nfEtCG-UB5SQ+CaqS8oDvrL52+bw@mail.gmail.com>
 <CADiQ+QAYDA4WU-Ej2OOn5+b=QQx6T2nqgeu+x7pfw9uo9OMkMQ@mail.gmail.com>
 <CAOTD34bL3rMUPwysNWYsCnEhGwuuk0pOMqgNQitpeXncTMQVHg@mail.gmail.com>
 <5733B859.6020406@canterbury.ac.nz>
 <CAOTD34a5gwnxffoDpLpo6bGmqwmgUzAWMpfr_2v+=+RDP=_a7Q@mail.gmail.com>
 <CADiQ+QBMRHdTWXEQOoskv7+nvGroYYwqnJew59=aFDSDCmcHzQ@mail.gmail.com>
Message-ID: <CAOTD34bMReJhy52YudBZo2W1bkGnPMem4Y_bB5jzzvVk9qV1qA@mail.gmail.com>

On Wed, May 18, 2016 at 12:22 AM, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Fri, May 13, 2016 at 4:26 AM, Erik Bray <erik.m.bray at gmail.com> wrote:
>> On Thu, May 12, 2016 at 12:55 AM, Greg Ewing
>> <greg.ewing at canterbury.ac.nz> wrote:
>>> Erik Bray wrote:
>>>>
>>>> A developer who writes some code that
>>>> happens to use long double isn't going to think themselves "Gosh, I
>>>> guess I can't accept `long double` here because it may cause Cython to
>>>> generate code that contains "truncl"
>>>
>>>
>>> Sounds to me like Cython shouldn't be taking it upon
>>> itself to generate calls to truncl just because long
>>> double is being used. The programmer should have to
>>> call it explicitly if that's what they want.
>>
>> Since it's just a small optimization it could be disabled without
>> trouble too.  Preferably with a #define that can be set or unset at
>> compile time.
>
> https://github.com/cython/cython/commit/46d3efc4ff9123c73889bcb54b2b200d6be39ff4

The commit looks good to me.  Sorry I somehow missed this earlier.

The other cases are a bit hairier, but this is the one I was most
concerned about since it doesn't explicitly involve any operations
with long doubles.

Thanks,
Erik

From abdealikothari at gmail.com  Wed Jun  8 02:28:30 2016
From: abdealikothari at gmail.com (Abdeali Kothari)
Date: Wed, 8 Jun 2016 11:58:30 +0530
Subject: [Cython] Suggested action for "Python.h not found"
Message-ID: <CA+BKPDVSJQOQxvmGsPB=vqcPrZpj1S5Y=tiU8yXKt5+y69LoyQ@mail.gmail.com>

Hi,

Currently, when a user installs cython, if the Python headers are not
found, an error message saying "Python.h: no such file or directory" is
shown (Example: https://justpaste.it/v0gz). Would it be possible to suggest
an action to install the headers ?

I found http://trac.cython.org/ticket/247 which is quite old, and makes the
error message easier to find, but doesn't recommend a solution.

`sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to
find which OS, platform, etc is and use that to suggest a command like:

    if sys.version_info < (3, ) and sys.platform.startswith("linux") and
platform.linux_distribution()[0] in ("ubuntu", "debian"):
        print("Python headers were not found. On Debian/Ubuntu, `sudo
apt-get install python-dev` should install the Python headers.")

    elif sys.version_info < (3, ) and sys.platform.startswith("linux") and
platform.linux_distribution()[0] in ("ubuntu", "debian"):
        print("Python headers were not found. On Debian/Ubuntu, `sudo
apt-get install python3-dev` should install the Python3 headers.")

This would help so that non-developers can use cython based packages easily
as it provides helpful instructions on what to do. Currently, you
essentially search for the error message and go to one of the forums/blogs
explaining what can be done.

Or maybe by default it can just point the user to an Installation Page
which explains in detail what needs to be done to get those headers for
various systems ?

[1] - https://pypi.python.org/pypi/distro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160608/1c505b35/attachment.html>

From robertwb at gmail.com  Wed Jun  8 13:25:06 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 8 Jun 2016 10:25:06 -0700
Subject: [Cython] Suggested action for "Python.h not found"
In-Reply-To: <CA+BKPDVSJQOQxvmGsPB=vqcPrZpj1S5Y=tiU8yXKt5+y69LoyQ@mail.gmail.com>
References: <CA+BKPDVSJQOQxvmGsPB=vqcPrZpj1S5Y=tiU8yXKt5+y69LoyQ@mail.gmail.com>
Message-ID: <CADiQ+QDO3qv-3n+uM=7FFj1hAM1eCoexeL+xksGVy4XWZE6hGA@mail.gmail.com>

+1, want to submit a pull request?

On Tue, Jun 7, 2016 at 11:28 PM, Abdeali Kothari
<abdealikothari at gmail.com> wrote:
> Hi,
>
> Currently, when a user installs cython, if the Python headers are not found,
> an error message saying "Python.h: no such file or directory" is shown
> (Example: https://justpaste.it/v0gz). Would it be possible to suggest an
> action to install the headers ?
>
> I found http://trac.cython.org/ticket/247 which is quite old, and makes the
> error message easier to find, but doesn't recommend a solution.
>
> `sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to
> find which OS, platform, etc is and use that to suggest a command like:
>
>     if sys.version_info < (3, ) and sys.platform.startswith("linux") and
> platform.linux_distribution()[0] in ("ubuntu", "debian"):
>         print("Python headers were not found. On Debian/Ubuntu, `sudo
> apt-get install python-dev` should install the Python headers.")
>
>     elif sys.version_info < (3, ) and sys.platform.startswith("linux") and
> platform.linux_distribution()[0] in ("ubuntu", "debian"):
>         print("Python headers were not found. On Debian/Ubuntu, `sudo
> apt-get install python3-dev` should install the Python3 headers.")
>
> This would help so that non-developers can use cython based packages easily
> as it provides helpful instructions on what to do. Currently, you
> essentially search for the error message and go to one of the forums/blogs
> explaining what can be done.
>
> Or maybe by default it can just point the user to an Installation Page which
> explains in detail what needs to be done to get those headers for various
> systems ?
>
> [1] - https://pypi.python.org/pypi/distro
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From elizabeth.fischer at columbia.edu  Thu Jun  9 01:37:00 2016
From: elizabeth.fischer at columbia.edu (Elizabeth A. Fischer)
Date: Thu, 9 Jun 2016 01:37:00 -0400
Subject: [Cython] Suggested action for "Python.h not found"
In-Reply-To: <CADiQ+QDO3qv-3n+uM=7FFj1hAM1eCoexeL+xksGVy4XWZE6hGA@mail.gmail.com>
References: <CA+BKPDVSJQOQxvmGsPB=vqcPrZpj1S5Y=tiU8yXKt5+y69LoyQ@mail.gmail.com>
 <CADiQ+QDO3qv-3n+uM=7FFj1hAM1eCoexeL+xksGVy4XWZE6hGA@mail.gmail.com>
Message-ID: <CAC_jL3ynPVnquT2bqg-KhzvxDDhFqjp96XUTfs9CoqFS__zM3w@mail.gmail.com>

>
> Or maybe by default it can just point the user to an Installation Page
> which explains in detail what needs to be done to get those headers for
> various systems ?


I think having this information on an installation instruction page is
definitely a good idea.  I don't think it makes sense to put in every
single error message that users should consult the installation
instructions; that should be a given.

This would help so that non-developers can use cython based packages easily


It is my belief that non-developers should not be manually building
software.  Nor should developers be doing that; if you're saying
"./configure" yourself, it's like programming assembly code.

There are many ways to install Cython without manually building:

* Use your Linux distro's Cython, if you're happy with that.  This is often
not ideal.  Because the Linux distro you're using, or forced to use by your
IT department, doesn't contain the compilers / Python / etc. needed for
your project.  Or upgrading your Linux ends up breaking your applications
that depend on it.  Best choice is to not depend on your distro for
software you really care about.

* If you're on a Mac, install with MacPorts.

* If you're on Linux (and maybe Mac too), install with Spack.  This can be
as easy as typing:
         spack install py-cython

https://github.com/llnl/spack

Spack gives you complete control over your compiler.  This is important if
you're  building Python extensions, especially C++-based extensions, which
must be built with the same compilers use to build Python.

I use Spack to build my complete software stack: the application software I
work on requires about 50 dependencies to build, and another ~20 Python
dependencies to analyze the output.  That includes the full Numpy/Scipy
stack.  Spack makes it easy for me to use consistent compiler and libraries
for Python and my libraries.  I've had success making Spack-based install
instructions that others (non-developers) can follow and get my software
stack installed on their systems:
    https://github.com/citibeth/icebin


--- Elizabeth


On Wed, Jun 8, 2016 at 1:25 PM, Robert Bradshaw <robertwb at gmail.com> wrote:

> +1, want to submit a pull request?
>
> On Tue, Jun 7, 2016 at 11:28 PM, Abdeali Kothari
> <abdealikothari at gmail.com> wrote:
> > Hi,
> >
> > Currently, when a user installs cython, if the Python headers are not
> found,
> > an error message saying "Python.h: no such file or directory" is shown
> > (Example: https://justpaste.it/v0gz). Would it be possible to suggest an
> > action to install the headers ?
> >
> > I found http://trac.cython.org/ticket/247 which is quite old, and makes
> the
> > error message easier to find, but doesn't recommend a solution.
> >
> > `sys.platform`, `platform.linux_distribution` or `distro`[1] can be used
> to
> > find which OS, platform, etc is and use that to suggest a command like:
> >
> >     if sys.version_info < (3, ) and sys.platform.startswith("linux") and
> > platform.linux_distribution()[0] in ("ubuntu", "debian"):
> >         print("Python headers were not found. On Debian/Ubuntu, `sudo
> > apt-get install python-dev` should install the Python headers.")
> >
> >     elif sys.version_info < (3, ) and sys.platform.startswith("linux")
> and
> > platform.linux_distribution()[0] in ("ubuntu", "debian"):
> >         print("Python headers were not found. On Debian/Ubuntu, `sudo
> > apt-get install python3-dev` should install the Python3 headers.")
> >
> > This would help so that non-developers can use cython based packages
> easily
> > as it provides helpful instructions on what to do. Currently, you
> > essentially search for the error message and go to one of the
> forums/blogs
> > explaining what can be done.
> >
> > Or maybe by default it can just point the user to an Installation Page
> which
> > explains in detail what needs to be done to get those headers for various
> > systems ?
> >
> > [1] - https://pypi.python.org/pypi/distro
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160609/f3ed4c69/attachment.html>

From njs at vorpus.org  Thu Jun  9 02:49:22 2016
From: njs at vorpus.org (Nathaniel Smith)
Date: Wed, 8 Jun 2016 23:49:22 -0700
Subject: [Cython] Suggested action for "Python.h not found"
In-Reply-To: <CAC_jL3ynPVnquT2bqg-KhzvxDDhFqjp96XUTfs9CoqFS__zM3w@mail.gmail.com>
References: <CA+BKPDVSJQOQxvmGsPB=vqcPrZpj1S5Y=tiU8yXKt5+y69LoyQ@mail.gmail.com>
 <CADiQ+QDO3qv-3n+uM=7FFj1hAM1eCoexeL+xksGVy4XWZE6hGA@mail.gmail.com>
 <CAC_jL3ynPVnquT2bqg-KhzvxDDhFqjp96XUTfs9CoqFS__zM3w@mail.gmail.com>
Message-ID: <CAPJVwBnYsr1m1bPcdSb_E3sGHwOwZL5VVB65wZoco0Nj7os73w@mail.gmail.com>

On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer
<elizabeth.fischer at columbia.edu> wrote:
> Spack gives you complete control over your compiler.  This is important if
> you're  building Python extensions, especially C++-based extensions, which
> must be built with the same compilers use to build Python.

Just to hopefully avoid confusing people -- you mostly only need to
worry about this on Windows. On OS X and Linux, I can't think of any
situation where using a different compiler for your extension and for
Python will cause a problem.

The one thing to watch out for these days is that if you have two
different C++ libraries that are being linked together (so this
doesn't apply to CPython itself -- it's not a C++ library), and the
interface between the libraries uses std::string or std::list, and
you're on Linux, then you need to be careful about the GCC 5 / C++11
ABI transition. The short version is that if you consistently use
versions of GCC <5 OR consistently use versions of GCC >=5, then you
should be fine; the long version can be found by googling :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From erik.m.bray at gmail.com  Thu Jun  9 03:36:24 2016
From: erik.m.bray at gmail.com (Erik Bray)
Date: Thu, 9 Jun 2016 09:36:24 +0200
Subject: [Cython] Suggested action for "Python.h not found"
In-Reply-To: <CAC_jL3ynPVnquT2bqg-KhzvxDDhFqjp96XUTfs9CoqFS__zM3w@mail.gmail.com>
References: <CA+BKPDVSJQOQxvmGsPB=vqcPrZpj1S5Y=tiU8yXKt5+y69LoyQ@mail.gmail.com>
 <CADiQ+QDO3qv-3n+uM=7FFj1hAM1eCoexeL+xksGVy4XWZE6hGA@mail.gmail.com>
 <CAC_jL3ynPVnquT2bqg-KhzvxDDhFqjp96XUTfs9CoqFS__zM3w@mail.gmail.com>
Message-ID: <CAOTD34bYuSP_U09Ke-RkKOx-T5Y3pKY=CSuPYedJkukJOFF7Ow@mail.gmail.com>

On Thu, Jun 9, 2016 at 7:37 AM, Elizabeth A. Fischer
<elizabeth.fischer at columbia.edu> wrote:
>> Or maybe by default it can just point the user to an Installation Page
>> which explains in detail what needs to be done to get those headers for
>> various systems ?
>
>
> I think having this information on an installation instruction page is
> definitely a good idea.  I don't think it makes sense to put in every single
> error message that users should consult the installation instructions; that
> should be a given.

I'm inclined to agree that putting instructions for several different
systems in the message is overkill, and is a slippery slope, but a
friendlier error message with links to documentation (and maybe
general, non-Cython-specific docs on building Python extension modules
too).

> It is my belief that non-developers should not be manually building
> software.  Nor should developers be doing that; if you're saying
> "./configure" yourself, it's like programming assembly code.
>
> There are many ways to install Cython without manually building:

Unfortunately sometimes, especially with Python software, the line
between "developer" and "non-developer" is blurry.  This is especially
true of scientific software where Cython finds its most use.  It's not
uncommon, for example, for Cython to be pulled in as a build
dependency of something else.  Of course as pointed out downthread
this is less of an issue on most platforms now that binary wheels are
available for so many (including now manylinux!).

So I guess point taken that when it comes to just building Cython
itself this is less of an issue than it used to be.

> * If you're on Linux (and maybe Mac too), install with Spack.  This can be
> as easy as typing:
>          spack install py-cython
>
> https://github.com/llnl/spack
>
> Spack gives you complete control over your compiler.  This is important if
> you're  building Python extensions, especially C++-based extensions, which
> must be built with the same compilers use to build Python.
>
> I use Spack to build my complete software stack: the application software I
> work on requires about 50 dependencies to build, and another ~20 Python
> dependencies to analyze the output.  That includes the full Numpy/Scipy
> stack.  Spack makes it easy for me to use consistent compiler and libraries
> for Python and my libraries.  I've had success making Spack-based install
> instructions that others (non-developers) can follow and get my software
> stack installed on their systems:
>     https://github.com/citibeth/icebin

This one's new to me.  Will have to investigate--this could be
relevant to my interests.

Thanks,
Erik

> On Wed, Jun 8, 2016 at 1:25 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>
>> +1, want to submit a pull request?
>>
>> On Tue, Jun 7, 2016 at 11:28 PM, Abdeali Kothari
>> <abdealikothari at gmail.com> wrote:
>> > Hi,
>> >
>> > Currently, when a user installs cython, if the Python headers are not
>> > found,
>> > an error message saying "Python.h: no such file or directory" is shown
>> > (Example: https://justpaste.it/v0gz). Would it be possible to suggest an
>> > action to install the headers ?
>> >
>> > I found http://trac.cython.org/ticket/247 which is quite old, and makes
>> > the
>> > error message easier to find, but doesn't recommend a solution.
>> >
>> > `sys.platform`, `platform.linux_distribution` or `distro`[1] can be used
>> > to
>> > find which OS, platform, etc is and use that to suggest a command like:
>> >
>> >     if sys.version_info < (3, ) and sys.platform.startswith("linux") and
>> > platform.linux_distribution()[0] in ("ubuntu", "debian"):
>> >         print("Python headers were not found. On Debian/Ubuntu, `sudo
>> > apt-get install python-dev` should install the Python headers.")
>> >
>> >     elif sys.version_info < (3, ) and sys.platform.startswith("linux")
>> > and
>> > platform.linux_distribution()[0] in ("ubuntu", "debian"):
>> >         print("Python headers were not found. On Debian/Ubuntu, `sudo
>> > apt-get install python3-dev` should install the Python3 headers.")
>> >
>> > This would help so that non-developers can use cython based packages
>> > easily
>> > as it provides helpful instructions on what to do. Currently, you
>> > essentially search for the error message and go to one of the
>> > forums/blogs
>> > explaining what can be done.
>> >
>> > Or maybe by default it can just point the user to an Installation Page
>> > which
>> > explains in detail what needs to be done to get those headers for
>> > various
>> > systems ?
>> >
>> > [1] - https://pypi.python.org/pypi/distro
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From elizabeth.fischer at columbia.edu  Thu Jun  9 09:28:06 2016
From: elizabeth.fischer at columbia.edu (Elizabeth A. Fischer)
Date: Thu, 9 Jun 2016 09:28:06 -0400
Subject: [Cython] Suggested action for "Python.h not found"
In-Reply-To: <CAPJVwBnYsr1m1bPcdSb_E3sGHwOwZL5VVB65wZoco0Nj7os73w@mail.gmail.com>
References: <CA+BKPDVSJQOQxvmGsPB=vqcPrZpj1S5Y=tiU8yXKt5+y69LoyQ@mail.gmail.com>
 <CADiQ+QDO3qv-3n+uM=7FFj1hAM1eCoexeL+xksGVy4XWZE6hGA@mail.gmail.com>
 <CAC_jL3ynPVnquT2bqg-KhzvxDDhFqjp96XUTfs9CoqFS__zM3w@mail.gmail.com>
 <CAPJVwBnYsr1m1bPcdSb_E3sGHwOwZL5VVB65wZoco0Nj7os73w@mail.gmail.com>
Message-ID: <CAC_jL3zXPB3t6=q068vBi0t_i2rCzfF10uozA2TnCk0yAr6_mA@mail.gmail.com>

On Thu, Jun 9, 2016 at 2:49 AM, Nathaniel Smith <njs at vorpus.org> wrote:

> On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer
> <elizabeth.fischer at columbia.edu> wrote:
> > Spack gives you complete control over your compiler.  This is important
> if
> > you're  building Python extensions, especially C++-based extensions,
> which
> > must be built with the same compilers use to build Python.
>
> Just to hopefully avoid confusing people -- you mostly only need to
> worry about this on Windows. On OS X and Linux, I can't think of any
> situation where using a different compiler for your extension and for
> Python will cause a problem.
>

It could cause a problem if your extension is a wrapper around a library
that you wrote in  C++ or Fortran --- whether you wrote your extension by
hand, or especially if you used the Cython variant that generates  C++ code
(a logical choice, if you're interfacing to a C++ library).  If your C++
Cython extension requires a different compiler than that used to build
Cython, you will also find that standard  Cython ways to building your
extension break.

Maybe you're right --- maybe the extension itself is built in C, and all
 C++ elements are safely "encapsulated" away from Python.  Maybe you can
coax Cython into building extensions with your chosen compiler.  But who
wants to spend time finding out exactly which heterogeneous compilers work
together, and how to avoid the standard Cython build paths?

The problem is more general than compilers --- it can actually bite you ANY
time you use two different versions of a shared library in a build that you
ultimately link together.  Suppose your Python stack was built with NetCDF4
but you built your extension with NetCDF3?  In a software stack of 50
packages, the possibilities for this kind of problem are endless --- or
more precisely, there are 50 opportunities for this kind of problem.  And
they are almost impossible to prevent if you're doing things by hand.
Sometimes you can get away with it, and sometimes these problems result in
strange, time-sucking bugs.  Spack ensures that only ONE version of each
package is linked in your final binary.

and the
> interface between the libraries uses std::string or std::list, and
> you're on Linux, then you need to be careful about the GCC 5 / C++11
> ABI transition. The short version is that if you consistently use
> versions of GCC <5 OR consistently use versions of GCC >=5, then you
> should be fine; the long version can be found by googling :-).
>

That has never been my experience.  My actual experience is every time I
upgraded my MacPorts, I had to specifically re-build all my C++ libraries.
This was after running into strange, time-sucking bugs.  Advise on GCC's
C++ supports this:

http://stackoverflow.com/questions/23895081/can-you-mix-c-compiled-with-different-versions-of-the-same-compiler

And then there's Fortran... in which the .mod files are incompatible
between every release.

In theory, theory and practice are the same.  In practice, the only things
that work are those than are used commonly.  People test that software
stacks build and work together with a single compiler.  They don't test
random heterogeneous combinations of compilers, libraries, etc.

At the end of the day, the advice is simple: build your entire software
stack with the same compiler (and same set of libraries too).  When you
upgrade compilers, re-build your entire software stack.  If you use an
auto-builder like Spack or MacPorts, this is easier than upgrading to
Windows 10 ;-).

-- Elizabeth
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160609/43cb24c2/attachment.html>

From njs at vorpus.org  Thu Jun  9 13:25:36 2016
From: njs at vorpus.org (Nathaniel Smith)
Date: Thu, 9 Jun 2016 10:25:36 -0700
Subject: [Cython] Suggested action for "Python.h not found"
In-Reply-To: <CAC_jL3zXPB3t6=q068vBi0t_i2rCzfF10uozA2TnCk0yAr6_mA@mail.gmail.com>
References: <CA+BKPDVSJQOQxvmGsPB=vqcPrZpj1S5Y=tiU8yXKt5+y69LoyQ@mail.gmail.com>
 <CADiQ+QDO3qv-3n+uM=7FFj1hAM1eCoexeL+xksGVy4XWZE6hGA@mail.gmail.com>
 <CAC_jL3ynPVnquT2bqg-KhzvxDDhFqjp96XUTfs9CoqFS__zM3w@mail.gmail.com>
 <CAPJVwBnYsr1m1bPcdSb_E3sGHwOwZL5VVB65wZoco0Nj7os73w@mail.gmail.com>
 <CAC_jL3zXPB3t6=q068vBi0t_i2rCzfF10uozA2TnCk0yAr6_mA@mail.gmail.com>
Message-ID: <CAPJVwBkR9++me_q7V_iXC2FQjh_Xmnw++2jnyCRzvkGBD-osmQ@mail.gmail.com>

On Jun 9, 2016 6:28 AM, "Elizabeth A. Fischer" <
elizabeth.fischer at columbia.edu> wrote:
>
>
>
> On Thu, Jun 9, 2016 at 2:49 AM, Nathaniel Smith <njs at vorpus.org> wrote:
>>
>> On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer
>> <elizabeth.fischer at columbia.edu> wrote:
>> > Spack gives you complete control over your compiler.  This is
important if
>> > you're  building Python extensions, especially C++-based extensions,
which
>> > must be built with the same compilers use to build Python.
>>
>> Just to hopefully avoid confusing people -- you mostly only need to
>> worry about this on Windows. On OS X and Linux, I can't think of any
>> situation where using a different compiler for your extension and for
>> Python will cause a problem.
>
>
> It could cause a problem if your extension is a wrapper around a library
that you wrote in  C++ or Fortran --- whether you wrote your extension by
hand, or especially if you used the Cython variant that generates  C++ code
(a logical choice, if you're interfacing to a C++ library).  If your C++
Cython extension requires a different compiler than that used to build
Cython, you will also find that standard  Cython ways to building your
extension break.

Right, if you're compiling cython to C++, and then that C++ is calling into
a different piece of C++, then you have two C++ modules that are calling
each other, and my caveat about that applies.

> Maybe you're right --- maybe the extension itself is built in C, and all
 C++ elements are safely "encapsulated" away from Python.

They are: Python itself has no C++ interfaces, so the interface between
Python and your code is totally stable and compiler-independent in all
cases I know of on Linux and OS X.

> Maybe you can coax Cython into building extensions with your chosen
compiler.  But who wants to spend time finding out exactly which
heterogeneous compilers work together, and how to avoid the standard Cython
build paths?

Sure, if you have the means to easily use the same compiler everywhere then
it's a reasonable choice and definitely safe. But there are also lots of
people who can't build everything from source all the time or who wish to
distribute binaries to users who can't build from source all the time, and
I want to make sure those people have accurate information available.

Lots of urban legends tend to accrete around this stuff. For example, it's
widely believed that on Windows, msvc uses the ".lib" format for describing
.dll interfaces, and mingw-w64 uses the ".a" format, and so every time you
want to use mingw-w64 and link against a msvc-compiled library, you have to
throw away the .lib that shipped shipped with the .dll and write some nasty
code to regenerate it from scratch in .a format.

It turns out that this is totally wrong: the truth is that both msvc and
mingw-w64 are happy to work with both .lib and .a files; the whole legend
mostly arose because of a tiny bug in how mingw-w64 was handling .lib files
(literally there was a flag field that could have two possible values but
it only checked for one of them -- fixing it was a ~6 line patch). But this
bug persisted for a decade because all the users who hit this bug decided
that it wasn't supposed to work so they never filed a bug report.

> The problem is more general than compilers --- it can actually bite you
ANY time you use two different versions of a shared library in a build that
you ultimately link together.  Suppose your Python stack was built with
NetCDF4 but you built your extension with NetCDF3?  In a software stack of
50 packages, the possibilities for this kind of problem are endless --- or
more precisely, there are 50 opportunities for this kind of problem.  And
they are almost impossible to prevent if you're doing things by hand.
Sometimes you can get away with it, and sometimes these problems result in
strange, time-sucking bugs.  Spack ensures that only ONE version of each
package is linked in your final binary.

Yeah, all major platforms make it *possible* to handle this, but it's
tricky and requires some platform specific knowledge that's very very
poorly documented. Definitely the easiest approach if you control your
whole stack is to just not use multiple different libraries versions
together. But if someone is reading this message who has this problem and
doesn't control their entire stack, then they should come ask on the
wheel-builders at python.org mailing list and we can help them figure out how
to make things work right. :-)

>> and the
>> interface between the libraries uses std::string or std::list, and
>> you're on Linux, then you need to be careful about the GCC 5 / C++11
>> ABI transition. The short version is that if you consistently use
>> versions of GCC <5 OR consistently use versions of GCC >=5, then you
>> should be fine; the long version can be found by googling :-).
>
>
> That has never been my experience.  My actual experience is every time I
upgraded my MacPorts, I had to specifically re-build all my C++ libraries.
This was after running into strange, time-sucking bugs.  Advise on GCC's
C++ supports this:
>
>
http://stackoverflow.com/questions/23895081/can-you-mix-c-compiled-with-different-versions-of-the-same-compiler

Ah interesting -- the issue cited there is the same one as I was talking
about, except that I hadn't realized that they accidentally released the
breaking change in 4.7.{0,1} (for code that requested the non-default C++11
mode), before reverting it in 4.7.2. So, yeah, fair point, you also should
avoid 4.7.{0,1} :-). I'm not saying that dealing with C++ modules that link
against each other is easy -- it's actually super annoying right now. But
it's definitely doable. And it doesn't apply to pure C interfaces, like the
python abi.

> And then there's Fortran... in which the .mod files are incompatible
between every release.
>
> In theory, theory and practice are the same.  In practice, the only
things that work are those than are used commonly.  People test that
software stacks build and work together with a single compiler.  They don't
test random heterogeneous combinations of compilers, libraries, etc.

But they do... every single person using binary wheels or conda on Linux or
OS X is testing exactly this case every day. (At least the heterogenous
compilers part, and often the heterogenous libraries part too.) It works
fine for millions of people.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160609/ea96516d/attachment.html>

From nathan12343 at gmail.com  Fri Jun 10 12:18:19 2016
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Fri, 10 Jun 2016 11:18:19 -0500
Subject: [Cython] Static checker for cython extension dependencies?
Message-ID: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>

Hi all,

I'm working on a pretty large python/cython codebase (yt) that has many
interdependent C extensions written in cython.

I've found it to be pretty hit or miss to depend on contributors to
manually update cython dependency information in our setup.py file. The
dependency information seems to only be used by setuptools to trigger
recompilation, but critically setuptools will happily compile a C extension
for the first time if the depends information is incomplete. This means
that during development, if I update a cython routine, there's no way to
ensure that other cython routines that cimport the one I modified will be
recompiled unless I manually ensure the depends information is updated
whenever cython code gains or loses a cimport.

To make that more concrete, here's a pull request I just made to yt that
adds missing dependencies for a cython header. Without this pull request,
setuptools fails to recompile these routines when selection_routines.pxd
changes, causing a build failure.

https://bitbucket.org/yt_analysis/yt/pull-requests/2220

I think it should be possible to write a test for this by defining the
dependency information outside of setup.py and parsing grep and looking for
all cython files that cimport other cython files defined inside yt.
However, before I do that, I'm curious whether anyone has done something
similar, or if there is some other way of forcing the dependency
information to be complete on the first compilation, rather than just for
subsequent incremental recompilations during development.

Thanks for your help!

-Nathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160610/148d0113/attachment.html>

From robertwb at gmail.com  Fri Jun 10 13:49:29 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 10 Jun 2016 10:49:29 -0700
Subject: [Cython] Static checker for cython extension dependencies?
In-Reply-To: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
References: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
Message-ID: <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>

You should be using cythonize rather than listing and maintaining the
Extension definitions themselves.

http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing

On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> Hi all,
>
> I'm working on a pretty large python/cython codebase (yt) that has many
> interdependent C extensions written in cython.
>
> I've found it to be pretty hit or miss to depend on contributors to manually
> update cython dependency information in our setup.py file. The dependency
> information seems to only be used by setuptools to trigger recompilation,
> but critically setuptools will happily compile a C extension for the first
> time if the depends information is incomplete. This means that during
> development, if I update a cython routine, there's no way to ensure that
> other cython routines that cimport the one I modified will be recompiled
> unless I manually ensure the depends information is updated whenever cython
> code gains or loses a cimport.
>
> To make that more concrete, here's a pull request I just made to yt that
> adds missing dependencies for a cython header. Without this pull request,
> setuptools fails to recompile these routines when selection_routines.pxd
> changes, causing a build failure.
>
> https://bitbucket.org/yt_analysis/yt/pull-requests/2220
>
> I think it should be possible to write a test for this by defining the
> dependency information outside of setup.py and parsing grep and looking for
> all cython files that cimport other cython files defined inside yt. However,
> before I do that, I'm curious whether anyone has done something similar, or
> if there is some other way of forcing the dependency information to be
> complete on the first compilation, rather than just for subsequent
> incremental recompilations during development.
>
> Thanks for your help!
>
> -Nathan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From nathan12343 at gmail.com  Fri Jun 10 13:55:59 2016
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Fri, 10 Jun 2016 12:55:59 -0500
Subject: [Cython] Static checker for cython extension dependencies?
In-Reply-To: <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>
References: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
 <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>
Message-ID: <CAJXewOmMCmrX1Kb==JSAyC_okufJoUXfExvrAZ+-fmf8yQVVNg@mail.gmail.com>

The reason we haven't done that is we would like our setup.py script to be
runnable without cython being installed. I think cythonize is being invoked
(or something similar to it) by setuptools, using a feature added in
setuptools 18.0:
https://setuptools.readthedocs.io/en/latest/history.html#id60

Is there a way to use cythonize for this build workflow without importing
it at the top-level in our setup.py file?

FWIW, our setup.py file is here:
https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default

On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb at gmail.com>
wrote:

> You should be using cythonize rather than listing and maintaining the
> Extension definitions themselves.
>
>
> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
>
> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
> > Hi all,
> >
> > I'm working on a pretty large python/cython codebase (yt) that has many
> > interdependent C extensions written in cython.
> >
> > I've found it to be pretty hit or miss to depend on contributors to
> manually
> > update cython dependency information in our setup.py file. The dependency
> > information seems to only be used by setuptools to trigger recompilation,
> > but critically setuptools will happily compile a C extension for the
> first
> > time if the depends information is incomplete. This means that during
> > development, if I update a cython routine, there's no way to ensure that
> > other cython routines that cimport the one I modified will be recompiled
> > unless I manually ensure the depends information is updated whenever
> cython
> > code gains or loses a cimport.
> >
> > To make that more concrete, here's a pull request I just made to yt that
> > adds missing dependencies for a cython header. Without this pull request,
> > setuptools fails to recompile these routines when selection_routines.pxd
> > changes, causing a build failure.
> >
> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
> >
> > I think it should be possible to write a test for this by defining the
> > dependency information outside of setup.py and parsing grep and looking
> for
> > all cython files that cimport other cython files defined inside yt.
> However,
> > before I do that, I'm curious whether anyone has done something similar,
> or
> > if there is some other way of forcing the dependency information to be
> > complete on the first compilation, rather than just for subsequent
> > incremental recompilations during development.
> >
> > Thanks for your help!
> >
> > -Nathan
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160610/8437931c/attachment.html>

From robertwb at gmail.com  Fri Jun 10 14:49:02 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 10 Jun 2016 11:49:02 -0700
Subject: [Cython] Static checker for cython extension dependencies?
In-Reply-To: <CAJXewOmMCmrX1Kb==JSAyC_okufJoUXfExvrAZ+-fmf8yQVVNg@mail.gmail.com>
References: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
 <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>
 <CAJXewOmMCmrX1Kb==JSAyC_okufJoUXfExvrAZ+-fmf8yQVVNg@mail.gmail.com>
Message-ID: <CADiQ+QDNmorkOTPcgPbP76d+m_vLNYTN-Mi1o+E7CgVb8p7+rQ@mail.gmail.com>

We write metadata in the generated C files for just this reason. You
can fall back to something like
https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if
cython is not available.


On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> The reason we haven't done that is we would like our setup.py script to be
> runnable without cython being installed. I think cythonize is being invoked
> (or something similar to it) by setuptools, using a feature added in
> setuptools 18.0:
> https://setuptools.readthedocs.io/en/latest/history.html#id60
>
> Is there a way to use cythonize for this build workflow without importing it
> at the top-level in our setup.py file?
>
> FWIW, our setup.py file is here:
> https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
>
> On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb at gmail.com>
> wrote:
>>
>> You should be using cythonize rather than listing and maintaining the
>> Extension definitions themselves.
>>
>>
>> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
>> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
>>
>> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>> > Hi all,
>> >
>> > I'm working on a pretty large python/cython codebase (yt) that has many
>> > interdependent C extensions written in cython.
>> >
>> > I've found it to be pretty hit or miss to depend on contributors to
>> > manually
>> > update cython dependency information in our setup.py file. The
>> > dependency
>> > information seems to only be used by setuptools to trigger
>> > recompilation,
>> > but critically setuptools will happily compile a C extension for the
>> > first
>> > time if the depends information is incomplete. This means that during
>> > development, if I update a cython routine, there's no way to ensure that
>> > other cython routines that cimport the one I modified will be recompiled
>> > unless I manually ensure the depends information is updated whenever
>> > cython
>> > code gains or loses a cimport.
>> >
>> > To make that more concrete, here's a pull request I just made to yt that
>> > adds missing dependencies for a cython header. Without this pull
>> > request,
>> > setuptools fails to recompile these routines when selection_routines.pxd
>> > changes, causing a build failure.
>> >
>> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
>> >
>> > I think it should be possible to write a test for this by defining the
>> > dependency information outside of setup.py and parsing grep and looking
>> > for
>> > all cython files that cimport other cython files defined inside yt.
>> > However,
>> > before I do that, I'm curious whether anyone has done something similar,
>> > or
>> > if there is some other way of forcing the dependency information to be
>> > complete on the first compilation, rather than just for subsequent
>> > incremental recompilations during development.
>> >
>> > Thanks for your help!
>> >
>> > -Nathan
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From nathan12343 at gmail.com  Fri Jun 10 16:18:52 2016
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Fri, 10 Jun 2016 15:18:52 -0500
Subject: [Cython] Static checker for cython extension dependencies?
In-Reply-To: <CADiQ+QDNmorkOTPcgPbP76d+m_vLNYTN-Mi1o+E7CgVb8p7+rQ@mail.gmail.com>
References: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
 <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>
 <CAJXewOmMCmrX1Kb==JSAyC_okufJoUXfExvrAZ+-fmf8yQVVNg@mail.gmail.com>
 <CADiQ+QDNmorkOTPcgPbP76d+m_vLNYTN-Mi1o+E7CgVb8p7+rQ@mail.gmail.com>
Message-ID: <CAJXewO=xP9ZbK7frNYF7OxQZhSD7bPQ6Xy_Xh4Qc7qD38LX9tQ@mail.gmail.com>

Hmm, so I've looked into this a bit further, and it looks like the metadata
isn't going to be useful for us. Many of our extensions can't be
autogenerated by cython because they have non-trivial dependencies or
depend directly on C code. For these extensions, cythonize must be passed
an instantiated Extension object, and the metadata embedded in the compiled
C file just uses whatever metadata is attached to the Extension object.

I think my original idea - to parse pyx files for cimports and add a test -
should allow us to manage this more safely in the future while still
maintaining fine-grained control over dependencies of C extensions in our
setup.py file.

That said, I'd love to hear alternate ideas.

On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb at gmail.com> wrote:

> We write metadata in the generated C files for just this reason. You
> can fall back to something like
> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if
> cython is not available.
>
>
> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
> > The reason we haven't done that is we would like our setup.py script to
> be
> > runnable without cython being installed. I think cythonize is being
> invoked
> > (or something similar to it) by setuptools, using a feature added in
> > setuptools 18.0:
> > https://setuptools.readthedocs.io/en/latest/history.html#id60
> >
> > Is there a way to use cythonize for this build workflow without
> importing it
> > at the top-level in our setup.py file?
> >
> > FWIW, our setup.py file is here:
> >
> https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
> >
> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb at gmail.com>
> > wrote:
> >>
> >> You should be using cythonize rather than listing and maintaining the
> >> Extension definitions themselves.
> >>
> >>
> >>
> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
> >>
> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
> >>
> >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343 at gmail.com
> >
> >> wrote:
> >> > Hi all,
> >> >
> >> > I'm working on a pretty large python/cython codebase (yt) that has
> many
> >> > interdependent C extensions written in cython.
> >> >
> >> > I've found it to be pretty hit or miss to depend on contributors to
> >> > manually
> >> > update cython dependency information in our setup.py file. The
> >> > dependency
> >> > information seems to only be used by setuptools to trigger
> >> > recompilation,
> >> > but critically setuptools will happily compile a C extension for the
> >> > first
> >> > time if the depends information is incomplete. This means that during
> >> > development, if I update a cython routine, there's no way to ensure
> that
> >> > other cython routines that cimport the one I modified will be
> recompiled
> >> > unless I manually ensure the depends information is updated whenever
> >> > cython
> >> > code gains or loses a cimport.
> >> >
> >> > To make that more concrete, here's a pull request I just made to yt
> that
> >> > adds missing dependencies for a cython header. Without this pull
> >> > request,
> >> > setuptools fails to recompile these routines when
> selection_routines.pxd
> >> > changes, causing a build failure.
> >> >
> >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
> >> >
> >> > I think it should be possible to write a test for this by defining the
> >> > dependency information outside of setup.py and parsing grep and
> looking
> >> > for
> >> > all cython files that cimport other cython files defined inside yt.
> >> > However,
> >> > before I do that, I'm curious whether anyone has done something
> similar,
> >> > or
> >> > if there is some other way of forcing the dependency information to be
> >> > complete on the first compilation, rather than just for subsequent
> >> > incremental recompilations during development.
> >> >
> >> > Thanks for your help!
> >> >
> >> > -Nathan
> >> >
> >> > _______________________________________________
> >> > cython-devel mailing list
> >> > cython-devel at python.org
> >> > https://mail.python.org/mailman/listinfo/cython-devel
> >> >
> >> _______________________________________________
> >> cython-devel mailing list
> >> cython-devel at python.org
> >> https://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160610/f143f86b/attachment.html>

From robertwb at gmail.com  Fri Jun 10 18:04:57 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 10 Jun 2016 15:04:57 -0700
Subject: [Cython] Static checker for cython extension dependencies?
In-Reply-To: <CAJXewO=xP9ZbK7frNYF7OxQZhSD7bPQ6Xy_Xh4Qc7qD38LX9tQ@mail.gmail.com>
References: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
 <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>
 <CAJXewOmMCmrX1Kb==JSAyC_okufJoUXfExvrAZ+-fmf8yQVVNg@mail.gmail.com>
 <CADiQ+QDNmorkOTPcgPbP76d+m_vLNYTN-Mi1o+E7CgVb8p7+rQ@mail.gmail.com>
 <CAJXewO=xP9ZbK7frNYF7OxQZhSD7bPQ6Xy_Xh4Qc7qD38LX9tQ@mail.gmail.com>
Message-ID: <CADiQ+QA=CNsaMY9=AcB60J854k6URTGtOHWrTPwnUeb8XOtr9Q@mail.gmail.com>

On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> Hmm, so I've looked into this a bit further, and it looks like the metadata
> isn't going to be useful for us. Many of our extensions can't be
> autogenerated by cython because they have non-trivial dependencies

meaning? In my cursory glance, these all look fine, but perhaps you
could point me at a case that isn't very nice. Also, cythonize isn't
necessarily all-or-nothing--you can always manually add Extensions to
your list.

> or depend directly on C code.

cythonize handles dependencies on C code just fine

> For these extensions, cythonize must be passed an
> instantiated Extension object, and the metadata embedded in the compiled C
> file just uses whatever metadata is attached to the Extension object.

The embedded metadata is the union of what was in the Extension object
and the extra dependencies it found.

> I think my original idea - to parse pyx files for cimports and add a test -
> should allow us to manage this more safely in the future while still
> maintaining fine-grained control over dependencies of C extensions in our
> setup.py file.
>
> That said, I'd love to hear alternate ideas.

"Parse pyx files for cimports" (recursively, handing includes and cdef
extern from ...) is essentially re-implementing cythonize. What is the
advantage of maintaining them by hand? Cythonize was invented because
people are really bad at maintaining these lists manually...and they
can be entirely deduced.

> On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>
>> We write metadata in the generated C files for just this reason. You
>> can fall back to something like
>> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if
>> cython is not available.
>>
>>
>> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>> > The reason we haven't done that is we would like our setup.py script to
>> > be
>> > runnable without cython being installed. I think cythonize is being
>> > invoked
>> > (or something similar to it) by setuptools, using a feature added in
>> > setuptools 18.0:
>> > https://setuptools.readthedocs.io/en/latest/history.html#id60
>> >
>> > Is there a way to use cythonize for this build workflow without
>> > importing it
>> > at the top-level in our setup.py file?
>> >
>> > FWIW, our setup.py file is here:
>> >
>> > https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
>> >
>> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb at gmail.com>
>> > wrote:
>> >>
>> >> You should be using cythonize rather than listing and maintaining the
>> >> Extension definitions themselves.
>> >>
>> >>
>> >>
>> >> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
>> >>
>> >> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
>> >>
>> >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum
>> >> <nathan12343 at gmail.com>
>> >> wrote:
>> >> > Hi all,
>> >> >
>> >> > I'm working on a pretty large python/cython codebase (yt) that has
>> >> > many
>> >> > interdependent C extensions written in cython.
>> >> >
>> >> > I've found it to be pretty hit or miss to depend on contributors to
>> >> > manually
>> >> > update cython dependency information in our setup.py file. The
>> >> > dependency
>> >> > information seems to only be used by setuptools to trigger
>> >> > recompilation,
>> >> > but critically setuptools will happily compile a C extension for the
>> >> > first
>> >> > time if the depends information is incomplete. This means that during
>> >> > development, if I update a cython routine, there's no way to ensure
>> >> > that
>> >> > other cython routines that cimport the one I modified will be
>> >> > recompiled
>> >> > unless I manually ensure the depends information is updated whenever
>> >> > cython
>> >> > code gains or loses a cimport.
>> >> >
>> >> > To make that more concrete, here's a pull request I just made to yt
>> >> > that
>> >> > adds missing dependencies for a cython header. Without this pull
>> >> > request,
>> >> > setuptools fails to recompile these routines when
>> >> > selection_routines.pxd
>> >> > changes, causing a build failure.
>> >> >
>> >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
>> >> >
>> >> > I think it should be possible to write a test for this by defining
>> >> > the
>> >> > dependency information outside of setup.py and parsing grep and
>> >> > looking
>> >> > for
>> >> > all cython files that cimport other cython files defined inside yt.
>> >> > However,
>> >> > before I do that, I'm curious whether anyone has done something
>> >> > similar,
>> >> > or
>> >> > if there is some other way of forcing the dependency information to
>> >> > be
>> >> > complete on the first compilation, rather than just for subsequent
>> >> > incremental recompilations during development.
>> >> >
>> >> > Thanks for your help!
>> >> >
>> >> > -Nathan
>> >> >
>> >> > _______________________________________________
>> >> > cython-devel mailing list
>> >> > cython-devel at python.org
>> >> > https://mail.python.org/mailman/listinfo/cython-devel
>> >> >
>> >> _______________________________________________
>> >> cython-devel mailing list
>> >> cython-devel at python.org
>> >> https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> >
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From nathan12343 at gmail.com  Fri Jun 10 18:21:57 2016
From: nathan12343 at gmail.com (Nathan Goldbaum)
Date: Fri, 10 Jun 2016 17:21:57 -0500
Subject: [Cython] Static checker for cython extension dependencies?
In-Reply-To: <CADiQ+QA=CNsaMY9=AcB60J854k6URTGtOHWrTPwnUeb8XOtr9Q@mail.gmail.com>
References: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
 <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>
 <CAJXewOmMCmrX1Kb==JSAyC_okufJoUXfExvrAZ+-fmf8yQVVNg@mail.gmail.com>
 <CADiQ+QDNmorkOTPcgPbP76d+m_vLNYTN-Mi1o+E7CgVb8p7+rQ@mail.gmail.com>
 <CAJXewO=xP9ZbK7frNYF7OxQZhSD7bPQ6Xy_Xh4Qc7qD38LX9tQ@mail.gmail.com>
 <CADiQ+QA=CNsaMY9=AcB60J854k6URTGtOHWrTPwnUeb8XOtr9Q@mail.gmail.com>
Message-ID: <CAJXewO=4r0+1qvU4BBMM8F0+59+Z+OE-p-v9S1HpO+deTUmVxg@mail.gmail.com>

On Fri, Jun 10, 2016 at 5:04 PM, Robert Bradshaw <robertwb at gmail.com> wrote:

> On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
> > Hmm, so I've looked into this a bit further, and it looks like the
> metadata
> > isn't going to be useful for us. Many of our extensions can't be
> > autogenerated by cython because they have non-trivial dependencies
>
> meaning? In my cursory glance, these all look fine, but perhaps you
> could point me at a case that isn't very nice. Also, cythonize isn't
> necessarily all-or-nothing--you can always manually add Extensions to
> your list.


> > or depend directly on C code.
>
> cythonize handles dependencies on C code just fine
>
> > For these extensions, cythonize must be passed an
> > instantiated Extension object, and the metadata embedded in the compiled
> C
> > file just uses whatever metadata is attached to the Extension object.
>
> The embedded metadata is the union of what was in the Extension object
> and the extra dependencies it found.
>

Hmm, maybe I made a mistake, but when I passed an Extension object to
cythonize earlier, I found that the metadata that got written out to the
autogenerated C only included the dependencies that were listed in the
Extension object's declaration, not additional ones inferred by cython.
Perhaps this is a bug?


>
> > I think my original idea - to parse pyx files for cimports and add a
> test -
> > should allow us to manage this more safely in the future while still
> > maintaining fine-grained control over dependencies of C extensions in our
> > setup.py file.
> >
> > That said, I'd love to hear alternate ideas.
>
> "Parse pyx files for cimports" (recursively, handing includes and cdef
> extern from ...) is essentially re-implementing cythonize. What is the
> advantage of maintaining them by hand? Cythonize was invented because
> people are really bad at maintaining these lists manually...and they
> can be entirely deduced.


The advantage here is that in my couple hours of playing with using
cythonize for our setup.py script, I couldn't get it to correctly generate
the metadata I needed to validate the dependencies, so it wasn't providing
any added value. Again, it's entirely possible I was making a mistake
somewhere.

Also, separately, the fake_cythonize script you suggested will error out if
cython isn't installed and the cython-generated C files do not exist yet.
That certainly makes sense if you expect developers to have cython
available, but it's a downgrade with respect to our current setup. Right
now we have it set up so that setuptools will install cython into the build
environment while it is processing C extensions if it is not already
installed. This means if new developers or users come along and want to
build yt from source, they won't need to explicitly install cython first -
setuptools will take care of it for us. This makes our installation
instructions simpler and causes fewer head-scratching crashes when people
try to compile yt from source.


>
> > On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb at gmail.com>
> wrote:
> >>
> >> We write metadata in the generated C files for just this reason. You
> >> can fall back to something like
> >> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if
> >> cython is not available.
> >>
> >>
> >> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <
> nathan12343 at gmail.com>
> >> wrote:
> >> > The reason we haven't done that is we would like our setup.py script
> to
> >> > be
> >> > runnable without cython being installed. I think cythonize is being
> >> > invoked
> >> > (or something similar to it) by setuptools, using a feature added in
> >> > setuptools 18.0:
> >> > https://setuptools.readthedocs.io/en/latest/history.html#id60
> >> >
> >> > Is there a way to use cythonize for this build workflow without
> >> > importing it
> >> > at the top-level in our setup.py file?
> >> >
> >> > FWIW, our setup.py file is here:
> >> >
> >> >
> https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
> >> >
> >> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb at gmail.com
> >
> >> > wrote:
> >> >>
> >> >> You should be using cythonize rather than listing and maintaining the
> >> >> Extension definitions themselves.
> >> >>
> >> >>
> >> >>
> >> >>
> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
> >> >>
> >> >>
> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
> >> >>
> >> >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum
> >> >> <nathan12343 at gmail.com>
> >> >> wrote:
> >> >> > Hi all,
> >> >> >
> >> >> > I'm working on a pretty large python/cython codebase (yt) that has
> >> >> > many
> >> >> > interdependent C extensions written in cython.
> >> >> >
> >> >> > I've found it to be pretty hit or miss to depend on contributors to
> >> >> > manually
> >> >> > update cython dependency information in our setup.py file. The
> >> >> > dependency
> >> >> > information seems to only be used by setuptools to trigger
> >> >> > recompilation,
> >> >> > but critically setuptools will happily compile a C extension for
> the
> >> >> > first
> >> >> > time if the depends information is incomplete. This means that
> during
> >> >> > development, if I update a cython routine, there's no way to ensure
> >> >> > that
> >> >> > other cython routines that cimport the one I modified will be
> >> >> > recompiled
> >> >> > unless I manually ensure the depends information is updated
> whenever
> >> >> > cython
> >> >> > code gains or loses a cimport.
> >> >> >
> >> >> > To make that more concrete, here's a pull request I just made to yt
> >> >> > that
> >> >> > adds missing dependencies for a cython header. Without this pull
> >> >> > request,
> >> >> > setuptools fails to recompile these routines when
> >> >> > selection_routines.pxd
> >> >> > changes, causing a build failure.
> >> >> >
> >> >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
> >> >> >
> >> >> > I think it should be possible to write a test for this by defining
> >> >> > the
> >> >> > dependency information outside of setup.py and parsing grep and
> >> >> > looking
> >> >> > for
> >> >> > all cython files that cimport other cython files defined inside yt.
> >> >> > However,
> >> >> > before I do that, I'm curious whether anyone has done something
> >> >> > similar,
> >> >> > or
> >> >> > if there is some other way of forcing the dependency information to
> >> >> > be
> >> >> > complete on the first compilation, rather than just for subsequent
> >> >> > incremental recompilations during development.
> >> >> >
> >> >> > Thanks for your help!
> >> >> >
> >> >> > -Nathan
> >> >> >
> >> >> > _______________________________________________
> >> >> > cython-devel mailing list
> >> >> > cython-devel at python.org
> >> >> > https://mail.python.org/mailman/listinfo/cython-devel
> >> >> >
> >> >> _______________________________________________
> >> >> cython-devel mailing list
> >> >> cython-devel at python.org
> >> >> https://mail.python.org/mailman/listinfo/cython-devel
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > cython-devel mailing list
> >> > cython-devel at python.org
> >> > https://mail.python.org/mailman/listinfo/cython-devel
> >> >
> >> _______________________________________________
> >> cython-devel mailing list
> >> cython-devel at python.org
> >> https://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160610/9ce24552/attachment.html>

From robertwb at gmail.com  Fri Jun 10 18:45:09 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 10 Jun 2016 15:45:09 -0700
Subject: [Cython] Static checker for cython extension dependencies?
In-Reply-To: <CAJXewO=4r0+1qvU4BBMM8F0+59+Z+OE-p-v9S1HpO+deTUmVxg@mail.gmail.com>
References: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
 <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>
 <CAJXewOmMCmrX1Kb==JSAyC_okufJoUXfExvrAZ+-fmf8yQVVNg@mail.gmail.com>
 <CADiQ+QDNmorkOTPcgPbP76d+m_vLNYTN-Mi1o+E7CgVb8p7+rQ@mail.gmail.com>
 <CAJXewO=xP9ZbK7frNYF7OxQZhSD7bPQ6Xy_Xh4Qc7qD38LX9tQ@mail.gmail.com>
 <CADiQ+QA=CNsaMY9=AcB60J854k6URTGtOHWrTPwnUeb8XOtr9Q@mail.gmail.com>
 <CAJXewO=4r0+1qvU4BBMM8F0+59+Z+OE-p-v9S1HpO+deTUmVxg@mail.gmail.com>
Message-ID: <CADiQ+QC07x79Ptoq2M6R+7QPfFHEkeFwp0nJOTfz8DVrwACgvw@mail.gmail.com>

On Fri, Jun 10, 2016 at 3:21 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>
> On Fri, Jun 10, 2016 at 5:04 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>
>> On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>> > Hmm, so I've looked into this a bit further, and it looks like the
>> > metadata
>> > isn't going to be useful for us. Many of our extensions can't be
>> > autogenerated by cython because they have non-trivial dependencies
>>
>> meaning? In my cursory glance, these all look fine, but perhaps you
>> could point me at a case that isn't very nice. Also, cythonize isn't
>> necessarily all-or-nothing--you can always manually add Extensions to
>> your list.
>>
>>
>> > or depend directly on C code.
>>
>> cythonize handles dependencies on C code just fine
>>
>> > For these extensions, cythonize must be passed an
>> > instantiated Extension object, and the metadata embedded in the compiled
>> > C
>> > file just uses whatever metadata is attached to the Extension object.
>>
>> The embedded metadata is the union of what was in the Extension object
>> and the extra dependencies it found.
>
>
> Hmm, maybe I made a mistake, but when I passed an Extension object to
> cythonize earlier, I found that the metadata that got written out to the
> autogenerated C only included the dependencies that were listed in the
> Extension object's declaration, not additional ones inferred by cython.
> Perhaps this is a bug?

Sounds like a bug to me.

>> > I think my original idea - to parse pyx files for cimports and add a
>> > test -
>> > should allow us to manage this more safely in the future while still
>> > maintaining fine-grained control over dependencies of C extensions in
>> > our
>> > setup.py file.
>> >
>> > That said, I'd love to hear alternate ideas.
>>
>> "Parse pyx files for cimports" (recursively, handing includes and cdef
>> extern from ...) is essentially re-implementing cythonize. What is the
>> advantage of maintaining them by hand? Cythonize was invented because
>> people are really bad at maintaining these lists manually...and they
>> can be entirely deduced.
>
> The advantage here is that in my couple hours of playing with using
> cythonize for our setup.py script, I couldn't get it to correctly generate
> the metadata I needed to validate the dependencies, so it wasn't providing
> any added value. Again, it's entirely possible I was making a mistake
> somewhere.

:(. I'll take a quick stab at it.

> Also, separately, the fake_cythonize script you suggested will error out if
> cython isn't installed and the cython-generated C files do not exist yet.

The fake_cythonize script (and metadata) is intended for the case that
the user doesn't have Cython, but the C files are shipped. Sounds like
you require Cython (installing it for the user if they don't have it).

> That certainly makes sense if you expect developers to have cython
> available, but it's a downgrade with respect to our current setup. Right now
> we have it set up so that setuptools will install cython into the build
> environment while it is processing C extensions if it is not already
> installed. This means if new developers or users come along and want to
> build yt from source, they won't need to explicitly install cython first -
> setuptools will take care of it for us. This makes our installation
> instructions simpler and causes fewer head-scratching crashes when people
> try to compile yt from source.

Makes sense. In this case, you would have to avoid importing and
calling cythonize until after your setup.py invoked setup.

>> > On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb at gmail.com>
>> > wrote:
>> >>
>> >> We write metadata in the generated C files for just this reason. You
>> >> can fall back to something like
>> >> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if
>> >> cython is not available.
>> >>
>> >>
>> >> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum
>> >> <nathan12343 at gmail.com>
>> >> wrote:
>> >> > The reason we haven't done that is we would like our setup.py script
>> >> > to
>> >> > be
>> >> > runnable without cython being installed. I think cythonize is being
>> >> > invoked
>> >> > (or something similar to it) by setuptools, using a feature added in
>> >> > setuptools 18.0:
>> >> > https://setuptools.readthedocs.io/en/latest/history.html#id60
>> >> >
>> >> > Is there a way to use cythonize for this build workflow without
>> >> > importing it
>> >> > at the top-level in our setup.py file?
>> >> >
>> >> > FWIW, our setup.py file is here:
>> >> >
>> >> >
>> >> > https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
>> >> >
>> >> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw
>> >> > <robertwb at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> You should be using cythonize rather than listing and maintaining
>> >> >> the
>> >> >> Extension definitions themselves.
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
>> >> >>
>> >> >>
>> >> >> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
>> >> >>
>> >> >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum
>> >> >> <nathan12343 at gmail.com>
>> >> >> wrote:
>> >> >> > Hi all,
>> >> >> >
>> >> >> > I'm working on a pretty large python/cython codebase (yt) that has
>> >> >> > many
>> >> >> > interdependent C extensions written in cython.
>> >> >> >
>> >> >> > I've found it to be pretty hit or miss to depend on contributors
>> >> >> > to
>> >> >> > manually
>> >> >> > update cython dependency information in our setup.py file. The
>> >> >> > dependency
>> >> >> > information seems to only be used by setuptools to trigger
>> >> >> > recompilation,
>> >> >> > but critically setuptools will happily compile a C extension for
>> >> >> > the
>> >> >> > first
>> >> >> > time if the depends information is incomplete. This means that
>> >> >> > during
>> >> >> > development, if I update a cython routine, there's no way to
>> >> >> > ensure
>> >> >> > that
>> >> >> > other cython routines that cimport the one I modified will be
>> >> >> > recompiled
>> >> >> > unless I manually ensure the depends information is updated
>> >> >> > whenever
>> >> >> > cython
>> >> >> > code gains or loses a cimport.
>> >> >> >
>> >> >> > To make that more concrete, here's a pull request I just made to
>> >> >> > yt
>> >> >> > that
>> >> >> > adds missing dependencies for a cython header. Without this pull
>> >> >> > request,
>> >> >> > setuptools fails to recompile these routines when
>> >> >> > selection_routines.pxd
>> >> >> > changes, causing a build failure.
>> >> >> >
>> >> >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
>> >> >> >
>> >> >> > I think it should be possible to write a test for this by defining
>> >> >> > the
>> >> >> > dependency information outside of setup.py and parsing grep and
>> >> >> > looking
>> >> >> > for
>> >> >> > all cython files that cimport other cython files defined inside
>> >> >> > yt.
>> >> >> > However,
>> >> >> > before I do that, I'm curious whether anyone has done something
>> >> >> > similar,
>> >> >> > or
>> >> >> > if there is some other way of forcing the dependency information
>> >> >> > to
>> >> >> > be
>> >> >> > complete on the first compilation, rather than just for subsequent
>> >> >> > incremental recompilations during development.
>> >> >> >
>> >> >> > Thanks for your help!
>> >> >> >
>> >> >> > -Nathan
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > cython-devel mailing list
>> >> >> > cython-devel at python.org
>> >> >> > https://mail.python.org/mailman/listinfo/cython-devel
>> >> >> >
>> >> >> _______________________________________________
>> >> >> cython-devel mailing list
>> >> >> cython-devel at python.org
>> >> >> https://mail.python.org/mailman/listinfo/cython-devel
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > cython-devel mailing list
>> >> > cython-devel at python.org
>> >> > https://mail.python.org/mailman/listinfo/cython-devel
>> >> >
>> >> _______________________________________________
>> >> cython-devel mailing list
>> >> cython-devel at python.org
>> >> https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> >
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at gmail.com  Fri Jun 10 20:00:09 2016
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 10 Jun 2016 17:00:09 -0700
Subject: [Cython] Static checker for cython extension dependencies?
In-Reply-To: <CADiQ+QC07x79Ptoq2M6R+7QPfFHEkeFwp0nJOTfz8DVrwACgvw@mail.gmail.com>
References: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
 <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>
 <CAJXewOmMCmrX1Kb==JSAyC_okufJoUXfExvrAZ+-fmf8yQVVNg@mail.gmail.com>
 <CADiQ+QDNmorkOTPcgPbP76d+m_vLNYTN-Mi1o+E7CgVb8p7+rQ@mail.gmail.com>
 <CAJXewO=xP9ZbK7frNYF7OxQZhSD7bPQ6Xy_Xh4Qc7qD38LX9tQ@mail.gmail.com>
 <CADiQ+QA=CNsaMY9=AcB60J854k6URTGtOHWrTPwnUeb8XOtr9Q@mail.gmail.com>
 <CAJXewO=4r0+1qvU4BBMM8F0+59+Z+OE-p-v9S1HpO+deTUmVxg@mail.gmail.com>
 <CADiQ+QC07x79Ptoq2M6R+7QPfFHEkeFwp0nJOTfz8DVrwACgvw@mail.gmail.com>
Message-ID: <CADiQ+QC7dCstw=RM7GYHtyMnEfHxOxqSX9N4JLF8CGQU1LqstA@mail.gmail.com>

https://bitbucket.org/yt_analysis/yt/pull-requests/2225/use-cythonize-to-manage-pxd-dependencies/diff

On Fri, Jun 10, 2016 at 3:45 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Fri, Jun 10, 2016 at 3:21 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>>
>> On Fri, Jun 10, 2016 at 5:04 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>>
>>> On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>>> wrote:
>>> > Hmm, so I've looked into this a bit further, and it looks like the
>>> > metadata
>>> > isn't going to be useful for us. Many of our extensions can't be
>>> > autogenerated by cython because they have non-trivial dependencies
>>>
>>> meaning? In my cursory glance, these all look fine, but perhaps you
>>> could point me at a case that isn't very nice. Also, cythonize isn't
>>> necessarily all-or-nothing--you can always manually add Extensions to
>>> your list.
>>>
>>>
>>> > or depend directly on C code.
>>>
>>> cythonize handles dependencies on C code just fine
>>>
>>> > For these extensions, cythonize must be passed an
>>> > instantiated Extension object, and the metadata embedded in the compiled
>>> > C
>>> > file just uses whatever metadata is attached to the Extension object.
>>>
>>> The embedded metadata is the union of what was in the Extension object
>>> and the extra dependencies it found.
>>
>>
>> Hmm, maybe I made a mistake, but when I passed an Extension object to
>> cythonize earlier, I found that the metadata that got written out to the
>> autogenerated C only included the dependencies that were listed in the
>> Extension object's declaration, not additional ones inferred by cython.
>> Perhaps this is a bug?
>
> Sounds like a bug to me.
>
>>> > I think my original idea - to parse pyx files for cimports and add a
>>> > test -
>>> > should allow us to manage this more safely in the future while still
>>> > maintaining fine-grained control over dependencies of C extensions in
>>> > our
>>> > setup.py file.
>>> >
>>> > That said, I'd love to hear alternate ideas.
>>>
>>> "Parse pyx files for cimports" (recursively, handing includes and cdef
>>> extern from ...) is essentially re-implementing cythonize. What is the
>>> advantage of maintaining them by hand? Cythonize was invented because
>>> people are really bad at maintaining these lists manually...and they
>>> can be entirely deduced.
>>
>> The advantage here is that in my couple hours of playing with using
>> cythonize for our setup.py script, I couldn't get it to correctly generate
>> the metadata I needed to validate the dependencies, so it wasn't providing
>> any added value. Again, it's entirely possible I was making a mistake
>> somewhere.
>
> :(. I'll take a quick stab at it.
>
>> Also, separately, the fake_cythonize script you suggested will error out if
>> cython isn't installed and the cython-generated C files do not exist yet.
>
> The fake_cythonize script (and metadata) is intended for the case that
> the user doesn't have Cython, but the C files are shipped. Sounds like
> you require Cython (installing it for the user if they don't have it).
>
>> That certainly makes sense if you expect developers to have cython
>> available, but it's a downgrade with respect to our current setup. Right now
>> we have it set up so that setuptools will install cython into the build
>> environment while it is processing C extensions if it is not already
>> installed. This means if new developers or users come along and want to
>> build yt from source, they won't need to explicitly install cython first -
>> setuptools will take care of it for us. This makes our installation
>> instructions simpler and causes fewer head-scratching crashes when people
>> try to compile yt from source.
>
> Makes sense. In this case, you would have to avoid importing and
> calling cythonize until after your setup.py invoked setup.
>
>>> > On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb at gmail.com>
>>> > wrote:
>>> >>
>>> >> We write metadata in the generated C files for just this reason. You
>>> >> can fall back to something like
>>> >> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if
>>> >> cython is not available.
>>> >>
>>> >>
>>> >> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum
>>> >> <nathan12343 at gmail.com>
>>> >> wrote:
>>> >> > The reason we haven't done that is we would like our setup.py script
>>> >> > to
>>> >> > be
>>> >> > runnable without cython being installed. I think cythonize is being
>>> >> > invoked
>>> >> > (or something similar to it) by setuptools, using a feature added in
>>> >> > setuptools 18.0:
>>> >> > https://setuptools.readthedocs.io/en/latest/history.html#id60
>>> >> >
>>> >> > Is there a way to use cythonize for this build workflow without
>>> >> > importing it
>>> >> > at the top-level in our setup.py file?
>>> >> >
>>> >> > FWIW, our setup.py file is here:
>>> >> >
>>> >> >
>>> >> > https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
>>> >> >
>>> >> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw
>>> >> > <robertwb at gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> You should be using cythonize rather than listing and maintaining
>>> >> >> the
>>> >> >> Extension definitions themselves.
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
>>> >> >>
>>> >> >>
>>> >> >> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
>>> >> >>
>>> >> >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum
>>> >> >> <nathan12343 at gmail.com>
>>> >> >> wrote:
>>> >> >> > Hi all,
>>> >> >> >
>>> >> >> > I'm working on a pretty large python/cython codebase (yt) that has
>>> >> >> > many
>>> >> >> > interdependent C extensions written in cython.
>>> >> >> >
>>> >> >> > I've found it to be pretty hit or miss to depend on contributors
>>> >> >> > to
>>> >> >> > manually
>>> >> >> > update cython dependency information in our setup.py file. The
>>> >> >> > dependency
>>> >> >> > information seems to only be used by setuptools to trigger
>>> >> >> > recompilation,
>>> >> >> > but critically setuptools will happily compile a C extension for
>>> >> >> > the
>>> >> >> > first
>>> >> >> > time if the depends information is incomplete. This means that
>>> >> >> > during
>>> >> >> > development, if I update a cython routine, there's no way to
>>> >> >> > ensure
>>> >> >> > that
>>> >> >> > other cython routines that cimport the one I modified will be
>>> >> >> > recompiled
>>> >> >> > unless I manually ensure the depends information is updated
>>> >> >> > whenever
>>> >> >> > cython
>>> >> >> > code gains or loses a cimport.
>>> >> >> >
>>> >> >> > To make that more concrete, here's a pull request I just made to
>>> >> >> > yt
>>> >> >> > that
>>> >> >> > adds missing dependencies for a cython header. Without this pull
>>> >> >> > request,
>>> >> >> > setuptools fails to recompile these routines when
>>> >> >> > selection_routines.pxd
>>> >> >> > changes, causing a build failure.
>>> >> >> >
>>> >> >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
>>> >> >> >
>>> >> >> > I think it should be possible to write a test for this by defining
>>> >> >> > the
>>> >> >> > dependency information outside of setup.py and parsing grep and
>>> >> >> > looking
>>> >> >> > for
>>> >> >> > all cython files that cimport other cython files defined inside
>>> >> >> > yt.
>>> >> >> > However,
>>> >> >> > before I do that, I'm curious whether anyone has done something
>>> >> >> > similar,
>>> >> >> > or
>>> >> >> > if there is some other way of forcing the dependency information
>>> >> >> > to
>>> >> >> > be
>>> >> >> > complete on the first compilation, rather than just for subsequent
>>> >> >> > incremental recompilations during development.
>>> >> >> >
>>> >> >> > Thanks for your help!
>>> >> >> >
>>> >> >> > -Nathan
>>> >> >> >
>>> >> >> > _______________________________________________
>>> >> >> > cython-devel mailing list
>>> >> >> > cython-devel at python.org
>>> >> >> > https://mail.python.org/mailman/listinfo/cython-devel
>>> >> >> >
>>> >> >> _______________________________________________
>>> >> >> cython-devel mailing list
>>> >> >> cython-devel at python.org
>>> >> >> https://mail.python.org/mailman/listinfo/cython-devel
>>> >> >
>>> >> >
>>> >> >
>>> >> > _______________________________________________
>>> >> > cython-devel mailing list
>>> >> > cython-devel at python.org
>>> >> > https://mail.python.org/mailman/listinfo/cython-devel
>>> >> >
>>> >> _______________________________________________
>>> >> cython-devel mailing list
>>> >> cython-devel at python.org
>>> >> https://mail.python.org/mailman/listinfo/cython-devel
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > cython-devel mailing list
>>> > cython-devel at python.org
>>> > https://mail.python.org/mailman/listinfo/cython-devel
>>> >
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>>

From elizabeth.fischer at columbia.edu  Fri Jun 10 21:02:01 2016
From: elizabeth.fischer at columbia.edu (Elizabeth A. Fischer)
Date: Fri, 10 Jun 2016 21:02:01 -0400
Subject: [Cython] Static checker for cython extension dependencies?
In-Reply-To: <CAJXewO=xP9ZbK7frNYF7OxQZhSD7bPQ6Xy_Xh4Qc7qD38LX9tQ@mail.gmail.com>
References: <CAJXewOk6Qo0YMheXrKqS97mT98nbrcQZjCi5_F=e9XPBgcTO7g@mail.gmail.com>
 <CADiQ+QDv9dC8XGjY6OGQjqzkoqqcmT_J+q+fQf7yo1gWHD8zmw@mail.gmail.com>
 <CAJXewOmMCmrX1Kb==JSAyC_okufJoUXfExvrAZ+-fmf8yQVVNg@mail.gmail.com>
 <CADiQ+QDNmorkOTPcgPbP76d+m_vLNYTN-Mi1o+E7CgVb8p7+rQ@mail.gmail.com>
 <CAJXewO=xP9ZbK7frNYF7OxQZhSD7bPQ6Xy_Xh4Qc7qD38LX9tQ@mail.gmail.com>
Message-ID: <CAC_jL3wsLWEf_LH-aauRgApBfqaX5EnuHzZTEXPMZ6c4he80-Q@mail.gmail.com>

It's not clear to me what you're doing, but it all sounds too complicated
to me.  I have Cython extensions that depend on C++ code, and I don't have
to resort to these kinds of contortions.

One model that works well is to build your C code as a shared library.
Make each Cython module in a single file, and have it depend on your C
library.  In my case, I don't use any of the Python/Cython build systems; I
just call cythonize to get C++ code, and then build that C++ code as I
would build any other C++ code (in CMake).  There's then a bit of
CMake/Python stuff needed to get CMake to install in the correct Python
locations.

Another way would be to build your supporting C code as a separate package
(with CMake or Autotools, etc.) --- and then use the Python build system
for the actual Cython extension.  Now you have two packages instead of one,
but that doesn't bother me much.

I chose the all-CMake way because I put a lot of time and effort into
learning CMake, and didn't want to have to put more time and effort into
learning another build system, just so I could build Python.  But either
way should work.

-- Elizabeth


On Fri, Jun 10, 2016 at 4:18 PM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:

> Hmm, so I've looked into this a bit further, and it looks like the
> metadata isn't going to be useful for us. Many of our extensions can't be
> autogenerated by cython because they have non-trivial dependencies or
> depend directly on C code. For these extensions, cythonize must be passed
> an instantiated Extension object, and the metadata embedded in the compiled
> C file just uses whatever metadata is attached to the Extension object.
>
> I think my original idea - to parse pyx files for cimports and add a test
> - should allow us to manage this more safely in the future while still
> maintaining fine-grained control over dependencies of C extensions in our
> setup.py file.
>
> That said, I'd love to hear alternate ideas.
>
> On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb at gmail.com>
> wrote:
>
>> We write metadata in the generated C files for just this reason. You
>> can fall back to something like
>> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if
>> cython is not available.
>>
>>
>> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>> > The reason we haven't done that is we would like our setup.py script to
>> be
>> > runnable without cython being installed. I think cythonize is being
>> invoked
>> > (or something similar to it) by setuptools, using a feature added in
>> > setuptools 18.0:
>> > https://setuptools.readthedocs.io/en/latest/history.html#id60
>> >
>> > Is there a way to use cythonize for this build workflow without
>> importing it
>> > at the top-level in our setup.py file?
>> >
>> > FWIW, our setup.py file is here:
>> >
>> https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
>> >
>> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb at gmail.com>
>> > wrote:
>> >>
>> >> You should be using cythonize rather than listing and maintaining the
>> >> Extension definitions themselves.
>> >>
>> >>
>> >>
>> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
>> >>
>> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
>> >>
>> >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <
>> nathan12343 at gmail.com>
>> >> wrote:
>> >> > Hi all,
>> >> >
>> >> > I'm working on a pretty large python/cython codebase (yt) that has
>> many
>> >> > interdependent C extensions written in cython.
>> >> >
>> >> > I've found it to be pretty hit or miss to depend on contributors to
>> >> > manually
>> >> > update cython dependency information in our setup.py file. The
>> >> > dependency
>> >> > information seems to only be used by setuptools to trigger
>> >> > recompilation,
>> >> > but critically setuptools will happily compile a C extension for the
>> >> > first
>> >> > time if the depends information is incomplete. This means that during
>> >> > development, if I update a cython routine, there's no way to ensure
>> that
>> >> > other cython routines that cimport the one I modified will be
>> recompiled
>> >> > unless I manually ensure the depends information is updated whenever
>> >> > cython
>> >> > code gains or loses a cimport.
>> >> >
>> >> > To make that more concrete, here's a pull request I just made to yt
>> that
>> >> > adds missing dependencies for a cython header. Without this pull
>> >> > request,
>> >> > setuptools fails to recompile these routines when
>> selection_routines.pxd
>> >> > changes, causing a build failure.
>> >> >
>> >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
>> >> >
>> >> > I think it should be possible to write a test for this by defining
>> the
>> >> > dependency information outside of setup.py and parsing grep and
>> looking
>> >> > for
>> >> > all cython files that cimport other cython files defined inside yt.
>> >> > However,
>> >> > before I do that, I'm curious whether anyone has done something
>> similar,
>> >> > or
>> >> > if there is some other way of forcing the dependency information to
>> be
>> >> > complete on the first compilation, rather than just for subsequent
>> >> > incremental recompilations during development.
>> >> >
>> >> > Thanks for your help!
>> >> >
>> >> > -Nathan
>> >> >
>> >> > _______________________________________________
>> >> > cython-devel mailing list
>> >> > cython-devel at python.org
>> >> > https://mail.python.org/mailman/listinfo/cython-devel
>> >> >
>> >> _______________________________________________
>> >> cython-devel mailing list
>> >> cython-devel at python.org
>> >> https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> >
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160610/0f80d7f9/attachment-0001.html>

From jdemeyer at cage.ugent.be  Mon Jun 20 09:08:51 2016
From: jdemeyer at cage.ugent.be (Jeroen Demeyer)
Date: Mon, 20 Jun 2016 15:08:51 +0200
Subject: [Cython] Virtual cpdef methods
Message-ID: <5767EAE3.4060607@cage.ugent.be>

Hello,

I would like to have a "virtual" cpdef method: something which allows a 
fast Cython call *when implemented* but without a default implementation.

The most simple implementation would be a "cpdef" method but without an 
entry for the method in the Python method table.

Of course, the "cdef" part needs to be implemented by the user (you need 
to put something in the vtab). For my purposes, something like "raise 
AttributeError" would be sufficient.

1. Can this be accomplished currently in Cython?

2. Is this something that makes sense to add to Cython?

3. What do you think of the following syntax?

cdef class MyClass(object):
     cpdef virtual_method(self):
         # implementation of "cdef" part ...

     del virtual_method  # don't put virtual_method in the method table

From robertwb at math.washington.edu  Mon Jun 20 11:31:05 2016
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 20 Jun 2016 08:31:05 -0700
Subject: [Cython] Virtual cpdef methods
In-Reply-To: <5767EAE3.4060607@cage.ugent.be>
References: <5767EAE3.4060607@cage.ugent.be>
Message-ID: <CADiQ+QAgOVcsA0Uutu8MX7qQozK9kdAdAmGyGQwDTZkEuDuFVw@mail.gmail.com>

All methods (cdef, cpdef, and def) are virtual by default in Cython,
just like Python.

Sounds like you want a cdef function to me. You can override a cdef
function with a cpdef function for any subclass that wishing to expose
it to Python.

On Mon, Jun 20, 2016 at 6:08 AM, Jeroen Demeyer <jdemeyer at cage.ugent.be> wrote:
> Hello,
>
> I would like to have a "virtual" cpdef method: something which allows a fast
> Cython call *when implemented* but without a default implementation.
>
> The most simple implementation would be a "cpdef" method but without an
> entry for the method in the Python method table.
>
> Of course, the "cdef" part needs to be implemented by the user (you need to
> put something in the vtab). For my purposes, something like "raise
> AttributeError" would be sufficient.
>
> 1. Can this be accomplished currently in Cython?
>
> 2. Is this something that makes sense to add to Cython?
>
> 3. What do you think of the following syntax?
>
> cdef class MyClass(object):
>     cpdef virtual_method(self):
>         # implementation of "cdef" part ...
>
>     del virtual_method  # don't put virtual_method in the method table
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From greg.ewing at canterbury.ac.nz  Mon Jun 20 18:14:17 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 21 Jun 2016 10:14:17 +1200
Subject: [Cython] Virtual cpdef methods
In-Reply-To: <CADiQ+QAgOVcsA0Uutu8MX7qQozK9kdAdAmGyGQwDTZkEuDuFVw@mail.gmail.com>
References: <5767EAE3.4060607@cage.ugent.be>
 <CADiQ+QAgOVcsA0Uutu8MX7qQozK9kdAdAmGyGQwDTZkEuDuFVw@mail.gmail.com>
Message-ID: <57686AB9.50808@canterbury.ac.nz>

Robert Bradshaw wrote:
> All methods (cdef, cpdef, and def) are virtual by default in Cython,
> just like Python.
> 
> On Mon, Jun 20, 2016 at 6:08 AM, Jeroen Demeyer <jdemeyer at cage.ugent.be> wrote:
> 
>>I would like to have a "virtual" cpdef method: something which allows a fast
>>Cython call *when implemented* but without a default implementation.

It sounds like Jeroen really means "abstract" here, not "virtual".

The usual way to do this kind of thing in Python is to write
a stub method that raises NotImplementedError or such like.

-- 
Greg

From jdemeyer at cage.ugent.be  Tue Jun 21 02:17:55 2016
From: jdemeyer at cage.ugent.be (Jeroen Demeyer)
Date: Tue, 21 Jun 2016 08:17:55 +0200
Subject: [Cython] Virtual cpdef methods
In-Reply-To: <CADiQ+QAgOVcsA0Uutu8MX7qQozK9kdAdAmGyGQwDTZkEuDuFVw@mail.gmail.com>
References: <5767EAE3.4060607@cage.ugent.be>
 <CADiQ+QAgOVcsA0Uutu8MX7qQozK9kdAdAmGyGQwDTZkEuDuFVw@mail.gmail.com>
Message-ID: <5768DC13.70909@cage.ugent.be>

On 2016-06-20 17:31, Robert Bradshaw wrote:
> All methods (cdef, cpdef, and def) are virtual by default in Cython,
> just like Python.

I meant "pure virtual" or "abstract": just declared but without an 
implementation.

> Sounds like you want a cdef function to me.  You can override a cdef
> function with a cpdef function for any subclass that wishing to expose
> it to Python.

I was totally not aware that you could override cdef with cpdef.

But hang on, are you sure that this works? The C functions for cdef and 
cpdef functions have a different signature: cpdef functions have a 
__pyx_skip_dispatch argument while cdef functions don't have such an 
argument.

From jdemeyer at cage.ugent.be  Tue Jun 21 07:35:33 2016
From: jdemeyer at cage.ugent.be (Jeroen Demeyer)
Date: Tue, 21 Jun 2016 13:35:33 +0200
Subject: [Cython] Virtual cpdef methods
In-Reply-To: <5768DC13.70909@cage.ugent.be>
References: <5767EAE3.4060607@cage.ugent.be>
 <CADiQ+QAgOVcsA0Uutu8MX7qQozK9kdAdAmGyGQwDTZkEuDuFVw@mail.gmail.com>
 <5768DC13.70909@cage.ugent.be>
Message-ID: <57692685.7070004@cage.ugent.be>

On 2016-06-21 08:17, Jeroen Demeyer wrote:
> But hang on, are you sure that this works? The C functions for cdef and
> cpdef functions have a different signature: cpdef functions have a
> __pyx_skip_dispatch argument while cdef functions don't have such an
> argument.

Answering my own question, it seems that Cython uses different vtab 
entries for the cdef and cpdef functions, where the cdef one calls the 
cpdef one with __pyx_skip_dispatch=0.

The only difference with my proposal is that the cdef method cannot 
automatically be overridden by a def method. But that can be solved by

cdef class MyClass(object):
     cdef foo(self):
         return (<object>self).foo()

So this overriding of cdef methods by cpdef methods might completely 
solve my use case. Thanks!