From ralf.gommers at gmail.com Wed Aug 1 01:11:00 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 31 Jul 2018 22:11:00 -0700 Subject: [Numpy-discussion] Roadmap proposal, v3 In-Reply-To: References: <20180724190449.pxud4eebr7juktzq@carbo> <20180725225457.eccwrgyslc2rdugv@carbo> Message-ID: On Wed, Jul 25, 2018 at 9:19 PM, Ralf Gommers wrote: > > > On Thu, Jul 26, 2018 at 12:41 AM, Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > >> > Obviously the string dtype proposal in the roadmap is only a sketch at >> this point :). >> > >> > I do think that options listed currently (encoded strings with >> fixed-width storage and variable length strings) cover the breadth of >> proposals from last time. We may not want to implement all of them in >> NumPy, but I think we can agree that there are use cases for all them, even >> if only as external dtypes? >> >> Maybe :-) ? but I totally agree that more complete handling of strings >> should be on the roadmap. >> >> > Would it help to add "and/or" after the first bullet? Mostly I care >> about having like to have "improve string dtypes" in some form on the >> roadmap, and thought it would be helpful to list the concrete proposals >> that I recall. >> >> Sure, something like and/or that makes it clear that the details are >> yet to be determined would be great. >> >> > The actual design choices (especially if we proposal to change any >> default behavior) will certainly need a NEP. >> > > +1 the roadmap just contains topics/directions of interest. It's not the > place for any technical decisions. > > Related note: we are now using the "wish list" issue label for anything > that is too small to put on a roadmap or write a NEP for. Right now there's > a lot of random stuff in that label though ([1]), so I think we have to > clean that up. Examples of good wish list items that are on there now: > > - Document API generation with setup.py, genapi.py, etc. (gh-9203) > - Feature request: signal broadcasting is OK over core dimension (gh-8811) > - Multidimensional fftfreq/rfftfreq (gh-9094) > > I plan to go through and remove the label from issues that don't fit in > the next couple of days. > This is done now. Reduced from 36 to 10 issues. Would be good if we could try for a while to use this label consistently for tasks/enhancements that fall below the "put on the roadmap" threshold but are larger than, e.g., adding a keyword to one function. Cheers, Ralf > [1] https://github.com/numpy/numpy/issues?q=is%3Aopen+is% > 3Aissue+label%3A%2223+-+Wish+List%22 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Wed Aug 1 03:20:43 2018 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Wed, 1 Aug 2018 02:20:43 -0500 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: I realize this was probably brought up in the discussions about the scipy code of conduct which I have not looked at, but I?m troubled by the inclusion of ?political beliefs? in the document. See e.g. https://github.com/jupyter/governance/pull/5 As a thought experiment, what if someone?s political beliefs imply that other contributors are not deserving of human rights? Increasingly ideas like this are coming into the mainstream worldwide and I think this is a real concern that should be considered. On Mon, Jul 30, 2018 at 8:25 PM Charles R Harris wrote: > On Fri, Jul 27, 2018 at 4:02 PM, Stefan van der Walt > wrote: > >> Hi everyone, >> >> A while ago, SciPy (the library) adopted its Code of Conduct: >> >> https://docs.scipy.org/doc/scipy/reference/dev/conduct/code_of_conduct.html >> >> We worked hard to make that document friendly, while at the same time >> stating clearly the kinds of behavior that would and would not be >> tolerated. >> >> I propose that we adopt the SciPy code of conduct for NumPy as well. It >> is a good way to signal to newcomers that this is a community that cares >> about how people are treated. And I think we should do anything in our >> power to make NumPy as attractive as possible! >> >> If we adopt this document as policy, we will need to select a Code of >> Conduct committee, to whom potential transgressions can be reported. >> The individuals doing this for SciPy may very well be happy to do the >> same for NumPy, but the community should decide whom will best serve >> those roles. >> >> Let me know your thoughts. >> > > +1 from me. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Aug 1 10:49:21 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 1 Aug 2018 07:49:21 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 12:20 AM, Nathan Goldbaum wrote: > I realize this was probably brought up in the discussions about the scipy > code of conduct which I have not looked at, but I?m troubled by the > inclusion of ?political beliefs? in the document. > It was not brought up explicitly as far as I remember. > See e.g. > https://github.com/jupyter/governance/pull/5 > That's about moving names around. I don't see any mention of political beliefs? > As a thought experiment, what if someone?s political beliefs imply that > other contributors are not deserving of human rights? Increasingly ideas > like this are coming into the mainstream worldwide and I think this is a > real concern that should be considered. > There is a difference between having beliefs, and expressing those beliefs in ways that offends others. I don't see any problem with saying that we welcome anyone, irrespective of political belief. However, if someone starts expressing things that are intolerant (like someone else not deserving human rights) on any of our communication forums or in an in-person meeting, that would be a clear violation of the CoC. Which can be dealt with via the reporting and enforcement mechanism in the CoC. I don't see a problem here, but I would see a real problem with removing the "political beliefs" phrase. Cheers, Ralf > > On Mon, Jul 30, 2018 at 8:25 PM Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> On Fri, Jul 27, 2018 at 4:02 PM, Stefan van der Walt < >> stefanv at berkeley.edu> wrote: >> >>> Hi everyone, >>> >>> A while ago, SciPy (the library) adopted its Code of Conduct: >>> https://docs.scipy.org/doc/scipy/reference/dev/conduct/ >>> code_of_conduct.html >>> >>> We worked hard to make that document friendly, while at the same time >>> stating clearly the kinds of behavior that would and would not be >>> tolerated. >>> >>> I propose that we adopt the SciPy code of conduct for NumPy as well. It >>> is a good way to signal to newcomers that this is a community that cares >>> about how people are treated. And I think we should do anything in our >>> power to make NumPy as attractive as possible! >>> >>> If we adopt this document as policy, we will need to select a Code of >>> Conduct committee, to whom potential transgressions can be reported. >>> The individuals doing this for SciPy may very well be happy to do the >>> same for NumPy, but the community should decide whom will best serve >>> those roles. >>> >>> Let me know your thoughts. >>> >> >> +1 from me. >> >> Chuck >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Wed Aug 1 11:12:54 2018 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Wed, 1 Aug 2018 10:12:54 -0500 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 9:49 AM, Ralf Gommers wrote: > > > On Wed, Aug 1, 2018 at 12:20 AM, Nathan Goldbaum > wrote: > >> I realize this was probably brought up in the discussions about the scipy >> code of conduct which I have not looked at, but I?m troubled by the >> inclusion of ?political beliefs? in the document. >> > > It was not brought up explicitly as far as I remember. > > >> See e.g. >> https://github.com/jupyter/governance/pull/5 >> > > That's about moving names around. I don't see any mention of political > beliefs? > Sorry about that, I elided the 6. This is the correct link: https://github.com/jupyter/governance/pull/56 > > >> As a thought experiment, what if someone?s political beliefs imply that >> other contributors are not deserving of human rights? Increasingly ideas >> like this are coming into the mainstream worldwide and I think this is a >> real concern that should be considered. >> > > There is a difference between having beliefs, and expressing those beliefs > in ways that offends others. I don't see any problem with saying that we > welcome anyone, irrespective of political belief. However, if someone > starts expressing things that are intolerant (like someone else not > deserving human rights) on any of our communication forums or in an > in-person meeting, that would be a clear violation of the CoC. Which can be > dealt with via the reporting and enforcement mechanism in the CoC. > > I don't see a problem here, but I would see a real problem with removing > the "political beliefs" phrase. > For another perspective on this issue see https://where.coraline.codes/blog/oscon/, where Coraline Ada describes her reasons for not speaking at OSCON this year due to a similar clause in the code of conduct. Cheers, > Ralf > > > >> >> On Mon, Jul 30, 2018 at 8:25 PM Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> On Fri, Jul 27, 2018 at 4:02 PM, Stefan van der Walt < >>> stefanv at berkeley.edu> wrote: >>> >>>> Hi everyone, >>>> >>>> A while ago, SciPy (the library) adopted its Code of Conduct: >>>> https://docs.scipy.org/doc/scipy/reference/dev/conduct/code_ >>>> of_conduct.html >>>> >>>> We worked hard to make that document friendly, while at the same time >>>> stating clearly the kinds of behavior that would and would not be >>>> tolerated. >>>> >>>> I propose that we adopt the SciPy code of conduct for NumPy as well. It >>>> is a good way to signal to newcomers that this is a community that cares >>>> about how people are treated. And I think we should do anything in our >>>> power to make NumPy as attractive as possible! >>>> >>>> If we adopt this document as policy, we will need to select a Code of >>>> Conduct committee, to whom potential transgressions can be reported. >>>> The individuals doing this for SciPy may very well be happy to do the >>>> same for NumPy, but the community should decide whom will best serve >>>> those roles. >>>> >>>> Let me know your thoughts. >>>> >>> >>> +1 from me. >>> >>> Chuck >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Aug 1 11:33:25 2018 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 1 Aug 2018 16:33:25 +0100 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: Hi, On Wed, Aug 1, 2018 at 4:12 PM, Nathan Goldbaum wrote: > > > On Wed, Aug 1, 2018 at 9:49 AM, Ralf Gommers wrote: >> >> >> >> On Wed, Aug 1, 2018 at 12:20 AM, Nathan Goldbaum >> wrote: >>> >>> I realize this was probably brought up in the discussions about the scipy >>> code of conduct which I have not looked at, but I?m troubled by the >>> inclusion of ?political beliefs? in the document. >> >> >> It was not brought up explicitly as far as I remember. >> >>> >>> See e.g. >>> https://github.com/jupyter/governance/pull/5 >> >> >> That's about moving names around. I don't see any mention of political >> beliefs? > > > Sorry about that, I elided the 6. This is the correct link: > > https://github.com/jupyter/governance/pull/56 > >> >> >>> >>> As a thought experiment, what if someone?s political beliefs imply that >>> other contributors are not deserving of human rights? Increasingly ideas >>> like this are coming into the mainstream worldwide and I think this is a >>> real concern that should be considered. >> >> >> There is a difference between having beliefs, and expressing those beliefs >> in ways that offends others. I don't see any problem with saying that we >> welcome anyone, irrespective of political belief. However, if someone starts >> expressing things that are intolerant (like someone else not deserving human >> rights) on any of our communication forums or in an in-person meeting, that >> would be a clear violation of the CoC. Which can be dealt with via the >> reporting and enforcement mechanism in the CoC. >> >> I don't see a problem here, but I would see a real problem with removing >> the "political beliefs" phrase. > > > For another perspective on this issue see > https://where.coraline.codes/blog/oscon/, where Coraline Ada describes her > reasons for not speaking at OSCON this year due to a similar clause in the > code of conduct. I agree with Ralf. From your link: """ But the inclusion of this language, making political affiliation a protected class, leads me to believe that alt-right technologists would be as welcome at the conference as I would be. Including alt-right technologists who display on their clothing, for example, neo-Nazi insignia. Or t-shirts printed with anti-trans or anti-Black slogans. These could easily be interpreted as protected political speech. """ That's the point. If you wear a t-shirt with anti-trans or anti-Black slogans to a Scipy event covered by the code of conduct, that would qualify as 'expressing things that are intolerant', as Ralf put it. Cheers, Matthew From ralf.gommers at gmail.com Wed Aug 1 11:37:54 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 1 Aug 2018 08:37:54 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 8:12 AM, Nathan Goldbaum wrote: > > > On Wed, Aug 1, 2018 at 9:49 AM, Ralf Gommers > wrote: > >> >> >> On Wed, Aug 1, 2018 at 12:20 AM, Nathan Goldbaum >> wrote: >> >>> I realize this was probably brought up in the discussions about the >>> scipy code of conduct which I have not looked at, but I?m troubled by the >>> inclusion of ?political beliefs? in the document. >>> >> >> It was not brought up explicitly as far as I remember. >> >> >>> See e.g. >>> https://github.com/jupyter/governance/pull/5 >>> >> >> That's about moving names around. I don't see any mention of political >> beliefs? >> > > Sorry about that, I elided the 6. This is the correct link: > > https://github.com/jupyter/governance/pull/56 > Thanks, that's useful context for your question. I'm personally not too attached to "political belief", but I think the discussion in that PR and in the OSCON context is very US-centric and reflective of the polarized atmosphere there. If everyone is fine with removing political beliefs then I'm fine with that, but I don't think that the argument itself (from a non-US perspective) has much merit. > > >> >> >>> As a thought experiment, what if someone?s political beliefs imply that >>> other contributors are not deserving of human rights? Increasingly ideas >>> like this are coming into the mainstream worldwide and I think this is a >>> real concern that should be considered. >>> >> >> There is a difference between having beliefs, and expressing those >> beliefs in ways that offends others. I don't see any problem with saying >> that we welcome anyone, irrespective of political belief. However, if >> someone starts expressing things that are intolerant (like someone else not >> deserving human rights) on any of our communication forums or in an >> in-person meeting, that would be a clear violation of the CoC. Which can be >> dealt with via the reporting and enforcement mechanism in the CoC. >> >> I don't see a problem here, but I would see a real problem with removing >> the "political beliefs" phrase. >> > > For another perspective on this issue see https://where.coraline.codes/ > blog/oscon/, where Coraline Ada describes her reasons for not speaking at > OSCON this year due to a similar clause in the code of conduct. > There's a lot of very unrealistic examples in that post. Plus retracting a week in advance of a conference is, to put it mildly, questionable. So not sure what to think of the rest of that post. There may be good points in there, but they're obscured by the obvious flaws in thinking and choice of examples. Cheers, Ralf > > > Cheers, >> Ralf >> >> >> >>> >>> On Mon, Jul 30, 2018 at 8:25 PM Charles R Harris < >>> charlesr.harris at gmail.com> wrote: >>> >>>> On Fri, Jul 27, 2018 at 4:02 PM, Stefan van der Walt < >>>> stefanv at berkeley.edu> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> A while ago, SciPy (the library) adopted its Code of Conduct: >>>>> https://docs.scipy.org/doc/scipy/reference/dev/conduct/code_ >>>>> of_conduct.html >>>>> >>>>> We worked hard to make that document friendly, while at the same time >>>>> stating clearly the kinds of behavior that would and would not be >>>>> tolerated. >>>>> >>>>> I propose that we adopt the SciPy code of conduct for NumPy as well. >>>>> It >>>>> is a good way to signal to newcomers that this is a community that >>>>> cares >>>>> about how people are treated. And I think we should do anything in our >>>>> power to make NumPy as attractive as possible! >>>>> >>>>> If we adopt this document as policy, we will need to select a Code of >>>>> Conduct committee, to whom potential transgressions can be reported. >>>>> The individuals doing this for SciPy may very well be happy to do the >>>>> same for NumPy, but the community should decide whom will best serve >>>>> those roles. >>>>> >>>>> Let me know your thoughts. >>>>> >>>> >>>> +1 from me. >>>> >>>> Chuck >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Wed Aug 1 11:55:10 2018 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Wed, 1 Aug 2018 17:55:10 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: I agree with Ralf. That thread is more towards a US based separation. Actually we briefly touched upon these on the SciPy side but indeed there was no real discussion. Political beliefs (especially communism in US for a practical example) can offend some people and that's OK because being offended by itself doesn't have a merit. There will always be someone getting offended by anything. But racism, sexism etc. are not "political" stances and deserve to be united against regardless whether someone is offended or not. Actually calling these discriminating doctrines as "political beliefs" is making me quite nervous instead. On Wed, Aug 1, 2018 at 5:37 PM, Ralf Gommers wrote: > > > On Wed, Aug 1, 2018 at 8:12 AM, Nathan Goldbaum > wrote: > >> >> >> On Wed, Aug 1, 2018 at 9:49 AM, Ralf Gommers >> wrote: >> >>> >>> >>> On Wed, Aug 1, 2018 at 12:20 AM, Nathan Goldbaum >>> wrote: >>> >>>> I realize this was probably brought up in the discussions about the >>>> scipy code of conduct which I have not looked at, but I?m troubled by the >>>> inclusion of ?political beliefs? in the document. >>>> >>> >>> It was not brought up explicitly as far as I remember. >>> >>> >>>> See e.g. >>>> https://github.com/jupyter/governance/pull/5 >>>> >>> >>> That's about moving names around. I don't see any mention of political >>> beliefs? >>> >> >> Sorry about that, I elided the 6. This is the correct link: >> >> https://github.com/jupyter/governance/pull/56 >> > > Thanks, that's useful context for your question. > > I'm personally not too attached to "political belief", but I think the > discussion in that PR and in the OSCON context is very US-centric and > reflective of the polarized atmosphere there. > > If everyone is fine with removing political beliefs then I'm fine with > that, but I don't think that the argument itself (from a non-US > perspective) has much merit. > > >> >> >>> >>> >>>> As a thought experiment, what if someone?s political beliefs imply that >>>> other contributors are not deserving of human rights? Increasingly ideas >>>> like this are coming into the mainstream worldwide and I think this is a >>>> real concern that should be considered. >>>> >>> >>> There is a difference between having beliefs, and expressing those >>> beliefs in ways that offends others. I don't see any problem with saying >>> that we welcome anyone, irrespective of political belief. However, if >>> someone starts expressing things that are intolerant (like someone else not >>> deserving human rights) on any of our communication forums or in an >>> in-person meeting, that would be a clear violation of the CoC. Which can be >>> dealt with via the reporting and enforcement mechanism in the CoC. >>> >>> I don't see a problem here, but I would see a real problem with removing >>> the "political beliefs" phrase. >>> >> >> For another perspective on this issue see https://where.coraline.codes/b >> log/oscon/, where Coraline Ada describes her reasons for not speaking at >> OSCON this year due to a similar clause in the code of conduct. >> > > There's a lot of very unrealistic examples in that post. Plus retracting a > week in advance of a conference is, to put it mildly, questionable. So not > sure what to think of the rest of that post. There may be good points in > there, but they're obscured by the obvious flaws in thinking and choice of > examples. > > Cheers, > Ralf > > >> >> >> Cheers, >>> Ralf >>> >>> >>> >>>> >>>> On Mon, Jul 30, 2018 at 8:25 PM Charles R Harris < >>>> charlesr.harris at gmail.com> wrote: >>>> >>>>> On Fri, Jul 27, 2018 at 4:02 PM, Stefan van der Walt < >>>>> stefanv at berkeley.edu> wrote: >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> A while ago, SciPy (the library) adopted its Code of Conduct: >>>>>> https://docs.scipy.org/doc/scipy/reference/dev/conduct/code_ >>>>>> of_conduct.html >>>>>> >>>>>> We worked hard to make that document friendly, while at the same time >>>>>> stating clearly the kinds of behavior that would and would not be >>>>>> tolerated. >>>>>> >>>>>> I propose that we adopt the SciPy code of conduct for NumPy as well. >>>>>> It >>>>>> is a good way to signal to newcomers that this is a community that >>>>>> cares >>>>>> about how people are treated. And I think we should do anything in >>>>>> our >>>>>> power to make NumPy as attractive as possible! >>>>>> >>>>>> If we adopt this document as policy, we will need to select a Code of >>>>>> Conduct committee, to whom potential transgressions can be reported. >>>>>> The individuals doing this for SciPy may very well be happy to do the >>>>>> same for NumPy, but the community should decide whom will best serve >>>>>> those roles. >>>>>> >>>>>> Let me know your thoughts. >>>>>> >>>>> >>>>> +1 from me. >>>>> >>>>> Chuck >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at python.org >>>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 1 11:57:21 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Aug 2018 09:57:21 -0600 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 9:37 AM, Ralf Gommers wrote: > > > On Wed, Aug 1, 2018 at 8:12 AM, Nathan Goldbaum > wrote: > >> >> >> On Wed, Aug 1, 2018 at 9:49 AM, Ralf Gommers >> wrote: >> >>> >>> >>> On Wed, Aug 1, 2018 at 12:20 AM, Nathan Goldbaum >>> wrote: >>> >>>> I realize this was probably brought up in the discussions about the >>>> scipy code of conduct which I have not looked at, but I?m troubled by the >>>> inclusion of ?political beliefs? in the document. >>>> >>> >>> It was not brought up explicitly as far as I remember. >>> >>> >>>> See e.g. >>>> https://github.com/jupyter/governance/pull/5 >>>> >>> >>> That's about moving names around. I don't see any mention of political >>> beliefs? >>> >> >> Sorry about that, I elided the 6. This is the correct link: >> >> https://github.com/jupyter/governance/pull/56 >> > > Thanks, that's useful context for your question. > > I'm personally not too attached to "political belief", but I think the > discussion in that PR and in the OSCON context is very US-centric and > reflective of the polarized atmosphere there. > > If everyone is fine with removing political beliefs then I'm fine with > that, but I don't think that the argument itself (from a non-US > perspective) has much merit. > I'm strongly opposed to removing it. The last thing I want is to have politics brought into NumPy development, which is where this discussion is already headed. It could get ugly fast. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From insertinterestingnamehere at gmail.com Wed Aug 1 12:48:09 2018 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Wed, 1 Aug 2018 11:48:09 -0500 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: This. Even from a US perspective, we really need to not let political division into even more apparently non-political things. As far as I can tell, the current language seems to be there to specifically avoid that. It isn't there to allow discrimination if someone tries to claim they're making a political statement. Best, Ian On Wed, Aug 1, 2018 at 10:58 AM Charles R Harris wrote: > On Wed, Aug 1, 2018 at 9:37 AM, Ralf Gommers > wrote: > >> >> >> On Wed, Aug 1, 2018 at 8:12 AM, Nathan Goldbaum >> wrote: >> >>> >>> >>> On Wed, Aug 1, 2018 at 9:49 AM, Ralf Gommers >>> wrote: >>> >>>> >>>> >>>> On Wed, Aug 1, 2018 at 12:20 AM, Nathan Goldbaum >>> > wrote: >>>> >>>>> I realize this was probably brought up in the discussions about the >>>>> scipy code of conduct which I have not looked at, but I?m troubled by the >>>>> inclusion of ?political beliefs? in the document. >>>>> >>>> >>>> It was not brought up explicitly as far as I remember. >>>> >>>> >>>>> See e.g. >>>>> https://github.com/jupyter/governance/pull/5 >>>>> >>>> >>>> That's about moving names around. I don't see any mention of political >>>> beliefs? >>>> >>> >>> Sorry about that, I elided the 6. This is the correct link: >>> >>> https://github.com/jupyter/governance/pull/56 >>> >> >> Thanks, that's useful context for your question. >> >> I'm personally not too attached to "political belief", but I think the >> discussion in that PR and in the OSCON context is very US-centric and >> reflective of the polarized atmosphere there. >> >> If everyone is fine with removing political beliefs then I'm fine with >> that, but I don't think that the argument itself (from a non-US >> perspective) has much merit. >> > > I'm strongly opposed to removing it. The last thing I want is to have > politics brought into NumPy development, which is where this discussion is > already headed. It could get ugly fast. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Wed Aug 1 20:27:03 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Wed, 1 Aug 2018 17:27:03 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol Message-ID: I propose to accept NEP-18, "A dispatch mechanism for NumPy?s high level array functions": http://www.numpy.org/neps/nep-0018-array-function-protocol.html Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is out of scope for the initial proposal in the NEP. If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details. Cheers, Stpehan -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Aug 1 20:57:32 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 1 Aug 2018 17:57:32 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 8:37 AM, Ralf Gommers wrote: > > On Wed, Aug 1, 2018 at 8:12 AM, Nathan Goldbaum > wrote: >> >> On Wed, Aug 1, 2018 at 9:49 AM, Ralf Gommers >> wrote: >>> >>> >>> >>> On Wed, Aug 1, 2018 at 12:20 AM, Nathan Goldbaum >>> wrote: >>>> >>>> I realize this was probably brought up in the discussions about the >>>> scipy code of conduct which I have not looked at, but I?m troubled by the >>>> inclusion of ?political beliefs? in the document. >>> >>> >>> It was not brought up explicitly as far as I remember. >>> >>>> >>>> See e.g. >>>> https://github.com/jupyter/governance/pull/5 >>> >>> >>> That's about moving names around. I don't see any mention of political >>> beliefs? >> >> >> Sorry about that, I elided the 6. This is the correct link: >> >> https://github.com/jupyter/governance/pull/56 > > > Thanks, that's useful context for your question. > > I'm personally not too attached to "political belief", but I think the > discussion in that PR and in the OSCON context is very US-centric and > reflective of the polarized atmosphere there. > > If everyone is fine with removing political beliefs then I'm fine with that, > but I don't think that the argument itself (from a non-US perspective) has > much merit. Probably a plurality of contributors are from the US though, so we can't exactly hand-wave away how it reads in the US... >>>> As a thought experiment, what if someone?s political beliefs imply that >>>> other contributors are not deserving of human rights? Increasingly ideas >>>> like this are coming into the mainstream worldwide and I think this is a >>>> real concern that should be considered. >>> >>> >>> There is a difference between having beliefs, and expressing those >>> beliefs in ways that offends others. I don't see any problem with saying >>> that we welcome anyone, irrespective of political belief. However, if >>> someone starts expressing things that are intolerant (like someone else not >>> deserving human rights) on any of our communication forums or in an >>> in-person meeting, that would be a clear violation of the CoC. Which can be >>> dealt with via the reporting and enforcement mechanism in the CoC. >>> >>> I don't see a problem here, but I would see a real problem with removing >>> the "political beliefs" phrase. I think there's two separate issues here: (a) whether things people express outside our forums should be considered relevant within our forums, (b) what we're communicating by calling out "political belief" as a thing that we won't exclude people over. This isn't happening in a vacuum either... if you search for "lambdaconf" you can see a ton of discussion about a programming conference that decided to invite a guy who's a high-profile leader in the pro-feudalism / pro-white-supremacism community (I can't believe I'm typing that), and then conference justified it by saying things like "we're supporting diversity of beliefs" and "this talk is purely about his technical work, let's keep politics out of it". As it turns out, the technical work in question is actually super tied to his political activism -- like in the original version of the system, his crypto key was the "king" that gave him ultimate authority, which he could delegate to "dukes", etc. (IIRC -- I'm not looking up the details, but it was something about that blatant.) Also all kinds of cult-y stuff about how contributors were special chosen ones, them against the world -- it was super creepy. But then he got VC funding, because of course he did, and went through and renamed things as a figleaf, so they could claim it was "purely technical". FWIW, my personal opinion is that I'm categorically uninterested in working with anyone like that. I don't care if they only say the ghastly things in non-numpy channels or claim that it's "merely a political disagreement", I'm still not interested. >> For another perspective on this issue see >> https://where.coraline.codes/blog/oscon/, where Coraline Ada describes her >> reasons for not speaking at OSCON this year due to a similar clause in the >> code of conduct. > > There's a lot of very unrealistic examples in that post. Plus retracting a > week in advance of a conference is, to put it mildly, questionable. So not > sure what to think of the rest of that post. There may be good points in > there, but they're obscured by the obvious flaws in thinking and choice of > examples. Ralf, I love you, but this paragraph sounds like a parody from "How to suppress women's writing" or something. Coraline Ada is a prominent expert on code-of-conduct issues, and also a trans woman, so she gets death threats and other harassment constantly and "will the conference organizers protect me if someone comes after me?" is a real question she has to ask. She wrote a blog post about how O'Reilly's handling of this (not just the language, but the totality of circumstances -- the way it was added, the response to her queries, etc.) made her feel that attending would be unsafe for her, so she didn't attend. (And about how distressed she was to realize this just a week before the conference.) It seems like you're taking her post as some logical argument about CoCs in the abstract, with the withdrawal as some kind of brinksmanship, and judging it by those standards? FWIW, the Sage Sharp who's quoted at the beginning of Ada's post and initially raised the issue, is also well-known expert on CoC issues, e.g. PyCon hired them to help revamp their policies and respond to incidents this year [1]. When experts say that something is a bad idea, and when the people who a CoC is supposed to protect says it makes them feel unsafe, I feel like we should listen to that. I also thought that the points made in the Jupyter discussion thread made a lot of sense: of course it's possible for people to start harassing each other over any excuse, and a CoC can, should, and does make clear that that's not OK. But if you specifically *call out* political affiliation as a protected class, at a time when lots of the people who the CoC is trying to protect are facing governmental harassment justified as "mere political disagreement", then it really sends the wrong message. Besides, uh... isn't the whole definition of politics that it's topics where there is active debate? Not really sure why it's even in that list to start with. -n [1] https://pycon.blogspot.com/2018/04/code-of-conduct-updates-for-pycon-2018.html -- Nathaniel J. Smith -- https://vorpus.org From rmay31 at gmail.com Wed Aug 1 23:34:08 2018 From: rmay31 at gmail.com (Ryan May) Date: Wed, 1 Aug 2018 21:34:08 -0600 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: > > When experts say that something is a bad idea, and when the people who > a CoC is supposed to protect says it makes them feel unsafe, I feel > like we should listen to that. > > I also thought that the points made in the Jupyter discussion thread > made a lot of sense: of course it's possible for people to start > harassing each other over any excuse, and a CoC can, should, and does > make clear that that's not OK. But if you specifically *call out* > political affiliation as a protected class, at a time when lots of the > people who the CoC is trying to protect are facing governmental > harassment justified as "mere political disagreement", then it really > sends the wrong message. > > Besides, uh... isn't the whole definition of politics that it's topics > where there is active debate? Not really sure why it's even in that > list to start with. > So I hear all the arguments about people feeling unsafe due to some truly despicable, discriminatory behavior, and I want absolutely no parts of protecting that. However, I also recognize that we in the U.S. are in a particularly divisive atmosphere, and people of varied political persuasions want absolutely nothing to do with those who share differing views. So, as a concrete example, if someone were to show up at a NumPy developer summit with a MAGA ("Make America Great Again") hat, or talks about their support for the president in non-numpy channels, WITHOUT expressing anything discriminatory or support for such views, if "political beliefs" is not in the CoC, is this person welcome? I'm not worried about my own views, but I have friends of widely varying views, and I truly wonder if they would be welcome. With differing "political beliefs" listed as something welcomed, I feel ok for them; if this language is removed, I'm much less certain. IMO, "political beliefs" encompasses so much more things than a handful of very specific, hateful views. People can disagree about a wide array of "political beliefs" and it is important that we as a community welcome a wide array of such views. If the CoC needs to protect against the wide array of discriminatory views and behavior that make up U.S. politics right now, how about specifically calling those behaviors out as not-welcome, rather than completely ignoring the fact that 99% of "political beliefs" are perfectly welcome within the community? The CoC is about spelling out the community norms--how about just spelling out that we welcome everyone, but, in the words of Will Wheaton, "Don't be a dick"? Ryan -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Aug 2 00:26:35 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 1 Aug 2018 21:26:35 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 8:34 PM, Ryan May wrote: > So I hear all the arguments about people feeling unsafe due to some truly > despicable, discriminatory behavior, and I want absolutely no parts of > protecting that. However, I also recognize that we in the U.S. are in a > particularly divisive atmosphere, and people of varied political persuasions > want absolutely nothing to do with those who share differing views. So, as a > concrete example, if someone were to show up at a NumPy developer summit > with a MAGA ("Make America Great Again") hat, or talks about their support > for the president in non-numpy channels, WITHOUT expressing anything > discriminatory or support for such views, if "political beliefs" is not in > the CoC, is this person welcome? I'm not worried about my own views, but I > have friends of widely varying views, and I truly wonder if they would be > welcome. With differing "political beliefs" listed as something welcomed, I > feel ok for them; if this language is removed, I'm much less certain. This is a difficult scenario. I do know lots of people who are uncomfortable with MAGA hats, and it's not because of they disagree about the details of some farm bill or whatever, it's because it's increasingly impossible to use that slogan without also expressing support for racism, sexism, transphobia, etc. -- i.e., all the other things that the CoC lists as unacceptable. So I feel like... to the extent that some political position *isn't* tied up with those things, I can't see why people would have a problem with it, or why the CoC would need to bother mentioning it. (There are all kinds of things we don't mention ? "we welcome people with odd *and* even telephone numbers!") And to the extent that some political position *is* tied up with those things... it would be pretty contradictory to list it as a protected class alongside race, sex, etc. I'm not going to go track people down on facebook and try to guess how they voted or something, but if someone wore a MAGA hat to a numpy sprint then I'd be totally fine with asking them to take it off in consideration of the other participants. -n -- Nathaniel J. Smith -- https://vorpus.org From robert.kern at gmail.com Thu Aug 2 01:04:34 2018 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 1 Aug 2018 22:04:34 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 8:35 PM Ryan May wrote: > When experts say that something is a bad idea, and when the people who >> a CoC is supposed to protect says it makes them feel unsafe, I feel >> like we should listen to that. >> >> I also thought that the points made in the Jupyter discussion thread >> made a lot of sense: of course it's possible for people to start >> harassing each other over any excuse, and a CoC can, should, and does >> make clear that that's not OK. But if you specifically *call out* >> political affiliation as a protected class, at a time when lots of the >> people who the CoC is trying to protect are facing governmental >> harassment justified as "mere political disagreement", then it really >> sends the wrong message. >> >> Besides, uh... isn't the whole definition of politics that it's topics >> where there is active debate? Not really sure why it's even in that >> list to start with. >> > > So I hear all the arguments about people feeling unsafe due to some truly > despicable, discriminatory behavior, and I want absolutely no parts of > protecting that. However, I also recognize that we in the U.S. are in a > particularly divisive atmosphere, and people of varied political > persuasions want absolutely nothing to do with those who share differing > views. So, as a concrete example, if someone were to show up at a NumPy > developer summit with a MAGA ("Make America Great Again") hat, or talks > about their support for the president in non-numpy channels, WITHOUT > expressing anything discriminatory or support for such views, if "political > beliefs" is not in the CoC, is this person welcome? I'm not worried about > my own views, but I have friends of widely varying views, and I truly > wonder if they would be welcome. With differing "political beliefs" listed > as something welcomed, I feel ok for them; if this language is removed, I'm > much less certain. > > IMO, "political beliefs" encompasses so much more things than a handful of > very specific, hateful views. People can disagree about a wide array of > "political beliefs" and it is important that we as a community welcome a > wide array of such views. If the CoC needs to protect against the wide > array of discriminatory views and behavior that make up U.S. politics right > now, how about specifically calling those behaviors out as not-welcome, > rather than completely ignoring the fact that 99% of "political beliefs" > are perfectly welcome within the community? > > The CoC is about spelling out the community norms--how about just spelling > out that we welcome everyone, but, in the words of Will Wheaton, "Don't be > a dick"? > I agree that it's worth clarifying in the text what this clause is intended to do. I think it has been misinterpreted as defining a legalistic set of protected classes along the lines of anti-discrimination laws and can be interpreted by itself outside of the context of the CoC as a whole. But it's not that. It's an aspirational statement, and a high one, at that, if we interpret it at its broadest. We will fail to meet it, in its entirety, and that's *okay* if the spirit of the CoC is being defended. I am perfectly happy to keep "political beliefs" explicit in the CoC and still boot the neo-feudalist for making the project's/conference's environment unwelcoming for a more vulnerable group of people, even if just by their presence. I *am* sensitive to how nominally well-intentioned "viewpoint diversity" efforts get hijacked by regressives looking to (re)assert their traditional power. But that problem is mostly confined to conferences who need to seek speakers and has less relevance to numpy, which largely doesn't run much except sprints. I think we can resolve that elsewhere, if not another document, then at least another clause. A CoC has to pull a kind of double duty: be friendly enough to digest for a newcomer and also be helpful to project organizers to make tough balancing decisions. We don't have to expect each sentence to pull that double duty on its own. I don't quite know what the phrasing would be (because, again, we don't run conferences), but I think we could make a statement that explicitly disclaims that we will be using "viewpoint diversity" to provide a platform for viewpoints antithetical to the CoC. None of these categorizations listed should be interpreted as get-out-of-jail-free cards for otherwise unwelcoming behavior, and I think maybe we should be explicit about that. Our diversity statement is an aspiration, not a suicide pact. Religion, neurotype, national origin, and subculture (4chan is a subculture, God help us), at minimum, are all items on that list that I have personally seen used to justify shitty behavior. Political belief is far from unique (nor the most common excuse, in my experience) in that list. But they all deserve to be on that list. I want the somewhat fringy progressive hacktivist to feel comfortable here as well as people more mainstream. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.corlay at gmail.com Thu Aug 2 06:04:09 2018 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Thu, 2 Aug 2018 12:04:09 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: The "political belief" was recently removed from the Jupyter CoC. One reason for this decision is that *Racism and Sexism are increasingly considered as mainstream "political beliefs"*, and we wanted to make it clear that people can still be sanctioned for e.g. sexist or racist behavior when engaging with the project (at events, on the mailing list or GitHub...) even if their racism and sexism is corresponds to a "political belief". It is still not OK for people to be excluded or discriminated against because of their political affiliation. The CoC statement reads "*This includes, but is not limited to...*". Also we* don't wish to prioritize or elevate any members of a particular political belief to the same level as any members of the examples remaining in the document*. Ultimately, the CoC committee uses their own judgement to assess reports and the appropriate response. Best, Sylvain On Thu, Aug 2, 2018 at 7:04 AM, Robert Kern wrote: > On Wed, Aug 1, 2018 at 8:35 PM Ryan May wrote: > >> When experts say that something is a bad idea, and when the people who >>> a CoC is supposed to protect says it makes them feel unsafe, I feel >>> like we should listen to that. >>> >>> I also thought that the points made in the Jupyter discussion thread >>> made a lot of sense: of course it's possible for people to start >>> harassing each other over any excuse, and a CoC can, should, and does >>> make clear that that's not OK. But if you specifically *call out* >>> political affiliation as a protected class, at a time when lots of the >>> people who the CoC is trying to protect are facing governmental >>> harassment justified as "mere political disagreement", then it really >>> sends the wrong message. >>> >>> Besides, uh... isn't the whole definition of politics that it's topics >>> where there is active debate? Not really sure why it's even in that >>> list to start with. >>> >> >> So I hear all the arguments about people feeling unsafe due to some truly >> despicable, discriminatory behavior, and I want absolutely no parts of >> protecting that. However, I also recognize that we in the U.S. are in a >> particularly divisive atmosphere, and people of varied political >> persuasions want absolutely nothing to do with those who share differing >> views. So, as a concrete example, if someone were to show up at a NumPy >> developer summit with a MAGA ("Make America Great Again") hat, or talks >> about their support for the president in non-numpy channels, WITHOUT >> expressing anything discriminatory or support for such views, if "political >> beliefs" is not in the CoC, is this person welcome? I'm not worried about >> my own views, but I have friends of widely varying views, and I truly >> wonder if they would be welcome. With differing "political beliefs" listed >> as something welcomed, I feel ok for them; if this language is removed, I'm >> much less certain. >> >> IMO, "political beliefs" encompasses so much more things than a handful >> of very specific, hateful views. People can disagree about a wide array of >> "political beliefs" and it is important that we as a community welcome a >> wide array of such views. If the CoC needs to protect against the wide >> array of discriminatory views and behavior that make up U.S. politics right >> now, how about specifically calling those behaviors out as not-welcome, >> rather than completely ignoring the fact that 99% of "political beliefs" >> are perfectly welcome within the community? >> >> The CoC is about spelling out the community norms--how about just >> spelling out that we welcome everyone, but, in the words of Will Wheaton, >> "Don't be a dick"? >> > > I agree that it's worth clarifying in the text what this clause is > intended to do. I think it has been misinterpreted as defining a legalistic > set of protected classes along the lines of anti-discrimination laws and > can be interpreted by itself outside of the context of the CoC as a whole. > But it's not that. It's an aspirational statement, and a high one, at that, > if we interpret it at its broadest. We will fail to meet it, in its > entirety, and that's *okay* if the spirit of the CoC is being defended. I > am perfectly happy to keep "political beliefs" explicit in the CoC and > still boot the neo-feudalist for making the project's/conference's > environment unwelcoming for a more vulnerable group of people, even if just > by their presence. I *am* sensitive to how nominally well-intentioned > "viewpoint diversity" efforts get hijacked by regressives looking to > (re)assert their traditional power. But that problem is mostly confined to > conferences who need to seek speakers and has less relevance to numpy, > which largely doesn't run much except sprints. I think we can resolve that > elsewhere, if not another document, then at least another clause. A CoC has > to pull a kind of double duty: be friendly enough to digest for a newcomer > and also be helpful to project organizers to make tough balancing > decisions. We don't have to expect each sentence to pull that double duty > on its own. I don't quite know what the phrasing would be (because, again, > we don't run conferences), but I think we could make a statement that > explicitly disclaims that we will be using "viewpoint diversity" to provide > a platform for viewpoints antithetical to the CoC. > > None of these categorizations listed should be interpreted as > get-out-of-jail-free cards for otherwise unwelcoming behavior, and I think > maybe we should be explicit about that. Our diversity statement is an > aspiration, not a suicide pact. Religion, neurotype, national origin, and > subculture (4chan is a subculture, God help us), at minimum, are all items > on that list that I have personally seen used to justify shitty behavior. > Political belief is far from unique (nor the most common excuse, in my > experience) in that list. But they all deserve to be on that list. I want > the somewhat fringy progressive hacktivist to feel comfortable here as well > as people more mainstream. > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Thu Aug 2 06:35:36 2018 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Thu, 2 Aug 2018 06:35:36 -0400 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: I think a legalistic focus on the letter rather than the spirit of the code of conduct is not that helpful (and probably what makes if feel US centric - funny how court systems end up shaping countries). My preference would be to keep exactly the scipy version, so that at least for these two highly related project there is just one code of conduct. But for what it is worth, it does seem to me political views belongs on the list - certainly as likely to give problems historically as for any of the others on the list. If we do end up with a different version, I'd prefer a really short one, like just stating the golden rule (treat others as one would like to be treated oneself). -- Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Aug 2 08:47:04 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 2 Aug 2018 05:47:04 -0700 Subject: [Numpy-discussion] Roadmap proposal, v3 In-Reply-To: <20180724190449.pxud4eebr7juktzq@carbo> References: <20180724190449.pxud4eebr7juktzq@carbo> Message-ID: On Tue, Jul 24, 2018 at 12:04 PM, Stefan van der Walt wrote: > Hi everyone, > > Please take a look at the latest roadmap proposal: > > https://github.com/numpy/numpy/pull/11611 > > This is a living document, so can easily be modified in the future, but > we'd like to get in place a document that corresponds fairly closely > with current community priorities. > The green button was pressed, the roadmap is now live on http://www.numpy.org/neps/. Thanks all! Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Aug 2 08:58:33 2018 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 02 Aug 2018 14:58:33 +0200 Subject: [Numpy-discussion] Roadmap proposal, v3 In-Reply-To: References: <20180724190449.pxud4eebr7juktzq@carbo> Message-ID: <24aff01026547d3b10b5d1f4e7c25a7bee679b39.camel@sipsolutions.net> On Thu, 2018-08-02 at 05:47 -0700, Ralf Gommers wrote: > > > On Tue, Jul 24, 2018 at 12:04 PM, Stefan van der Walt ey.edu> wrote: > > Hi everyone, > > > > Please take a look at the latest roadmap proposal: > > > > https://github.com/numpy/numpy/pull/11611 > > > > This is a living document, so can easily be modified in the future, > > but > > we'd like to get in place a document that corresponds fairly > > closely > > with current community priorities. > > The green button was pressed, the roadmap is now live on http://www.n > umpy.org/neps/. Thanks all! > Great, I hope we can check off some of them soon! :) - Sebastian > Cheers, > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Thu Aug 2 08:57:23 2018 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 02 Aug 2018 14:57:23 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: <1136f7ac8bab52527e28999170ac977b6a292c00.camel@sipsolutions.net> On Thu, 2018-08-02 at 12:04 +0200, Sylvain Corlay wrote: > The "political belief" was recently removed from the Jupyter CoC. > > One reason for this decision is that Racism and Sexism are > increasingly considered as mainstream "political beliefs", and we > wanted to make it clear that people can still be sanctioned for e.g. > sexist or racist behavior when engaging with the project (at events, > on the mailing list or GitHub...) even if their racism and sexism is > corresponds to a "political belief". > > It is still not OK for people to be excluded or discriminated against > because of their political affiliation. The CoC statement reads "This > includes, but is not limited to...". Also we don't wish to prioritize > or elevate any members of a particular political belief to the same > level as any members of the examples remaining in the > document. Ultimately, the CoC committee uses their own judgement to > assess reports and the appropriate response. > TL;DR: I don't think it matters, as for the CoC as such, it seems fine to me, lets just put it in and be done with it. I do not think we should have a long discussion (about that list), and in case it might go there, would suggest we try to find a way to refuse to have it. Maybe by letting the committee that is in the CoC decide. Actually: I am good with the people currently listed for SciPy if they will do it, or anyone else wants to jump in? I won't really follow the discussion much more (except for reading) and do not feel like I really know enough about CoCs, but my point is, that I do not care much. The CoC as suggested seems pretty uncontroversial to me (it does not draw any hard lines worth fighting over). And that is probably the only current believe I have, that I think it should not really draw those lines. Political opinion being included or not? I am not sure I care, because as I read it and you point out, it does not really matter whether or not it is included, including it would just raise awareness for a specific issue. This is not about freedom to express political believes (on numpy channels), I suppose there may be a point where even a boycott can be discriminatory and things may be tricky to assess [1], but again those cases need careful weighing (by the committee mostly), a CoC might bias this a little, but not much, and if we decide which way to bias it we might end up fighting, so lets refuse to do it outside specific cases? Freedom of expression is always limited by the protection of other individuals rights (note that I believe in the US this freedom tends to be held very high when weighing the two). But, since there is normally no reason for voicing political opinions on numpy, it seems obvious to me that it will tend to lose when weighed against the other persons rights being protected [2]. Weighing different "rights" is always tricky, but cannot be avoided or really formalized too much IMO [3,4]. Which comes to the point that I think the list is one to raise awareness for and be welcoming to specific people (either very general or minority), who have in the past (or currently) not felt welcome. And such a list always will be set in the current time/mentality. We are maybe in an odd spot where political discussion/judicial progress feels lagging behind social development (and some fronts are hardening :(), which makes things a bit trickier. Overall, all it would do is to maybe suggested that "political opinion" is currently not something that need special raised awareness. It does not mean this defines a "bias", nor that the list cannot change at some point. Either way, I do not read the list as giving any additional protection for *voicing* your opinion. In fact, I would argue the opposite may be the case. If you voice it you make the opposite (political) opinion feel less welcome, and since there is no reason for voicing a political opinion *on a numpy channel* when weighing those against each other it seems like a hard case [5]. At some points lines may have to be drawn (and drawing them once, does not set them in stone for the next time!). I do not think we draw or should draw them (much) with this statement itself, the statement says that they will be drawn if and when necessary and then it will be done so carefully. Plus it generally raises awareness and gives a bit guidelines. It seems to me that this may be the actual discussion with many of those other discussions. Not so much the wording, but over how exactly lines were drawn in practice. Sure, probably we set a bit of bias with the list, but I doubt it is enough to fight over. And hopefully we can avoid a huge discussion :) (for now looks like it). Best, Sebastian PS: I do not mind synchronizing numpy and scipy (or numpy and Jupyter or all three) as much as possible. I guess you could sum it up to, maybe I am even -0 on removal (I am not sure ;)), but anything is fine and even something like "Jupyter removed it, lets stay synchronized" seems like a fair enough argument to me. PPS: Since this is a CoC thing... if anyone finds my opinion strange/offensive, please write me, because I honestly would want to know! [1] Strangely enough, I have done this in a very mild form, by ignoring an issue (or similar) on github, because I understood their github picture as an offensive symbol. If they were seriously active/representative, I might have asked to at least change it (e.g. before being on the steering council). [2] There are difficult lines when it comes to representative persons maybe, but I do not think you can really set up hard rules for it. It might be that my perspective is a bit different because german judicial system to my knowledge tends to draw few hard lines (two courts can easily decide differently on almost identical cases). [3] We could formalize it, but that would seem to be the exact opposite of what a CoC tries to achieve. [4] In germany there was a discussion recently with some politicians suggesting that "safety" should be a fundamental right or even "super fundamental right". As far as I know judiciary thought it was a very bad idea, because the "super" would mean that you strangely and definitely prefer safety over e.g. privacy, leading to an impossibility to thoughtfully weigh things against each other (not in specific areas, but in general). [5] Yes, there may be some tough points where a person (without doing anything else) practically represents something so much that it could be offensive. First, that probably will not happen, ever, second, I do not see how anyone can hope to anticipate it. There is a reason fundamental laws/constitutions/human rights all start very general like "The dignity (of a human) is unimpeachable.". > Best, > > Sylvain > > > On Thu, Aug 2, 2018 at 7:04 AM, Robert Kern > wrote: > > On Wed, Aug 1, 2018 at 8:35 PM Ryan May wrote: > > > > When experts say that something is a bad idea, and when the > > > > people who > > > > a CoC is supposed to protect says it makes them feel unsafe, I > > > > feel > > > > like we should listen to that. > > > > > > > > I also thought that the points made in the Jupyter discussion > > > > thread > > > > made a lot of sense: of course it's possible for people to > > > > start > > > > harassing each other over any excuse, and a CoC can, should, > > > > and does > > > > make clear that that's not OK. But if you specifically *call > > > > out* > > > > political affiliation as a protected class, at a time when lots > > > > of the > > > > people who the CoC is trying to protect are facing governmental > > > > harassment justified as "mere political disagreement", then it > > > > really > > > > sends the wrong message. > > > > > > > > Besides, uh... isn't the whole definition of politics that it's > > > > topics > > > > where there is active debate? Not really sure why it's even in > > > > that > > > > list to start with. > > > > > > So I hear all the arguments about people feeling unsafe due to > > > some truly despicable, discriminatory behavior, and I want > > > absolutely no parts of protecting that. However, I also recognize > > > that we in the U.S. are in a particularly divisive atmosphere, > > > and people of varied political persuasions want absolutely > > > nothing to do with those who share differing views. So, as a > > > concrete example, if someone were to show up at a NumPy developer > > > summit with a MAGA ("Make America Great Again") hat, or talks > > > about their support for the president in non-numpy channels, > > > WITHOUT expressing anything discriminatory or support for such > > > views, if "political beliefs" is not in the CoC, is this person > > > welcome? I'm not worried about my own views, but I have friends > > > of widely varying views, and I truly wonder if they would be > > > welcome. With differing "political beliefs" listed as something > > > welcomed, I feel ok for them; if this language is removed, I'm > > > much less certain. > > > > > > IMO, "political beliefs" encompasses so much more things than a > > > handful of very specific, hateful views. People can disagree > > > about a wide array of "political beliefs" and it is important > > > that we as a community welcome a wide array of such views. If the > > > CoC needs to protect against the wide array of discriminatory > > > views and behavior that make up U.S. politics right now, how > > > about specifically calling those behaviors out as not-welcome, > > > rather than completely ignoring the fact that 99% of "political > > > beliefs" are perfectly welcome within the community? > > > > > > The CoC is about spelling out the community norms--how about just > > > spelling out that we welcome everyone, but, in the words of Will > > > Wheaton, "Don't be a dick"? > > > > I agree that it's worth clarifying in the text what this clause is > > intended to do. I think it has been misinterpreted as defining a > > legalistic set of protected classes along the lines of anti- > > discrimination laws and can be interpreted by itself outside of the > > context of the CoC as a whole. But it's not that. It's an > > aspirational statement, and a high one, at that, if we interpret it > > at its broadest. We will fail to meet it, in its entirety, and > > that's *okay* if the spirit of the CoC is being defended. I am > > perfectly happy to keep "political beliefs" explicit in the CoC and > > still boot the neo-feudalist for making the project's/conference's > > environment unwelcoming for a more vulnerable group of people, even > > if just by their presence. I *am* sensitive to how nominally well- > > intentioned "viewpoint diversity" efforts get hijacked by > > regressives looking to (re)assert their traditional power. But that > > problem is mostly confined to conferences who need to seek speakers > > and has less relevance to numpy, which largely doesn't run much > > except sprints. I think we can resolve that elsewhere, if not > > another document, then at least another clause. A CoC has to pull a > > kind of double duty: be friendly enough to digest for a newcomer > > and also be helpful to project organizers to make tough balancing > > decisions. We don't have to expect each sentence to pull that > > double duty on its own. I don't quite know what the phrasing would > > be (because, again, we don't run conferences), but I think we could > > make a statement that explicitly disclaims that we will be using > > "viewpoint diversity" to provide a platform for viewpoints > > antithetical to the CoC. > > > > None of these categorizations listed should be interpreted as get- > > out-of-jail-free cards for otherwise unwelcoming behavior, and I > > think maybe we should be explicit about that. Our diversity > > statement is an aspiration, not a suicide pact. Religion, > > neurotype, national origin, and subculture (4chan is a subculture, > > God help us), at minimum, are all items on that list that I have > > personally seen used to justify shitty behavior. Political belief > > is far from unique (nor the most common excuse, in my experience) > > in that list. But they all deserve to be on that list. I want the > > somewhat fringy progressive hacktivist to feel comfortable here as > > well as people more mainstream. > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From charlesr.harris at gmail.com Thu Aug 2 09:02:58 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Aug 2018 07:02:58 -0600 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 11:04 PM, Robert Kern wrote: > On Wed, Aug 1, 2018 at 8:35 PM Ryan May wrote: > >> When experts say that something is a bad idea, and when the people who >>> a CoC is supposed to protect says it makes them feel unsafe, I feel >>> like we should listen to that. >>> >>> I also thought that the points made in the Jupyter discussion thread >>> made a lot of sense: of course it's possible for people to start >>> harassing each other over any excuse, and a CoC can, should, and does >>> make clear that that's not OK. But if you specifically *call out* >>> political affiliation as a protected class, at a time when lots of the >>> people who the CoC is trying to protect are facing governmental >>> harassment justified as "mere political disagreement", then it really >>> sends the wrong message. >>> >>> Besides, uh... isn't the whole definition of politics that it's topics >>> where there is active debate? Not really sure why it's even in that >>> list to start with. >>> >> >> So I hear all the arguments about people feeling unsafe due to some truly >> despicable, discriminatory behavior, and I want absolutely no parts of >> protecting that. However, I also recognize that we in the U.S. are in a >> particularly divisive atmosphere, and people of varied political >> persuasions want absolutely nothing to do with those who share differing >> views. So, as a concrete example, if someone were to show up at a NumPy >> developer summit with a MAGA ("Make America Great Again") hat, or talks >> about their support for the president in non-numpy channels, WITHOUT >> expressing anything discriminatory or support for such views, if "political >> beliefs" is not in the CoC, is this person welcome? I'm not worried about >> my own views, but I have friends of widely varying views, and I truly >> wonder if they would be welcome. With differing "political beliefs" listed >> as something welcomed, I feel ok for them; if this language is removed, I'm >> much less certain. >> >> IMO, "political beliefs" encompasses so much more things than a handful >> of very specific, hateful views. People can disagree about a wide array of >> "political beliefs" and it is important that we as a community welcome a >> wide array of such views. If the CoC needs to protect against the wide >> array of discriminatory views and behavior that make up U.S. politics right >> now, how about specifically calling those behaviors out as not-welcome, >> rather than completely ignoring the fact that 99% of "political beliefs" >> are perfectly welcome within the community? >> >> The CoC is about spelling out the community norms--how about just >> spelling out that we welcome everyone, but, in the words of Will Wheaton, >> "Don't be a dick"? >> > > I agree that it's worth clarifying in the text what this clause is > intended to do. I think it has been misinterpreted as defining a legalistic > set of protected classes along the lines of anti-discrimination laws and > can be interpreted by itself outside of the context of the CoC as a whole. > But it's not that. It's an aspirational statement, and a high one, at that, > if we interpret it at its broadest. We will fail to meet it, in its > entirety, and that's *okay* if the spirit of the CoC is being defended. I > am perfectly happy to keep "political beliefs" explicit in the CoC and > still boot the neo-feudalist for making the project's/conference's > environment unwelcoming for a more vulnerable group of people, even if just > by their presence. I *am* sensitive to how nominally well-intentioned > "viewpoint diversity" efforts get hijacked by regressives looking to > (re)assert their traditional power. But that problem is mostly confined to > conferences who need to seek speakers and has less relevance to numpy, > which largely doesn't run much except sprints. I think we can resolve that > elsewhere, if not another document, then at least another clause. A CoC has > to pull a kind of double duty: be friendly enough to digest for a newcomer > and also be helpful to project organizers to make tough balancing > decisions. We don't have to expect each sentence to pull that double duty > on its own. I don't quite know what the phrasing would be (because, again, > we don't run conferences), but I think we could make a statement that > explicitly disclaims that we will be using "viewpoint diversity" to provide > a platform for viewpoints antithetical to the CoC. > > None of these categorizations listed should be interpreted as > get-out-of-jail-free cards for otherwise unwelcoming behavior, and I think > maybe we should be explicit about that. Our diversity statement is an > aspiration, not a suicide pact. Religion, neurotype, national origin, and > subculture (4chan is a subculture, God help us), at minimum, are all items > on that list that I have personally seen used to justify shitty behavior. > Political belief is far from unique (nor the most common excuse, in my > experience) in that list. But they all deserve to be on that list. I want > the somewhat fringy progressive hacktivist to feel comfortable here as well > as people more mainstream. > > I was opposed to having a list in the first place, because the longer such a list is, the more significant the omissions become. And indeed, the arguments I have seen for omitting "politics" are that one should be allowed to discriminate on the basis of politics, because, reasons. One could marshal the same arguments to support discrimination on the basis of religion, nationality, or culture. Such discrimination is always a temptation and historically has led to conflict; the possibility of conflict is precisely why they are put these lists. The list is a declaration that we will *not* argue about these things because such arguments are well known to be contentious and lead to bad feelings. There are many places for such contention, but I would argue that NumPy development is not one of them. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Aug 2 09:24:55 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 2 Aug 2018 06:24:55 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 5:57 PM, Nathaniel Smith wrote: > On Wed, Aug 1, 2018 at 8:37 AM, Ralf Gommers > wrote: > > > > >> For another perspective on this issue see > >> https://where.coraline.codes/blog/oscon/, where Coraline Ada describes > her > >> reasons for not speaking at OSCON this year due to a similar clause in > the > >> code of conduct. > > > > There's a lot of very unrealistic examples in that post. Plus retracting > a > > week in advance of a conference is, to put it mildly, questionable. So > not > > sure what to think of the rest of that post. There may be good points in > > there, but they're obscured by the obvious flaws in thinking and choice > of > > examples. > > Ralf, I love you, but this paragraph sounds like a parody from "How to > suppress women's writing" or something. > > Coraline Ada is a prominent expert on code-of-conduct issues, and also > a trans woman, so she gets death threats and other harassment > constantly and "will the conference organizers protect me if someone > comes after me?" is a real question she has to ask. She wrote a blog > post about how O'Reilly's handling of this (not just the language, but > the totality of circumstances -- the way it was added, the response to > her queries, etc.) made her feel that attending would be unsafe for > her, so she didn't attend. (And about how distressed she was to > realize this just a week before the conference.) > > It seems like you're taking her post as some logical argument about > CoCs in the abstract, with the withdrawal as some kind of > brinksmanship, and judging it by those standards? > No. For one, from experience as a previous EuroSciPy program chair where we had a pretty similar case. Keynote speaker accepted invitation, then shortly before the event says "I cannot speak unless you introduce a CoC". There was little discussion possible. It felt like blackmail to the whole committee. Because, well, that's what it was. If existence or exact wording of a CoC is that important to you as a speaker, you should check it carefully before accepting an invitation. (and for the record, a CoC was adding the next year after there was time for a serious discussion) Also, I probably agree with all or almost all of her political views. However, starting with unrealistic hypotheticals like people with neo-Nazi insignia just ruins the credibility of the rest of the post for me. I'm not too interested in continuing this particular discussion, it won't be very productive. For the record, I don't much appreciate the parody comment. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Aug 2 09:25:02 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 2 Aug 2018 06:25:02 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 10:04 PM, Robert Kern wrote: > On Wed, Aug 1, 2018 at 8:35 PM Ryan May wrote: > >> When experts say that something is a bad idea, and when the people who >>> a CoC is supposed to protect says it makes them feel unsafe, I feel >>> like we should listen to that. >>> >>> I also thought that the points made in the Jupyter discussion thread >>> made a lot of sense: of course it's possible for people to start >>> harassing each other over any excuse, and a CoC can, should, and does >>> make clear that that's not OK. But if you specifically *call out* >>> political affiliation as a protected class, at a time when lots of the >>> people who the CoC is trying to protect are facing governmental >>> harassment justified as "mere political disagreement", then it really >>> sends the wrong message. >>> >>> Besides, uh... isn't the whole definition of politics that it's topics >>> where there is active debate? Not really sure why it's even in that >>> list to start with. >>> >> >> So I hear all the arguments about people feeling unsafe due to some truly >> despicable, discriminatory behavior, and I want absolutely no parts of >> protecting that. However, I also recognize that we in the U.S. are in a >> particularly divisive atmosphere, and people of varied political >> persuasions want absolutely nothing to do with those who share differing >> views. So, as a concrete example, if someone were to show up at a NumPy >> developer summit with a MAGA ("Make America Great Again") hat, or talks >> about their support for the president in non-numpy channels, WITHOUT >> expressing anything discriminatory or support for such views, if "political >> beliefs" is not in the CoC, is this person welcome? I'm not worried about >> my own views, but I have friends of widely varying views, and I truly >> wonder if they would be welcome. With differing "political beliefs" listed >> as something welcomed, I feel ok for them; if this language is removed, I'm >> much less certain. >> >> IMO, "political beliefs" encompasses so much more things than a handful >> of very specific, hateful views. People can disagree about a wide array of >> "political beliefs" and it is important that we as a community welcome a >> wide array of such views. If the CoC needs to protect against the wide >> array of discriminatory views and behavior that make up U.S. politics right >> now, how about specifically calling those behaviors out as not-welcome, >> rather than completely ignoring the fact that 99% of "political beliefs" >> are perfectly welcome within the community? >> >> The CoC is about spelling out the community norms--how about just >> spelling out that we welcome everyone, but, in the words of Will Wheaton, >> "Don't be a dick"? >> > > I agree that it's worth clarifying in the text what this clause is > intended to do. I think it has been misinterpreted as defining a legalistic > set of protected classes along the lines of anti-discrimination laws and > can be interpreted by itself outside of the context of the CoC as a whole. > But it's not that. It's an aspirational statement, and a high one, at that, > if we interpret it at its broadest. We will fail to meet it, in its > entirety, and that's *okay* if the spirit of the CoC is being defended. I > am perfectly happy to keep "political beliefs" explicit in the CoC and > still boot the neo-feudalist for making the project's/conference's > environment unwelcoming for a more vulnerable group of people, even if just > by their presence. I *am* sensitive to how nominally well-intentioned > "viewpoint diversity" efforts get hijacked by regressives looking to > (re)assert their traditional power. But that problem is mostly confined to > conferences who need to seek speakers and has less relevance to numpy, > which largely doesn't run much except sprints. I think we can resolve that > elsewhere, if not another document, then at least another clause. A CoC has > to pull a kind of double duty: be friendly enough to digest for a newcomer > and also be helpful to project organizers to make tough balancing > decisions. We don't have to expect each sentence to pull that double duty > on its own. I don't quite know what the phrasing would be (because, again, > we don't run conferences), but I think we could make a statement that > explicitly disclaims that we will be using "viewpoint diversity" to provide > a platform for viewpoints antithetical to the CoC. > This sounds like a good idea. Should be possible to express that in a couple of sentences, which will make the CoC better. Cheers, Ralf > > None of these categorizations listed should be interpreted as > get-out-of-jail-free cards for otherwise unwelcoming behavior, and I think > maybe we should be explicit about that. Our diversity statement is an > aspiration, not a suicide pact. Religion, neurotype, national origin, and > subculture (4chan is a subculture, God help us), at minimum, are all items > on that list that I have personally seen used to justify shitty behavior. > Political belief is far from unique (nor the most common excuse, in my > experience) in that list. But they all deserve to be on that list. I want > the somewhat fringy progressive hacktivist to feel comfortable here as well > as people more mainstream. > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Aug 2 09:25:06 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 2 Aug 2018 06:25:06 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Thu, Aug 2, 2018 at 3:35 AM, Marten van Kerkwijk < m.h.vankerkwijk at gmail.com> wrote: > I think a legalistic focus on the letter rather than the spirit of the > code of conduct is not that helpful (and probably what makes if feel US > centric - funny how court systems end up shaping countries). > > My preference would be to keep exactly the scipy version, so that at least > for these two highly related project there is just one code of conduct. But > for what it is worth, it does seem to me political views belongs on the > list - certainly as likely to give problems historically as for any of the > others on the list. > +1 to all of that. If as a result of this discussion we'd want to amend the CoC, let's do that to the SciPy version first (or as well) to keep things in sync. Cheers, Ralf > If we do end up with a different version, I'd prefer a really short one, > like just stating the golden rule (treat others as one would like to be > treated oneself). > > -- Marten > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Aug 2 13:07:36 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 2 Aug 2018 19:07:36 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: <20180802190736.326705db@fsol> On Wed, 1 Aug 2018 21:26:35 -0700 Nathaniel Smith wrote: > On Wed, Aug 1, 2018 at 8:34 PM, Ryan May wrote: > > So I hear all the arguments about people feeling unsafe due to some truly > > despicable, discriminatory behavior, and I want absolutely no parts of > > protecting that. However, I also recognize that we in the U.S. are in a > > particularly divisive atmosphere, and people of varied political persuasions > > want absolutely nothing to do with those who share differing views. So, as a > > concrete example, if someone were to show up at a NumPy developer summit > > with a MAGA ("Make America Great Again") hat, or talks about their support > > for the president in non-numpy channels, WITHOUT expressing anything > > discriminatory or support for such views, if "political beliefs" is not in > > the CoC, is this person welcome? I'm not worried about my own views, but I > > have friends of widely varying views, and I truly wonder if they would be > > welcome. With differing "political beliefs" listed as something welcomed, I > > feel ok for them; if this language is removed, I'm much less certain. > > This is a difficult scenario. I do know lots of people who are > uncomfortable with MAGA hats, and it's not because of they disagree > about the details of some farm bill or whatever, it's because it's > increasingly impossible to use that slogan without also expressing > support for racism, sexism, transphobia, etc. -- i.e., all the other > things that the CoC lists as unacceptable. > > So I feel like... to the extent that some political position *isn't* > tied up with those things, I can't see why people would have a problem > with it, or why the CoC would need to bother mentioning it. (There are > all kinds of things we don't mention ? "we welcome people with odd > *and* even telephone numbers!") And to the extent that some political > position *is* tied up with those things... it would be pretty > contradictory to list it as a protected class alongside race, sex, > etc. > > I'm not going to go track people down on facebook and try to guess how > they voted or something, but if someone wore a MAGA hat to a numpy > sprint then I'd be totally fine with asking them to take it off in > consideration of the other participants. What if someone is wearing a religious symbol? If one is concerned about horrible beliefs or opinions, there are a good deal of them in many religious ("holy") books (*), yet CoCs are generally meant to prohibit discrimination based on religious beliefs... So the argument that CoCs should not protect political beliefs is starting to become flimsy (horrible beliefs are ok if they are part of a religious system, not otherwise?). And it's not just theoretical, because people are physically repressed, in various parts of the world, in the name of religious beliefs. Regards Antoine. From chris.barker at noaa.gov Thu Aug 2 13:08:00 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 2 Aug 2018 10:08:00 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Thu, Aug 2, 2018 at 3:35 AM, Marten van Kerkwijk < m.h.vankerkwijk at gmail.com> wrote: > If we do end up with a different version, I'd prefer a really short one, > like just stating the golden rule (treat others as one would like to be > treated oneself). > Unfortunately, the golden rule is sometime used as a justification for bad behaviour -- "I don't mind it if someone calls my ideas stupid -- I can defend ideas just fine, thank you" So a more specified CoC is worthwhile. As for the issue at hand: A code of conduct is about, well, conduct -- the whole idea is that we are defining appropriate conduct, but NOT discriminating at all based on who or what you are or how you identify yourself. So is wearing a MAGA hat any different than wearing a cross, or a yarmulke, or a hijab ? Or a Bernie T-shirt? (Sorry for the US-centric examples) Note the debate in France about burkinis: https://www.washingtonpost.com/world/europe/frances-burkini-debate-about-a-bathing-suit-and-a-countrys-peculiar-secularism/2016/08/26/48ec273e-6bad-11e6-91cb-ecb5418830e9_story.html?utm_term=.50789110d06a So it's not an easy answer. In the end, though, wearing a particular item of clothing is a behavior, not an identity per se --- so saying that people of and political persuasion are welcome is not the same as saying you can express any political opinion you like publicly in this forum. But it's a really slippery slope: Some religions require the devout to wear particular items of clothing (or hair styles, or...) And there are a lot of issue with people saying: "I don't care if you are gay, just don't talk about it at work" -- but straight people get to talk about their personal lives at work -- so of course everyone should be able to. This is why (in the US anyway) there is the legal concept of a "protected class" -- it needs to be clear exactly what one can can't "discriminate" based on. If an employee is required to wear a particular item of clothing in order to adhere to their religion, then you can't ban that type of clothing -- but you can ban other types of clothing. For this issue, maybe we could get some guidance from the "Hatch Act" -- it is a law that regulates what types of political activity a US federal employee can participate in. It bans some activities even when off the job, but the part that might be relevant is what is banned while on the job. That is, as a US federal employee, you can belong to any political party you like, you can hold any political opinion you like, but you can't freely express those on the job -- i.e. "engage in political activity". Hmm -- I found this: " Hatch Act regulations define political activity as one ?directed toward the success or failure of a political party, candidate for partisan political office, or partisan political group.? Interesting -- I'm pretty sure I'm not allowed to promote white supremacy on the job -- though that's not a partisan political group per se (can't find the definition of partisan, either) TL;DR: I think "political beliefs" should be included, but it should be clear somehow that that doesn't mean you can express any political belief in the context of the project. Honestly, the really horrible people often can't help themselves -- they will actually *do* something inappropriate in the context of the project. And if they don't, then how d we even know what horrible ideas they may promote elsewhere? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Aug 2 17:58:05 2018 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 2 Aug 2018 14:58:05 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: <20180802190736.326705db@fsol> References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <20180802190736.326705db@fsol> Message-ID: On Thu, Aug 2, 2018 at 10:07 AM, Antoine Pitrou wrote: > What if someone is wearing a religious symbol? > > If one is concerned about horrible beliefs or opinions, there are a > good deal of them in many religious ("holy") books (*), yet CoCs are > generally meant to prohibit discrimination based on religious > beliefs... So the argument that CoCs should not protect political > beliefs is starting to become flimsy (horrible beliefs are ok if they > are part of a religious system, not otherwise?). > > And it's not just theoretical, because people are physically repressed, > in various parts of the world, in the name of religious beliefs. Yeah, it's difficult. But to address specifically the question of "what's the difference between listing religion and listing political affiliation": Say you're considering whether to participate in a project. Maybe you're marginalized on some axis, or have been harassed before, and before joining in you want to know how the community will respond if something happens. Of course it's impossible to know, but maybe reading the CoC can help you make some imperfect guess. Now if you see "religion" there, then what does that tell you? Maybe it means that these people are really excited about protecting oppressive religions. Or... maybe it means that they're opposed to anti-semitism, Islamophobia, etc. That would be a pretty obvious interpretation too, and makes a lot more sense in the context of the rest of the text. Of course you're not certain, and yeah, maybe someone will harass you and then claim it's because of their religion and then the community will point at the CoC and take their side. It's possible. But seeing that word isn't a huge red flag either. What about "political affiliation"? Well, if it's the US in the 1950s, obviously they're taking a brave stand against McCarthyism... but that's probably not what jumps to anyone's mind today :-). Context matters! Especially in the OSCON case, where apparently they slipped "political affiliation" into their CoC immediately after the US election in 2016, without telling anyone or giving any explanation. That's like... perfectly designed to make people nervous and suspicious about their intentions. What does this mean for NumPy's CoC? Not sure -- obviously the whole "secretly added to the CoC at the same time everyone is freaking out" part doesn't apply. I think Robert's message made some good points. -n -- Nathaniel J. Smith -- https://vorpus.org From allanhaldane at gmail.com Thu Aug 2 17:58:19 2018 From: allanhaldane at gmail.com (Allan Haldane) Date: Thu, 2 Aug 2018 17:58:19 -0400 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On 08/02/2018 09:25 AM, Ralf Gommers wrote: > > > On Thu, Aug 2, 2018 at 3:35 AM, Marten van Kerkwijk > > wrote: > > I think a legalistic focus on the letter rather than the spirit of > the code of conduct is not that helpful (and probably what makes if > feel US centric - funny how court systems end up shaping countries). > > My preference would be to keep exactly the scipy version, so that at > least for these two highly related project there is just one code of > conduct. But for what it is worth, it does seem to me political > views belongs on the list - certainly as likely to give problems > historically as for any of the others on the list. > > > +1 to all of that. If as a result of this discussion we'd want to amend > the CoC, let's do that to the SciPy version first (or as well) to keep > things in sync. +1, I've just read through all this discussion, and I favor this approach too. I overall agree with Scipy CoC and would be fine accepting it as is. I might prefer it to be shorter and less legalistic as well. One of the goals of the CoC is to attract new and diverse contributors, but we don't want to push anyone away with a scary legal document either. But CoCs are a fairly new and somewhat untested phenomenon in open source, so given that the scipy CoC seems like a good and reasonable effort. By the way, I thought these articles on CoCs were interesting [1][2], including the interviews with open-source CoC creators on their experience in [1] on pros and cons of "rule-based" CoCs. Allan [1] Tourani, Adams and Serebrenik, "Code of conduct in open source projects," 2017 IEEE 24th International Conference on SANER, Klagenfurt, 2017, pp. 24-33. https://www.win.tue.nl/~aserebre/SANER2017.pdf [2] https://modelviewculture.com/pieces/the-new-normal-codes-of-conduct-in-2015-and-beyond > Cheers, > Ralf > > > If we do end up with a different version, I'd prefer a really short > one, like just stating the golden rule (treat others as one would > like to be treated oneself). > > -- Marten> _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > From njs at pobox.com Thu Aug 2 18:27:12 2018 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 2 Aug 2018 15:27:12 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Wed, Aug 1, 2018 at 10:04 PM, Robert Kern wrote: > A CoC has to pull a kind of double > duty: be friendly enough to digest for a newcomer and also be helpful to > project organizers to make tough balancing decisions. We don't have to > expect each sentence to pull that double duty on its own. I don't quite know > what the phrasing would be (because, again, we don't run conferences), but I > think we could make a statement that explicitly disclaims that we will be > using "viewpoint diversity" to provide a platform for viewpoints > antithetical to the CoC. > > None of these categorizations listed should be interpreted as > get-out-of-jail-free cards for otherwise unwelcoming behavior, and I think > maybe we should be explicit about that. Our diversity statement is an > aspiration, not a suicide pact. Religion, neurotype, national origin, and > subculture (4chan is a subculture, God help us), at minimum, are all items > on that list that I have personally seen used to justify shitty behavior. > Political belief is far from unique (nor the most common excuse, in my > experience) in that list. But they all deserve to be on that list. I want > the somewhat fringy progressive hacktivist to feel comfortable here as well > as people more mainstream. This all seems very sensible to me. In personal projects I use the WeAllJS CoC, because I think it does a good job of giving clear guidance on behavior and non-scary enforcement examples, while also avoiding legalism and being clear that trying to game the rules won't work. It might be a good source of inspiration here: https://github.com/WeAllJS/weallbehave/blob/latest/CODE_OF_CONDUCT.md -n -- Nathaniel J. Smith -- https://vorpus.org From einstein.edison at gmail.com Fri Aug 3 01:37:24 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Fri, 3 Aug 2018 07:37:24 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: <20180727220258.fdrtqxuk5t5xfqns@carbo> References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: <50440A7C-D690-493E-9685-1B05FE852DF7@gmail.com> > On 28. Jul 2018, at 00:02, Stefan van der Walt wrote: > > Hi everyone, > > A while ago, SciPy (the library) adopted its Code of Conduct: > https://docs.scipy.org/doc/scipy/reference/dev/conduct/code_of_conduct.html > > We worked hard to make that document friendly, while at the same time > stating clearly the kinds of behavior that would and would not be > tolerated. > > I propose that we adopt the SciPy code of conduct for NumPy as well. It > is a good way to signal to newcomers that this is a community that cares > about how people are treated. And I think we should do anything in our > power to make NumPy as attractive as possible! > > If we adopt this document as policy, we will need to select a Code of > Conduct committee, to whom potential transgressions can be reported. > The individuals doing this for SciPy may very well be happy to do the > same for NumPy, but the community should decide whom will best serve > those roles. > > Let me know your thoughts. > > Thanks! > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Hi all! I?ve been following this thread mainly from the sidelines and thought I?d give a few of my thoughts. I like the idea that one of the rules or ?protected classes? such as people of a certain race, gender, political affiliation etc. can?t use their ?class status? to break any of the other rules. I believe we can make that clear in not so many words. Nathaniel?s WeAllJS CoC seems a bit too conservative, and might promote an overly uptight and formal atmosphere, cruising through some of the examples. People should be allowed to joke and express themselves, so long as it isn?t derogatory towards others. Use of the word ?crazy? should be allowed if it isn?t directed towards a person/group/work, or if it expresses extremes rather than a mental condition. However, I do agree that a some people do like to insult people/groups/work out of habit and then just call it ?jokes? or ?shitposting?. No version of this should be allowed, even in humour. Best Regards, Hameer Abbasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Aug 3 02:01:59 2018 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 2 Aug 2018 23:01:59 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Thu, Aug 2, 2018 at 6:04 AM Charles R Harris wrote: > I was opposed to having a list in the first place, because the longer such > a list is, the more significant the omissions become. And indeed, the > arguments I have seen for omitting "politics" are that one should be > allowed to discriminate on the basis of politics, because, reasons. > I don't think that's quite fair; I think Nathaniel's example, at least, was relatively clear, but let me see if I can summarize concisely (it's not my strength, as these parentheticals attest, but let's give it a whirl regardless). The main purpose of diversity statements is to signal that we will make a special effort to ensure that the less powerful, more vulnerable, or simply traditionally excluded are able to participate fully, safely, and comfortably. At the same time, we are not trying to exclude the more powerful, less vulnerable, or traditionally included that are already there; they just don't need the extra effort. So diversity statements end up more or less facially neutral. Bad actors sometimes take advantage of that facial neutrality, under the guise of "viewpoint diversity" or similar claims, in bad faith, to make use of the community's platform to reinforce or reassert the traditional structures that make the community less welcoming to the less powerful, more vulnerable, and traditionally excluded individuals. Sometimes the community is well-meaning and being taken advantage of by the individual bad actor, but sometimes the community itself is exercising bad faith. Sometimes the community wants the public cover of a diversity statement in name only but continue to be unwelcoming, using the facial neutrality of the diversity statement ot undermine the diversity goals. "Political belief", like "viewpoint diversity", is one of those common weak points that are exploited by these bad actors. Those bad-faith actors and bad-faith communities are not theoretical; we have examples. By including "political belief" in that list, we look like we might possibly be one of those bad-faith communities. The less powerful, more vulnerable, and traditionally excluded individuals may rightly want more of a commitment from us that they would be truly be supported, protected, and welcomed here. Nope, concision is definitely not my strength. But I hope I made the argument clear, at least. Now, I'm not particularly in favor of just dropping "political belief" from the CoC. I think the concerns are valid, but I think that those concerns expose a structural weakness in the CoC that is better addressed with other statements about how we deal with bad actors. "Political belief" isn't the only exploitable item in that list, and "political belief" is also an axis along which less powerful, etc. such that I think it's worth keeping on the list as a reminder. Removing the list entirely for a "we welcome everyone" message is also exploitable, as bad-faith actors will just read in whatever they feel like they need. Personally, I view the list best as not defining a legalistic set of protected classes, but rather as a helpful set of examples for community members to keep in mind as they interact with the community. This is why I don't like a simple "follow the Golden Rule" or "Don't be a dick". It gives absolutely no guidance to the reader. Everyone is a good person in their own head. Telling them to be a good person doesn't give them any tools to be better at welcoming a broader diversity in our community. They will read that and carry on with their own personal status quo with no more reflection. And this requires reflection and work. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Aug 3 03:32:56 2018 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 3 Aug 2018 00:32:56 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: I think I'm going to leave it there for the time being and mute this thread until I get back from vacation. I know that's terribly rude, and you all have my abject apologies. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Aug 3 03:49:23 2018 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 3 Aug 2018 00:49:23 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: On Thu, Aug 2, 2018 at 11:01 PM Robert Kern wrote: > Nope, concision is definitely not my strength. But I hope I > made the argument clear, at least. > No, wait. I got it: Bad actors use "diversity of political beliefs" in bad faith as cover for undermining the goals of the diversity statement. Marginalized groups want more assurance that our community (1) isn't one of those bad actors and (2) is willing and capable of resisting those bad actors when they come. There. Okay. Vacation time. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Aug 3 03:51:01 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Aug 2018 09:51:01 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <20180802190736.326705db@fsol> Message-ID: <20180803095101.497ff180@fsol> On Thu, 2 Aug 2018 14:58:05 -0700 Nathaniel Smith wrote: > > Now if you see "religion" there, then what does that tell you? Maybe > it means that these people are really excited about protecting > oppressive religions. Or... maybe it means that they're opposed to > anti-semitism, Islamophobia, etc. That would be a pretty obvious > interpretation too, and makes a lot more sense in the context of the > rest of the text. Of course you're not certain, and yeah, maybe > someone will harass you and then claim it's because of their religion > and then the community will point at the CoC and take their side. It's > possible. But seeing that word isn't a huge red flag either. > > What about "political affiliation"? Well, if it's the US in the 1950s, > obviously they're taking a brave stand against McCarthyism... but > that's probably not what jumps to anyone's mind today :-). Context > matters! Especially in the OSCON case, where apparently they slipped > "political affiliation" into their CoC immediately after the US > election in 2016, without telling anyone or giving any explanation. > That's like... perfectly designed to make people nervous and > suspicious about their intentions. I think we're coming back to what other posters said. If "political opinion" sends more of a red flag than "religion", then it probably says a lot about US society. Now the question is whether the CoC should be American or global. Personally, I'm ok with an American CoC, as long as it only applies to Americans ;-) Regards Antoine. From stefanv at berkeley.edu Fri Aug 3 04:35:03 2018 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Fri, 03 Aug 2018 10:35:03 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> Message-ID: <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> On August 3, 2018 09:50:38 Robert Kern wrote: > On Thu, Aug 2, 2018 at 11:01 PM Robert Kern wrote: > Nope, concision is definitely not my strength. But I hope I > made the argument clear, at least. > > No, wait. I got it: > > Bad actors use "diversity of political beliefs" in bad faith as cover for > undermining the goals of the diversity statement. Marginalized groups want > more assurance that our community (1) isn't one of those bad actors and (2) > is willing and capable of resisting those bad actors when they come. That's a very useful summary; thank you. I think we can fairly easily add a sentence that encourages participation from a wide diversity of people, while making it clear that including someone in the conversation does not give them free reigns in contradiction with the rest of the guidelines. Ralf, if you agree, shall we do this for SciPy, and use the new version for NumPy too? Best regards, St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Fri Aug 3 04:57:46 2018 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Fri, 03 Aug 2018 10:57:46 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: <164ff01cf10.27ae.d9ac9213ada4a807322f99081296784b@berkeley.edu> On August 3, 2018 10:35:57 Stefan van der Walt wrote: > On August 3, 2018 09:50:38 Robert Kern wrote: >> On Thu, Aug 2, 2018 at 11:01 PM Robert Kern wrote: >> Nope, concision is definitely not my strength. But I hope I >> made the argument clear, at least. >> >> No, wait. I got it: >> >> Bad actors use "diversity of political beliefs" in bad faith as cover for >> undermining the goals of the diversity statement. Marginalized groups want >> more assurance that our community (1) isn't one of those bad actors and (2) >> is willing and capable of resisting those bad actors when they come. > > That's a very useful summary; thank you. > > I think we can fairly easily add a sentence that encourages participation > from a wide diversity of people, while making it clear that including > someone in the conversation does not give them free reigns in contradiction > with the rest of the guidelines. > > Ralf, if you agree, shall we do this for SciPy, and use the new version for > NumPy too? Although, perhaps, a better question to answer is how many people feel that the current document is deficient, and does not go far enough in stating explicitly what we want from our community interactions. It is always hard to tell the opinion of the sometimes silent majority? Best regards, St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Fri Aug 3 05:00:32 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Fri, 3 Aug 2018 11:00:32 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: <164ff01cf10.27ae.d9ac9213ada4a807322f99081296784b@berkeley.edu> References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <164ff01cf10.27ae.d9ac9213ada4a807322f99081296784b@berkeley.edu> Message-ID: <44834010-5A74-46F7-977C-DD02355E4DD8@gmail.com> Hi! I feel we should make it clear that being in any specific category does not give you a free pass on any of the other rules. We should make that explicit, otherwise this looks fine to me. Best Regards, Hameer Abbasi > On 3. Aug 2018, at 10:57, Stefan van der Walt wrote: > > > > On August 3, 2018 10:35:57 Stefan van der Walt wrote: > >> On August 3, 2018 09:50:38 Robert Kern wrote: >>> On Thu, Aug 2, 2018 at 11:01 PM Robert Kern > wrote: >>> Nope, concision is definitely not my strength. But I hope I made the argument clear, at least. >>> >>> No, wait. I got it: >>> >>> Bad actors use "diversity of political beliefs" in bad faith as cover for undermining the goals of the diversity statement. Marginalized groups want more assurance that our community (1) isn't one of those bad actors and (2) is willing and capable of resisting those bad actors when they come. >> >> >> That's a very useful summary; thank you. >> >> I think we can fairly easily add a sentence that encourages participation from a wide diversity of people, while making it clear that including someone in the conversation does not give them free reigns in contradiction with the rest of the guidelines. >> >> Ralf, if you agree, shall we do this for SciPy, and use the new version for NumPy too? > > > Although, perhaps, a better question to answer is how many people feel that the current document is deficient, and does not go far enough in stating explicitly what we want from our community interactions. > > It is always hard to tell the opinion of the sometimes silent majority? > > Best regards, > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From spointer at humdai.net Fri Aug 3 05:02:34 2018 From: spointer at humdai.net (Steve Pointer) Date: Fri, 03 Aug 2018 10:02:34 +0100 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: <44834010-5A74-46F7-977C-DD02355E4DD8@gmail.com> References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <164ff01cf10.27ae.d9ac9213ada4a807322f99081296784b@berkeley.edu> <44834010-5A74-46F7-977C-DD02355E4DD8@gmail.com> Message-ID: <1533286954.4149981.1462160520.2F351AAE@webmail.messagingengine.com> less is more. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Aug 3 05:04:37 2018 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 3 Aug 2018 10:04:37 +0100 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: Hi, On Fri, Aug 3, 2018 at 9:35 AM, Stefan van der Walt wrote: > On August 3, 2018 09:50:38 Robert Kern wrote: >> >> On Thu, Aug 2, 2018 at 11:01 PM Robert Kern wrote: >>> >>> Nope, concision is definitely not my strength. But I hope I >>> made the argument clear, at least. >> >> >> No, wait. I got it: >> >> Bad actors use "diversity of political beliefs" in bad faith as cover for >> undermining the goals of the diversity statement. Marginalized groups want >> more assurance that our community (1) isn't one of those bad actors and (2) >> is willing and capable of resisting those bad actors when they come. > > > That's a very useful summary; thank you. > > I think we can fairly easily add a sentence that encourages participation > from a wide diversity of people, while making it clear that including > someone in the conversation does not give them free reigns in contradiction > with the rest of the guidelines. > > Ralf, if you agree, shall we do this for SciPy, and use the new version for > NumPy too? I must say, I disagree. I think we're already treading close to the edge with the current document, and it's more likely we'd get closer still with virtually any addition on this line. I'm in favor of keeping the political beliefs in there, on the basis it's really not too hard to distinguish good-faith political beliefs, and the current atmosphere is so repellent to people who would not identify as progressive, that I would like them to feel they have some protection. If you will not allow me "no change" and you offered me a) paragraph by group of the not-discriminated trying to imagine something comforting to imagined extremely sensitive and progressive (name your other group here) or b) no stated defense for not-progressive persons, I'd take b). Cheers, Matthew From charlesr.harris at gmail.com Fri Aug 3 09:49:23 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 3 Aug 2018 07:49:23 -0600 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: On Fri, Aug 3, 2018 at 3:04 AM, Matthew Brett wrote: > Hi, > > On Fri, Aug 3, 2018 at 9:35 AM, Stefan van der Walt > wrote: > > On August 3, 2018 09:50:38 Robert Kern wrote: > >> > >> On Thu, Aug 2, 2018 at 11:01 PM Robert Kern > wrote: > >>> > >>> Nope, concision is definitely not my strength. But I hope > I > >>> made the argument clear, at least. > >> > >> > >> No, wait. I got it: > >> > >> Bad actors use "diversity of political beliefs" in bad faith as cover > for > >> undermining the goals of the diversity statement. Marginalized groups > want > >> more assurance that our community (1) isn't one of those bad actors and > (2) > >> is willing and capable of resisting those bad actors when they come. > > > > > > That's a very useful summary; thank you. > > > > I think we can fairly easily add a sentence that encourages participation > > from a wide diversity of people, while making it clear that including > > someone in the conversation does not give them free reigns in > contradiction > > with the rest of the guidelines. > > > > Ralf, if you agree, shall we do this for SciPy, and use the new version > for > > NumPy too? > > I must say, I disagree. I think we're already treading close to the > edge with the current document, and it's more likely we'd get closer > still with virtually any addition on this line. I'm in favor of > keeping the political beliefs in there, on the basis it's really not > too hard to distinguish good-faith political beliefs, and the current > atmosphere is so repellent to people who would not identify as > progressive, that I would like them to feel they have some protection. > If you will not allow me "no change" and you offered me a) paragraph > by group of the not-discriminated trying to imagine something > comforting to imagined extremely sensitive and progressive (name your > other group here) or b) no stated defense for not-progressive persons, > I'd take b). > > I propose that we accept the CoC as is. It seems fine to me and there seems to be general support for it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Aug 3 11:44:16 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 3 Aug 2018 08:44:16 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: On Fri, Aug 3, 2018 at 2:04 AM, Matthew Brett wrote: > Hi, > > On Fri, Aug 3, 2018 at 9:35 AM, Stefan van der Walt > wrote: > > On August 3, 2018 09:50:38 Robert Kern wrote: > >> > >> On Thu, Aug 2, 2018 at 11:01 PM Robert Kern > wrote: > >>> > >>> Nope, concision is definitely not my strength. But I hope > I > >>> made the argument clear, at least. > >> > >> > >> No, wait. I got it: > >> > >> Bad actors use "diversity of political beliefs" in bad faith as cover > for > >> undermining the goals of the diversity statement. Marginalized groups > want > >> more assurance that our community (1) isn't one of those bad actors and > (2) > >> is willing and capable of resisting those bad actors when they come. > > > > > > That's a very useful summary; thank you. > > > > I think we can fairly easily add a sentence that encourages participation > > from a wide diversity of people, while making it clear that including > > someone in the conversation does not give them free reigns in > contradiction > > with the rest of the guidelines. > > > > Ralf, if you agree, shall we do this for SciPy, and use the new version > for > > NumPy too? > If someone with good wordsmithing skills could draft 1-2 sentences and send a PR to the SciPy repo, so we have something concrete to discuss/approve, that would be great. If not, I can take a stab at it early next week. > I must say, I disagree. I think we're already treading close to the > edge with the current document, and it's more likely we'd get closer > still with virtually any addition on this line. I'm in favor of > keeping the political beliefs in there, on the basis There's a much more straightforward basis one can think of. There are many countries in the world that have dictatorships or one-party rule. This includes countries that we get regular contributions from. Expressing support for, e.g., democratic elections, can land you in all sorts of trouble there. For a US conference it may be okay to take a purely US perspective, and even then the inclusion/removal of "political beliefs" can be argued (as evidenced by this thread). For a project with a global reach like NumPy it's really not very good to take into account only US/Western voices. it's really not > too hard to distinguish good-faith political beliefs, and the current > atmosphere is so repellent to people who would not identify as > progressive, that I would like them to feel they have some protection. > If you will not allow me "no change" I think "not allow" is too strong. Your opinion matters as well, so I'm happy to have/facilitate a higher bandwidth discussion on this if you want (after Monday). > and you offered me a) paragraph > by group of the not-discriminated trying to imagine something > comforting to imagined extremely sensitive and progressive (name your > other group here) or b) no stated defense for not-progressive persons, > I'd take b). > Imho Robert made a very compelling argument here, so I don't completely understand the choice. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Fri Aug 3 11:59:01 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Fri, 3 Aug 2018 17:59:01 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> > On 3. Aug 2018, at 17:44, Ralf Gommers wrote: > > > > On Fri, Aug 3, 2018 at 2:04 AM, Matthew Brett > wrote: > Hi, > > On Fri, Aug 3, 2018 at 9:35 AM, Stefan van der Walt > > wrote: > > On August 3, 2018 09:50:38 Robert Kern > wrote: > >> > >> On Thu, Aug 2, 2018 at 11:01 PM Robert Kern > wrote: > >>> > >>> Nope, concision is definitely not my strength. But I hope I > >>> made the argument clear, at least. > >> > >> > >> No, wait. I got it: > >> > >> Bad actors use "diversity of political beliefs" in bad faith as cover for > >> undermining the goals of the diversity statement. Marginalized groups want > >> more assurance that our community (1) isn't one of those bad actors and (2) > >> is willing and capable of resisting those bad actors when they come. > > > > > > That's a very useful summary; thank you. > > > > I think we can fairly easily add a sentence that encourages participation > > from a wide diversity of people, while making it clear that including > > someone in the conversation does not give them free reigns in contradiction > > with the rest of the guidelines. > > > > Ralf, if you agree, shall we do this for SciPy, and use the new version for > > NumPy too? > > If someone with good wordsmithing skills could draft 1-2 sentences and send a PR to the SciPy repo, so we have something concrete to discuss/approve, that would be great. If not, I can take a stab at it early next week. > > > I must say, I disagree. I think we're already treading close to the > edge with the current document, and it's more likely we'd get closer > still with virtually any addition on this line. I'm in favor of > keeping the political beliefs in there, on the basis > > There's a much more straightforward basis one can think of. There are many countries in the world that have dictatorships or one-party rule. This includes countries that we get regular contributions from. Expressing support for, e.g., democratic elections, can land you in all sorts of trouble there. > > For a US conference it may be okay to take a purely US perspective, and even then the inclusion/removal of "political beliefs" can be argued (as evidenced by this thread). For a project with a global reach like NumPy it's really not very good to take into account only US/Western voices. > > it's really not > too hard to distinguish good-faith political beliefs, and the current > atmosphere is so repellent to people who would not identify as > progressive, that I would like them to feel they have some protection. > If you will not allow me "no change" > > I think "not allow" is too strong. Your opinion matters as well, so I'm happy to have/facilitate a higher bandwidth discussion on this if you want (after Monday). > > and you offered me a) paragraph > by group of the not-discriminated trying to imagine something > comforting to imagined extremely sensitive and progressive (name your > other group here) or b) no stated defense for not-progressive persons, > I'd take b). > > Imho Robert made a very compelling argument here, so I don't completely understand the choice. > > Cheers, > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion I?ve created a PR, and I?ve kept the language ?not too stern?. https://github.com/scipy/scipy/pull/9109 Hameer Abbasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Aug 3 12:44:32 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 3 Aug 2018 09:44:32 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> Message-ID: On Fri, Aug 3, 2018 at 8:59 AM, Hameer Abbasi wrote > > > I?ve created a PR, and I?ve kept the language ?not too stern?. > https://github.com/scipy/scipy/pull/9109 > Thanks -- for ease of this thread, the sentence Hameer added is: "We expect that you will extend the same courtesy and open-mindedness towards other members of the SciPy community." LGTM -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Aug 3 14:20:42 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 3 Aug 2018 11:20:42 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> Message-ID: One other thought: Given Jupyter, numpy, scipy, matplotlib?, etc, are all working on a CoC -- maybe we could have NumFocus take a lead on this for the whole community? I think most (all?) of the NumFocus projects have essentially the same goals in this regard. -CHB On Fri, Aug 3, 2018 at 9:44 AM, Chris Barker wrote: > On Fri, Aug 3, 2018 at 8:59 AM, Hameer Abbasi > wrote >> >> >> I?ve created a PR, and I?ve kept the language ?not too stern?. >> https://github.com/scipy/scipy/pull/9109 >> > > Thanks -- for ease of this thread, the sentence Hameer added is: > > "We expect that you will extend the same courtesy and open-mindedness > towards other members of the SciPy community." > > LGTM > > -CHB > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From nelle.varoquaux at gmail.com Fri Aug 3 14:33:59 2018 From: nelle.varoquaux at gmail.com (Nelle Varoquaux) Date: Fri, 3 Aug 2018 11:33:59 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> Message-ID: On 3 August 2018 at 11:20, Chris Barker wrote: > One other thought: > > Given Jupyter, numpy, scipy, matplotlib?, etc, are all working on a CoC -- > maybe we could have NumFocus take a lead on this for the whole community? > > I think most (all?) of the NumFocus projects have essentially the same > goals in this regard. > I think what matters in code of conduct is community buy-in and the discussions around it, more than the document itself. By off-loading the discussion and writing process to someone else, you are missing most of the benefits of codes of conducts. This is also the reason why I think codes of conduct should be revisited regularly. My 2 cents, N > -CHB > > > > > > On Fri, Aug 3, 2018 at 9:44 AM, Chris Barker > wrote: > >> On Fri, Aug 3, 2018 at 8:59 AM, Hameer Abbasi >> wrote >>> >>> >>> I?ve created a PR, and I?ve kept the language ?not too stern?. >>> https://github.com/scipy/scipy/pull/9109 >>> >> >> Thanks -- for ease of this thread, the sentence Hameer added is: >> >> "We expect that you will extend the same courtesy and open-mindedness >> towards other members of the SciPy community." >> >> LGTM >> >> -CHB >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Aug 3 14:39:41 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 3 Aug 2018 11:39:41 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> Message-ID: On Fri, Aug 3, 2018 at 11:20 AM, Chris Barker wrote: > Given Jupyter, numpy, scipy, matplotlib?, etc, are all working on a CoC -- > maybe we could have NumFocus take a lead on this for the whole community? > or adopt an existing one, like maybe: The Contributor Covenant was adopted by several prominent open source projects, including Atom, AngularJS, Eclipse, and even Rails. According to Github, total adoption of the Contributor Covenant is nearing an astounding ten thousand open source projects. I'm trying to figure out why numpy (Or any project, really) has either unique needs or people better qualified to write a CoC than any other project or community. So much like OSS licences -- it's much better to pick an established one than write your own. For the record, the Covenant does have a laundry list of "classes", that does not include political belief, but does mention "political" here: """ Examples of unacceptable behavior by participants include: ... Trolling, insulting/derogatory comments, and personal or political attacks ... """ -CHB Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Aug 3 14:48:05 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 3 Aug 2018 11:48:05 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> Message-ID: On Fri, Aug 3, 2018 at 11:39 AM, Chris Barker wrote: > On Fri, Aug 3, 2018 at 11:20 AM, Chris Barker > wrote: > >> Given Jupyter, numpy, scipy, matplotlib?, etc, are all working on a CoC >> -- maybe we could have NumFocus take a lead on this for the whole community? >> > Nelle is right about the process and community buy-in. > > or adopt an existing one, like maybe: > > The Contributor Covenant was > adopted by several prominent open source projects, including Atom, > AngularJS, Eclipse, and even Rails. According to Github, total adoption of > the Contributor Covenant is nearing an astounding ten thousand open source > projects. > The tone of the Contributor Covenant is far from good. All of this and more was extensively discussed when introducing the SciPy CoC. Can you please read the mailing list discussion on scipy-dev before suggesting a major change in direction? Also keep in mind that the SciPy and NumPy communities strongly overlap, and everyone was okay with the SciPy CoC. We're discussing one tweak to that; removing two words or adding 1-2 sentences. It is counter-productive to start from scratch. Cheers, Ralf > > I'm trying to figure out why numpy (Or any project, really) has either > unique needs or people better qualified to write a CoC than any other > project or community. So much like OSS licences -- it's much better to pick > an established one than write your own. > > For the record, the Covenant does have a laundry list of "classes", that > does not include political belief, but does mention "political" here: > > """ > Examples of unacceptable behavior by participants include: > ... > Trolling, insulting/derogatory comments, and personal or political attacks > ... > """ > > -CHB > > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Aug 3 14:49:32 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 3 Aug 2018 11:49:32 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> Message-ID: On Fri, Aug 3, 2018 at 11:33 AM, Nelle Varoquaux wrote: I think what matters in code of conduct is community buy-in and the > discussions around it, more than the document itself. > This is a really good point. Though I think a community could still have that discussion around whether and which CoC to adopt, rather than the bike-shedding of the document itself. And the reality is that a small sub-fraction of eh community takes part in the conversation anyway. I'm very much on the fence about whether this thread has been truly helpful, for instance, though it's certainly got me trolling the web reading about the issue -- which I probably would not have if this were simply a: "should we adopt the NumFocos CoC" thread... By off-loading the discussion and writing process to someone else, you are > missing most of the benefits of codes of conducts. > well, when reading about CoCs, it seem a large part of their benefit is not to the existing community, but rather what it projects to the rest of the world, particularly possible new contributors. > This is also the reason why I think codes of conduct should be revisited > regularly. > That is a good idea, yes. I'll note that at least the Contributor Covenant is pretty vague about enforcement: """ All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. """ I'd think refining THAT part for the project may provide the benefits of discussion... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Fri Aug 3 15:45:00 2018 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Fri, 03 Aug 2018 21:45:00 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> Message-ID: <16501525e60.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> On August 3, 2018 20:51:00 Chris Barker wrote: > I'll note that at least the Contributor Covenant is pretty vague about > enforcement: > > """ > All complaints will be reviewed and investigated and will result in a > response that is deemed necessary and appropriate to the circumstances. > """ > > I'd think refining THAT part for the project may provide the benefits of > discussion... But the SciPy CoC has a whole additional document that goes into further detail on this specific issue, so let's not concern ourselves with the weaknesses of the Covenant (there are plenty), but rather on what is being proposed for adoption here: https://docs.scipy.org/doc/scipy/reference/dev/conduct/report_handling_manual.html#coc-reporting-manual Best regards, St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.e.creasey.00 at googlemail.com Fri Aug 3 15:45:59 2018 From: p.e.creasey.00 at googlemail.com (Peter Creasey) Date: Fri, 3 Aug 2018 12:45:59 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct Message-ID: +1 for keeping the same CoC as Scipy, making a new thing just seems a bigger surface area to maintain. Personally I already assumed Scipy's "honour[ing] diversity in..." did not imply any protection of behaviours that violate the CoC *itself*, but if you wanted to be really explicit you could add "to the extent that these do not conflict with this code of conduct." to that line. Best, Peter From chris.barker at noaa.gov Fri Aug 3 15:55:15 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 3 Aug 2018 12:55:15 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: <16501525e60.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> References: <20180727220258.fdrtqxuk5t5xfqns@carbo> <164feed02d8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> <999A90B4-BADE-4F12-A7E9-EF37C23486F3@gmail.com> <16501525e60.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: On Fri, Aug 3, 2018 at 12:45 PM, Stefan van der Walt wrote: > I'll note that at least the Contributor Covenant is pretty vague about >> enforcement: >> >> """ >> All complaints will be reviewed and investigated and will result in a >> response that is deemed necessary and appropriate to the circumstances. >> """ >> >> I'd think refining THAT part for the project may provide the benefits of >> discussion... >> > > But the SciPy CoC has a whole additional document that goes into further > detail on this specific issue, so let's not concern ourselves with the > weaknesses of the Covenant (there are plenty), > Actually, I did not indent that to be highlighting a limitation in the Covenant, but rather pointing out that there is plenty to discuss, even if one does adopt an existing CoC. But at Ralf points out, that discussion has been had in the context of scipy, so I agree -- numpy should adopt scipy's CoC and be done with it. In fact, if someone still feels strongly that "political beliefs" should be removed, then it's probably better to bring that up in the context of scipy, rather than numpy -- as has been said, it is practically the same community. To the point where the scipy developers guide and the numpy developers guide are published on the same web site. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Aug 3 16:02:33 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 3 Aug 2018 14:02:33 -0600 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: Message-ID: On Fri, Aug 3, 2018 at 1:45 PM, Peter Creasey wrote: > +1 for keeping the same CoC as Scipy, making a new thing just seems a > bigger surface area to maintain. Personally I already assumed Scipy's > "honour[ing] diversity in..." did not imply any protection of > behaviours that violate the CoC *itself*, but if you wanted to be > really explicit you could add "to the extent that these do not > conflict with this code of conduct." to that line. > I prefer that to the proposed modification, short and sweet. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From harrigan.matthew at gmail.com Fri Aug 3 18:22:57 2018 From: harrigan.matthew at gmail.com (Matthew Harrigan) Date: Fri, 3 Aug 2018 18:22:57 -0400 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: Message-ID: One concern I have is the phrase "explicitly honour" in "we explicitly honour diversity in: age, culture, ...". Honour is a curious word choice. honour is defined as, among other things, "to worship", "high public esteem; fame; glory", and "a source of credit or distinction". I would object to some of those interpretations. Also its not clear to me how honouring diversity relates to conduct. I would definitely agree to follow the other parts of the CoC and also to welcome others regardless of where they fall on the various axes of diversity. "Explicitly welcome" is better and much more closely related to conduct IMO. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Sat Aug 4 01:36:33 2018 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Sat, 04 Aug 2018 07:36:33 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: Message-ID: <165036ff2e8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Hi Matthew, On August 4, 2018 00:23:44 Matthew Harrigan wrote: > One concern I have is the phrase "explicitly honour" in "we explicitly > honour diversity in: age, culture, ...". Honour is a curious word choice. > honour is defined as, among other things, "to worship", "high public > esteem; fame; glory", and "a source of credit or distinction". I would > object to some of those interpretations. Also its not clear to me how > honouring diversity relates to conduct. I would definitely agree to follow > the other parts of the CoC and also to welcome others regardless of where > they fall on the various axes of diversity. "Explicitly welcome" is better > and much more closely related to conduct IMO. While honor may be a slightly strange choice, I don't think it is as strange as this specific definition makes it out to be. You also say "I honor my promise", i.e., I take it seriously, and it has meaning to me. Diversity has meaning to our community (it enriches us, both intellectually and otherwise) and should be cherished. How does honoring diversity relate to the CoC? It is part of the motivation for having a CoC. You cannot build diverse communities without providing a friendly environment for interaction. Best regards, St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Aug 6 13:58:29 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 6 Aug 2018 10:58:29 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: <165036ff2e8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> References: <165036ff2e8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: > On August 4, 2018 00:23:44 Matthew Harrigan > wrote: > >> One concern I have is the phrase "explicitly honour" in "we explicitly >> honour diversity in: age, culture, ...". Honour is a curious word choice. >> honour is defined as, among >> other things, "to worship", "high public esteem; fame; glory", and "a >> source of credit or distinction". I would object to some of those >> interpretations. Also its not clear to me how honouring diversity relates >> to conduct. I would definitely agree to follow the other parts of the >> CoC and also to welcome others regardless of where they fall on the various >> axes of diversity. "Explicitly welcome" is better and much more closely >> related to conduct IMO. >> > > While honor may be a slightly strange choice, I don't think it is as > strange as this specific definition makes it out to be. You also say "I > honor my promise", i.e., I take it seriously, and it has meaning to me. > > Diversity has meaning to our community (it enriches us, both > intellectually and otherwise) and should be cherished. > It's also key to note the specific phrasing -- it is *diversity* that is honored, whereas we would (and do) welcome diverse individuals. So I like the phasing as it is. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From harrigan.matthew at gmail.com Mon Aug 6 20:30:36 2018 From: harrigan.matthew at gmail.com (Matthew Harrigan) Date: Mon, 6 Aug 2018 20:30:36 -0400 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <165036ff2e8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: > > It's also key to note the specific phrasing -- it is *diversity* that is > honored, whereas we would (and do) welcome diverse individuals. > I'm afraid I miss your point. I understand that diversity is what is being honoured in the current CoC, and that is my central issue. My issue is not so much diversity, but more that honour is not the right word. We all agree (I think/hope) that we should and do welcome diverse individuals. That actually paraphrases my suggested edit: Though no list can hope to be comprehensive, we explicitly *welcome* diversity in: age, culture, ethnicity, genotype, gender identity or expression, language, national origin, neurotype, phenotype, political beliefs, profession, race, religion, sexual orientation, socioeconomic status, subculture and technical ability. Practically speaking I don't think my edit means much. I can't think of a situation where someone is friendly, welcoming, and respectful to everyone yet should be referred referred to CoC committee for failing to honour diversity. One goal of the CoC should be to make sure that diverse people from potentially marginalized or targeted groups feel welcome and my edit addresses that more directly than the original. But in principle the difference, to me at least, is stark. Thank you for considering my view. On Mon, Aug 6, 2018 at 1:58 PM, Chris Barker wrote: > > On August 4, 2018 00:23:44 Matthew Harrigan >> wrote: >> >>> One concern I have is the phrase "explicitly honour" in "we explicitly >>> honour diversity in: age, culture, ...". Honour is a curious word choice. >>> honour is defined as, among >>> other things, "to worship", "high public esteem; fame; glory", and "a >>> source of credit or distinction". I would object to some of those >>> interpretations. Also its not clear to me how honouring diversity relates >>> to conduct. I would definitely agree to follow the other parts of the >>> CoC and also to welcome others regardless of where they fall on the various >>> axes of diversity. "Explicitly welcome" is better and much more closely >>> related to conduct IMO. >>> >> >> While honor may be a slightly strange choice, I don't think it is as >> strange as this specific definition makes it out to be. You also say "I >> honor my promise", i.e., I take it seriously, and it has meaning to me. >> >> Diversity has meaning to our community (it enriches us, both >> intellectually and otherwise) and should be cherished. >> > > It's also key to note the specific phrasing -- it is *diversity* that is > honored, whereas we would (and do) welcome diverse individuals. > > So I like the phasing as it is. > > -CHB > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE > > (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Mon Aug 6 22:02:17 2018 From: matti.picus at gmail.com (Matti Picus) Date: Mon, 6 Aug 2018 19:02:17 -0700 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel Message-ID: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> Over the past few days spambots have been hitting freenode's IRC channels[0, 1]. It turns out the #numpy channel has no operator, so we cannot make the channel mode "|+q?$~a"[2] - i.e. only registered freenode users can talk but anyone can listen. I was in touch with the freenode staff, they requested that someone from the steering council reach out to them at ||projects at freenode.net, here is the quote from the discussion: " it's pretty much a matter of them sending an email telling us who they'd like to represent them on freenode, which channels and cloak namespaces they want, and any info we might need on the project " In the mean time they set the channel mode appropriately, so this is also a notice that if you want to chat on the #numpy IRC channel you need to register. Hope someone from the council picks this up and reaches out to them, and will decide who is to able to become channel operators (the recommended practice is to use it like sudo, only assume the role when needed then turn it back off). Matti [0] https://freenode.net/news/spambot-attack [1] https://freenode.net/news/spam-shake [2] https://nedbatchelder.com/blog/201808/fighting_spam_on_freenode.html | From nathan12343 at gmail.com Mon Aug 6 22:15:00 2018 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Mon, 6 Aug 2018 21:15:00 -0500 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel In-Reply-To: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> References: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> Message-ID: Hi, I idle in #scipy and have op in there. I?m happy start idling in #numpy and be op if the community is willing to let me. I?m also in the process of getting ops for #matplotlib for similar spam-related reasons. I?d say all the scientific python IRC channels I?m in get a decent amount of traffic (perhaps 10% of the number of questions that get asked on StackOverflow) and it?s a good venue for asking quick questions. Let?s hope that forcing people to register doesn?t kill that, although there?s not much we can do given the spam attack. Nathan On Mon, Aug 6, 2018 at 9:03 PM Matti Picus wrote: > Over the past few days spambots have been hitting freenode's IRC > channels[0, 1]. It turns out the #numpy channel has no operator, so we > cannot make the channel mode "|+q $~a"[2] - i.e. only registered > freenode users can talk but anyone can listen. > > I was in touch with the freenode staff, they requested that someone from > the steering council reach out to them at ||projects at freenode.net, here > is the quote from the discussion: > > " > it's pretty much a matter of them sending an email telling us who they'd > like to represent them on freenode, which channels and cloak namespaces > they want, and any info we might need on the project > " > > In the mean time they set the channel mode appropriately, so this is > also a notice that if you want to chat on the #numpy IRC channel you > need to register. > > Hope someone from the council picks this up and reaches out to them, and > will decide who is to able to become channel operators (the recommended > practice is to use it like sudo, only assume the role when needed then > turn it back off). > > Matti > > [0] https://freenode.net/news/spambot-attack > [1] https://freenode.net/news/spam-shake > [2] https://nedbatchelder.com/blog/201808/fighting_spam_on_freenode.html > | > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Aug 7 00:52:23 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 6 Aug 2018 21:52:23 -0700 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel In-Reply-To: References: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> Message-ID: On Mon, Aug 6, 2018 at 7:15 PM, Nathan Goldbaum wrote: > Hi, > > I idle in #scipy and have op in there. I?m happy start idling in #numpy > and be op if the community is willing to let me. > Thanks Nathan. Sounds useful. There's also a Gitter numpy channel. AFAIK few/none core devs are regularly active on either IRC or Gitter. I would suggest that we document both these channels as community-run at https://scipy.org/scipylib/mailing-lists.html, and give Nathan and others who are interested the permissions they need. I think our official recommendation for usage questions is StackOverflow. Cheers, Ralf I?m also in the process of getting ops for #matplotlib for similar > spam-related reasons. I?d say all the scientific python IRC channels I?m in > get a decent amount of traffic (perhaps 10% of the number of questions that > get asked on StackOverflow) and it?s a good venue for asking quick > questions. Let?s hope that forcing people to register doesn?t kill that, > although there?s not much we can do given the spam attack. > > Nathan > > On Mon, Aug 6, 2018 at 9:03 PM Matti Picus wrote: > >> Over the past few days spambots have been hitting freenode's IRC >> channels[0, 1]. It turns out the #numpy channel has no operator, so we >> cannot make the channel mode "|+q $~a"[2] - i.e. only registered >> freenode users can talk but anyone can listen. >> >> I was in touch with the freenode staff, they requested that someone from >> the steering council reach out to them at ||projects at freenode.net, here >> is the quote from the discussion: >> >> " >> it's pretty much a matter of them sending an email telling us who they'd >> like to represent them on freenode, which channels and cloak namespaces >> they want, and any info we might need on the project >> " >> >> In the mean time they set the channel mode appropriately, so this is >> also a notice that if you want to chat on the #numpy IRC channel you >> need to register. >> >> Hope someone from the council picks this up and reaches out to them, and >> will decide who is to able to become channel operators (the recommended >> practice is to use it like sudo, only assume the role when needed then >> turn it back off). >> >> Matti >> >> [0] https://freenode.net/news/spambot-attack >> [1] https://freenode.net/news/spam-shake >> [2] https://nedbatchelder.com/blog/201808/fighting_spam_on_freenode.html >> | >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Aug 7 07:34:29 2018 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 07 Aug 2018 13:34:29 +0200 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel In-Reply-To: References: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> Message-ID: <611f2fd6a4861e1eafe47ca86c250e1bf0698999.camel@sipsolutions.net> On Mon, 2018-08-06 at 21:52 -0700, Ralf Gommers wrote: > > > On Mon, Aug 6, 2018 at 7:15 PM, Nathan Goldbaum m> wrote: > > Hi, > > > > I idle in #scipy and have op in there. I?m happy start idling in > > #numpy and be op if the community is willing to let me. > > > > Thanks Nathan. Sounds useful. > Sounds good. I haven't really hung out there for a long time (frankly, I never hung out in #numpy, I thought people just use #scipy). Can we just give a few names (such as Matti, Nathan, maybe me, anyone else right now?) and add others later ourselves? I can get in contact with freenode (unless someone already did). > There's also a Gitter numpy channel. AFAIK few/none core devs are > regularly active on either IRC or Gitter. I would suggest that we > document both these channels as community-run at > https://scipy.org/scipylib/mailing-lists.html, and give Nathan and > others who are interested the permissions they need. > Yeah, the gitter seems pretty inactive as well. But I guess it doesn't hurt to mention them. - Sebastian > I think our official recommendation for usage questions is > StackOverflow. > > Cheers, > Ralf > > > > I?m also in the process of getting ops for #matplotlib for similar > > spam-related reasons. I?d say all the scientific python IRC > > channels I?m in get a decent amount of traffic (perhaps 10% of the > > number of questions that get asked on StackOverflow) and it?s a > > good venue for asking quick questions. Let?s hope that forcing > > people to register doesn?t kill that, although there?s not much we > > can do given the spam attack. > > > > Nathan > > > > On Mon, Aug 6, 2018 at 9:03 PM Matti Picus > > wrote: > > > Over the past few days spambots have been hitting freenode's IRC > > > channels[0, 1]. It turns out the #numpy channel has no operator, > > > so we > > > cannot make the channel mode "|+q $~a"[2] - i.e. only registered > > > freenode users can talk but anyone can listen. > > > > > > I was in touch with the freenode staff, they requested that > > > someone from > > > the steering council reach out to them at ||projects at freenode.net > > > , here > > > is the quote from the discussion: > > > > > > " > > > it's pretty much a matter of them sending an email telling us who > > > they'd > > > like to represent them on freenode, which channels and cloak > > > namespaces > > > they want, and any info we might need on the project > > > " > > > > > > In the mean time they set the channel mode appropriately, so this > > > is > > > also a notice that if you want to chat on the #numpy IRC channel > > > you > > > need to register. > > > > > > Hope someone from the council picks this up and reaches out to > > > them, and > > > will decide who is to able to become channel operators (the > > > recommended > > > practice is to use it like sudo, only assume the role when needed > > > then > > > turn it back off). > > > > > > Matti > > > > > > [0] https://freenode.net/news/spambot-attack > > > [1] https://freenode.net/news/spam-shake > > > [2] https://nedbatchelder.com/blog/201808/fighting_spam_on_freeno > > > de.html > > > | > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From chris.barker at noaa.gov Tue Aug 7 12:31:36 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 7 Aug 2018 09:31:36 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <165036ff2e8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: On Mon, Aug 6, 2018 at 5:30 PM, Matthew Harrigan wrote: > It's also key to note the specific phrasing -- it is *diversity* that is >> honored, whereas we would (and do) welcome diverse individuals. >> > > I'm afraid I miss your point. I understand that diversity is what is > being honoured in the current CoC, and that is my central issue. My issue > is not so much diversity, but more that honour is not the right word. We > all agree (I think/hope) that we should and do welcome diverse > individuals. That actually paraphrases my suggested edit: > > Though no list can hope to be comprehensive, we explicitly *welcome* > diversity in: age, culture, ethnicity, genotype, gender identity or > expression, language, national origin, neurotype, phenotype, political > beliefs, profession, race, religion, sexual orientation, socioeconomic > status, subculture and technical ability. > I think the authors were explicitly using a stronger word: diversity is not jstu welcome, it is more than welcome -- it is honored -- that is, it's a good thing that we explicitly want to support. > Practically speaking I don't think my edit means much. I can't think of a > situation where someone is friendly, welcoming, and respectful to everyone > yet should be referred referred to CoC committee for failing to honour > diversity. One goal of the CoC should be to make sure that diverse people > from potentially marginalized or targeted groups feel welcome and my edit > addresses that more directly than the original. But in principle the > difference, to me at least, is stark. Thank you for considering my view. > > > On Mon, Aug 6, 2018 at 1:58 PM, Chris Barker > wrote: > >> >> On August 4, 2018 00:23:44 Matthew Harrigan >>> wrote: >>> >>>> One concern I have is the phrase "explicitly honour" in "we explicitly >>>> honour diversity in: age, culture, ...". Honour is a curious word choice. >>>> honour is defined as, among >>>> other things, "to worship", "high public esteem; fame; glory", and "a >>>> source of credit or distinction". >>>> >>> I think that last one is, in fact, the point. Anyway, I for one think it's fine either way, but would suggest that any minor changes like this be made to the SciPy CoC (of at all), and that numpy uses the same one. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Aug 8 01:07:29 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 7 Aug 2018 22:07:29 -0700 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel In-Reply-To: <611f2fd6a4861e1eafe47ca86c250e1bf0698999.camel@sipsolutions.net> References: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> <611f2fd6a4861e1eafe47ca86c250e1bf0698999.camel@sipsolutions.net> Message-ID: On Tue, Aug 7, 2018 at 4:34 AM, Sebastian Berg wrote: > On Mon, 2018-08-06 at 21:52 -0700, Ralf Gommers wrote: > > > > > > On Mon, Aug 6, 2018 at 7:15 PM, Nathan Goldbaum > m> wrote: > > > Hi, > > > > > > I idle in #scipy and have op in there. I?m happy start idling in > > > #numpy and be op if the community is willing to let me. > > > > > > > Thanks Nathan. Sounds useful. > > > > Sounds good. I haven't really hung out there for a long time (frankly, > I never hung out in #numpy, I thought people just use #scipy). > > Can we just give a few names (such as Matti, Nathan, maybe me, anyone > else right now?) and add others later ourselves? > I can get in contact with freenode (unless someone already did). > Thanks Sebastian. Go ahead I'd say. Ralf > > There's also a Gitter numpy channel. AFAIK few/none core devs are > > regularly active on either IRC or Gitter. I would suggest that we > > document both these channels as community-run at > > https://scipy.org/scipylib/mailing-lists.html, and give Nathan and > > others who are interested the permissions they need. > > > > Yeah, the gitter seems pretty inactive as well. But I guess it doesn't > hurt to mention them. > > - Sebastian > > > > I think our official recommendation for usage questions is > > StackOverflow. > > > > Cheers, > > Ralf > > > > > > > I?m also in the process of getting ops for #matplotlib for similar > > > spam-related reasons. I?d say all the scientific python IRC > > > channels I?m in get a decent amount of traffic (perhaps 10% of the > > > number of questions that get asked on StackOverflow) and it?s a > > > good venue for asking quick questions. Let?s hope that forcing > > > people to register doesn?t kill that, although there?s not much we > > > can do given the spam attack. > > > > > > Nathan > > > > > > On Mon, Aug 6, 2018 at 9:03 PM Matti Picus > > > wrote: > > > > Over the past few days spambots have been hitting freenode's IRC > > > > channels[0, 1]. It turns out the #numpy channel has no operator, > > > > so we > > > > cannot make the channel mode "|+q $~a"[2] - i.e. only registered > > > > freenode users can talk but anyone can listen. > > > > > > > > I was in touch with the freenode staff, they requested that > > > > someone from > > > > the steering council reach out to them at ||projects at freenode.net > > > > , here > > > > is the quote from the discussion: > > > > > > > > " > > > > it's pretty much a matter of them sending an email telling us who > > > > they'd > > > > like to represent them on freenode, which channels and cloak > > > > namespaces > > > > they want, and any info we might need on the project > > > > " > > > > > > > > In the mean time they set the channel mode appropriately, so this > > > > is > > > > also a notice that if you want to chat on the #numpy IRC channel > > > > you > > > > need to register. > > > > > > > > Hope someone from the council picks this up and reaches out to > > > > them, and > > > > will decide who is to able to become channel operators (the > > > > recommended > > > > practice is to use it like sudo, only assume the role when needed > > > > then > > > > turn it back off). > > > > > > > > Matti > > > > > > > > [0] https://freenode.net/news/spambot-attack > > > > [1] https://freenode.net/news/spam-shake > > > > [2] https://nedbatchelder.com/blog/201808/fighting_spam_on_freeno > > > > de.html > > > > | > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Aug 8 04:07:10 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 8 Aug 2018 01:07:10 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: I'm sorry, I've been trying to find the time to read what you ended up with, and haven't managed yet ? could I get a few days extension? :-) -n On Wed, Aug 1, 2018 at 5:27 PM, Stephan Hoyer wrote: > I propose to accept NEP-18, "A dispatch mechanism for NumPy?s high level > array functions": > http://www.numpy.org/neps/nep-0018-array-function-protocol.html > > Since the last round of discussion, we added a new section on "Callable > objects generated at runtime" clarifying that to handle such objects is out > of scope for the initial proposal in the NEP. > > If there are no substantive objections within 7 days from this email, then > the NEP will be accepted; see NEP 0 for more details. > > Cheers, > Stpehan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Nathaniel J. Smith -- https://vorpus.org From sebastian at sipsolutions.net Wed Aug 8 04:23:15 2018 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 08 Aug 2018 10:23:15 +0200 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel In-Reply-To: References: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> <611f2fd6a4861e1eafe47ca86c250e1bf0698999.camel@sipsolutions.net> Message-ID: <4d36ef13b80b58543ed7c3189af8b17fa9852a77.camel@sipsolutions.net> On Tue, 2018-08-07 at 22:07 -0700, Ralf Gommers wrote: > > > On Tue, Aug 7, 2018 at 4:34 AM, Sebastian Berg s.net> wrote: > > On Mon, 2018-08-06 at 21:52 -0700, Ralf Gommers wrote: > > > > > > > > > On Mon, Aug 6, 2018 at 7:15 PM, Nathan Goldbaum > l.co > > > m> wrote: > > > > Hi, > > > > > > > > I idle in #scipy and have op in there. I?m happy start idling > > in > > > > #numpy and be op if the community is willing to let me. > > > > > > > > > > Thanks Nathan. Sounds useful. > > > > > > > Sounds good. I haven't really hung out there for a long time > > (frankly, > > I never hung out in #numpy, I thought people just use #scipy). > > > > Can we just give a few names (such as Matti, Nathan, maybe me, > > anyone > > else right now?) and add others later ourselves? > > I can get in contact with freenode (unless someone already did). > > Thanks Sebastian. Go ahead I'd say. Will do, just realized looking at it. The Steering Council list, etc. has a list of names, but not email addresses (or PGP keys). I do not remember, was that intentional or not? Also I am not sure if the steering council email address is published anywhere, IIRC it was possible for anyone to send an email to it (OTOH, it would be nice to not catch spam there, so maybe it is fine to ask for the address first). - Sebastian > > Ralf > > > > There's also a Gitter numpy channel. AFAIK few/none core devs are > > > regularly active on either IRC or Gitter. I would suggest that we > > > document both these channels as community-run at > > > https://scipy.org/scipylib/mailing-lists.html, and give Nathan > > and > > > others who are interested the permissions they need. > > > > > > > Yeah, the gitter seems pretty inactive as well. But I guess it > > doesn't > > hurt to mention them. > > > > - Sebastian > > > > > > > I think our official recommendation for usage questions is > > > StackOverflow. > > > > > > Cheers, > > > Ralf > > > > > > > > > > I?m also in the process of getting ops for #matplotlib for > > similar > > > > spam-related reasons. I?d say all the scientific python IRC > > > > channels I?m in get a decent amount of traffic (perhaps 10% of > > the > > > > number of questions that get asked on StackOverflow) and it?s a > > > > good venue for asking quick questions. Let?s hope that forcing > > > > people to register doesn?t kill that, although there?s not much > > we > > > > can do given the spam attack. > > > > > > > > Nathan > > > > > > > > On Mon, Aug 6, 2018 at 9:03 PM Matti Picus > om> > > > > wrote: > > > > > Over the past few days spambots have been hitting freenode's > > IRC > > > > > channels[0, 1]. It turns out the #numpy channel has no > > operator, > > > > > so we > > > > > cannot make the channel mode "|+q $~a"[2] - i.e. only > > registered > > > > > freenode users can talk but anyone can listen. > > > > > > > > > > I was in touch with the freenode staff, they requested that > > > > > someone from > > > > > the steering council reach out to them at ||projects at freenode > > .net > > > > > , here > > > > > is the quote from the discussion: > > > > > > > > > > " > > > > > it's pretty much a matter of them sending an email telling us > > who > > > > > they'd > > > > > like to represent them on freenode, which channels and cloak > > > > > namespaces > > > > > they want, and any info we might need on the project > > > > > " > > > > > > > > > > In the mean time they set the channel mode appropriately, so > > this > > > > > is > > > > > also a notice that if you want to chat on the #numpy IRC > > channel > > > > > you > > > > > need to register. > > > > > > > > > > Hope someone from the council picks this up and reaches out > > to > > > > > them, and > > > > > will decide who is to able to become channel operators (the > > > > > recommended > > > > > practice is to use it like sudo, only assume the role when > > needed > > > > > then > > > > > turn it back off). > > > > > > > > > > Matti > > > > > > > > > > [0] https://freenode.net/news/spambot-attack > > > > > [1] https://freenode.net/news/spam-shake > > > > > [2] https://nedbatchelder.com/blog/201808/fighting_spam_on_fr > > eeno > > > > > de.html > > > > > | > > > > > _______________________________________________ > > > > > NumPy-Discussion mailing list > > > > > NumPy-Discussion at python.org > > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From charlesr.harris at gmail.com Wed Aug 8 11:23:45 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Aug 2018 09:23:45 -0600 Subject: [Numpy-discussion] pytest, fixture and parametrize Message-ID: Hi All, I'm writing some tests and would like to come to a common understanding as to whether we are should use pytest fixtures and parametrized classes and functions. The immediate problem is testing over all types that should be supported, which something we should to much more of and is most easily achieved using one of those pytest tools. The advantage is code clarity and more informative error messages, the disadvantage is being dependent on pytest as unittest does not support that functionality. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Wed Aug 8 11:27:05 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Wed, 8 Aug 2018 17:27:05 +0200 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: References: Message-ID: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> > On 8. Aug 2018, at 17:23, Charles R Harris wrote: > > Hi All, > > I'm writing some tests and would like to come to a common understanding as to whether we are should use pytest fixtures and parametrized classes and functions. The immediate problem is testing over all types that should be supported, which something we should to much more of and is most easily achieved using one of those pytest tools. The advantage is code clarity and more informative error messages, the disadvantage is being dependent on pytest as unittest does not support that functionality. > > Thoughts? > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Hi all! I think it mainly depends on how likely we are to move away from Pytest. And I?m assuming not very likely. Anything likely to replace it (which won?t happen short-term in any case) will already have something similar, IMO. So my vote is we go ahead and use them. Hameer Abbasi From ralf.gommers at gmail.com Wed Aug 8 11:43:35 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 8 Aug 2018 08:43:35 -0700 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> Message-ID: On Wed, Aug 8, 2018 at 8:27 AM, Hameer Abbasi wrote: > > On 8. Aug 2018, at 17:23, Charles R Harris > wrote: > > > > Hi All, > > > > I'm writing some tests and would like to come to a common understanding > as to whether we are should use pytest fixtures and parametrized classes > and functions. The immediate problem is testing over all types that should > be supported, which something we should to much more of and is most easily > achieved using one of those pytest tools. The advantage is code clarity and > more informative error messages, the disadvantage is being dependent on > pytest as unittest does not support that functionality. > > > > Thoughts? > > > > Chuck > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > Hi all! > > I think it mainly depends on how likely we are to move away from Pytest. > And I?m assuming not very likely. Anything likely to replace it (which > won?t happen short-term in any case) will already have something similar, > IMO. > > So my vote is we go ahead and use them. > I agree. There's no other viable test framework at the moment, so spending effort reinventing pytest features is a waste of time right now. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Aug 8 11:55:22 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 8 Aug 2018 08:55:22 -0700 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel In-Reply-To: <4d36ef13b80b58543ed7c3189af8b17fa9852a77.camel@sipsolutions.net> References: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> <611f2fd6a4861e1eafe47ca86c250e1bf0698999.camel@sipsolutions.net> <4d36ef13b80b58543ed7c3189af8b17fa9852a77.camel@sipsolutions.net> Message-ID: On Wed, Aug 8, 2018 at 1:23 AM, Sebastian Berg wrote: > On Tue, 2018-08-07 at 22:07 -0700, Ralf Gommers wrote: > > > > > > On Tue, Aug 7, 2018 at 4:34 AM, Sebastian Berg > s.net> wrote: > > > On Mon, 2018-08-06 at 21:52 -0700, Ralf Gommers wrote: > > > > > > > > > > > > On Mon, Aug 6, 2018 at 7:15 PM, Nathan Goldbaum > > l.co > > > > m> wrote: > > > > > Hi, > > > > > > > > > > I idle in #scipy and have op in there. I?m happy start idling > > > in > > > > > #numpy and be op if the community is willing to let me. > > > > > > > > > > > > > Thanks Nathan. Sounds useful. > > > > > > > > > > Sounds good. I haven't really hung out there for a long time > > > (frankly, > > > I never hung out in #numpy, I thought people just use #scipy). > > > > > > Can we just give a few names (such as Matti, Nathan, maybe me, > > > anyone > > > else right now?) and add others later ourselves? > > > I can get in contact with freenode (unless someone already did). > > > > Thanks Sebastian. Go ahead I'd say. > > > Will do, just realized looking at it. The Steering Council list, etc. > has a list of names, but not email addresses (or PGP keys). I do not > remember, was that intentional or not? I have a vague memory of that being intentional, but not sure. I don't mind making email addresses public; they can be found from git commit logs and mailing lists anyway, so why make life difficult for whomever wants to reach us. > Also I am not sure if the > steering council email address is published anywhere, IIRC it was > possible for anyone to send an email to it (OTOH, it would be nice to > not catch spam there, so maybe it is fine to ask for the address > first). > Google's spam filters are pretty good. For the record, it is numpy-steering-council at googlegroups.com Cheers, Ralf > > - Sebastian > > > > > Ralf > > > > > > There's also a Gitter numpy channel. AFAIK few/none core devs are > > > > regularly active on either IRC or Gitter. I would suggest that we > > > > document both these channels as community-run at > > > > https://scipy.org/scipylib/mailing-lists.html, and give Nathan > > > and > > > > others who are interested the permissions they need. > > > > > > > > > > Yeah, the gitter seems pretty inactive as well. But I guess it > > > doesn't > > > hurt to mention them. > > > > > > - Sebastian > > > > > > > > > > I think our official recommendation for usage questions is > > > > StackOverflow. > > > > > > > > Cheers, > > > > Ralf > > > > > > > > > > > > > I?m also in the process of getting ops for #matplotlib for > > > similar > > > > > spam-related reasons. I?d say all the scientific python IRC > > > > > channels I?m in get a decent amount of traffic (perhaps 10% of > > > the > > > > > number of questions that get asked on StackOverflow) and it?s a > > > > > good venue for asking quick questions. Let?s hope that forcing > > > > > people to register doesn?t kill that, although there?s not much > > > we > > > > > can do given the spam attack. > > > > > > > > > > Nathan > > > > > > > > > > On Mon, Aug 6, 2018 at 9:03 PM Matti Picus > > om> > > > > > wrote: > > > > > > Over the past few days spambots have been hitting freenode's > > > IRC > > > > > > channels[0, 1]. It turns out the #numpy channel has no > > > operator, > > > > > > so we > > > > > > cannot make the channel mode "|+q $~a"[2] - i.e. only > > > registered > > > > > > freenode users can talk but anyone can listen. > > > > > > > > > > > > I was in touch with the freenode staff, they requested that > > > > > > someone from > > > > > > the steering council reach out to them at ||projects at freenode > > > .net > > > > > > , here > > > > > > is the quote from the discussion: > > > > > > > > > > > > " > > > > > > it's pretty much a matter of them sending an email telling us > > > who > > > > > > they'd > > > > > > like to represent them on freenode, which channels and cloak > > > > > > namespaces > > > > > > they want, and any info we might need on the project > > > > > > " > > > > > > > > > > > > In the mean time they set the channel mode appropriately, so > > > this > > > > > > is > > > > > > also a notice that if you want to chat on the #numpy IRC > > > channel > > > > > > you > > > > > > need to register. > > > > > > > > > > > > Hope someone from the council picks this up and reaches out > > > to > > > > > > them, and > > > > > > will decide who is to able to become channel operators (the > > > > > > recommended > > > > > > practice is to use it like sudo, only assume the role when > > > needed > > > > > > then > > > > > > turn it back off). > > > > > > > > > > > > Matti > > > > > > > > > > > > [0] https://freenode.net/news/spambot-attack > > > > > > [1] https://freenode.net/news/spam-shake > > > > > > [2] https://nedbatchelder.com/blog/201808/fighting_spam_on_fr > > > eeno > > > > > > de.html > > > > > > | > > > > > > _______________________________________________ > > > > > > NumPy-Discussion mailing list > > > > > > NumPy-Discussion at python.org > > > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > > _______________________________________________ > > > > > NumPy-Discussion mailing list > > > > > NumPy-Discussion at python.org > > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Aug 8 12:05:10 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 8 Aug 2018 09:05:10 -0700 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: References: Message-ID: > as to whether we are should use pytest fixtures and parametrized classes and functions. Absolutely! > the disadvantage is being dependent on pytest as unittest does not support that functionality. Which is the whole point of using pytest, yes? I?m very opinionated about this, but I really dislike unittest ? it?s simply way too Java-y ? makes the easy things harder than they should be, and is missing critical should-be-easy features. I moved to pure pytest a few years ago, and have been very happy about it. In fact, I recently converted some d unittest code to pure pytest, and it was literally about 1/4 as much code. The only reason I can see to avoid pytest features is that it?s not in the standard lib ? but it?s not a run-time dependency, and it?s easy to install and well supported ? so all good. I suppose we may want to avoid some of pytest?s more magical esoteric features, but certainly not core functionality like fixtures and parameterized tests. -CHB From sebastian at sipsolutions.net Wed Aug 8 12:06:36 2018 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 08 Aug 2018 18:06:36 +0200 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel In-Reply-To: References: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> <611f2fd6a4861e1eafe47ca86c250e1bf0698999.camel@sipsolutions.net> <4d36ef13b80b58543ed7c3189af8b17fa9852a77.camel@sipsolutions.net> Message-ID: <174d4f22275ddd4eb7136201df2c4999d96000dd.camel@sipsolutions.net> On Wed, 2018-08-08 at 08:55 -0700, Ralf Gommers wrote: > > > On Wed, Aug 8, 2018 at 1:23 AM, Sebastian Berg s.net> wrote: > > On Tue, 2018-08-07 at 22:07 -0700, Ralf Gommers wrote: > > > > > > > > > On Tue, Aug 7, 2018 at 4:34 AM, Sebastian Berg > tion > > > s.net> wrote: > > > > On Mon, 2018-08-06 at 21:52 -0700, Ralf Gommers wrote: > > > > > > > > > > > > > > > On Mon, Aug 6, 2018 at 7:15 PM, Nathan Goldbaum > gmai > > > > l.co > > > > > m> wrote: > > > > > > Hi, > > > > > > > > > > > > I idle in #scipy and have op in there. I?m happy start > > idling > > > > in > > > > > > #numpy and be op if the community is willing to let me. > > > > > > > > > > > > > > > > Thanks Nathan. Sounds useful. > > > > > > > > > > > > > Sounds good. I haven't really hung out there for a long time > > > > (frankly, > > > > I never hung out in #numpy, I thought people just use #scipy). > > > > > > > > Can we just give a few names (such as Matti, Nathan, maybe me, > > > > anyone > > > > else right now?) and add others later ourselves? > > > > I can get in contact with freenode (unless someone already > > did). > > > > > > Thanks Sebastian. Go ahead I'd say. > > > > > > Will do, just realized looking at it. The Steering Council list, > > etc. > > has a list of names, but not email addresses (or PGP keys). I do > > not > > remember, was that intentional or not? > > I have a vague memory of that being intentional, but not sure. I > don't mind making email addresses public; they can be found from git > commit logs and mailing lists anyway, so why make life difficult for > whomever wants to reach us. > Yeah, well, I find PGP keys a good idea, even if they might outdate once in a while.... That means if someone wants to check you can easily sign an email and they can be pretty sure you can claim to have some sway in NumPy (right now freenode did it by seeing I have power on github, but it is not really quite ideal). On a general note about IRC, we have claimed #numpy now: If anyone wants anything #numpy related on IRC now (new channels, cloak namespaces, ...), please contact me or Matti (I assume you are happy with that role!). If someone is unhappy with us two being the main contact/people who have those right on freenode, also contact us so we can get it changed. - Sebastian > > Also I am not sure if the > > steering council email address is published anywhere, IIRC it was > > possible for anyone to send an email to it (OTOH, it would be nice > > to > > not catch spam there, so maybe it is fine to ask for the address > > first). > > Google's spam filters are pretty good. For the record, it is numpy-st > eering-council at googlegroups.com > > Cheers, > Ralf > > > > - Sebastian > > > > > > > > Ralf > > > > > > > > There's also a Gitter numpy channel. AFAIK few/none core devs > > are > > > > > regularly active on either IRC or Gitter. I would suggest > > that we > > > > > document both these channels as community-run at > > > > > https://scipy.org/scipylib/mailing-lists.html, and give > > Nathan > > > > and > > > > > others who are interested the permissions they need. > > > > > > > > > > > > > Yeah, the gitter seems pretty inactive as well. But I guess it > > > > doesn't > > > > hurt to mention them. > > > > > > > > - Sebastian > > > > > > > > > > > > > I think our official recommendation for usage questions is > > > > > StackOverflow. > > > > > > > > > > Cheers, > > > > > Ralf > > > > > > > > > > > > > > > > I?m also in the process of getting ops for #matplotlib for > > > > similar > > > > > > spam-related reasons. I?d say all the scientific python IRC > > > > > > channels I?m in get a decent amount of traffic (perhaps 10% > > of > > > > the > > > > > > number of questions that get asked on StackOverflow) and > > it?s a > > > > > > good venue for asking quick questions. Let?s hope that > > forcing > > > > > > people to register doesn?t kill that, although there?s not > > much > > > > we > > > > > > can do given the spam attack. > > > > > > > > > > > > Nathan > > > > > > > > > > > > On Mon, Aug 6, 2018 at 9:03 PM Matti Picus > il.c > > > > om> > > > > > > wrote: > > > > > > > Over the past few days spambots have been hitting > > freenode's > > > > IRC > > > > > > > channels[0, 1]. It turns out the #numpy channel has no > > > > operator, > > > > > > > so we > > > > > > > cannot make the channel mode "|+q $~a"[2] - i.e. only > > > > registered > > > > > > > freenode users can talk but anyone can listen. > > > > > > > > > > > > > > I was in touch with the freenode staff, they requested > > that > > > > > > > someone from > > > > > > > the steering council reach out to them at ||projects at free > > node > > > > .net > > > > > > > , here > > > > > > > is the quote from the discussion: > > > > > > > > > > > > > > " > > > > > > > it's pretty much a matter of them sending an email > > telling us > > > > who > > > > > > > they'd > > > > > > > like to represent them on freenode, which channels and > > cloak > > > > > > > namespaces > > > > > > > they want, and any info we might need on the project > > > > > > > " > > > > > > > > > > > > > > In the mean time they set the channel mode appropriately, > > so > > > > this > > > > > > > is > > > > > > > also a notice that if you want to chat on the #numpy IRC > > > > channel > > > > > > > you > > > > > > > need to register. > > > > > > > > > > > > > > Hope someone from the council picks this up and reaches > > out > > > > to > > > > > > > them, and > > > > > > > will decide who is to able to become channel operators > > (the > > > > > > > recommended > > > > > > > practice is to use it like sudo, only assume the role > > when > > > > needed > > > > > > > then > > > > > > > turn it back off). > > > > > > > > > > > > > > Matti > > > > > > > > > > > > > > [0] https://freenode.net/news/spambot-attack > > > > > > > [1] https://freenode.net/news/spam-shake > > > > > > > [2] https://nedbatchelder.com/blog/201808/fighting_spam_o > > n_fr > > > > eeno > > > > > > > de.html > > > > > > > | > > > > > > > _______________________________________________ > > > > > > > NumPy-Discussion mailing list > > > > > > > NumPy-Discussion at python.org > > > > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > _______________________________________________ > > > > > > NumPy-Discussion mailing list > > > > > > NumPy-Discussion at python.org > > > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > > > > > _______________________________________________ > > > > > NumPy-Discussion mailing list > > > > > NumPy-Discussion at python.org > > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From chris.barker at noaa.gov Wed Aug 8 12:08:14 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 8 Aug 2018 09:08:14 -0700 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> Message-ID: BTW: with pytest.raises(AnException): .... Is another nice feature. -CHB From tyler.je.reddy at gmail.com Wed Aug 8 12:25:23 2018 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Wed, 8 Aug 2018 09:25:23 -0700 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> Message-ID: +1 for pytest parametrization and some of the more straight forward context managers like pytest.raises and so on +0.5 for the fancy fixture stuff in pytest (have to concede that this stuff can be really hard to reason about sometimes, especially with custom scoping decorators, but if it helps to mock something like recently-discovered network dependency for some unit tests it may be better than adding a new dependency or reimplementing effectively the same thing) On Wed, 8 Aug 2018 at 09:09, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > BTW: > > with pytest.raises(AnException): > .... > > Is another nice feature. > > -CHB > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wieser.eric+numpy at gmail.com Wed Aug 8 12:29:44 2018 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Wed, 8 Aug 2018 09:29:44 -0700 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> Message-ID: Is another nice feature You forget that we already have that feature in our testing layer, with np.testing.assert_raises(AnException): pass ? On Wed, 8 Aug 2018 at 09:08 Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > BTW: > > with pytest.raises(AnException): > .... > > Is another nice feature. > > -CHB > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Wed Aug 8 12:38:04 2018 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Wed, 8 Aug 2018 19:38:04 +0300 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> Message-ID: This is actually available from both nose (old-style np.testing framework), and pytest. Stdlib unittest supports self.assertRaises context manager from python 3.1 On Wed, Aug 8, 2018, 7:30 PM Eric Wieser wrote: > Is another nice feature > > You forget that we already have that feature in our testing layer, > > with np.testing.assert_raises(AnException): > pass > > ? > > On Wed, 8 Aug 2018 at 09:08 Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > >> BTW: >> >> with pytest.raises(AnException): >> .... >> >> Is another nice feature. >> >> -CHB >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Aug 8 16:15:51 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 8 Aug 2018 13:15:51 -0700 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> Message-ID: On Wed, Aug 8, 2018 at 9:38 AM, Evgeni Burovski wrote: > Stdlib unittest supports self.assertRaises context manager from python 3.1 > but that requires using unittest :-) On Wed, Aug 8, 2018, 7:30 PM Eric Wieser > wrote: > >> You forget that we already have that feature in our testing layer, >> >> with np.testing.assert_raises(AnException): >> pass >> >> fair enough -- I wouldn't re-write that now, but as its there already, it may make sense to use it. Perhaps we need a doc that lays out the prefered testing utilities. Worthy of a NEP? Or is just a README or something in the code base. Personally, I think a commitment to pytest is the best way to go -- but there are a lot of legacy tests, so there will be a jumble. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Wed Aug 8 18:16:56 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Wed, 8 Aug 2018 15:16:56 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: OK, I am can give you a few more days. I'll be camping through the weekend, but hope to accept it when I get back on Monday! On Wed, Aug 8, 2018 at 1:07 AM Nathaniel Smith wrote: > I'm sorry, I've been trying to find the time to read what you ended up > with, and haven't managed yet ? could I get a few days extension? :-) > > -n > > On Wed, Aug 1, 2018 at 5:27 PM, Stephan Hoyer wrote: > > I propose to accept NEP-18, "A dispatch mechanism for NumPy?s high level > > array functions": > > http://www.numpy.org/neps/nep-0018-array-function-protocol.html > > > > Since the last round of discussion, we added a new section on "Callable > > objects generated at runtime" clarifying that to handle such objects is > out > > of scope for the initial proposal in the NEP. > > > > If there are no substantive objections within 7 days from this email, > then > > the NEP will be accepted; see NEP 0 for more details. > > > > Cheers, > > Stpehan > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Wed Aug 8 20:15:29 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Wed, 8 Aug 2018 17:15:29 -0700 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> Message-ID: On Wed, Aug 8, 2018 at 9:25 AM Tyler Reddy wrote: > +1 for pytest parametrization and some of the more straight forward > context managers like pytest.raises and so on > +1 from me, too. > +0.5 for the fancy fixture stuff in pytest (have to concede that this > stuff can be really hard to reason about sometimes, > especially with custom scoping decorators, but if it helps to mock > something like recently-discovered network dependency > for some unit tests it may be better than adding a new dependency or > reimplementing effectively the same thing) > I'm -0 on fixtures, but unfortunately there don't seem to be viable alternatives in pytest. I find the way that pytest automatically match *argument names* with the names of fixtures very distasteful and un-Pythonic. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Wed Aug 8 20:34:02 2018 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Thu, 09 Aug 2018 10:34:02 +1000 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> Message-ID: <1533774842.544520.1468149528.41832EAE@webmail.messagingengine.com> > I find the way that pytest automatically match *argument names* with > the names of fixtures very distasteful and un-Pythonic. THIS! I've been wanting to articulate this for a while but couldn't quite put my finger on it. Nevertheless, I agree with the sentiment in this thread, specifically that, just like pytest supports unittest tests, I think any framework that comes along to replace it will have to support pytest syntax to be viable. A NumPy document on preferred testing practices would be very valuable even to projects beyond NumPy, just like HOW_TO_DOCUMENT.txt was valuable to the entire ecosystem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From harrigan.matthew at gmail.com Wed Aug 8 20:35:36 2018 From: harrigan.matthew at gmail.com (Matthew Harrigan) Date: Wed, 8 Aug 2018 20:35:36 -0400 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: <165036ff2e8.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: > > One concern I have is the phrase "explicitly honour" in "we explicitly >>>> honour diversity in: age, culture, ...". Honour is a curious word choice. >>>> honour is defined as, among >>>> other things, "to worship", "high public esteem; fame; glory", and "a >>>> source of credit or distinction". >>>> >>> > I think that last one is, in fact, the point. > So I'll use the last one, honour = "a source of credit or distinction". The simplest definition of diversity is a range of different things. What is the range? If it just minimum, i.e. more than one, the honouring diversity loses its power. If its up to each individual to decide, then its just a trivial statement that each person can honour what they want to honour. Hypothetically if someone defined gender identification diversity as only a traditional male and female as decided at birth, that is "a range of things", but that would be blatantly against the point of the CoC. An arbiter to decide the range has another large set of problems. Its clear to me at least that it must be a maximal range. Political diversity now has obvious issues. Some political views are abhorrent to me, and a range of political views that includes them is not at all a source of credit or distinction to me. And this problem is broader than just politics. Back to gender identification, hypothetically if someone identified as a female on odd days of the month and male on even days of the month, I would probably think they are just making a mockery of an important issue and therefore not believe it to be a source of credit or distinction. My point is that no matter what I would welcome them, be respectful, and be friendly. That is why i suggested replacing honour with welcome. Finally I strongly believe that for CoC's to result in positive change they must be carefully read and reflected upon. That was my goal here. I hope it has been worth the time. But if not take comfort that this is my last email on this topic. On Tue, Aug 7, 2018 at 12:31 PM, Chris Barker wrote: > On Mon, Aug 6, 2018 at 5:30 PM, Matthew Harrigan < > harrigan.matthew at gmail.com> wrote: > >> It's also key to note the specific phrasing -- it is *diversity* that is >>> honored, whereas we would (and do) welcome diverse individuals. >>> >> >> I'm afraid I miss your point. I understand that diversity is what is >> being honoured in the current CoC, and that is my central issue. My issue >> is not so much diversity, but more that honour is not the right word. We >> all agree (I think/hope) that we should and do welcome diverse >> individuals. That actually paraphrases my suggested edit: >> >> Though no list can hope to be comprehensive, we explicitly *welcome* >> diversity in: age, culture, ethnicity, genotype, gender identity or >> expression, language, national origin, neurotype, phenotype, political >> beliefs, profession, race, religion, sexual orientation, socioeconomic >> status, subculture and technical ability. >> > > I think the authors were explicitly using a stronger word: diversity is > not jstu welcome, it is more than welcome -- it is honored -- that is, it's > a good thing that we explicitly want to support. > > >> Practically speaking I don't think my edit means much. I can't think of >> a situation where someone is friendly, welcoming, and respectful to >> everyone yet should be referred referred to CoC committee for failing to >> honour diversity. One goal of the CoC should be to make sure that diverse >> people from potentially marginalized or targeted groups feel welcome and my >> edit addresses that more directly than the original. But in principle the >> difference, to me at least, is stark. Thank you for considering my view. >> >> >> On Mon, Aug 6, 2018 at 1:58 PM, Chris Barker >> wrote: >> >>> >>> On August 4, 2018 00:23:44 Matthew Harrigan >>>> wrote: >>>> >>>>> One concern I have is the phrase "explicitly honour" in "we explicitly >>>>> honour diversity in: age, culture, ...". Honour is a curious word choice. >>>>> honour is defined as, >>>>> among other things, "to worship", "high public esteem; fame; glory", >>>>> and "a source of credit or distinction". >>>>> >>>> > I think that last one is, in fact, the point. > > Anyway, I for one think it's fine either way, but would suggest that any > minor changes like this be made to the SciPy CoC (of at all), and that > numpy uses the same one. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE > > (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed Aug 8 20:40:00 2018 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 8 Aug 2018 17:40:00 -0700 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: <1533774842.544520.1468149528.41832EAE@webmail.messagingengine.com> References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> <1533774842.544520.1468149528.41832EAE@webmail.messagingengine.com> Message-ID: <090c20dd-6c56-6c4a-2231-26c4ce4867f2@gmail.com> On 08/08/18 17:34, Juan Nunez-Iglesias wrote: > A NumPy document on preferred testing practices would be very valuable > even to projects beyond NumPy, just like HOW_TO_DOCUMENT.txt was > valuable to the entire ecosystem. > We have these guidelines http://www.numpy.org/devdocs/reference/testing.html, or if you prefer the source https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt. It was updated for pytest in the 1.15 release, but could use some more editing and refinement. Matti From chris.barker at noaa.gov Thu Aug 9 17:27:03 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 9 Aug 2018 17:27:03 -0400 Subject: [Numpy-discussion] pytest, fixture and parametrize In-Reply-To: <090c20dd-6c56-6c4a-2231-26c4ce4867f2@gmail.com> References: <52BFD25A-6F96-4C25-A5BD-3397A03B0B27@gmail.com> <1533774842.544520.1468149528.41832EAE@webmail.messagingengine.com> <090c20dd-6c56-6c4a-2231-26c4ce4867f2@gmail.com> Message-ID: > On Aug 8, 2018, at 5:40 PM, Matti Picus wrote: > We have these guidelines http://www.numpy.org/devdocs/reference/testing.html, Thanks Matti ? that?s clearly the place to document best practices. > It was updated for pytest in the 1.15 release, but could use some more editing and refinement. Giving it s quick read right now, it clearly has not embraced pytest (yet?) Before we try to update the doc, we should have some guidelines as to the goals. At the broad level ? do we want to make use of the pytest testing Framework, or just use pytest as a test runner? I would say that it?s a no brainer, except that he numpy testing framework already has a lot of the key features of pytest that we need. Options: 1) Use pytest only for test running 2) Use pytest features that numpy.testing does not have a good replacement for. 3) prefer pytest features for new tests. In all cases, I think we should limit ourselves to the less magical pytest features. BTW, I see this: ?Setup and teardown functions to functions and methods are known as ?fixtures?, and their use is not encouraged.? First, is that a typo? I don?t understand what it means. Though I think I know what a fixture is. Second ? why are fixtures discouraged? Third, if fixtures are discouraged, rather than banned, we should say when it is OK to use one. -CHB > Matti > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Fri Aug 10 18:05:24 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Aug 2018 16:05:24 -0600 Subject: [Numpy-discussion] Stacklevel for warnings. Message-ID: Hi All, Do we have a policy for the stacklevel that should be used in NumPy? How far back should the stack be displayed? I note that the optimum stacklevel may vary between users and developers. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.s.farimani at gmail.com Fri Aug 10 18:24:57 2018 From: f.s.farimani at gmail.com (Foad Sojoodi Farimani) Date: Sat, 11 Aug 2018 00:24:57 +0200 Subject: [Numpy-discussion] Convolution of NumPy arrays of arbitrary dimension Message-ID: Hello everyone, My first email here, so sorry in advance if I'm violating any rules. I'm trying to implement the convolution of NumPy arrays of arbitrary dimension in order to get Cauchy product of multivariate power series. However I have some issues. I was wondering maybe you can help and this might also be of your interest. I have explained everything here in this StackOverflow post: http://bit.ly/2MhJCex I would appreciate if you could help me figure this out. Thanks in advance. Best, Foad -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Aug 10 23:43:40 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 10 Aug 2018 20:43:40 -0700 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel In-Reply-To: <174d4f22275ddd4eb7136201df2c4999d96000dd.camel@sipsolutions.net> References: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> <611f2fd6a4861e1eafe47ca86c250e1bf0698999.camel@sipsolutions.net> <4d36ef13b80b58543ed7c3189af8b17fa9852a77.camel@sipsolutions.net> <174d4f22275ddd4eb7136201df2c4999d96000dd.camel@sipsolutions.net> Message-ID: On Wed, Aug 8, 2018 at 9:06 AM, Sebastian Berg wrote: > On Wed, 2018-08-08 at 08:55 -0700, Ralf Gommers wrote: > > > > > > On Wed, Aug 8, 2018 at 1:23 AM, Sebastian Berg > s.net> wrote: > > > > > > > > > Will do, just realized looking at it. The Steering Council list, > > > etc. > > > has a list of names, but not email addresses (or PGP keys). I do > > > not > > > remember, was that intentional or not? > > > > I have a vague memory of that being intentional, but not sure. I > > don't mind making email addresses public; they can be found from git > > commit logs and mailing lists anyway, so why make life difficult for > > whomever wants to reach us. > > > > > Yeah, well, I find PGP keys a good idea, even if they might outdate > once in a while.... That means if someone wants to check you can easily > sign an email and they can be pretty sure you can claim to have some > sway in NumPy (right now freenode did it by seeing I have power on > github, but it is not really quite ideal). > Hmm, happy to add PGP keys (optionally), but it doesn't really replace an email address for me. E.g. I don't have PGP keys on all the machines I work on. It's just not worth the trouble syncing that for the few times a year I use a PGP key. How about adding both, optionally? If yes, then it would be nice to collect those in a single PR to limit the churn to the governance doc files. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sat Aug 11 00:11:45 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 10 Aug 2018 21:11:45 -0700 Subject: [Numpy-discussion] Taking back control of the #numpy irc channel In-Reply-To: <174d4f22275ddd4eb7136201df2c4999d96000dd.camel@sipsolutions.net> References: <91a983a8-b3d5-e702-1af2-9a38ed3e8bf8@gmail.com> <611f2fd6a4861e1eafe47ca86c250e1bf0698999.camel@sipsolutions.net> <4d36ef13b80b58543ed7c3189af8b17fa9852a77.camel@sipsolutions.net> <174d4f22275ddd4eb7136201df2c4999d96000dd.camel@sipsolutions.net> Message-ID: On Wed, Aug 8, 2018 at 9:06 AM, Sebastian Berg > If someone is unhappy with us two being the main > contact/people who have those right on freenode, On the contrary, thanks much for taking this on! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Sat Aug 11 01:38:38 2018 From: einstein.edison at gmail.com (einstein.edison at gmail.com) Date: Sat, 11 Aug 2018 07:38:38 +0200 Subject: [Numpy-discussion] Convolution of NumPy arrays of arbitrary dimension In-Reply-To: References: Message-ID: > On 11. Aug 2018, at 00:24, Foad Sojoodi Farimani wrote: > > Hello everyone, > > My first email here, so sorry in advance if I'm violating any rules. > I'm trying to implement the convolution of NumPy arrays of arbitrary dimension in order to get Cauchy product of multivariate power series. However I have some issues. I was wondering maybe you can help and this might also be of your interest. I have explained everything here in this StackOverflow post: > > http://bit.ly/2MhJCex > > I would appreciate if you could help me figure this out. Thanks in advance. > > Best, > Foad > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Hi Foad, you can use scipy.signal.convolve. It works on N-D arrays. Best Regards Hameer Abbasi Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.s.farimani at gmail.com Sat Aug 11 03:44:27 2018 From: f.s.farimani at gmail.com (Foad Sojoodi Farimani) Date: Sat, 11 Aug 2018 09:44:27 +0200 Subject: [Numpy-discussion] Convolution of NumPy arrays of arbitrary dimension In-Reply-To: References: Message-ID: Hi Hameer, Thanks for the reply. I have indeed seen scipy.ndimage.convolve and have mentioned it in the OP . but some questions: 1. although there is nothing about the dimension of the ndarrays in its official page , but I haven't seen any examples showing it works with higher dimensions. 2. the only generalization I know of is astropy.convolution which works up to 3D. 3. what is the difference between seen scipy.ndimage and scipy.signal.convolve? It seems to me the later is for function analysis not array arithmetics. 4. As I can see the term convolution, even for array, arithmetics has different meanings. For example there is also scipy.ndimage.filters.convolve which apparently calculate things in a different way. My final goal is do finite multivariate formal power series multiplication (Cauchy product). I think I have figured the formula out here , but I'm not sure if it is correct completely. questions are: 1. is my formula correct? - if not what is the correct one? 2. if yes has this been done before? - if yes where? does any of the above functions do the job? 3. if not is my implementation correct so far? 4. how to finish the final step to populate the ndarray using the conv function? Thanks a gain and looking forwards to hearing back. Best, Foad On Sat, Aug 11, 2018 at 7:39 AM wrote: > On 11. Aug 2018, at 00:24, Foad Sojoodi Farimani > wrote: > > Hello everyone, > > My first email here, so sorry in advance if I'm violating any rules. > I'm trying to implement the convolution of NumPy arrays of arbitrary > dimension in order to get Cauchy product of multivariate power series. > However I have some issues. I was wondering maybe you can help and this > might also be of your interest. I have explained everything here in this > StackOverflow post: > > http://bit.ly/2MhJCex > > I would appreciate if you could help me figure this out. Thanks in advance. > > Best, > Foad > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > > Hi Foad, you can use scipy.signal.convolve. It works on N-D arrays. > > Best Regards > Hameer Abbasi > Sent from my iPhone > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sat Aug 11 04:22:50 2018 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 11 Aug 2018 10:22:50 +0200 Subject: [Numpy-discussion] Stacklevel for warnings. In-Reply-To: References: Message-ID: On Fri, 2018-08-10 at 16:05 -0600, Charles R Harris wrote: > Hi All, > > Do we have a policy for the stacklevel that should be used in NumPy? > How far back should the stack be displayed? I note that the optimum > stacklevel may vary between users and developers. > I thought it was so that it will point to the correct user line (or tend to point there). So stacklevel=2 for exposed and higher for private (python) functions IIRC. As for developers, I would hope they are OK with (and know how to) turning the warning into an error. Not sure we discussed it much, I seem to have a vague memory of asking if we are sure this is what we want and at least Ralf agreeing. Also I don't know how consistent it is overall. - Sebastian > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at gmail.com Sat Aug 11 14:11:01 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 11 Aug 2018 11:11:01 -0700 Subject: [Numpy-discussion] Stacklevel for warnings. In-Reply-To: References: Message-ID: On Sat, Aug 11, 2018 at 1:22 AM, Sebastian Berg wrote: > On Fri, 2018-08-10 at 16:05 -0600, Charles R Harris wrote: > > Hi All, > > > > Do we have a policy for the stacklevel that should be used in NumPy? > > How far back should the stack be displayed? I note that the optimum > > stacklevel may vary between users and developers. > > > > I thought it was so that it will point to the correct user line (or > tend to point there). So stacklevel=2 for exposed and higher for > private (python) functions IIRC. > As for developers, I would hope they are OK with (and know how to) > turning the warning into an error. > > Not sure we discussed it much, I seem to have a vague memory of asking > if we are sure this is what we want and at least Ralf agreeing. Also I > don't know how consistent it is overall. > That sounds right to me. I think when it was introduced it was quite consistent, because Sebastian replace warning filters everywhere with suppress_warnings. Would be good to document this in the devguide. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Aug 11 14:29:34 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 11 Aug 2018 12:29:34 -0600 Subject: [Numpy-discussion] Does numpy need a *.dist-info directory? What is the status of setuptools? Message-ID: Hi All, A question for any packaging gurus out there, should NumPy have a *.dist-info directory? I note that currently we have a *.egg-info directory. I also have a question as to what role setuptools should have going forward. Should we still rely on them, or has pip matured to the degree that we no longer need it. Note that we will only be supporting Python >= 3.5 in the near future. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sat Aug 11 15:05:29 2018 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 11 Aug 2018 21:05:29 +0200 Subject: [Numpy-discussion] Stacklevel for warnings. In-Reply-To: References: Message-ID: On Sat, 2018-08-11 at 11:11 -0700, Ralf Gommers wrote: > > > On Sat, Aug 11, 2018 at 1:22 AM, Sebastian Berg ns.net> wrote: > > On Fri, 2018-08-10 at 16:05 -0600, Charles R Harris wrote: > > > Hi All, > > > > > > Do we have a policy for the stacklevel that should be used in > > NumPy? > > > How far back should the stack be displayed? I note that the > > optimum > > > stacklevel may vary between users and developers. > > > > > > > I thought it was so that it will point to the correct user line (or > > tend to point there). So stacklevel=2 for exposed and higher for > > private (python) functions IIRC. > > As for developers, I would hope they are OK with (and know how to) > > turning the warning into an error. > > > > Not sure we discussed it much, I seem to have a vague memory of > > asking > > if we are sure this is what we want and at least Ralf agreeing. > > Also I > > don't know how consistent it is overall. > > That sounds right to me. I think when it was introduced it was quite > consistent, because Sebastian replace warning filters everywhere with > suppress_warnings. Would be good to document this in the devguide. Yeah, probably reasonably consistent, but I only added a test to check that the stacklevel argument is never missing entirely, it is up to the author to figure out what is the right level (or best easily possible, since sometimes it would be pretty ugly to make it always right). The warning testing (suppress_warnings, etc.) or any of our tests never actually check the stacklevel that I am aware of, or maybe I forgot :), could be something to think about though. I guess we did it around the same time of the general warning testing cleanup probably. - Sebastian > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at gmail.com Sat Aug 11 15:13:45 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 11 Aug 2018 12:13:45 -0700 Subject: [Numpy-discussion] Does numpy need a *.dist-info directory? What is the status of setuptools? In-Reply-To: References: Message-ID: On Sat, Aug 11, 2018 at 11:29 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > Hi All, > > A question for any packaging gurus out there, should NumPy have a > *.dist-info directory? I note > that currently we have a *.egg-info directory. > Those are both autogenerated, we don't have those in the repo. The current state of our packaging is fine in that respect. > I also have a question as to what role setuptools should have going > forward. Should we still rely on them, or has pip matured to the degree > that we no longer need it. Note that we will only be supporting Python >= > 3.5 in the near future. > Pip is not a build tool, it invokes setuptools. The only viable alternative to setuptools at the moment is scikit-build. But I'm not in a hurry to switch, I'd rather try scikit-build on less critical packages than numpy first. Cheers, Ralf > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Aug 11 15:17:07 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 11 Aug 2018 13:17:07 -0600 Subject: [Numpy-discussion] Does numpy need a *.dist-info directory? What is the status of setuptools? In-Reply-To: References: Message-ID: On Sat, Aug 11, 2018 at 1:13 PM, Ralf Gommers wrote: > > > On Sat, Aug 11, 2018 at 11:29 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> Hi All, >> >> A question for any packaging gurus out there, should NumPy have a >> *.dist-info directory? I >> note that currently we have a *.egg-info directory. >> > > Those are both autogenerated, we don't have those in the repo. The current > state of our packaging is fine in that respect. > > >> I also have a question as to what role setuptools should have going >> forward. Should we still rely on them, or has pip matured to the degree >> that we no longer need it. Note that we will only be supporting Python >= >> 3.5 in the near future. >> > > Pip is not a build tool, it invokes setuptools. The only viable > alternative to setuptools at the moment is scikit-build. But I'm not in a > hurry to switch, I'd rather try scikit-build on less critical packages than > numpy first. > I was more thinking of distutils vs setuptools, but maybe I am misunderstanding the relationship between the two. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Aug 11 15:24:34 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 11 Aug 2018 12:24:34 -0700 Subject: [Numpy-discussion] Does numpy need a *.dist-info directory? What is the status of setuptools? In-Reply-To: References: Message-ID: On Sat, Aug 11, 2018 at 12:17 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Sat, Aug 11, 2018 at 1:13 PM, Ralf Gommers > wrote: > >> >> >> On Sat, Aug 11, 2018 at 11:29 AM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> Hi All, >>> >>> A question for any packaging gurus out there, should NumPy have a >>> *.dist-info directory? I >>> note that currently we have a *.egg-info directory. >>> >> >> Those are both autogenerated, we don't have those in the repo. The >> current state of our packaging is fine in that respect. >> >> >>> I also have a question as to what role setuptools should have going >>> forward. Should we still rely on them, or has pip matured to the degree >>> that we no longer need it. Note that we will only be supporting Python >= >>> 3.5 in the near future. >>> >> >> Pip is not a build tool, it invokes setuptools. The only viable >> alternative to setuptools at the moment is scikit-build. But I'm not in a >> hurry to switch, I'd rather try scikit-build on less critical packages than >> numpy first. >> > > I was more thinking of distutils vs setuptools, but maybe I am > misunderstanding the relationship between the two. > We don't want to go back to distutils-only; there are some advantages to setuptools and it's gotten less bad at breaking numpy.distutils over the last years. And anyway, even if we would switch away from setuptools, pip will enable setuptools before calling our own setup.py and at that point setuptools has already done its monkeypatch-distutils dance. So there's no point avoiding setuptools. Ralf > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robbmcleod at gmail.com Sun Aug 12 13:03:15 2018 From: robbmcleod at gmail.com (Robert McLeod) Date: Sun, 12 Aug 2018 10:03:15 -0700 Subject: [Numpy-discussion] ANN: Numexpr 2.6.7 Message-ID: ========================== Announcing Numexpr 2.6.7 ========================== Hi everyone, This is a bug-fix release. Thanks to Lehman Garrison for a fix that could result in memory leak-like behavior. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.6.6 to 2.6.7 --------------------------- - Thanks to Lehman Garrison for finding and fixing a bug that exhibited memory leak-like behavior. The use in `numexpr.evaluate` of `sys._getframe` combined with `.f_locals` from that frame object results an extra refcount on objects in the frame that calls `numexpr.evaluate`, and not `evaluate`'s frame. So if the calling frame remains in scope for a long time (such as a procedural script where `numexpr` is called from the base frame) garbage collection would never occur. - Imports for the `numexpr.test` submodule were made lazy in the `numexpr` module. What's Numexpr? --------------- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? ------------------------- The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience --------------------- Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod, Ph.D. robbmcleod at gmail.com robbmcleod at protonmail.com robert.mcleod at hitachi-hhtc.ca www.entropyreduction.al -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Aug 13 05:44:01 2018 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 13 Aug 2018 02:44:01 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: Hey all, So I've finally read through NEP 18 (__array_function__). Sorry again for the delay! It's an impressive piece of work! Thanks to the many authors; there's clearly been a lot of thought put into this. # The trade-off between comprehensive APIs versus clean APIs At a high-level, what makes me nervous about this proposal is that it reminds me of a classic software design pattern that... I don't know a name for. You might call it the "structured monkeypatching" approach to extensibility. The pattern is: a project decides they want to allow some kind of extensions or addons or plugins or something, but defining a big structured API for this is too difficult. So they take their current API surface area and declare that that's the plugin API (plus some mechanism for plugins to hook in etc.). Is this pattern good? It's... hard to say. What generally happens is: 1. You get a very complete, powerful, flexible plugin API with minimal work. 2. This quickly leads to a rich system of powerful plugins, which drives quick uptake of the project, sometimes even driving out competitors. 3. The maintainers slowly realize that committing to such a large and unstructured API is horribly unwieldy and makes changes difficult. 4. The maintainers spend huge amounts of effort trying to crawl out from under the weight of their commitments, which mixed success. Examples: pytest, sphinx: For both of these projects, writing plugins is a miserable experience, and you never really know if they'll work with new releases or when composed with random other plugins. Both projects are absolutely the dominant players in their niche, far better than the competition, largely thanks to their rich plugin ecosystems. CPython: the C extension API is basically just... all of CPython's internals dumped into a header file. Without this numpy wouldn't exist. A key ingredient in Python's miraculous popularity. Also, at this point, possibly the largest millstone preventing further improvements in Python ? this is why we can't have multicore support, JITs, etc. etc.; all the most ambitious discussions at the Python language summit the last few years have circled back to "...but we can't do that b/c it will break the C API". See also: https://mail.python.org/pipermail/python-dev/2018-July/154814.html Firefox: their original extension API was basically just "our UI is written in javascript, extension modules get to throw more javascript in the pot". One of Firefox's original USPs, and a key part of like... how Mozilla even exists instead of having gone out of business a decade ago. Eventually the extension API started blocking critical architectural changes (e.g. for better sandboxing), and they had to go through an *immensely* painful migration to a properly designed API, which tooks years and burned huge amounts of goodwill. So this is like... an extreme version of technical debt. You're making a deal with the devil for wealth and fame, and then eventually the bill becomes due. It's hard for me to say categorically that this is a bad idea ? empirically, it can be very successful! But there are real trade-offs. And it makes me a bit nervous that Matt is the one proposing this, because I'm pretty sure if you asked him he'd say he's absolutely focused on how to get something working ASAP and has no plans to maintain numpy in the future. The other approach would be to incrementally add clean, well-defined dunder methods like __array_ufunc__, __array_concatenate__, etc. This way we end up putting some thought into each interface, making sure that it's something we can support, protecting downstream libraries from unnecessary complexity (e.g. they can implement __array_concatenate__ instead of hstack, vstack, row_stack, column_stack, ...), or avoiding adding new APIs entirely (e.g., by converting existing functions into ufuncs so __array_ufunc__ starts automagically working). And in the end we get a clean list of dunder methods that new array container implementations have to define. It's plausible to imagine a generic test suite for array containers. (I suspect that every library that tries to implement __array_function__ will end up with accidental behavioral differences, just because the numpy API is so vast and contains so many corner cases.) So the clean-well-defined-dunders approach has lots of upsides. The big downside is that this is a much longer road to go down. I am genuinely uncertain which of these approaches is better on net, or whether we should do both. But because I'm uncertain, I'm nervous about committing to the NEP 18 approach -- it feels risky. ## Can we mitigate that risk? One thing that helps is the way the proposal makes it all-or-nothing: if you have an __array_function__ method, then you are committing to reimplementing *all* the numpy API (or at least all the parts that you want to work at all). This is arguably a bad thing in the long run, because only large and well-resourced projects can realistically hope to implement __array_function__. But for now it does somewhat mitigate the risks, because the fewer users we have the easier it is to work with them to change course later. But that's probably not enough -- "don't worry, if we change it we'll only break large, important projects with lots of users" isn't actually *that* reassuring :-). The proposal also bills itself as an unstable, provisional experiment ("this protocol should be considered strictly experimental. We reserve the right to change the details of this protocol and how specific NumPy functions use it at any time in the future ? even in otherwise bug-fix only releases of NumPy."). This mitigates a lot of risk! If we aren't committing to anything, then sure, why not experiment. But... this is wishful thinking. No matter what the NEP says, I simply don't believe that we'll actually go break dask, sparse arrays, xarray, and sklearn in a numpy point release. Or any numpy release. Nor should we. If we're serious about keeping this experimental ? and I think that's an excellent idea for now! ? then IMO we need to do something more to avoid getting trapped by backwards compatibility. My suggestion: at numpy import time, check for an envvar, like say NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the __array_function__ dispatches turn into no-ops. This lets interested downstream libraries and users try this out, but makes sure that we won't have a hundred thousand end users depending on it without realizing. Other advantages: - makes it easy for end-users to check how much overhead this adds (by running their code with it enabled vs disabled) - if/when we decide to commit to supporting it for real, we just remove the envvar. With this change, I'm overall +1 on the proposal. Without it, I... would like more convincing, at least :-). # Minor quibbles I don't really understand the 'types' frozenset. The NEP says "it will be used by most __array_function__ methods, which otherwise would need to extract this information themselves"... but they still need to extract the information themselves, because they still have to examine each object and figure out what type it is. And, simply creating a frozenset costs ~0.2 ?s on my laptop, which is overhead that we can't possibly optimize later... -n On Wed, Aug 1, 2018 at 5:27 PM, Stephan Hoyer wrote: > I propose to accept NEP-18, "A dispatch mechanism for NumPy?s high level > array functions": > http://www.numpy.org/neps/nep-0018-array-function-protocol.html > > Since the last round of discussion, we added a new section on "Callable > objects generated at runtime" clarifying that to handle such objects is out > of scope for the initial proposal in the NEP. > > If there are no substantive objections within 7 days from this email, then > the NEP will be accepted; see NEP 0 for more details. > > Cheers, > Stpehan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Nathaniel J. Smith -- https://vorpus.org From einstein.edison at gmail.com Mon Aug 13 10:08:05 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Mon, 13 Aug 2018 16:08:05 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: Hi Nathaniel, Very well written summary, it provides a lot of perspective into the different ways that this could go wrong. Here is a little commentary. > On 13. Aug 2018, at 11:44, Nathaniel Smith wrote: > > Hey all, > > So I've finally read through NEP 18 (__array_function__). Sorry again > for the delay! > > It's an impressive piece of work! Thanks to the many authors; there's > clearly been a lot of thought put into this. > > # The trade-off between comprehensive APIs versus clean APIs > > At a high-level, what makes me nervous about this proposal is that it > reminds me of a classic software design pattern that... I don't know a > name for. You might call it the "structured monkeypatching" approach > to extensibility. The pattern is: a project decides they want to allow > some kind of extensions or addons or plugins or something, but > defining a big structured API for this is too difficult. So they take > their current API surface area and declare that that's the plugin API > (plus some mechanism for plugins to hook in etc.). Is this pattern > good? It's... hard to say. What generally happens is: > > 1. You get a very complete, powerful, flexible plugin API with minimal work. > 2. This quickly leads to a rich system of powerful plugins, which > drives quick uptake of the project, sometimes even driving out > competitors. > 3. The maintainers slowly realize that committing to such a large and > unstructured API is horribly unwieldy and makes changes difficult. > 4. The maintainers spend huge amounts of effort trying to crawl out > from under the weight of their commitments, which mixed success. > > Examples: > > pytest, sphinx: For both of these projects, writing plugins is a > miserable experience, and you never really know if they'll work with > new releases or when composed with random other plugins. Both projects > are absolutely the dominant players in their niche, far better than > the competition, largely thanks to their rich plugin ecosystems. Ah, yes. I?ve been an affectee of this in both instances. For example, there?s a bug that with doctests enabled, you can?t select tests to run, and with coverage enabled, the pydev debugger stops working. However, composition (at least) is better handled with this protocol. This is answered in more detail later on. > > CPython: the C extension API is basically just... all of CPython's > internals dumped into a header file. Without this numpy wouldn't > exist. A key ingredient in Python's miraculous popularity. Also, at > this point, possibly the largest millstone preventing further > improvements in Python ? this is why we can't have multicore support, > JITs, etc. etc.; all the most ambitious discussions at the Python > language summit the last few years have circled back to "...but we > can't do that b/c it will break the C API". See also: > https://mail.python.org/pipermail/python-dev/2018-July/154814.html > > Firefox: their original extension API was basically just "our UI is > written in javascript, extension modules get to throw more javascript > in the pot". One of Firefox's original USPs, and a key part of like... > how Mozilla even exists instead of having gone out of business a > decade ago. Eventually the extension API started blocking critical > architectural changes (e.g. for better sandboxing), and they had to go > through an *immensely* painful migration to a properly designed API, > which tooks years and burned huge amounts of goodwill. Ah, yes. I remember heated debates about this. A lot of good add-ons (as Mozilla calls them) were lost because of alienated developers or missing APIs. > > So this is like... an extreme version of technical debt. You're making > a deal with the devil for wealth and fame, and then eventually the > bill becomes due. It's hard for me to say categorically that this is a > bad idea ? empirically, it can be very successful! But there are real > trade-offs. And it makes me a bit nervous that Matt is the one > proposing this, because I'm pretty sure if you asked him he'd say he's > absolutely focused on how to get something working ASAP and has no > plans to maintain numpy in the future. > > The other approach would be to incrementally add clean, well-defined > dunder methods like __array_ufunc__, __array_concatenate__, etc. This > way we end up putting some thought into each interface, making sure > that it's something we can support, protecting downstream libraries > from unnecessary complexity (e.g. they can implement > __array_concatenate__ instead of hstack, vstack, row_stack, > column_stack, ...), or avoiding adding new APIs entirely (e.g., by > converting existing functions into ufuncs so __array_ufunc__ starts > automagically working). Yes, this is the way I?d prefer to go as well, but the machinery required for converting something to a ufunc is rather complex. Take, for example, the three-argument np.where. In order to preserve full backward compatibility, we need to support structured arrays and strings? which is rather hard to do. Same with the np.identity ufunc that was proposed for casting to a given dtype. This isn?t necessarily an argument against using ufuncs, I?d actually take it as an argument for better documenting these sorts of things. In fact, I wanted to do both of these myself at one point, but I found the documentation for writing ufuncs insufficient for handling these corner cases. > And in the end we get a clean list of dunder > methods that new array container implementations have to define. It's > plausible to imagine a generic test suite for array containers. (I > suspect that every library that tries to implement __array_function__ > will end up with accidental behavioral differences, just because the > numpy API is so vast and contains so many corner cases.) So the > clean-well-defined-dunders approach has lots of upsides. The big > downside is that this is a much longer road to go down. It all comes down to how stable we consider the NumPy API to be. And I?d say pretty stable. Once a function is overridden, we generally do not let NumPy handle it at all. We generally tend to preserve backward compatibility in the API, and the API is all we?re exposing. > > I am genuinely uncertain which of these approaches is better on net, > or whether we should do both. But because I'm uncertain, I'm nervous > about committing to the NEP 18 approach -- it feels risky. > > ## Can we mitigate that risk? > > One thing that helps is the way the proposal makes it all-or-nothing: > if you have an __array_function__ method, then you are committing to > reimplementing *all* the numpy API (or at least all the parts that you > want to work at all). This is arguably a bad thing in the long run, > because only large and well-resourced projects can realistically hope > to implement __array_function__. But for now it does somewhat mitigate > the risks, because the fewer users we have the easier it is to work > with them to change course later. But that's probably not enough -- > "don't worry, if we change it we'll only break large, important > projects with lots of users" isn't actually *that* reassuring :-). There was a proposal that we elected to leave out ? Maybe this is a reason to put it back in. The proposal was to have a sentinel (np.NotImplementedButCoercible) that would take the regular route via coercion, with NotImplemented raising a TypeError. This way, someone could do these things incrementally. Also, another thing we could do (as Stephan Hoyer, I, Travis Oliphant, Tyler Reddy, Saul Shanabrook and a few others discussed in the SciPy 2018 sprints) is to go through the NumPy API, convert as much of them into ufuncs as possible, and identify a ?base set? from the rest. Then duck array implementations would only need to implement this ?base set? of operations along with __array_ufunc__. My idea is that we get started on identifying these things as soon as possible, and only allow this ?base set? under __array_function__, and the rest should simply use these to create all NumPy functionality. I might take on identifying this ?base set? of functionality in early September if enough people are interested. I might even go as far as to say that we shouldn?t allow any function under this protocol unless it?s in the "base set?, and rewrite the rest of NumPy to use the ?base set?. It?s a big project for sure, but identifying the ?base set?/ufunc-able functions shouldn?t take too long. The rewriting part might. The downside would be that some things (such as np.stack) could have better implementations if a custom one was allowed, rather than for example, error checking + concatenate + reshape or error-checking + introduce extra dimension + concatenate. I?m willing to live with this downside, personally, in favour of a cleaner API. > > The proposal also bills itself as an unstable, provisional experiment > ("this protocol should be considered strictly experimental. We reserve > the right to change the details of this protocol and how specific > NumPy functions use it at any time in the future ? even in otherwise > bug-fix only releases of NumPy."). This mitigates a lot of risk! If we > aren't committing to anything, then sure, why not experiment. > > But... this is wishful thinking. No matter what the NEP says, I simply > don't believe that we'll actually go break dask, sparse arrays, > xarray, and sklearn in a numpy point release. Or any numpy release. > Nor should we. If we're serious about keeping this experimental ? and > I think that's an excellent idea for now! ? then IMO we need to do > something more to avoid getting trapped by backwards compatibility. > > My suggestion: at numpy import time, check for an envvar, like say > NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the > __array_function__ dispatches turn into no-ops. This lets interested > downstream libraries and users try this out, but makes sure that we > won't have a hundred thousand end users depending on it without > realizing. Other advantages: > > - makes it easy for end-users to check how much overhead this adds (by > running their code with it enabled vs disabled) > - if/when we decide to commit to supporting it for real, we just > remove the envvar. We also have to consider that this might hinder adoption. But I?m fine with that. Properly > Quickly, as long as it doesn?t take too long. I?m +0 on this until we properly hammer out this stuff, then we remove it and make this the default. However, I also realise that pydata/sparse is in the early stages, and can probably wait. Other duck array implementations such as Dask and XArray might need this soon-ish. > > With this change, I'm overall +1 on the proposal. Without it, I... > would like more convincing, at least :-). > > # Minor quibbles > > I don't really understand the 'types' frozenset. The NEP says "it will > be used by most __array_function__ methods, which otherwise would need > to extract this information themselves"... but they still need to > extract the information themselves, because they still have to examine > each object and figure out what type it is. And, simply creating a > frozenset costs ~0.2 ?s on my laptop, which is overhead that we can't > possibly optimize later? The rationale here is that most implementations would check if the types in the array are actually supported by their implementation. If not, they?d return NotImplemented. If it wasn?t done here, every input would need to do it individually, and this may take a lot of time. I do agree that it violates DRY a bit though? The types are already present in the passed-in arguments, and this can be inferred from those. > > -n > > On Wed, Aug 1, 2018 at 5:27 PM, Stephan Hoyer wrote: >> I propose to accept NEP-18, "A dispatch mechanism for NumPy?s high level >> array functions": >> http://www.numpy.org/neps/nep-0018-array-function-protocol.html >> >> Since the last round of discussion, we added a new section on "Callable >> objects generated at runtime" clarifying that to handle such objects is out >> of scope for the initial proposal in the NEP. >> >> If there are no substantive objections within 7 days from this email, then >> the NEP will be accepted; see NEP 0 for more details. >> >> Cheers, >> Stpehan >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Hope that clarifies things! Best regards, Hameer Abbasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Aug 13 18:27:52 2018 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 13 Aug 2018 15:27:52 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Mon, Aug 13, 2018 at 2:44 AM, Nathaniel Smith wrote: > So this is like... an extreme version of technical debt. You're making > a deal with the devil for wealth and fame, and then eventually the > bill becomes due. It's hard for me to say categorically that this is a > bad idea ? empirically, it can be very successful! But there are real > trade-offs. And it makes me a bit nervous that Matt is the one > proposing this, because I'm pretty sure if you asked him he'd say he's > absolutely focused on how to get something working ASAP and has no > plans to maintain numpy in the future. Rereading this today I realized that it could come across like I have an issue with Matt specifically. I apologize to anyone who got that impression (esp. Matt!) -- that definitely wasn't my intent. Matt is awesome. I should stop writing these things at 2 am. What I should have said is: We have an unusual decision to make here, where there are two plausible approaches that both have significant upsides and downsides, and whose effects are going to be distributed in a complicated way across different parts of our community over time. So the big challenge is to figure out how to take all that into account and weigh the needs of different stakeholders against each other. One major argument for the __array_function__ approach is that it has an actual NEP, which happened because we have a contributor who took the lead on making it happen, and who's deeply involved in some of the target projects like dask and sparse, so can make sure that the proposal will work well for them. That's a huge advantage! But... it also makes me a *little* nervous, because when you have really talented and productive contributors like this it's easy to get swept up in their perspective. So I want to double-check that we're also thinking about the stakeholders who can't be as active in the discussion, like "numpy maintainers from the future". (And I mostly mean this as a "we should keep this in mind" kind of thing ? like I said in my original post, I think moving forward implementing __array_function__ is a great idea; I just want to be cautious about getting more experience before committing.) -n -- Nathaniel J. Smith -- https://vorpus.org From mrocklin at gmail.com Mon Aug 13 23:32:00 2018 From: mrocklin at gmail.com (Matthew Rocklin) Date: Mon, 13 Aug 2018 21:32:00 -0600 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: Hi Nathaniel, I appreciate the clarification. Thank you for that. For what it's worth, I think that you may overestimate my involvement in the writing of that NEP. I sat down with Stephan during a Numpy dev meeting and we hacked something together. Afterwards several other people poured their thoughts into the process. I'd like to think that my perspective helped to inform this NEP, but it wasn't, by far, the driving force. If anyone had a strongest hand in the writing process it would probably be Stephan, who I find generally has a more conservative and careful perspective than I do. That being said, I do think that Numpy would be wise to move quickly here. I think that the growing fragmentation that we see in array computing in Numpy (Tensorflow, Torch, Dask, Sparse, CuPy) is largely due to Numpy moving slowly in the past. There is, I think, a systemic problem slowly erupting now that I think the community to respond to quickly if it is possible to do so safely. I believe that Numpy should absolutely be willing to try something experimental, and then say "nope, that was a bad idea" and retract it if it doesn't work out well. I think that figuring out all of __array_concatenate__, __array_stack__, __array_foo__, etc. for each of the many cases will take too long to respond to in an effective timeframe. I believe that we simply don't move quickly enough that this piece-by-piece careful handling of the API will result in Numpy's API becoming a meaningful standard in the broader community in the near-future. That being said, I think that we *should* engage in this piece-by-piece discussion, and as we figure them out we should slowly encroach on __array_function__ and remove functionality from it, much as __array_ufunc__ is not included in it in the current NEP. Ideally we should get to exactly where you want to get to. I perceive the __array_function__ protocol as a sort of necessary stop-gap. All that being said, this is just my personal stance. I suspect that each of the authors of the NEP and others who engaged in its careful review have a different perspective, which should probably carry more weight than my own. Best, -matt On Mon, Aug 13, 2018 at 4:29 PM Nathaniel Smith wrote: > On Mon, Aug 13, 2018 at 2:44 AM, Nathaniel Smith wrote: > > So this is like... an extreme version of technical debt. You're making > > a deal with the devil for wealth and fame, and then eventually the > > bill becomes due. It's hard for me to say categorically that this is a > > bad idea ? empirically, it can be very successful! But there are real > > trade-offs. And it makes me a bit nervous that Matt is the one > > proposing this, because I'm pretty sure if you asked him he'd say he's > > absolutely focused on how to get something working ASAP and has no > > plans to maintain numpy in the future. > > Rereading this today I realized that it could come across like I have > an issue with Matt specifically. I apologize to anyone who got that > impression (esp. Matt!) -- that definitely wasn't my intent. Matt is > awesome. I should stop writing these things at 2 am. > > What I should have said is: > > We have an unusual decision to make here, where there are two > plausible approaches that both have significant upsides and downsides, > and whose effects are going to be distributed in a complicated way > across different parts of our community over time. So the big > challenge is to figure out how to take all that into account and weigh > the needs of different stakeholders against each other. > > One major argument for the __array_function__ approach is that it has > an actual NEP, which happened because we have a contributor who took > the lead on making it happen, and who's deeply involved in some of the > target projects like dask and sparse, so can make sure that the > proposal will work well for them. That's a huge advantage! But... it > also makes me a *little* nervous, because when you have really > talented and productive contributors like this it's easy to get swept > up in their perspective. So I want to double-check that we're also > thinking about the stakeholders who can't be as active in the > discussion, like "numpy maintainers from the future". > > (And I mostly mean this as a "we should keep this in mind" kind of > thing ? like I said in my original post, I think moving forward > implementing __array_function__ is a great idea; I just want to be > cautious about getting more experience before committing.) > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Aug 14 05:17:31 2018 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 14 Aug 2018 10:17:31 +0100 Subject: [Numpy-discussion] Dropping 32-bit from macOS wheels Message-ID: Hi, Short version: We are planning to drop 32-bit compatibility from the numpy macOS wheels. This email is to ask for feedback. Long version: macOS has tooling to allow distribution of binaries that have objects for more than one architecture. These are called "fat" binaries, and they were useful when Macs could be PPC, 32-bit or 64-bit intel. Now of course, they are effectively all 64-bit Intel. Amusingly enough, you can use the "lipo" command to operate on fat binaries. For example, system Python (at /usr/bin/python) is a fat binary, containing objects for 32 and 64-bit. $ lipo -info /usr/bin/python Architectures in the fat file: /usr/bin/python are: i386 x86_64 It is possible to run Python in 32-bit mode with: $ arch -i386 /usr/bin/python Up until the last release of numpy, we have build fat (32 and 64 bit) binaries for numpy, so, if you do run Python in 32-bit mode, you can still import and use numpy. In the last release, I accidentally broke the 32-bit part of the build - see https://github.com/numpy/numpy/issues/11625 . We could fix this, but we suspect that there are very few people still using 32-bit on macOS. If you do use it - please let us know, and we'll consider the options. Cheers, Matthew From chris.barker at noaa.gov Tue Aug 14 18:47:41 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 14 Aug 2018 15:47:41 -0700 Subject: [Numpy-discussion] Dropping 32-bit from macOS wheels In-Reply-To: References: Message-ID: On Tue, Aug 14, 2018 at 2:17 AM, Matthew Brett wrote: > We are planning to drop 32-bit compatibility from the numpy macOS > wheels. +1 -- it really is time. I note that python.org has finally gone 64 bit only -- at least for the default download: """ For 3.7.0, we provide two binary installer options for download. The default variant is 64-bit-only and works on macOS 10.9 (Mavericks) and later systems. """ granted, it'll be quite a while before everyone is running 3.7+, but still. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Aug 15 00:30:24 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 14 Aug 2018 21:30:24 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: Message-ID: On Fri, Aug 3, 2018 at 1:02 PM, Charles R Harris wrote: > > > On Fri, Aug 3, 2018 at 1:45 PM, Peter Creasey < > p.e.creasey.00 at googlemail.com> wrote: > >> +1 for keeping the same CoC as Scipy, making a new thing just seems a >> bigger surface area to maintain. Personally I already assumed Scipy's >> "honour[ing] diversity in..." did not imply any protection of >> behaviours that violate the CoC *itself*, but if you wanted to be >> really explicit you could add "to the extent that these do not >> conflict with this code of conduct." to that line. >> > > I prefer that to the proposed modification, short and sweet. > This edit to the SciPy CoC has now been merged. It looks to me like we're good to go here and take over the SciPy CoC. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Wed Aug 15 00:49:37 2018 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 14 Aug 2018 21:49:37 -0700 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: Message-ID: +1 Jarrod On Tue, Aug 14, 2018 at 9:30 PM, Ralf Gommers wrote: > > > On Fri, Aug 3, 2018 at 1:02 PM, Charles R Harris > wrote: >> >> >> >> On Fri, Aug 3, 2018 at 1:45 PM, Peter Creasey >> wrote: >>> >>> +1 for keeping the same CoC as Scipy, making a new thing just seems a >>> bigger surface area to maintain. Personally I already assumed Scipy's >>> "honour[ing] diversity in..." did not imply any protection of >>> behaviours that violate the CoC *itself*, but if you wanted to be >>> really explicit you could add "to the extent that these do not >>> conflict with this code of conduct." to that line. >> >> >> I prefer that to the proposed modification, short and sweet. > > > This edit to the SciPy CoC has now been merged. > > It looks to me like we're good to go here and take over the SciPy CoC. > > Cheers, > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > From hans.dembinski at gmail.com Wed Aug 15 05:40:23 2018 From: hans.dembinski at gmail.com (Hans Dembinski) Date: Wed, 15 Aug 2018 11:40:23 +0200 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code Message-ID: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> Dear all, I just joined the numpy mailing list to suggest an enhancement of the docs about writing binding code https://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html I hope this is the right place to discuss this. So, this page is one of the first hits in Google when you search for "python bindings numpy". It is an important page for orientation. What I miss is a longer article about pybind11. http://pybind11.readthedocs.io/en/stable/ https://github.com/pybind/pybind11 pybind11 is currently the best tool on the market to wrap C++ code to Python. This is my professional opinion. When you look at the facts, it is hard to disagree. Pybind11 is based on the approach of Boost.Python, but is a compact project that doesn't require Boost and is developed independently. If you use it in a Python package, you can add it as a requirement and pip will happily install it. It doesn't require you to learn a new language, the bindings are generated using C++ meta-programming techniques under the hood. pybind11 has outstanding documentation and is extremely popular on github (3800+ stars). Refcounting is done automatically, which is why I would even use it to wrap C code. Naturally, it has excellent support for numpy arrays. Pybind11 is FOSS (BSD-style license). I have used Cython, Boost.Python, SWIG, and pybind11 in small to large projects, and pybind11 is by far the most pleasant and the most powerful. You can do really sophisticated things in pybind11, which I cannot imagine doing with other binding tools, and most importantly, it never chokes over your C++ code. Cython and SWIG both have trouble with certain C++ idioms, which is not surprising because C++ is notoriously difficult to parse and these tools were primarily developed to wrap C (which is much easier to parse). For C++, it is much better to not add a custom parser to the toolchain and just let the C++ compiler generate the low-level binding code. This is what pybind11 does. So far all these reasons and more, it should be mentioned and even highlighted here: https://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html I am happy to write a section about it. Disclaimer: I am not at all affiliated with the pybind11 developers, just a thankful user. Best regards, Hans -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Aug 15 06:50:27 2018 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Aug 2018 12:50:27 +0200 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> Message-ID: Hi, ke, 2018-08-15 kello 11:40 +0200, Hans Dembinski kirjoitti: [clip: pybind11] > So far all these reasons and more, it should be mentioned and even > highlighted here: > > https://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html < > https://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html> > > I am happy to write a section about it. Disclaimer: I am not at all > affiliated with the pybind11 developers, just a thankful user. Please go ahead --- the relevant source file is here: https://github.com/numpy/numpy/blob/master/doc/source/user/c-info.python-as-glue.rst It also does no mention CFFI either, mostly because the original text predates the project. -- Pauli Virtanen From charlesr.harris at gmail.com Wed Aug 15 08:33:17 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Aug 2018 06:33:17 -0600 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: Message-ID: On Tue, Aug 14, 2018 at 10:30 PM, Ralf Gommers wrote: > > > On Fri, Aug 3, 2018 at 1:02 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Fri, Aug 3, 2018 at 1:45 PM, Peter Creasey < >> p.e.creasey.00 at googlemail.com> wrote: >> >>> +1 for keeping the same CoC as Scipy, making a new thing just seems a >>> bigger surface area to maintain. Personally I already assumed Scipy's >>> "honour[ing] diversity in..." did not imply any protection of >>> behaviours that violate the CoC *itself*, but if you wanted to be >>> really explicit you could add "to the extent that these do not >>> conflict with this code of conduct." to that line. >>> >> >> I prefer that to the proposed modification, short and sweet. >> > > This edit to the SciPy CoC has now been merged. > > It looks to me like we're good to go here and take over the SciPy CoC. > > +1 Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Aug 15 12:25:13 2018 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 15 Aug 2018 17:25:13 +0100 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: Hi, Thanks Nathaniel for this thoughtful response. On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith wrote: ... > The other approach would be to incrementally add clean, well-defined > dunder methods like __array_ufunc__, __array_concatenate__, etc. This > way we end up putting some thought into each interface, making sure > that it's something we can support, protecting downstream libraries > from unnecessary complexity (e.g. they can implement > __array_concatenate__ instead of hstack, vstack, row_stack, > column_stack, ...), or avoiding adding new APIs entirely (e.g., by > converting existing functions into ufuncs so __array_ufunc__ starts > automagically working). And in the end we get a clean list of dunder > methods that new array container implementations have to define. It's > plausible to imagine a generic test suite for array containers. (I > suspect that every library that tries to implement __array_function__ > will end up with accidental behavioral differences, just because the > numpy API is so vast and contains so many corner cases.) So the > clean-well-defined-dunders approach has lots of upsides. The big > downside is that this is a much longer road to go down. Does everyone agree that, if we had infinite time and resources, this would be the better solution? If we devoted all the resources of the current Numpy grant to taking this track, could we complete it in a reasonable time? Cheers, Matthew From einstein.edison at gmail.com Wed Aug 15 12:36:53 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Wed, 15 Aug 2018 18:36:53 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: <064E6AAF-5488-4B39-8CAB-DE5A559C9E8A@gmail.com> > On 15. Aug 2018, at 18:25, Matthew Brett wrote: > > Hi, > > Thanks Nathaniel for this thoughtful response. > > On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith wrote: > ... >> The other approach would be to incrementally add clean, well-defined >> dunder methods like __array_ufunc__, __array_concatenate__, etc. This >> way we end up putting some thought into each interface, making sure >> that it's something we can support, protecting downstream libraries >> from unnecessary complexity (e.g. they can implement >> __array_concatenate__ instead of hstack, vstack, row_stack, >> column_stack, ...), or avoiding adding new APIs entirely (e.g., by >> converting existing functions into ufuncs so __array_ufunc__ starts >> automagically working). And in the end we get a clean list of dunder >> methods that new array container implementations have to define. It's >> plausible to imagine a generic test suite for array containers. (I >> suspect that every library that tries to implement __array_function__ >> will end up with accidental behavioral differences, just because the >> numpy API is so vast and contains so many corner cases.) So the >> clean-well-defined-dunders approach has lots of upsides. The big >> downside is that this is a much longer road to go down. > > Does everyone agree that, if we had infinite time and resources, this > would be the better solution? > More resources means (given NumPy?s consensus system), more people have to agree on the overall design, so in my mind, it might even be slower. > If we devoted all the resources of the current Numpy grant to taking > this track, could we complete it in a reasonable time? I somehow think just the design of all these different protocols, heck, even ironing all these different protocols out and ignoring implementation; would take an unreasonably long amount of time, as evidenced by this one NEP. I?m more in favour of using this one rather conservatively: Perhaps a mailing list consensus before actually adding a function to __array_function__, making sure it won?t hinder too much progress. I also differ with Nathaniel on one minor thing with his comparisons to Firefox, CPython, pytest and Sphinx: We?re not talking about monkey-patching NumPy internals, we?re just talking about monkey-patching the public API. Of course, this is still a cost and can still hinder development, but it?s definitely better than exposing all internals. > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Best Regards, Hameer Abbasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 15 12:40:09 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Aug 2018 10:40:09 -0600 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Wed, Aug 15, 2018 at 10:25 AM, Matthew Brett wrote: > Hi, > > Thanks Nathaniel for this thoughtful response. > > On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith wrote: > ... > > The other approach would be to incrementally add clean, well-defined > > dunder methods like __array_ufunc__, __array_concatenate__, etc. This > > way we end up putting some thought into each interface, making sure > > that it's something we can support, protecting downstream libraries > > from unnecessary complexity (e.g. they can implement > > __array_concatenate__ instead of hstack, vstack, row_stack, > > column_stack, ...), or avoiding adding new APIs entirely (e.g., by > > converting existing functions into ufuncs so __array_ufunc__ starts > > automagically working). And in the end we get a clean list of dunder > > methods that new array container implementations have to define. It's > > plausible to imagine a generic test suite for array containers. (I > > suspect that every library that tries to implement __array_function__ > > will end up with accidental behavioral differences, just because the > > numpy API is so vast and contains so many corner cases.) So the > > clean-well-defined-dunders approach has lots of upsides. The big > > downside is that this is a much longer road to go down. > > Does everyone agree that, if we had infinite time and resources, this > would be the better solution? > If we devoted all the resources of the current Numpy grant to taking > this track, could we complete it in a reasonable time? > I think it is further down the road than that. Determining a core set of functions would depend on feedback (need), as well be dependent on implementation details for other functions. I think a dependency list for common NumPy functions might be interesting. That said, I don't think the current proposal is orthogonal to this. I don't expect every NumPy function to make call through this API, and we should probably try to limit, and list, the functions that implement the proposed mechanism. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Aug 15 12:44:00 2018 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 15 Aug 2018 17:44:00 +0100 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: <064E6AAF-5488-4B39-8CAB-DE5A559C9E8A@gmail.com> References: <064E6AAF-5488-4B39-8CAB-DE5A559C9E8A@gmail.com> Message-ID: Hi, On Wed, Aug 15, 2018 at 5:36 PM, Hameer Abbasi wrote: > On 15. Aug 2018, at 18:25, Matthew Brett wrote: > > Hi, > > Thanks Nathaniel for this thoughtful response. > > On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith wrote: > ... > > The other approach would be to incrementally add clean, well-defined > dunder methods like __array_ufunc__, __array_concatenate__, etc. This > way we end up putting some thought into each interface, making sure > that it's something we can support, protecting downstream libraries > from unnecessary complexity (e.g. they can implement > __array_concatenate__ instead of hstack, vstack, row_stack, > column_stack, ...), or avoiding adding new APIs entirely (e.g., by > converting existing functions into ufuncs so __array_ufunc__ starts > automagically working). And in the end we get a clean list of dunder > methods that new array container implementations have to define. It's > plausible to imagine a generic test suite for array containers. (I > suspect that every library that tries to implement __array_function__ > will end up with accidental behavioral differences, just because the > numpy API is so vast and contains so many corner cases.) So the > clean-well-defined-dunders approach has lots of upsides. The big > downside is that this is a much longer road to go down. > > > Does everyone agree that, if we had infinite time and resources, this > would be the better solution? > > > More resources means (given NumPy?s consensus system), more people have to > agree on the overall design, so in my mind, it might even be slower. I don't think that's likely. As far as I can see, past discussions have been slow because several people need to get deep down into the details in order to understand the problem, and then stay focused through the discussion. When there is no-one working on that full-time, it's easy for the discussion to drift into the background, and the shared understanding is lost. My suspicion is, to the extent that Matti and Tyler can devote time and energy to shepherding the discussion, these will become quicker and more productive. Cheers, Matthew From shoyer at gmail.com Wed Aug 15 12:45:10 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Wed, 15 Aug 2018 09:45:10 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: Nathaniel, Thanks for raising these thoughtful concerns. Your independent review of this proposal is greatly appreciated! See my responses inline below: On Mon, Aug 13, 2018 at 2:44 AM Nathaniel Smith wrote: > The other approach would be to incrementally add clean, well-defined > dunder methods like __array_ufunc__, __array_concatenate__, etc. This > way we end up putting some thought into each interface, making sure > that it's something we can support, protecting downstream libraries > from unnecessary complexity (e.g. they can implement > __array_concatenate__ instead of hstack, vstack, row_stack, > column_stack, ...), or avoiding adding new APIs entirely (e.g., by > converting existing functions into ufuncs so __array_ufunc__ starts > automagically working). And in the end we get a clean list of dunder > methods that new array container implementations have to define. It's > plausible to imagine a generic test suite for array containers. (I > suspect that every library that tries to implement __array_function__ > will end up with accidental behavioral differences, just because the > numpy API is so vast and contains so many corner cases.) So the > clean-well-defined-dunders approach has lots of upsides. The big > downside is that this is a much longer road to go down. > RE: accidental differences in behavior: I actually think that the __array_function__ approach is *less* prone to accidental differences in behavior, because we require implementing every function directly (or it raises an error). This avoids a classic subclassing problem that has plagued NumPy for years, where overriding the behavior of method A causes apparently unrelated method B to break, because it relied on method A internally. In NumPy, this constrained our implementation of np.median(), because it needed to call np.mean() in order for subclasses implementing units to work properly. There will certainly be accidental differences in behavior for third-party code that *uses* NumPy, but this is basically inevitable for any proposal to allow's NumPy's public API to be overloaded. It's also avoided by default by third-party libraries that follow the current best practice of casting all input arrays with np.asarray(). -------------- RE: a hypothetical simplified interface: The need to implement everything you want to use in NumPy's public API could certainly be onerous, but on the other hand there are a long list of projects that have already done this today -- and these are the projects that most need __array_function__. I'm sure there are cases were simplification would be warranted, but in particular I don't think __array_concatenate__ has significant advantages over simply implementing __array_function__ for np.concatenate. It's a slightly different way of spelling, but it basically does the same thing. The level of complexity to implement hstack, vstack, row_stack and column_stack in terms of np.concatenate is pretty minimal. __array_function__ implementors could easily copy and paste code from NumPy or use a third-party helpers library (like NDArrayOperatorsMixin) that provides such implementations. I also have other concerns about the "simplified API" approach beyond the difficulty of figuring it out, but those are already mentioned in the NEP: http://www.numpy.org/neps/nep-0018-array-function-protocol.html#implementations-in-terms-of-a-limited-core-api But... this is wishful thinking. No matter what the NEP says, I simply > don't believe that we'll actually go break dask, sparse arrays, > xarray, and sklearn in a numpy point release. Or any numpy release. > Nor should we. If we're serious about keeping this experimental ? and > I think that's an excellent idea for now! ? then IMO we need to do > something more to avoid getting trapped by backwards compatibility. > I agree, but to be clear, development for dask, sparse and xarray (and even broadly supported machine learning libraries like TensorFlow) still happens at a much faster pace than is currently the case for "core" projects in the SciPy stack like NumPy. It would not be a big deal to encounter breaking changes in a "major" NumPy release (i.e., 1.X -> 1.(X+1)). (Side note: sklearn doesn't directly implement any array types, so I don't think it would make use of __array_function__ in any way, except possibly to implement overloadable functions.) > My suggestion: at numpy import time, check for an envvar, like say > NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the > __array_function__ dispatches turn into no-ops. This lets interested > downstream libraries and users try this out, but makes sure that we > won't have a hundred thousand end users depending on it without > realizing. - makes it easy for end-users to check how much overhead this adds (by > running their code with it enabled vs disabled) > - if/when we decide to commit to supporting it for real, we just > remove the envvar. > I'm slightly concerned that the cost of reading an environment variable with os.environ could exaggerate the performance cost of __array_function__. It takes about 1 microsecond to read an environment variable on my laptop, which is comparable to the full overhead of __array_function__. So we may want to switch to an explicit Python API instead, e.g., np.enable_experimental_array_function(). My bigger concern is when/how we decide to graduate __array_function__ from requiring an explicit opt-in. We don't need to make a final decision now, but it would be good to clear about what specifically we are waiting for. I see three types of likely scenarios for changing __array_function__: 1. We decide that the overloading the NumPy namespace in general is a bad idea, based on either performance or predictability consequences for third-party libraries. In this case, I imagine we would probably keep __array_function__, but revert to a separate namespace for explicitly overloaded functions, e.g., numpy.api. 2. We want to keep __array_function__, but need a breaking change to the interface (and we're really attached to keeping the name __array_function__). 3. We decide that specific functions should use a different interface (e.g., switch from __array_function__ to __array_ufunc__). (1) and (2) are the sort of major concerns that in my mind would warrant hiding a feature behind an experimental flag. For the most part, I expect (1) could be resolved relatively quickly by running benchmark suites after we have a working version of __array_function__. To be honest, I don't see either of these rollback scenarios as terribly likely, but the downside risk is large enough that we may want to protect ourselves for a major release or two (6-12 months). (3) will be a much longer process, likely to stretch out over years at the current pace of NumPy development. I don't think we'll want to keep an opt-in flag for this long of a period. Rather, we may want to accept a shorter deprecation cycle than usual. In most cases, I suspect we could incrementally switch to new overloads while preserving the __array_function__ overload for a release or two. I don't really understand the 'types' frozenset. The NEP says "it will > be used by most __array_function__ methods, which otherwise would need > to extract this information themselves"... but they still need to > extract the information themselves, because they still have to examine > each object and figure out what type it is. And, simply creating a > frozenset costs ~0.2 ?s on my laptop, which is overhead that we can't > possibly optimize later... > The most flexible alternative would be to just say that we provide an fixed-length iterable, and return a tuple object. (In my microbenchmarks, it's faster to make a tuple than a list or set.) In an early draft of the NEP, I proposed exactly this, but the speed difference seemed really marginal to me. I included 'types' in the interface because I really do think it's something that almost all __array_function__ implementations should use use. It preserves a nice separation of concerns between dispatching logic and implementations for a new type. At least as long as __array_function__ is experimental, I don't think we should be encouraging people to write functions that could return NotImplemented directly and to rely entirely on the NumPy interface. Many but not all implementations will need to look at argument types. This is only really essential for cases where mixed operations between NumPy arrays and another type are allowed. If you only implement the NumPy interface for MyArray objects, then in the usual Python style you wouldn't need isinstance checks. It's also important from an ecosystem perspective. If we don't make it easy to get type information, my guess is that many __array_function__ authors wouldn't bother to return NotImplemented for unexpected types, which means that __array_function__ will break in weird ways when used with objects from unrelated libraries. Cheers, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.corlay at gmail.com Wed Aug 15 13:38:18 2018 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Wed, 15 Aug 2018 19:38:18 +0200 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> Message-ID: Hi Hans, Pauli, If `pybind11` is included, it could be interesting to also include `xtensor` and `xtensor-python`. - Xtensor is a C++ dynamic N-d array library that offers numpy-like features including broadcasting and universal functions. It is also lazy evaluated and continuously benchmarked against numpy, eigen, pythran and numba. You can check out the numpy to xtensor cheat sheet: https://xtensor.readthedocs.io/en/latest/numpy.html. - Xtensor-python makes it possible to operate on numpy arrays inplace using the xtensor API. So that e.g. an xtensor reshape will result in a reshape on the python side (using the numpy C API under the hood). Xtensor-python is built upon pybind11, but brings it much closer to feature parity with NumPy. There is a vibrant community of users and developers, actively working to make xtensor faster and cover more of numpy APIs. I would argue that xtensor-python is one of the easiest ways to make use of numpy arrays from a C++ program, given the similar high level API, and tools to make ufuncs and bindings with one-liners. Resources: - xtensor: https://github.com/QuantStack/xtensor (documentation: https://xtensor.readthedocs.io/) - xtensor-python: https://github.com/QuantStack/xtensor-python (documentation: https://xtensor-python.readthedocs.io/) - xtensor-blas: https://github.com/QuantStack/xtensor-blas (documentation: https://xtensor-blas.readthedocs.io) - xtensor-io: https://github.com/QuantStack/xtensor-io (documentation: https://xtensor-io.readthedocs.io) for reading and writing various file formats Other language bindings: - xtensor-julia: https://github.com/QuantStack/xtensor-julia (documentation: https://xtensor-julia.readthedocs.io/en/latest/) - xtensor-r: https://github.com/QuantStack/xtensor-r (documentation: https://xtensor-r.readthedocs.io/en/latest/) Best, Sylvain On Wed, Aug 15, 2018 at 12:50 PM, Pauli Virtanen wrote: > Hi, > > ke, 2018-08-15 kello 11:40 +0200, Hans Dembinski kirjoitti: > [clip: pybind11] > > So far all these reasons and more, it should be mentioned and even > > highlighted here: > > > > https://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html < > > https://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html> > > > > I am happy to write a section about it. Disclaimer: I am not at all > > affiliated with the pybind11 developers, just a thankful user. > > Please go ahead --- the relevant source file is here: > > https://github.com/numpy/numpy/blob/master/doc/source/ > user/c-info.python-as-glue.rst > > It also does no mention CFFI either, mostly because the original text > predates the project. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Wed Aug 15 15:32:08 2018 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 15 Aug 2018 21:32:08 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: Message-ID: <1653dbbd2a0.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> On August 15, 2018 06:31:08 Ralf Gommers wrote: > > > On Fri, Aug 3, 2018 at 1:02 PM, Charles R Harris > wrote: > > I prefer that to the proposed modification, short and sweet. > > This edit to the SciPy CoC has now been merged. > > It looks to me like we're good to go here and take over the SciPy CoC. +1 St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Wed Aug 15 15:57:17 2018 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 15 Aug 2018 21:57:17 +0200 Subject: [Numpy-discussion] Adoption of a Code of Conduct In-Reply-To: References: Message-ID: <7fa6643ec8189e90f1600febf955d2244d696923.camel@sipsolutions.net> On Tue, 2018-08-14 at 21:30 -0700, Ralf Gommers wrote: > > > On Fri, Aug 3, 2018 at 1:02 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > > > > On Fri, Aug 3, 2018 at 1:45 PM, Peter Creasey < > > p.e.creasey.00 at googlemail.com> wrote: > > > +1 for keeping the same CoC as Scipy, making a new thing just > > > seems a > > > bigger surface area to maintain. Personally I already assumed > > > Scipy's > > > "honour[ing] diversity in..." did not imply any protection of > > > behaviours that violate the CoC *itself*, but if you wanted to be > > > really explicit you could add "to the extent that these do not > > > conflict with this code of conduct." to that line. > > > > I prefer that to the proposed modification, short and sweet. > > > > This edit to the SciPy CoC has now been merged. > > It looks to me like we're good to go here and take over the SciPy > CoC. Sounds good, so +1. I am happy with the committee as well, and I guess most/all are, but we might want to discuss it separately? - Sebastian > > Cheers, > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From anntzer.lee at gmail.com Wed Aug 15 16:27:12 2018 From: anntzer.lee at gmail.com (Antony Lee) Date: Wed, 15 Aug 2018 22:27:12 +0200 Subject: [Numpy-discussion] mplcursors 0.2 release Message-ID: Dear all, I am please to announce the release of mplcursors 0.2. # Description mplcursors provides interactive data selection cursors for Matplotlib. Features include a callback system allowing full customization of the actions when data selection is triggered, and the ability to add a data selection cursor to any preexisting program using Matplotlib by just setting the MPLCURSORS environment variable, without changes to the source code. # Changelog from mplcursors 0.1 to mplcursors 0.2 - Updated dependency to matplotlib 2.1 (2.0 gives more information about orientation of bar plots; 2.1 improves the handling of step plots). - Setting `MPLCURSORS` hooks `Figure.draw` (once per figure only) instead of `plt.show`, thus supporting figures created after the first call to `plt.show`. - Automatic positioning and alignment of annotation text. - Selections on images now have an index as well. - Selections created on `scatter` plots, `errorbar` plots, and `polar` plots can now be moved. - `PathCollection`\s not created by `plt.scatter` are now picked as paths, not as collections of points. - `Patch`\es now pick on their borders, not their interior. - Improved picking of `Container`\s. - In hover mode, annotations can still be removed by right-clicking. Enjoy, Antony Lee -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed Aug 15 19:43:24 2018 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 16 Aug 2018 02:43:24 +0300 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <064E6AAF-5488-4B39-8CAB-DE5A559C9E8A@gmail.com> Message-ID: An HTML attachment was scrubbed... URL: From hans.dembinski at gmail.com Thu Aug 16 04:06:11 2018 From: hans.dembinski at gmail.com (Hans Dembinski) Date: Thu, 16 Aug 2018 10:06:11 +0200 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> Message-ID: <4B77C595-18FE-4A3C-B138-D51777C81237@gmail.com> Dear Paul, > On 15. Aug 2018, at 12:50, Pauli Virtanen wrote: > > Please go ahead --- the relevant source file is here: > > https://github.com/numpy/numpy/blob/master/doc/source/user/c-info.python-as-glue.rst > > It also does no mention CFFI either, mostly because the original text > predates the project. great, I will prepare a pull-request then. Perhaps someone else can also write about CFFI. I would be happy to do it, but I don't know CFFI well enough to write about it. Best regards, Hans From hans.dembinski at gmail.com Thu Aug 16 04:51:43 2018 From: hans.dembinski at gmail.com (Hans Dembinski) Date: Thu, 16 Aug 2018 10:51:43 +0200 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> Message-ID: <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> Hi Sylvain, > On 15. Aug 2018, at 19:38, Sylvain Corlay wrote: > > If `pybind11` is included, it could be interesting to also include `xtensor` and `xtensor-python`. > > - Xtensor is a C++ dynamic N-d array library that offers numpy-like features including broadcasting and universal functions. It is also lazy evaluated and continuously benchmarked against numpy, eigen, pythran and numba. You can check out the numpy to xtensor cheat sheet: https://xtensor.readthedocs.io/en/latest/numpy.html . > > - Xtensor-python makes it possible to operate on numpy arrays inplace using the xtensor API. So that e.g. an xtensor reshape will result in a reshape on the python side (using the numpy C API under the hood). > > Xtensor-python is built upon pybind11, but brings it much closer to feature parity with NumPy. There is a vibrant community of users and developers, actively working to make xtensor faster and cover more of numpy APIs. > > I would argue that xtensor-python is one of the easiest ways to make use of numpy arrays from a C++ program, given the similar high level API, and tools to make ufuncs and bindings with one-liners. > > Resources: > > - xtensor: https://github.com/QuantStack/xtensor (documentation: https://xtensor.readthedocs.io/ ) > - xtensor-python: https://github.com/QuantStack/xtensor-python (documentation: https://xtensor-python.readthedocs.io/ ) > - xtensor-blas: https://github.com/QuantStack/xtensor-blas (documentation: https://xtensor-blas.readthedocs.io ) > - xtensor-io: https://github.com/QuantStack/xtensor-io (documentation: https://xtensor-io.readthedocs.io ) for reading and writing various file formats > > Other language bindings: > > - xtensor-julia: https://github.com/QuantStack/xtensor-julia (documentation: https://xtensor-julia.readthedocs.io/en/latest/ ) > - xtensor-r: https://github.com/QuantStack/xtensor-r (documentation: https://xtensor-r.readthedocs.io/en/latest/ ) sounds good, I think it should be mentioned in the pybind11 part. I just stumbled over xtensor yesterday. Based on your post I read a bit more about it. I like the expression engine and lazy evaluation, the concept is similar to Eigen. xtensor itself has nothing to do with binding, but makes working with numpy arrays on the C++ side easier - especially when you are familiar with the numpy API. The docs say: "Xtensor operations are continuously benchmarked, and are significantly improved at each new version. Current performances on statically dimensioned tensors match those of the Eigen library. Dynamically dimension tensors for which the shape is heap allocated come at a small additional cost." I couldn't find these benchmark results online, though, could you point me to the right page? Google only produced an outdated SO post where numpy performed better than xtensor. Best regards, Hans PS: A bit of nitpicking: you use the term "tensor" for an n-dimensional block of numbers - a generalisation of "matrix", but the term "tensor" in mathematics and physics is more specific. A tensor has well-defined transformation properties when you change the basis of your vector space, just like a "vector" (a vector is a one-dimensional tensor), while a general block of numbers does not. https://en.wikipedia.org/wiki/Tensor -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.corlay at gmail.com Thu Aug 16 07:29:05 2018 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Thu, 16 Aug 2018 13:29:05 +0200 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> Message-ID: Hi Hans, On Thu, Aug 16, 2018 at 10:51 AM, Hans Dembinski wrote: > Hi Sylvain, > > On 15. Aug 2018, at 19:38, Sylvain Corlay > wrote: > > If `pybind11` is included, it could be interesting to also include > `xtensor` and `xtensor-python`. > > - Xtensor is a C++ dynamic N-d array library that offers numpy-like > features including broadcasting and universal functions. It is also lazy > evaluated and continuously benchmarked against numpy, eigen, pythran and > numba. You can check out the numpy to xtensor cheat sheet: > https://xtensor.readthedocs.io/en/latest/numpy.html. > > - Xtensor-python makes it possible to operate on numpy arrays inplace > using the xtensor API. So that e.g. an xtensor reshape will result in a > reshape on the python side (using the numpy C API under the hood). > > Xtensor-python is built upon pybind11, but brings it much closer to > feature parity with NumPy. There is a vibrant community of users and > developers, actively working to make xtensor faster and cover more of numpy > APIs. > > I would argue that xtensor-python is one of the easiest ways to make use > of numpy arrays from a C++ program, given the similar high level API, and > tools to make ufuncs and bindings with one-liners. > > Resources: > > - xtensor: https://github.com/QuantStack/xtensor (documentation: > https://xtensor.readthedocs.io/) > - xtensor-python: https://github.com/QuantStack/xtensor-python > (documentation: https://xtensor-python.readthedocs.io/) > - xtensor-blas: https://github.com/QuantStack/xtensor-blas (documentation: > https://xtensor-blas.readthedocs.io) > - xtensor-io: https://github.com/QuantStack/xtensor-io (documentation: > https://xtensor-io.readthedocs.io) for reading and writing various file > formats > > Other language bindings: > > - xtensor-julia: https://github.com/QuantStack/xtensor-julia > (documentation: https://xtensor-julia.readthedocs.io/en/latest/) > - xtensor-r: https://github.com/QuantStack/xtensor-r (documentation: > https://xtensor-r.readthedocs.io/en/latest/) > > > sounds good, I think it should be mentioned in the pybind11 part. I just > stumbled over xtensor yesterday. Based on your post I read a bit more about > it. I like the expression engine and lazy evaluation, the concept is > similar to Eigen. xtensor itself has nothing to do with binding, but makes > working with numpy arrays on the C++ side easier - especially when you are > familiar with the numpy API. > Actually, xtensor-python does a lot more in terms of numpy bindings, as it uses the C APIs of numpy directly for a number of things. Plus, the integration into the xtensor expression system lets you do things such as view / broadcasting / newaxis / ufuncs directly from the C++ side (and all that is in the cheat sheets). > The docs say: > "Xtensor operations are continuously benchmarked, and are significantly > improved at each new version. Current performances on statically > dimensioned tensors match those of the Eigen library. Dynamically dimension > tensors for which the shape is heap allocated come at a small additional > cost." > > I couldn't find these benchmark results online, though, could you point me > to the right page? Google only produced an outdated SO post where numpy > performed better than xtensor. > > That is because we run the benchmarks on our own hardware. Since xtensor is explicitly SIMD accelerated for a variety of architectures including e.g. avx512, it is hard to have a consistent environment to run the benchmarks. We have a I9 machine that runs the benchmarks with various options, and manually run them on raspberry pis for the neon acceleration benchmarks (continuous testing of neon instruction sets are tested with an emulator on travisci in the xsimd project). Cheers, Sylvain > Best regards, > Hans > > PS: A bit of nitpicking: you use the term "tensor" for an n-dimensional > block of numbers - a generalisation of "matrix", but the term "tensor" in > mathematics and physics is more specific. A tensor has well-defined > transformation properties when you change the basis of your vector space, > just like a "vector" (a vector is a one-dimensional tensor), while a > general block of numbers does not. > > https://en.wikipedia.org/wiki/Tensor > > it is clearly a very overloaded term. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.dembinski at gmail.com Fri Aug 17 05:00:47 2018 From: hans.dembinski at gmail.com (Hans Dembinski) Date: Fri, 17 Aug 2018 11:00:47 +0200 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> Message-ID: <863F4C6B-60DA-4E8A-9946-DD9C6C514580@gmail.com> Hi Sylvain, > On 16. Aug 2018, at 13:29, Sylvain Corlay wrote: > > Actually, xtensor-python does a lot more in terms of numpy bindings, as it uses the C APIs of numpy directly for a number of things. > > Plus, the integration into the xtensor expression system lets you do things such as view / broadcasting / newaxis / ufuncs directly from the C++ side (and all that is in the cheat sheets). ok, good, but my point was different. The page in question is about Python as a glue language. The other solutions on that site are general purpose binding solutions for any kind of C++ code, while xtensor-python is xtensor-specific. xtensor in turn is a library that mimics the numpy API in C++. > The docs say: > "Xtensor operations are continuously benchmarked, and are significantly improved at each new version. Current performances on statically dimensioned tensors match those of the Eigen library. Dynamically dimension tensors for which the shape is heap allocated come at a small additional cost." > > I couldn't find these benchmark results online, though, could you point me to the right page? Google only produced an outdated SO post where numpy performed better than xtensor. > > > That is because we run the benchmarks on our own hardware. Since xtensor is explicitly SIMD accelerated for a variety of architectures including e.g. avx512, it is hard to have a consistent environment to run the benchmarks. We have a I9 machine that runs the benchmarks with various options, and manually run them on raspberry pis for the neon acceleration benchmarks (continuous testing of neon instruction sets are tested with an emulator on travisci in the xsimd project). Ok, but you can still put the results for everyone to see and judge by themselves on a web page. Just state on what kind of machine you ran the code. It is ok if the results on my machine differ, I am still interested in the results that you get on your machines and since you generate them anyway, I don't see why not. > [tensor] is clearly a very overloaded term. I agree that vector is a very overloaded term (the STL vector is particularly to blame). But until recently, tensor used to be a well-defined technical term which exclusively referred to a specific mathematical concept https://en.wikipedia.org/wiki/Tensor_(disambiguation) https://medium.com/@quantumsteinke/whats-the-difference-between-a-matrix-and-a-tensor-4505fbdc576c https://www.quora.com/What-is-a-tensor Then Google started to popularise the term wrongly in Tensorflow and now another well-defined technical term gets watered down. xtensor is going with the tensorflow, sad. Best regards, Hans -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Aug 17 17:44:01 2018 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 17 Aug 2018 14:44:01 -0700 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: <863F4C6B-60DA-4E8A-9946-DD9C6C514580@gmail.com> References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> <863F4C6B-60DA-4E8A-9946-DD9C6C514580@gmail.com> Message-ID: On Fri, Aug 17, 2018 at 2:00 AM Hans Dembinski wrote: > Hi Sylvain, > > On 16. Aug 2018, at 13:29, Sylvain Corlay > wrote: > > Actually, xtensor-python does a lot more in terms of numpy bindings, as it > uses the C APIs of numpy directly for a number of things. > > Plus, the integration into the xtensor expression system lets you do > things such as view / broadcasting / newaxis / ufuncs directly from the C++ > side (and all that is in the cheat sheets). > > ok, good, but my point was different. The page in question is about Python > as a glue language. The other solutions on that site are general purpose > binding solutions for any kind of C++ code, while xtensor-python is > xtensor-specific. xtensor in turn is a library that mimics the numpy API in > C++. > Even if you don't use the numpy-mimicking parts of the xtensor API, xtensor-python is a probably a net improvement over pybind11 for communicating arrays back and forth across the C++/Python boundary. Even if the rest of your C++ code doesn't use xtensor, you could profitably use xtensor-python at the interface. Also, though the article is generally framed as using Python as a glue language (i.e. communicating with existing C/C++/Fortran code), it is also relevant for the use case where you are writing the C/C++/Fortran code from scratch (perhaps just accelerating small kernels or whatever). Talking about the available options for that use case is perfectly on-topic for that article. You don't have to be the one that writes it, though, if you just want to cover pybind11. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Fri Aug 17 17:59:31 2018 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 17 Aug 2018 17:59:31 -0400 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> <863F4C6B-60DA-4E8A-9946-DD9C6C514580@gmail.com> Message-ID: The only way I've seen xtensor used is *with* pybind11 On Fri, Aug 17, 2018 at 5:45 PM Robert Kern wrote: > On Fri, Aug 17, 2018 at 2:00 AM Hans Dembinski > wrote: > >> Hi Sylvain, >> >> On 16. Aug 2018, at 13:29, Sylvain Corlay >> wrote: >> >> Actually, xtensor-python does a lot more in terms of numpy bindings, as >> it uses the C APIs of numpy directly for a number of things. >> >> Plus, the integration into the xtensor expression system lets you do >> things such as view / broadcasting / newaxis / ufuncs directly from the C++ >> side (and all that is in the cheat sheets). >> >> ok, good, but my point was different. The page in question is about >> Python as a glue language. The other solutions on that site are general >> purpose binding solutions for any kind of C++ code, while xtensor-python is >> xtensor-specific. xtensor in turn is a library that mimics the numpy API in >> C++. >> > > Even if you don't use the numpy-mimicking parts of the xtensor API, > xtensor-python is a probably a net improvement over pybind11 for > communicating arrays back and forth across the C++/Python boundary. Even if > the rest of your C++ code doesn't use xtensor, you could profitably use > xtensor-python at the interface. Also, though the article is generally > framed as using Python as a glue language (i.e. communicating with existing > C/C++/Fortran code), it is also relevant for the use case where you are > writing the C/C++/Fortran code from scratch (perhaps just accelerating > small kernels or whatever). Talking about the available options for that > use case is perfectly on-topic for that article. > > You don't have to be the one that writes it, though, if you just want to > cover pybind11. > > -- > Robert Kern > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Aug 18 10:43:24 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 18 Aug 2018 08:43:24 -0600 Subject: [Numpy-discussion] The http://planet.scipy.org/ site is down. Message-ID: Hi All, Anyone know who hosts http://planet.scipy.org/? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sat Aug 18 10:51:01 2018 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 18 Aug 2018 07:51:01 -0700 Subject: [Numpy-discussion] The http://planet.scipy.org/ site is down. In-Reply-To: References: Message-ID: It used to be enthought. Sorry, I'm traveling for vacations, and I cannot take care of this right now. Ga?l ?Sent from my phone. Please forgive typos and briefness.? On Aug 18, 2018, 07:44, at 07:44, Charles R Harris wrote: >Hi All, > >Anyone know who hosts http://planet.scipy.org/? > >Chuck > > >------------------------------------------------------------------------ > >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at python.org >https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From robbmcleod at gmail.com Sun Aug 19 14:58:12 2018 From: robbmcleod at gmail.com (Robert McLeod) Date: Sun, 19 Aug 2018 11:58:12 -0700 Subject: [Numpy-discussion] ANN: NumExpr 2.6.8 Message-ID: ========================== Announcing Numexpr 2.6.8 ========================== Hi everyone, Our attempt to fix the memory leak in 2.6.7 had an unforseen consequence that the `f_locals` from the top-most frame is actually `f_globals`, and clearing it to fix the extra reference count deletes all global variables. Needless to say this is undesired behavior. A check has been added to prevent clearing the globals dict, tested against both `python` and `ipython`. As such, we recommend skipping 2.6.7 and upgrading straight to 2.6.8 from 2.6.6. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.6.7 to 2.6.8 --------------------------- - Add check to make sure that `f_locals` is not actually `f_globals` when we do the `f_locals` clear to avoid the #310 memory leak issue. - Compare NumPy versions using `distutils.version.LooseVersion` to avoid issue #312 when working with NumPy development versions. - As part of `multibuild`, wheels for Python 3.7 for Linux and MacOSX are now available on PyPI. What's Numexpr? --------------- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? ------------------------- The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience --------------------- Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod, Ph.D. robbmcleod at gmail.com robbmcleod at protonmail.com robert.mcleod at hitachi-hhtc.ca www.entropyreduction.al -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Aug 19 16:53:56 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 19 Aug 2018 14:53:56 -0600 Subject: [Numpy-discussion] NumPy 1.15.1 release Message-ID: Hi All, The NumPy 1.15.1 release is ready to go, the release notes may be reviewed at gh-11787 . Pending comments, I will make the release next Tuesday. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dpinte at enthought.com Mon Aug 20 07:06:06 2018 From: dpinte at enthought.com (Didrik Pinte) Date: Mon, 20 Aug 2018 13:06:06 +0200 Subject: [Numpy-discussion] The http://planet.scipy.org/ site is down. In-Reply-To: References: Message-ID: Andy Terrel and I have been in the process of migrating the service out of Rackspace account. There was an issue when the Rackspace machine and its snapshots got deleted a little too fast. We're working on recovering the service asap. -- Didrik On Sat, 18 Aug 2018 at 16:52, Gael Varoquaux wrote: > It used to be enthought. > > Sorry, I'm traveling for vacations, and I cannot take care of this right > now. > > Ga?l > > Sent from my phone. Please forgive typos and briefness. > On Aug 18, 2018, at 07:44, Charles R Harris > wrote: >> >> Hi All, >> >> Anyone know who hosts http://planet.scipy.org/? >> >> Chuck >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From andy.terrel at gmail.com Mon Aug 20 07:39:08 2018 From: andy.terrel at gmail.com (Andy Ray Terrel) Date: Mon, 20 Aug 2018 06:39:08 -0500 Subject: [Numpy-discussion] The http://planet.scipy.org/ site is down. In-Reply-To: References: Message-ID: Unfortunately the machine that was hosting planet.scipy.org has died. The volunteer that was supporting it, Sheila Miguez, had warned the scipy leadership for over a year that it was going down. Over the last six months after the Rackspace open source program has ended she repeatedly reached out for help. I tried to save it but the machine wasn?t backed up or had any instructions on how to operate. The rackspace images couldn?t be transferred to the NumFOCUS account for some reason that was never understood. Sheila informed Rackspace that she couldn?t pay the bill of over $1200 for the machine and we would need to cancel the account. I paid the bill for her, and started the process of copying files over to a different box. Rackspace, despite telling us it would take 3 days to delete the account took less then 3hours. Since then I have been on numerous support calls but it appears the account is gone. ? Andy On Mon, Aug 20, 2018 at 6:07 AM Didrik Pinte wrote: > Andy Terrel and I have been in the process of migrating the service out of > Rackspace account. There was an issue when the Rackspace machine and its > snapshots got deleted a little too fast. We're working on recovering the > service asap. > > -- Didrik > > On Sat, 18 Aug 2018 at 16:52, Gael Varoquaux < > gael.varoquaux at normalesup.org> wrote: > >> It used to be enthought. >> >> Sorry, I'm traveling for vacations, and I cannot take care of this right >> now. >> >> Ga?l >> >> Sent from my phone. Please forgive typos and briefness. >> On Aug 18, 2018, at 07:44, Charles R Harris >> wrote: >>> >>> Hi All, >>> >>> Anyone know who hosts http://planet.scipy.org/? >>> >>> Chuck >>> >>> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Aug 20 11:26:21 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 20 Aug 2018 09:26:21 -0600 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: Ping to finish up this discussion so we can come to a conclusion. I'm in favor of the NEP, as I don't see it as orthogonal to Nathaniel's concerns. However, we might want to be selective as to which functions we expose via the `__array_function__` method. On Wed, Aug 15, 2018 at 10:45 AM, Stephan Hoyer wrote: > Nathaniel, > > Thanks for raising these thoughtful concerns. Your independent review of > this proposal is greatly appreciated! > > See my responses inline below: > > On Mon, Aug 13, 2018 at 2:44 AM Nathaniel Smith wrote: > >> The other approach would be to incrementally add clean, well-defined >> dunder methods like __array_ufunc__, __array_concatenate__, etc. This >> way we end up putting some thought into each interface, making sure >> that it's something we can support, protecting downstream libraries >> from unnecessary complexity (e.g. they can implement >> __array_concatenate__ instead of hstack, vstack, row_stack, >> column_stack, ...), or avoiding adding new APIs entirely (e.g., by >> converting existing functions into ufuncs so __array_ufunc__ starts >> automagically working). And in the end we get a clean list of dunder >> methods that new array container implementations have to define. It's >> plausible to imagine a generic test suite for array containers. (I >> suspect that every library that tries to implement __array_function__ >> will end up with accidental behavioral differences, just because the >> numpy API is so vast and contains so many corner cases.) So the >> clean-well-defined-dunders approach has lots of upsides. The big >> downside is that this is a much longer road to go down. >> > > RE: accidental differences in behavior: > > I actually think that the __array_function__ approach is *less* prone to > accidental differences in behavior, because we require implementing every > function directly (or it raises an error). > > This avoids a classic subclassing problem that has plagued NumPy for > years, where overriding the behavior of method A causes apparently > unrelated method B to break, because it relied on method A internally. In > NumPy, this constrained our implementation of np.median(), because it > needed to call np.mean() in order for subclasses implementing units to work > properly. > > There will certainly be accidental differences in behavior for third-party > code that *uses* NumPy, but this is basically inevitable for any proposal > to allow's NumPy's public API to be overloaded. It's also avoided by > default by third-party libraries that follow the current best practice of > casting all input arrays with np.asarray(). > > -------------- > > RE: a hypothetical simplified interface: > > The need to implement everything you want to use in NumPy's public API > could certainly be onerous, but on the other hand there are a long list of > projects that have already done this today -- and these are the projects > that most need __array_function__. > > I'm sure there are cases were simplification would be warranted, but in > particular I don't think __array_concatenate__ has significant advantages > over simply implementing __array_function__ for np.concatenate. It's a > slightly different way of spelling, but it basically does the same thing. > The level of complexity to implement hstack, vstack, row_stack and > column_stack in terms of np.concatenate is pretty minimal. > __array_function__ implementors could easily copy and paste code from NumPy > or use a third-party helpers library (like NDArrayOperatorsMixin) that > provides such implementations. > > I also have other concerns about the "simplified API" approach beyond the > difficulty of figuring it out, but those are already mentioned in the NEP: > http://www.numpy.org/neps/nep-0018-array-function-protocol. > html#implementations-in-terms-of-a-limited-core-api > > But... this is wishful thinking. No matter what the NEP says, I simply >> don't believe that we'll actually go break dask, sparse arrays, >> xarray, and sklearn in a numpy point release. Or any numpy release. >> Nor should we. If we're serious about keeping this experimental ? and >> I think that's an excellent idea for now! ? then IMO we need to do >> something more to avoid getting trapped by backwards compatibility. >> > > I agree, but to be clear, development for dask, sparse and xarray (and > even broadly supported machine learning libraries like TensorFlow) still > happens at a much faster pace than is currently the case for "core" > projects in the SciPy stack like NumPy. It would not be a big deal to > encounter breaking changes in a "major" NumPy release (i.e., 1.X -> > 1.(X+1)). > > (Side note: sklearn doesn't directly implement any array types, so I don't > think it would make use of __array_function__ in any way, except possibly > to implement overloadable functions.) > Here is Travis Oliphant's talk at PyBay , where he talks about the proliferation of arrays and interfaces in the ML/AI ecosystem among other things. I think that we should definitely try to get NumPy out there as an option in the near future. > >> My suggestion: at numpy import time, check for an envvar, like say >> NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the >> __array_function__ dispatches turn into no-ops. This lets interested >> downstream libraries and users try this out, but makes sure that we >> won't have a hundred thousand end users depending on it without >> realizing. > > > > - makes it easy for end-users to check how much overhead this adds (by >> running their code with it enabled vs disabled) >> - if/when we decide to commit to supporting it for real, we just >> remove the envvar. >> > > I'm slightly concerned that the cost of reading an environment variable > with os.environ could exaggerate the performance cost of > __array_function__. It takes about 1 microsecond to read an environment > variable on my laptop, which is comparable to the full overhead of > __array_function__. So we may want to switch to an explicit Python API > instead, e.g., np.enable_experimental_array_function(). > > My bigger concern is when/how we decide to graduate __array_function__ > from requiring an explicit opt-in. We don't need to make a final decision > now, but it would be good to clear about what specifically we are waiting > for. > > I see three types of likely scenarios for changing __array_function__: > 1. We decide that the overloading the NumPy namespace in general is a bad > idea, based on either performance or predictability consequences for > third-party libraries. In this case, I imagine we would probably keep > __array_function__, but revert to a separate namespace for explicitly > overloaded functions, e.g., numpy.api. > 2. We want to keep __array_function__, but need a breaking change to the > interface (and we're really attached to keeping the name > __array_function__). > 3. We decide that specific functions should use a different interface > (e.g., switch from __array_function__ to __array_ufunc__). > > (1) and (2) are the sort of major concerns that in my mind would warrant > hiding a feature behind an experimental flag. For the most part, I expect > (1) could be resolved relatively quickly by running benchmark suites after > we have a working version of __array_function__. To be honest, I don't see > either of these rollback scenarios as terribly likely, but the downside > risk is large enough that we may want to protect ourselves for a major > release or two (6-12 months). > > (3) will be a much longer process, likely to stretch out over years at the > current pace of NumPy development. I don't think we'll want to keep an > opt-in flag for this long of a period. Rather, we may want to accept a > shorter deprecation cycle than usual. In most cases, I suspect we could > incrementally switch to new overloads while preserving the > __array_function__ overload for a release or two. > > I don't really understand the 'types' frozenset. The NEP says "it will >> be used by most __array_function__ methods, which otherwise would need >> to extract this information themselves"... but they still need to >> extract the information themselves, because they still have to examine >> each object and figure out what type it is. And, simply creating a >> frozenset costs ~0.2 ?s on my laptop, which is overhead that we can't >> possibly optimize later... >> > > The most flexible alternative would be to just say that we provide an > fixed-length iterable, and return a tuple object. (In my microbenchmarks, > it's faster to make a tuple than a list or set.) In an early draft of the > NEP, I proposed exactly this, but the speed difference seemed really > marginal to me. > > I included 'types' in the interface because I really do think it's > something that almost all __array_function__ implementations should use > use. It preserves a nice separation of concerns between dispatching logic > and implementations for a new type. At least as long as __array_function__ > is experimental, I don't think we should be encouraging people to write > functions that could return NotImplemented directly and to rely entirely on > the NumPy interface. > > Many but not all implementations will need to look at argument types. This > is only really essential for cases where mixed operations between NumPy > arrays and another type are allowed. If you only implement the NumPy > interface for MyArray objects, then in the usual Python style you wouldn't > need isinstance checks. > > It's also important from an ecosystem perspective. If we don't make it > easy to get type information, my guess is that many __array_function__ > authors wouldn't bother to return NotImplemented for unexpected types, > which means that __array_function__ will break in weird ways when used with > objects from unrelated libraries. > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.dembinski at gmail.com Mon Aug 20 11:52:15 2018 From: hans.dembinski at gmail.com (Hans Dembinski) Date: Mon, 20 Aug 2018 17:52:15 +0200 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> <863F4C6B-60DA-4E8A-9946-DD9C6C514580@gmail.com> Message-ID: <8DA9F65B-0F7C-4728-A343-29E9E3B5C0E0@gmail.com> Dear Robert, > On 17. Aug 2018, at 23:44, Robert Kern wrote: > > Even if you don't use the numpy-mimicking parts of the xtensor API, xtensor-python is a probably a net improvement over pybind11 for communicating arrays back and forth across the C++/Python boundary. Even if the rest of your C++ code doesn't use xtensor, you could profitably use xtensor-python at the interface. Also, though the article is generally framed as using Python as a glue language (i.e. communicating with existing C/C++/Fortran code), it is also relevant for the use case where you are writing the C/C++/Fortran code from scratch (perhaps just accelerating small kernels or whatever). Talking about the available options for that use case is perfectly on-topic for that article. no objections here, xtensor should be highlighted in the pybind11 part for these reasons. I just think it should not be a separate section. Best regards, Hans From shoyer at gmail.com Mon Aug 20 11:57:09 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Mon, 20 Aug 2018 08:57:09 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Mon, Aug 20, 2018 at 8:27 AM Charles R Harris wrote: > Ping to finish up this discussion so we can come to a conclusion. I'm in > favor of the NEP, as I don't see it as orthogonal to Nathaniel's concerns. > However, we might want to be selective as to which functions we expose via > the `__array_function__` method. > Chunk -- thanks for bringing this back up. My proposal is to hide this feature behind an environment variable for now, or something morally equivalent to an environment variable if that's too slow (i.e., an explicit Python variable). I don't think Nathaniel's concerns are entirely warranted for the reasons I went into in my earlier reply, but I do really want to get this moving forward now in whatever way is necessary. We can figure out the rest down the road. Nathaniel -- are you OK with that? > Here is Travis Oliphant's talk at PyBay > , where he talks about > the proliferation of arrays and interfaces in the ML/AI ecosystem among > other things. I think that we should definitely try to get NumPy out there > as an option in the near future. > Yes, there is an urgent need for this :). Cheers, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Mon Aug 20 11:57:29 2018 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 20 Aug 2018 17:57:29 +0200 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: <8DA9F65B-0F7C-4728-A343-29E9E3B5C0E0@gmail.com> References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> <863F4C6B-60DA-4E8A-9946-DD9C6C514580@gmail.com> <8DA9F65B-0F7C-4728-A343-29E9E3B5C0E0@gmail.com> Message-ID: I'm confused, do you have a link or example showing how to use xtensor-python without pybind11? Thanks, Neal On Mon, Aug 20, 2018 at 5:50 PM Hans Dembinski wrote: > Dear Robert, > > > On 17. Aug 2018, at 23:44, Robert Kern wrote: > > > > Even if you don't use the numpy-mimicking parts of the xtensor API, > xtensor-python is a probably a net improvement over pybind11 for > communicating arrays back and forth across the C++/Python boundary. Even if > the rest of your C++ code doesn't use xtensor, you could profitably use > xtensor-python at the interface. Also, though the article is generally > framed as using Python as a glue language (i.e. communicating with existing > C/C++/Fortran code), it is also relevant for the use case where you are > writing the C/C++/Fortran code from scratch (perhaps just accelerating > small kernels or whatever). Talking about the available options for that > use case is perfectly on-topic for that article. > > no objections here, xtensor should be highlighted in the pybind11 part for > these reasons. I just think it should not be a separate section. > > Best regards, > Hans > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.corlay at gmail.com Mon Aug 20 12:04:09 2018 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Mon, 20 Aug 2018 12:04:09 -0400 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: <8DA9F65B-0F7C-4728-A343-29E9E3B5C0E0@gmail.com> References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> <863F4C6B-60DA-4E8A-9946-DD9C6C514580@gmail.com> <8DA9F65B-0F7C-4728-A343-29E9E3B5C0E0@gmail.com> Message-ID: Thanks guys. If we add pybind11 and xtensor, boost.python is also a good contender there. S. On Mon, Aug 20, 2018, 11:51 Hans Dembinski wrote: > Dear Robert, > > > On 17. Aug 2018, at 23:44, Robert Kern wrote: > > > > Even if you don't use the numpy-mimicking parts of the xtensor API, > xtensor-python is a probably a net improvement over pybind11 for > communicating arrays back and forth across the C++/Python boundary. Even if > the rest of your C++ code doesn't use xtensor, you could profitably use > xtensor-python at the interface. Also, though the article is generally > framed as using Python as a glue language (i.e. communicating with existing > C/C++/Fortran code), it is also relevant for the use case where you are > writing the C/C++/Fortran code from scratch (perhaps just accelerating > small kernels or whatever). Talking about the available options for that > use case is perfectly on-topic for that article. > > no objections here, xtensor should be highlighted in the pybind11 part for > these reasons. I just think it should not be a separate section. > > Best regards, > Hans > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Aug 20 13:17:49 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 20 Aug 2018 10:17:49 -0700 Subject: [Numpy-discussion] Add pybind11 to docs about writing binding code In-Reply-To: References: <38B997CD-79E8-40E0-9ACB-7DCB85E5753E@gmail.com> <8A45D352-66F6-4C7D-867F-DFC47026DFC6@gmail.com> <863F4C6B-60DA-4E8A-9946-DD9C6C514580@gmail.com> <8DA9F65B-0F7C-4728-A343-29E9E3B5C0E0@gmail.com> Message-ID: On Mon, Aug 20, 2018 at 8:57 AM, Neal Becker wrote: > I'm confused, do you have a link or example showing how to use > xtensor-python without pybind11? > I think you may have it backwards: """ The Python bindings for xtensor are based on the pybind11 C++ library, which enables seemless interoperability between C++ and Python. """ So no, yu can't use xtenson-python without pybind11 -- I think what was suggested was that you *could* use xtenson-python without using xtenson on the C++ side. i.e. xtensor-python is a higher level binding system than pybind11 alone, rather than just bindings for xtensor. And thus belongs in the docs about binding tools. Which makes me want to take a closer look at it... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Aug 20 13:18:44 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 20 Aug 2018 10:18:44 -0700 Subject: [Numpy-discussion] The http://planet.scipy.org/ site is down. In-Reply-To: References: Message-ID: On Mon, Aug 20, 2018 at 4:39 AM Andy Ray Terrel wrote: > Unfortunately the machine that was hosting planet.scipy.org has died. > > The volunteer that was supporting it, Sheila Miguez, had warned the scipy > leadership for over a year that it was going down. Over the last six months > after the Rackspace open source program has ended she repeatedly reached > out for help. I tried to save it but the machine wasn?t backed up or had > any instructions on how to operate. The rackspace images couldn?t be > transferred to the NumFOCUS account for some reason that was never > understood. > > Sheila informed Rackspace that she couldn?t pay the bill of over $1200 for > the machine and we would need to cancel the account. I paid the bill for > her, and started the process of copying files over to a different box. > Rackspace, despite telling us it would take 3 days to delete the account > took less then 3hours. Since then I have been on numerous support calls but > it appears the account is gone. > Thanks for your efforts Andy and Didrik. Planet SciPy was useful and fairly popular I think, so worth considering if and how we want to revive it. Besides the server that planet.scipy.org was living on (which was in bad shape), there's the issue that the Planet software hasn't been maintained for ages. Here's what I dug up a couple of years ago: "It took me forever to find the latest planet code, it's actually named Planet Venus now (Planet Planet is dead): http://thread.gmane.org/gmane.comp.web.planet.devel/2223 https://github.com/rubys/venus http://www.intertwingly.net/code/venus/docs/index.html " Planet Venus had its last update in 2011, so also not viable anymore. The only semi-maintained one (last update Nov 2017) is Planet Pluto now ( https://github.com/feedreader/), which is a Ruby project. So probably best to look for a non-planet solution (e.g. https://www.wprssaggregator.com/). If anyone has experience with something like this that works, would be great to hear. Number 1 on my wish list for an alternative is something that's either hosted or serverless; we always have problems with any server that needs maintaining. Right now there's on healthy server for scipy.org/docs.scipy.org, and I wouldn't want to see that run anything like Planet or Wordpress. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Aug 20 19:02:23 2018 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 21 Aug 2018 01:02:23 +0200 Subject: [Numpy-discussion] The http://planet.scipy.org/ site is down. In-Reply-To: References: Message-ID: <1774e779bd31b5dcb750c25b1c023fce9dbf691f.camel@iki.fi> ma, 2018-08-20 kello 10:18 -0700, Ralf Gommers kirjoitti: [clip] > Number 1 on my wish list for an alternative is something that's > either > hosted or serverless; we always have problems with any server that > needs > maintaining. Right now there's on healthy server for > scipy.org/docs.scipy.org, and I wouldn't want to see that run > anything like > Planet or Wordpress. There are some static site generators, including rawdog ( https://offog.org/code/rawdog/) which apparently is used for planet.kde.org Proof of concept: https://pv.github.io/rawdogplanetscipy/ https://github.com/pv/rawdogplanetscipy/ This has the advantage that it's sort of stateless --- anyone can just clone the git repository, and it should generate more or less the same page. Only static web server needed. Probably someone could put it into crontab and mostly forget about it. However, it's misuse of github pages to host this there, so some other static host would need to be found. Not volunteering to take this into completion, though. Pauli From njs at pobox.com Tue Aug 21 03:19:44 2018 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 21 Aug 2018 00:19:44 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Wed, Aug 15, 2018 at 9:45 AM, Stephan Hoyer wrote: > RE: accidental differences in behavior: > > I actually think that the __array_function__ approach is *less* prone to > accidental differences in behavior, because we require implementing every > function directly (or it raises an error). > > This avoids a classic subclassing problem that has plagued NumPy for years, > where overriding the behavior of method A causes apparently unrelated method > B to break, because it relied on method A internally. In NumPy, this > constrained our implementation of np.median(), because it needed to call > np.mean() in order for subclasses implementing units to work properly. I don't think I follow... if B uses A internally, then overriding A shouldn't cause B to break, unless the overridden A is buggy. The median() case was different: it wasn't overriding A that caused B to break, that part worked fine. It was when we changed the implementation of B that we had problems. ...actually, this made me realize that I was uncertain about what exactly happened in that case. I just checked, and AFAICT with current astropy the call to mean() is unnecessary. I tried modifying np.median to remove the call to mean, and it still gave the same result for np.median([1, 2, 3] * u.m). I also checked np.percentile, and it seems to work fine on units-arrays if you make it call np.asanyarray instead of np.asarray. The issue is here if anyone else wants to refresh their memory: https://github.com/numpy/numpy/issues/3846 Reading the comments there it looks like we might have hacked np.median so it (a) uses np.asanyarray, and (b) always calls np.mean, and actually the 'asanyarray' change is what astropy needed and the 'mean' part was just a red herring all along! Whoops. And here I've been telling this story for years now... > RE: a hypothetical simplified interface: > > The need to implement everything you want to use in NumPy's public API could > certainly be onerous, but on the other hand there are a long list of > projects that have already done this today -- and these are the projects > that most need __array_function__. > > I'm sure there are cases were simplification would be warranted, but in > particular I don't think __array_concatenate__ has significant advantages > over simply implementing __array_function__ for np.concatenate. It's a > slightly different way of spelling, but it basically does the same thing. > The level of complexity to implement hstack, vstack, row_stack and > column_stack in terms of np.concatenate is pretty minimal. > __array_function__ implementors could easily copy and paste code from NumPy > or use a third-party helpers library (like NDArrayOperatorsMixin) that > provides such implementations. And when we fix a bug in row_stack, this means we also have to fix it in all the copy-paste versions, which won't happen, so np.row_stack has different semantics on different objects, even if they started out matching. The NDArrayOperatorsMixin reduces the number of duplicate copies of the same code that need to be updated, but 2 copies is still a lot worse than 1 copy :-). > I also have other concerns about the "simplified API" approach beyond the > difficulty of figuring it out, but those are already mentioned in the NEP: > http://www.numpy.org/neps/nep-0018-array-function-protocol.html#implementations-in-terms-of-a-limited-core-api Yeah, there are definitely trade-offs. I don't have like, knock-down rebuttals to these or anything, but since I didn't comment on them before I might as well say a few words :-). > 1. The details of how NumPy implements a high-level function in terms of overloaded functions now becomes an implicit part of NumPy?s public API. For example, refactoring stack to use np.block() instead of np.concatenate() internally would now become a breaking change. The way I'm imagining this would work is, we guarantee not to take a function that used to be implemented in terms of overridable operations, and refactor it so it's implemented in terms of overridable operations. So long as people have correct implementations of __array_concatenate__ and __array_block__, they shouldn't care which one we use. In the interim period where we have __array_concatenate__ but there's no such thing as __array_block__, then that refactoring would indeed break things, so we shouldn't do that :-). But we could fix that by adding __array_block__. > 2. Array libraries may prefer to implement high level functions differently than NumPy. For example, a library might prefer to implement a fundamental operations like mean() directly rather than relying on sum() followed by division. More generally, it?s not clear yet what exactly qualifies as core functionality, and figuring this out could be a large project. True. And this is a very general problem... for example, the appropriate way to implement logistic regression is very different in-core versus out-of-core. You're never going to be able to take code written for ndarray, drop in an arbitrary new array object, and get optimal results in all cases -- that's just way too ambitious to hope for. There will be cases where reducing to operations like sum() and division is fine. There will be cases where you have a high-level operation like logistic regression, where reducing to sum() and division doesn't work, but reducing to slightly-higher-level operations like np.mean also doesn't work, because you need to redo the whole high-level operation. And then there will be cases where sum() and division are too low-level, but mean() is high-level enough to make the critical difference. It's that last one where it's important to be able to override mean() directly. Are there a lot of cases like this? For mean() in particular I doubt it. But then, mean() in particular is irrelevant here, because mean() is already directly overridable, regardless of __array_function__ :-). So really the question is about the larger landscape of numpy APIs: What traps are lurking in the underbrush that we don't know about? And yeah, the intuition behind the "simplified API" approach is that we need to do the work to clear out that underbrush, and the downside is exactly that that will be a lot of work and take time. So... I think this criticism is basically that restated? > 3. We don?t yet have an overloading system for attributes and methods on array objects, e.g., for accessing .dtype and .shape. This should be the subject of a future NEP, but until then we should be reluctant to rely on these properties. This one I don't understand. If you have a duck-array object, and you want to access its .dtype or .shape attributes, you just... write myobj.dtype or myobj.shape? That doesn't need a NEP though so I must be missing something :-). >> But... this is wishful thinking. No matter what the NEP says, I simply >> don't believe that we'll actually go break dask, sparse arrays, >> xarray, and sklearn in a numpy point release. Or any numpy release. >> Nor should we. If we're serious about keeping this experimental ? and >> I think that's an excellent idea for now! ? then IMO we need to do >> something more to avoid getting trapped by backwards compatibility. > > > I agree, but to be clear, development for dask, sparse and xarray (and even > broadly supported machine learning libraries like TensorFlow) still happens > at a much faster pace than is currently the case for "core" projects in the > SciPy stack like NumPy. It would not be a big deal to encounter breaking > changes in a "major" NumPy release (i.e., 1.X -> 1.(X+1)). > > (Side note: sklearn doesn't directly implement any array types, so I don't > think it would make use of __array_function__ in any way, except possibly to > implement overloadable functions.) They don't implement array types, but they do things like use sparse arrays internally, so from the user's point of view you could have some code that only uses numpy and sklearn, and then the new numpy release breaks sklearn (because it broke the sparse package that sklearn was using internally). >> >> My suggestion: at numpy import time, check for an envvar, like say >> NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the >> __array_function__ dispatches turn into no-ops. This lets interested >> downstream libraries and users try this out, but makes sure that we >> won't have a hundred thousand end users depending on it without >> realizing. >> >> >> >> - makes it easy for end-users to check how much overhead this adds (by >> running their code with it enabled vs disabled) >> - if/when we decide to commit to supporting it for real, we just >> remove the envvar. > > > I'm slightly concerned that the cost of reading an environment variable with > os.environ could exaggerate the performance cost of __array_function__. It > takes about 1 microsecond to read an environment variable on my laptop, > which is comparable to the full overhead of __array_function__. That's why I said "at numpy import time" :-). I was imagining we'd check it once at import, and then from then on it'd be stashed in some C global, so after that the overhead would just be a single predictable branch 'if (array_function_is_enabled) { ... }'. > So we may > want to switch to an explicit Python API instead, e.g., > np.enable_experimental_array_function(). If we do this, then libraries that want to use __array_function__ will just call it themselves at import time. The point of the env-var is that our policy is not to break end-users, so if we want an API to be provisional and experimental then it's end-users who need to be aware of that before using it. (This is also an advantage of checking the envvar only at import time: it means libraries can't easily just setenv() to enable the functionality behind users' backs.) > My bigger concern is when/how we decide to graduate __array_function__ from > requiring an explicit opt-in. We don't need to make a final decision now, > but it would be good to clear about what specifically we are waiting for. The motivation for keeping it provisional is that we'll know more after we have some implementation experience, so our future selves will be in a better position to make this decision. If I knew what I was waiting for, I might not need to wait :-). But yeah, to be clear, I'm totally OK with the possibility that we'll do this for a few releases and then look again and be like "eh... now that we have more experience, it looks like the original plan was fine after all, let's remove the envvar and document some kind of accelerated deprecation cycle". >> I don't really understand the 'types' frozenset. The NEP says "it will >> be used by most __array_function__ methods, which otherwise would need >> to extract this information themselves"... but they still need to >> extract the information themselves, because they still have to examine >> each object and figure out what type it is. And, simply creating a >> frozenset costs ~0.2 ?s on my laptop, which is overhead that we can't >> possibly optimize later... > > > The most flexible alternative would be to just say that we provide an > fixed-length iterable, and return a tuple object. (In my microbenchmarks, > it's faster to make a tuple than a list or set.) In an early draft of the > NEP, I proposed exactly this, but the speed difference seemed really > marginal to me. > > I included 'types' in the interface because I really do think it's something > that almost all __array_function__ implementations should use use. It > preserves a nice separation of concerns between dispatching logic and > implementations for a new type. At least as long as __array_function__ is > experimental, I don't think we should be encouraging people to write > functions that could return NotImplemented directly and to rely entirely on > the NumPy interface. > > Many but not all implementations will need to look at argument types. This > is only really essential for cases where mixed operations between NumPy > arrays and another type are allowed. If you only implement the NumPy > interface for MyArray objects, then in the usual Python style you wouldn't > need isinstance checks. > > It's also important from an ecosystem perspective. If we don't make it easy > to get type information, my guess is that many __array_function__ authors > wouldn't bother to return NotImplemented for unexpected types, which means > that __array_function__ will break in weird ways when used with objects from > unrelated libraries. This is much more of a detail as compared to the rest of the discussion, so I don't want to quibble too much about it. (Especially since if we keep things really-provisional, we can change our mind about the argument later :-).) Mostly I'm just confused, because there are lots of __dunder__ functions in Python (and NumPy), and none of them take a special 'types' argument... so what's special about __array_function__ that makes it necessary/worthwhile? Any implementation of, say, concatenate-via-array_function is going to involve iterating through all the arguments and looking at each of them to figure out what kind of object it is and how to handle it, right? That's true whether or not they've done a "pre-check" using the types set, so in theory it's just as easy to return NotImplemented at that point. But I guess your point in the last paragraph is that this means there will be lots of chances to mess up the NotImplemented-returning code in particular, especially since it's less likely to be tested than the happy path, which seems plausible. So basically the point of the types set is to let people factor out that little bit of lots of functions into one common place? I guess some careful devs might be unhappy with paying extra so that other lazier devs can get away with being lazy, but maybe it's a good tradeoff for us (esp. since as numpy devs, we'll be getting the bug reports regardless :-)). If that's the goal, then it does make me wonder if there might be a more direct way to accomplish it -- like, should we let classes define an __array_function_types__ attribute that numpy would check before even trying to dispatch to __array_function__? -n -- Nathaniel J. Smith -- https://vorpus.org From m.h.vankerkwijk at gmail.com Tue Aug 21 04:34:33 2018 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Tue, 21 Aug 2018 10:34:33 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: > >> I don't really understand the 'types' frozenset. The NEP says "it will > >> be used by most __array_function__ methods, which otherwise would need > >> to extract this information themselves"... but they still need to > >> extract the information themselves, because they still have to examine > >> each object and figure out what type it is. And, simply creating a > >> frozenset costs ~0.2 ?s on my laptop, which is overhead that we can't > >> possibly optimize later... > > > > > > The most flexible alternative would be to just say that we provide an > > fixed-length iterable, and return a tuple object. (In my microbenchmarks, > > it's faster to make a tuple than a list or set.) In an early draft of the > > NEP, I proposed exactly this, but the speed difference seemed really > > marginal to me. > > > > I included 'types' in the interface because I really do think it's > something > > that almost all __array_function__ implementations should use use. It > > preserves a nice separation of concerns between dispatching logic and > > implementations for a new type. At least as long as __array_function__ is > > experimental, I don't think we should be encouraging people to write > > functions that could return NotImplemented directly and to rely entirely > on > > the NumPy interface. > > > > Many but not all implementations will need to look at argument types. > This > > is only really essential for cases where mixed operations between NumPy > > arrays and another type are allowed. If you only implement the NumPy > > interface for MyArray objects, then in the usual Python style you > wouldn't > > need isinstance checks. > > > > It's also important from an ecosystem perspective. If we don't make it > easy > > to get type information, my guess is that many __array_function__ authors > > wouldn't bother to return NotImplemented for unexpected types, which > means > > that __array_function__ will break in weird ways when used with objects > from > > unrelated libraries. > > This is much more of a detail as compared to the rest of the > discussion, so I don't want to quibble too much about it. (Especially > since if we keep things really-provisional, we can change our mind > about the argument later :-).) Mostly I'm just confused, because there > are lots of __dunder__ functions in Python (and NumPy), and none of > them take a special 'types' argument... so what's special about > __array_function__ that makes it necessary/worthwhile? > > Any implementation of, say, concatenate-via-array_function is going to > involve iterating through all the arguments and looking at each of > them to figure out what kind of object it is and how to handle it, > right? That's true whether or not they've done a "pre-check" using the > types set, so in theory it's just as easy to return NotImplemented at > that point. But I guess your point in the last paragraph is that this > means there will be lots of chances to mess up the > NotImplemented-returning code in particular, especially since it's > less likely to be tested than the happy path, which seems plausible. > So basically the point of the types set is to let people factor out > that little bit of lots of functions into one common place? I guess > some careful devs might be unhappy with paying extra so that other > lazier devs can get away with being lazy, but maybe it's a good > tradeoff for us (esp. since as numpy devs, we'll be getting the bug > reports regardless :-)). > > If that's the goal, then it does make me wonder if there might be a > more direct way to accomplish it -- like, should we let classes define > an __array_function_types__ attribute that numpy would check before > even trying to dispatch to __array_function__? > > I quite like that idea; I've not been enchanted by the extra `types` either - it seems like `method` in `__array_ufunc__`, it could become quite superfluous. -- Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Tue Aug 21 04:53:54 2018 From: einstein.edison at gmail.com (einstein.edison at gmail.com) Date: Tue, 21 Aug 2018 10:53:54 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: I?m +0 on removing it, so mostly neutral, but slightly in favour. While I see the argument for having it, I also see it as a violation of DRY... The information is already available in relevant arguments. I doubt any people implementing this protocol are going to be lazy enough not to implement a type check. So far, we?ve been good on __array_ufunc__. The part of me that says it?s good to have it is the mostly ?squeeze every bit of performance out? part. Best Regards Hameer Abbasi Sent from my iPhone > On 21. Aug 2018, at 10:34, Marten van Kerkwijk wrote: > > >> >> I don't really understand the 'types' frozenset. The NEP says "it will >> >> be used by most __array_function__ methods, which otherwise would need >> >> to extract this information themselves"... but they still need to >> >> extract the information themselves, because they still have to examine >> >> each object and figure out what type it is. And, simply creating a >> >> frozenset costs ~0.2 ?s on my laptop, which is overhead that we can't >> >> possibly optimize later... >> > >> > >> > The most flexible alternative would be to just say that we provide an >> > fixed-length iterable, and return a tuple object. (In my microbenchmarks, >> > it's faster to make a tuple than a list or set.) In an early draft of the >> > NEP, I proposed exactly this, but the speed difference seemed really >> > marginal to me. >> > >> > I included 'types' in the interface because I really do think it's something >> > that almost all __array_function__ implementations should use use. It >> > preserves a nice separation of concerns between dispatching logic and >> > implementations for a new type. At least as long as __array_function__ is >> > experimental, I don't think we should be encouraging people to write >> > functions that could return NotImplemented directly and to rely entirely on >> > the NumPy interface. >> > >> > Many but not all implementations will need to look at argument types. This >> > is only really essential for cases where mixed operations between NumPy >> > arrays and another type are allowed. If you only implement the NumPy >> > interface for MyArray objects, then in the usual Python style you wouldn't >> > need isinstance checks. >> > >> > It's also important from an ecosystem perspective. If we don't make it easy >> > to get type information, my guess is that many __array_function__ authors >> > wouldn't bother to return NotImplemented for unexpected types, which means >> > that __array_function__ will break in weird ways when used with objects from >> > unrelated libraries. >> >> This is much more of a detail as compared to the rest of the >> discussion, so I don't want to quibble too much about it. (Especially >> since if we keep things really-provisional, we can change our mind >> about the argument later :-).) Mostly I'm just confused, because there >> are lots of __dunder__ functions in Python (and NumPy), and none of >> them take a special 'types' argument... so what's special about >> __array_function__ that makes it necessary/worthwhile? >> >> Any implementation of, say, concatenate-via-array_function is going to >> involve iterating through all the arguments and looking at each of >> them to figure out what kind of object it is and how to handle it, >> right? That's true whether or not they've done a "pre-check" using the >> types set, so in theory it's just as easy to return NotImplemented at >> that point. But I guess your point in the last paragraph is that this >> means there will be lots of chances to mess up the >> NotImplemented-returning code in particular, especially since it's >> less likely to be tested than the happy path, which seems plausible. >> So basically the point of the types set is to let people factor out >> that little bit of lots of functions into one common place? I guess >> some careful devs might be unhappy with paying extra so that other >> lazier devs can get away with being lazy, but maybe it's a good >> tradeoff for us (esp. since as numpy devs, we'll be getting the bug >> reports regardless :-)). >> >> If that's the goal, then it does make me wonder if there might be a >> more direct way to accomplish it -- like, should we let classes define >> an __array_function_types__ attribute that numpy would check before >> even trying to dispatch to __array_function__? >> > I quite like that idea; I've not been enchanted by the extra `types` either - it seems like `method` in `__array_ufunc__`, it could become quite superfluous. > > -- Marten > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Aug 21 12:39:01 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 21 Aug 2018 09:39:01 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: > On Wed, Aug 15, 2018 at 9:45 AM, Stephan Hoyer wrote: > > This avoids a classic subclassing problem that has plagued NumPy for > years, > > where overriding the behavior of method A causes apparently unrelated > method > > B to break, because it relied on method A internally. In NumPy, this > > constrained our implementation of np.median(), because it needed to call > > np.mean() in order for subclasses implementing units to work properly. > > I don't think I follow... if B uses A internally, then overriding A > shouldn't cause B to break, unless the overridden A is buggy. > Let me try another example with arrays with units. My understanding of the contract provided by unit implementations is their behavior should never deviate from NumPy unless an operation raises an error. (This is more explicit for arrays with units because they raise errors for operations with incompatible units, but practically speaking almost all duck arrays will have at least some unsupported operations in NumPy's giant API.) It is quite possible that NumPy functions could be (re)written in a way that is incompatible with some unit implementations but is perfectly valid for "full" duck arrays. We actually see this even within NumPy already -- for example, see this recent PR adding support for the datetime64 dtype to percentile: https://github.com/numpy/numpy/pull/11627 A lesser case of this are changes in NumPy causing performance issues for users of duck arrays, which is basically inevitable if we share implementations. I don't think it's possible to anticipate all of these cases, and I don't want NumPy to be unduly constrained in its internal design. I want our user support answer to be simple: if you care about performance for a particular array operations on your type of arrays, you should implement it yourself (i.e., with __array_function__). This definitely doesn't preclude the careful, systematic overriding approach. But I think we'll almost always want NumPy's external API to be overridable. And when we fix a bug in row_stack, this means we also have to fix it > in all the copy-paste versions, which won't happen, so np.row_stack > has different semantics on different objects, even if they started out > matching. The NDArrayOperatorsMixin reduces the number of duplicate > copies of the same code that need to be updated, but 2 copies is still > a lot worse than 1 copy :-). > I see your point, but in all seriousness if encounter a bug in np.row_stack at this point we might just call it a feature instead. > > 1. The details of how NumPy implements a high-level function in terms of > overloaded functions now becomes an implicit part of NumPy?s public API. > For example, refactoring stack to use np.block() instead of > np.concatenate() internally would now become a breaking change. > > The way I'm imagining this would work is, we guarantee not to take a > function that used to be implemented in terms of overridable > operations, and refactor it so it's implemented in terms of > overridable operations. So long as people have correct implementations > of __array_concatenate__ and __array_block__, they shouldn't care > which one we use. In the interim period where we have > __array_concatenate__ but there's no such thing as __array_block__, > then that refactoring would indeed break things, so we shouldn't do > that :-). But we could fix that by adding __array_block__. > ""we guarantee not to take a function that used to be implemented in terms of overridable operations, and refactor it so it's implemented in terms of overridable operations" Did you miss a "not" in here somewhere, e.g., "refactor it so it's NOT implemented"? If we ever tried to do something like this, I'm pretty sure that it just wouldn't happen -- unless we also change NumPy's extremely conservative approach to breaking third-party code. np.block() is much more complex to implement than np.concatenate(), and users would resist being forced to handle that complexity if they don't need it. (Example: TensorFlow has a concatenate function, but not block.) > > 2. Array libraries may prefer to implement high level functions > differently than NumPy. For example, a library might prefer to implement a > fundamental operations like mean() directly rather than relying on sum() > followed by division. More generally, it?s not clear yet what exactly > qualifies as core functionality, and figuring this out could be a large > project. > > True. And this is a very general problem... for example, the > appropriate way to implement logistic regression is very different > in-core versus out-of-core. You're never going to be able to take code > written for ndarray, drop in an arbitrary new array object, and get > optimal results in all cases -- that's just way too ambitious to hope > for. There will be cases where reducing to operations like sum() and > division is fine. There will be cases where you have a high-level > operation like logistic regression, where reducing to sum() and > division doesn't work, but reducing to slightly-higher-level > operations like np.mean also doesn't work, because you need to redo > the whole high-level operation. And then there will be cases where > sum() and division are too low-level, but mean() is high-level enough > to make the critical difference. It's that last one where it's > important to be able to override mean() directly. Are there a lot of > cases like this? > mean() is not entirely hypothetical. TensorFlow and Eigen actually do implement mean separately from sum, though to be honest it's not entirely clear to me why: https://github.com/tensorflow/tensorflow/blob/1c1dad105a57bb13711492a8ba5ab9d10c91b5df/tensorflow/core/kernels/reduction_ops_mean.cc https://eigen.tuxfamily.org/dox/unsupported/TensorFunctors_8h_source.html I do think this probably will come up with some frequency for other operations, but the bigger answer here really is consistency -- it allows projects and their users to have very clearly defined dependencies on NumPy's API. They don't need to worry about any implementation details from NumPy leaking into their override of a function. > > 3. We don?t yet have an overloading system for attributes and methods on > array objects, e.g., for accessing .dtype and .shape. This should be the > subject of a future NEP, but until then we should be reluctant to rely on > these properties. > > This one I don't understand. If you have a duck-array object, and you > want to access its .dtype or .shape attributes, you just... write > myobj.dtype or myobj.shape? That doesn't need a NEP though so I must > be missing something :-). > We don't have np.asduckarray() yet or whatever we'll end up calling our proposed casting function from NEP 22, so we don't have a fully fleshed out mechanism for NumPy to declare "this object needs to support .shape and .dtype, or I'm going to cast it into something that does". More comments on the environment variable and the interface to come in my next email... Cheers, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Aug 21 16:47:34 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 21 Aug 2018 14:47:34 -0600 Subject: [Numpy-discussion] NumPy 1.15.1 released. Message-ID: Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.15.1. This is a bugfix release for bugs and regressions reported following the 1.15.0 release. Noticeable fixes are - The annoying but harmless RuntimeWarning that "numpy.dtype size changed" has been suppressed. The long standing suppression was lost in the transition to pytest. - The update to Cython 0.28.3 exposed a problematic use of a gcc attribute used to prefer code size over speed in module initialization, possibly resulting in incorrect compiled code. This has been fixed in latest Cython but has been disabled here for safety. - Support for big-endian and ARMv8 architectures has been improved. The Python versions supported by this release are 2.7, 3.4-3.7. The wheels are linked with OpenBLAS v0.3.0, which should fix some of the linalg problems reported for NumPy 1.14. Wheels for this release can be downloaded from PyPI , source archives are available from Github . *Compatibility Note* The NumPy 1.15.x OS X wheels released on PyPI no longer contain 32-bit binaries. That will also be the case in future releases. See #11625 for the related discussion. Those needing 32-bit support on the Mac should look elsewhere or build from source. *Contributors* A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Charles Harris * Chris Billington * Elliott Sales de Andrade + * Eric Wieser * Jeremy Manning + * Matti Picus * Ralf Gommers *Pull requests merged* A total of 24 pull requests were merged for this release. * `#11647 `__: MAINT: Filter Cython warnings in ``__init__.py`` * `#11648 `__: BUG: Fix doc source links to unwrap decorators * `#11657 `__: BUG: Ensure singleton dimensions are not dropped when converting... * `#11661 `__: BUG: Warn on Nan in minimum,maximum for scalars * `#11665 `__: BUG: cython sometimes emits invalid gcc attribute * `#11682 `__: BUG: Fix regression in void_getitem * `#11698 `__: BUG: Make matrix_power again work for object arrays. * `#11700 `__: BUG: Add missing PyErr_NoMemory after failing malloc * `#11719 `__: BUG: Fix undefined functions on big-endian systems. * `#11720 `__: MAINT: Make einsum optimize default to False. * `#11746 `__: BUG: Fix regression in loadtxt for bz2 text files in Python 2. * `#11757 `__: BUG: Revert use of `console_scripts`. * `#11758 `__: BUG: Fix Fortran kind detection for aarch64 & s390x. * `#11759 `__: BUG: Fix printing of longdouble on ppc64le. * `#11760 `__: BUG: Fixes for unicode field names in Python 2 * `#11761 `__: BUG: Increase required cython version on python 3.7 * `#11763 `__: BUG: check return value of _buffer_format_string * `#11775 `__: MAINT: Make assert_array_compare more generic. * `#11776 `__: TST: Fix urlopen stubbing. * `#11777 `__: BUG: Fix regression in intersect1d. * `#11779 `__: BUG: Fix test sensitive to platform byte order. * `#11781 `__: BUG: Avoid signed overflow in histogram * `#11785 `__: BUG: Fix pickle and memoryview for datetime64, timedelta64 scalars * `#11786 `__: BUG: Deprecation triggers segfault Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Aug 21 21:12:25 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 21 Aug 2018 18:12:25 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: > >> My suggestion: at numpy import time, check for an envvar, like say > >> NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the > >> __array_function__ dispatches turn into no-ops. This lets interested > >> downstream libraries and users try this out, but makes sure that we > >> won't have a hundred thousand end users depending on it without > >> realizing. > >> > >> > >> > >> - makes it easy for end-users to check how much overhead this adds (by > >> running their code with it enabled vs disabled) > >> - if/when we decide to commit to supporting it for real, we just > >> remove the envvar. > > > > > > I'm slightly concerned that the cost of reading an environment variable > with > > os.environ could exaggerate the performance cost of __array_function__. > It > > takes about 1 microsecond to read an environment variable on my laptop, > > which is comparable to the full overhead of __array_function__. > > That's why I said "at numpy import time" :-). I was imagining we'd > check it once at import, and then from then on it'd be stashed in some > C global, so after that the overhead would just be a single > predictable branch 'if (array_function_is_enabled) { ... }'. > Indeed, I missed the "at numpy import time" bit :). In that case, I'm concerned that it isn't always possible to set environment variables once before importing NumPy. The environment variable solution works great if users have full control of their own Python binaries, but that isn't always the case today in this era of server-less infrastructure and online notebooks. One example offhand is Google's Colaboratory ( https://research.google.com/colaboratory), a web based Jupyter notebook. NumPy is always loaded when a notebook is opened, as you can check from inspecting sys.modules. Now, I work with the developers of Colaboratory, so we could probably figure out a work-around together, but I'm pretty sure this would also come up in the context of other tools. Another problem is unit testing. Does pytest use a separate Python process for running the tests in each file? I don't know and that feels like an implementation detail that I shouldn't have to know :). Yes, in principle I could use a subprocess in my __array_function__ for unit tests, but that would be really awkward. > So we may > > want to switch to an explicit Python API instead, e.g., > > np.enable_experimental_array_function(). > > If we do this, then libraries that want to use __array_function__ will > just call it themselves at import time. The point of the env-var is > that our policy is not to break end-users, so if we want an API to be > provisional and experimental then it's end-users who need to be aware > of that before using it. (This is also an advantage of checking the > envvar only at import time: it means libraries can't easily just > setenv() to enable the functionality behind users' backs.) > I'm in complete agreement that only authors of end-user applications should invoke this option, but just because something is technically possible doesn't mean that people will actually do it or that we need to support that use case :). numpy.seterr() is a good example. It allows users to globally set how NumPy does error handling, but well written libraries still don't do that. TensorFlow has similar function tf.enable_eager_execution() for enabling "eager mode" that is also worth examining: https://www.tensorflow.org/api_docs/python/tf/enable_eager_execution To solve the testing issue, they wrote decorator for using with tests, run_in_graph_and_eager_modes(): https://www.tensorflow.org/api_docs/python/tf/contrib/eager/run_test_in_graph_and_eager_modes -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Aug 21 21:46:12 2018 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 21 Aug 2018 18:46:12 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Tue, Aug 21, 2018 at 9:39 AM, Stephan Hoyer wrote: > On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: >> >> On Wed, Aug 15, 2018 at 9:45 AM, Stephan Hoyer wrote: >> > This avoids a classic subclassing problem that has plagued NumPy for >> > years, >> > where overriding the behavior of method A causes apparently unrelated >> > method >> > B to break, because it relied on method A internally. In NumPy, this >> > constrained our implementation of np.median(), because it needed to call >> > np.mean() in order for subclasses implementing units to work properly. >> >> I don't think I follow... if B uses A internally, then overriding A >> shouldn't cause B to break, unless the overridden A is buggy. > > > Let me try another example with arrays with units. My understanding of the > contract provided by unit implementations is their behavior should never > deviate from NumPy unless an operation raises an error. (This is more > explicit for arrays with units because they raise errors for operations with > incompatible units, but practically speaking almost all duck arrays will > have at least some unsupported operations in NumPy's giant API.) > > It is quite possible that NumPy functions could be (re)written in a way that > is incompatible with some unit implementations but is perfectly valid for > "full" duck arrays. We actually see this even within NumPy already -- for > example, see this recent PR adding support for the datetime64 dtype to > percentile: > https://github.com/numpy/numpy/pull/11627 I clicked the link, but I don't see anything about units? Of course units are a tricky example to make inferences from, because they aren't a good fit for the duck array concept in general. (In terms of numpy's core semantics, data-with-units is a special dtype, not a special container type.) >From your mention of "full" duck arrays I guess you're thinking of this distinction?: http://www.numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html#principle-1-focus-on-full-duck-arrays-but-dont-rule-out-partial-duck-arrays You're right: if numpy changes the implementation of some high-level function to use protocol-A instead of protocol-B, and there's some partial-duck-array that only implements protocol-B, then it gets broken. Of course, in general __array_function__ has the same problem: if sklearn changes their implementation of some function to call numpy function A instead of numpy function B, and there's a partial-duck-array that only implements numpy function B, then sklearn is broken. I think that as duck arrays roll out, we're just going to have to get used to dealing with breakage like this sometimes. The advantage of __array_function__ is that we get to ignore these issues within numpy itself. The advantage of having focused-protocols is that they make it easier to implement full duck arrays, and they give us a vocabulary for talking about degrees of partiality. For example, with __array_concatenate__, a duck array either supports all the concatenation/stacking operations or none of them ? so sklearn never has to worry that switching between np.row_stack and np.stack will cause issues. > A lesser case of this are changes in NumPy causing performance issues for > users of duck arrays, which is basically inevitable if we share > implementations. NumPy (and Python in general) is never going to make everything 100% optimized all the time. Over and over we choose to accept small inefficiencies in order to improve maintainability. How big are these inefficiencies ? 1% overhead, 10% overhead, 10x overhead...? Do they show up everywhere, or just for a few key functions? What's the maintenance cost of making NumPy's whole API overrideable, in terms of making it harder for us to evolve numpy? What about for users dealing with a proliferation of subtly incompatible implementations? You may be right that the tradeoffs work out so that every API needs to be individually overridable and the benefits are worth it, but we at least need to be asking these questions. >> And when we fix a bug in row_stack, this means we also have to fix it >> in all the copy-paste versions, which won't happen, so np.row_stack >> has different semantics on different objects, even if they started out >> matching. The NDArrayOperatorsMixin reduces the number of duplicate >> copies of the same code that need to be updated, but 2 copies is still >> a lot worse than 1 copy :-). > > > I see your point, but in all seriousness if encounter a bug in np.row_stack > at this point we might just call it a feature instead. Yeah, you're right, row_stack is a bad example :-). But of course the point is that it's literally any bug-fix or added feature in numpy's public API. Here's a better, more concrete example: back in 2015, you added np.stack (PR #5605), which was a great new feature. Its implementation was entirely in terms of np.concatenate and other basic APIs like .ndim, asanyarray, etc. In the smallish-set-of-designed-protocols world, as soon as that's merged into numpy, you're done: it works on sparse arrays, dask arrays, tensorflow tensors, etc. People can use it as soon as they upgrade their numpy. In the __array_function__ world, merging into numpy is only the beginning: now you have to go make new PRs to sparse, dask, tensorflow, etc., get them merged, released, etc. Downstream projects may refuse to use it until it's supported in multiple projects that have their own release cycles, etc. Or another example: at a workshop a few years ago, Matti put up some of the source code to numpypy to demonstrate what it looked like. I immediately spotted a subtle bug, because I happened to know that it was one we'd found and fixed recently. (IIRC it was the thing where arr[...] should return a view of arr, not arr itself.) Of course indexing for duck arrays is its own mess that's somewhat orthogonal to __array_function__, but the basic point is that numpy has a lot of complex error-prone semantics, and we are still actively finding and fixing issues in numpy's own implementations. >> >> > 1. The details of how NumPy implements a high-level function in terms of >> > overloaded functions now becomes an implicit part of NumPy?s public API. For >> > example, refactoring stack to use np.block() instead of np.concatenate() >> > internally would now become a breaking change. >> >> The way I'm imagining this would work is, we guarantee not to take a >> function that used to be implemented in terms of overridable >> operations, and refactor it so it's implemented in terms of >> overridable operations. So long as people have correct implementations >> of __array_concatenate__ and __array_block__, they shouldn't care >> which one we use. In the interim period where we have >> __array_concatenate__ but there's no such thing as __array_block__, >> then that refactoring would indeed break things, so we shouldn't do >> that :-). But we could fix that by adding __array_block__. > > > ""we guarantee not to take a function that used to be implemented in terms > of overridable operations, and refactor it so it's implemented in terms of > overridable operations" > Did you miss a "not" in here somewhere, e.g., "refactor it so it's NOT > implemented"? Yeah, sorry. > If we ever tried to do something like this, I'm pretty sure that it just > wouldn't happen -- unless we also change NumPy's extremely conservative > approach to breaking third-party code. np.block() is much more complex to > implement than np.concatenate(), and users would resist being forced to > handle that complexity if they don't need it. (Example: TensorFlow has a > concatenate function, but not block.) I agree, we probably wouldn't do this particular change. >> > 2. Array libraries may prefer to implement high level functions >> > differently than NumPy. For example, a library might prefer to implement a >> > fundamental operations like mean() directly rather than relying on sum() >> > followed by division. More generally, it?s not clear yet what exactly >> > qualifies as core functionality, and figuring this out could be a large >> > project. >> >> True. And this is a very general problem... for example, the >> appropriate way to implement logistic regression is very different >> in-core versus out-of-core. You're never going to be able to take code >> written for ndarray, drop in an arbitrary new array object, and get >> optimal results in all cases -- that's just way too ambitious to hope >> for. There will be cases where reducing to operations like sum() and >> division is fine. There will be cases where you have a high-level >> operation like logistic regression, where reducing to sum() and >> division doesn't work, but reducing to slightly-higher-level >> operations like np.mean also doesn't work, because you need to redo >> the whole high-level operation. And then there will be cases where >> sum() and division are too low-level, but mean() is high-level enough >> to make the critical difference. It's that last one where it's >> important to be able to override mean() directly. Are there a lot of >> cases like this? > > > mean() is not entirely hypothetical. TensorFlow and Eigen actually do > implement mean separately from sum, though to be honest it's not entirely > clear to me why: > https://github.com/tensorflow/tensorflow/blob/1c1dad105a57bb13711492a8ba5ab9d10c91b5df/tensorflow/core/kernels/reduction_ops_mean.cc > https://eigen.tuxfamily.org/dox/unsupported/TensorFunctors_8h_source.html > > I do think this probably will come up with some frequency for other > operations, but the bigger answer here really is consistency -- it allows > projects and their users to have very clearly defined dependencies on > NumPy's API. They don't need to worry about any implementation details from > NumPy leaking into their override of a function. When you say "consistency" here that means: "they can be sure that when they disagree with the numpy devs about the semantics/implementation of a numpy API, then the numpy API will consistently act the way they want, not the way the numpy devs want". Right? This is a very double-edged sword :-). >> > 3. We don?t yet have an overloading system for attributes and methods on >> > array objects, e.g., for accessing .dtype and .shape. This should be the >> > subject of a future NEP, but until then we should be reluctant to rely on >> > these properties. >> >> This one I don't understand. If you have a duck-array object, and you >> want to access its .dtype or .shape attributes, you just... write >> myobj.dtype or myobj.shape? That doesn't need a NEP though so I must >> be missing something :-). > > > We don't have np.asduckarray() yet or whatever we'll end up calling our > proposed casting function from NEP 22, so we don't have a fully fleshed out > mechanism for NumPy to declare "this object needs to support .shape and > .dtype, or I'm going to cast it into something that does". That's true, but it's just as big a problem for NEP 18, because __array_function__ is never going to do much if you've already coerced the thing to an ndarray. Some kind of asduckarray solution is basically a prerequisite to any other duck array features. -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Tue Aug 21 21:56:06 2018 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 21 Aug 2018 18:56:06 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Tue, Aug 21, 2018 at 6:12 PM, Stephan Hoyer wrote: > On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: >> >> >> My suggestion: at numpy import time, check for an envvar, like say >> >> NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the >> >> __array_function__ dispatches turn into no-ops. This lets interested >> >> downstream libraries and users try this out, but makes sure that we >> >> won't have a hundred thousand end users depending on it without >> >> realizing. >> >> >> >> >> >> >> >> - makes it easy for end-users to check how much overhead this adds (by >> >> running their code with it enabled vs disabled) >> >> - if/when we decide to commit to supporting it for real, we just >> >> remove the envvar. >> > >> > >> > I'm slightly concerned that the cost of reading an environment variable >> > with >> > os.environ could exaggerate the performance cost of __array_function__. >> > It >> > takes about 1 microsecond to read an environment variable on my laptop, >> > which is comparable to the full overhead of __array_function__. >> >> That's why I said "at numpy import time" :-). I was imagining we'd >> check it once at import, and then from then on it'd be stashed in some >> C global, so after that the overhead would just be a single >> predictable branch 'if (array_function_is_enabled) { ... }'. > > > Indeed, I missed the "at numpy import time" bit :). > > In that case, I'm concerned that it isn't always possible to set environment > variables once before importing NumPy. The environment variable solution > works great if users have full control of their own Python binaries, but > that isn't always the case today in this era of server-less infrastructure > and online notebooks. > > One example offhand is Google's Colaboratory > (https://research.google.com/colaboratory), a web based Jupyter notebook. > NumPy is always loaded when a notebook is opened, as you can check from > inspecting sys.modules. Now, I work with the developers of Colaboratory, so > we could probably figure out a work-around together, but I'm pretty sure > this would also come up in the context of other tools. I mean, the idea of the envvar is to be a temporary measure enable devs to experiment with a provisional feature, while being awkward enough that people don't build lots of stuff assuming its there. It doesn't have to 100% supported in every environment. > Another problem is unit testing. Does pytest use a separate Python process > for running the tests in each file? I don't know and that feels like an > implementation detail that I shouldn't have to know :). Yes, in principle I > could use a subprocess in my __array_function__ for unit tests, but that > would be really awkward. Set the envvar before invoking pytest? For numpy itself we'll need to write a few awkward tests involving subprocesses to make sure the envvar parsing is working properly, but I don't think this is a big deal. As long as we only have 1-2 places that __array_function__ dispatch funnels through, we just need to make sure that they work properly with/without the envvar; no need to test every API separately. Or if it is an issue we can have some private API that's only available to the numpy test suite... >> > So we may >> > want to switch to an explicit Python API instead, e.g., >> > np.enable_experimental_array_function(). >> >> If we do this, then libraries that want to use __array_function__ will >> just call it themselves at import time. The point of the env-var is >> that our policy is not to break end-users, so if we want an API to be >> provisional and experimental then it's end-users who need to be aware >> of that before using it. (This is also an advantage of checking the >> envvar only at import time: it means libraries can't easily just >> setenv() to enable the functionality behind users' backs.) > > > I'm in complete agreement that only authors of end-user applications should > invoke this option, but just because something is technically possible > doesn't mean that people will actually do it or that we need to support that > use case :). I didn't say "authors of end-user applications", I said "end-users" :-). That said, I dunno. My intuition is that if we have a function call like this then libraries that define __array_function__ will merrily call it in their package __init__ and it accomplishes nothing, but maybe I'm being too cynical and untrusting. -n -- Nathaniel J. Smith -- https://vorpus.org From einstein.edison at gmail.com Wed Aug 22 07:21:45 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Wed, 22 Aug 2018 13:21:45 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: <195B08D2-F6E8-48F9-9FD6-605EF5CE85DD@gmail.com> Hi Nathaniel and Stephan, Since this conversation is getting a bit lengthy and I see a lot of repeated stuff, I?ll summarise the arguments for everyone?s benefit and then present my own viewpoints: Nathaniel: Undue maintenance burden on NumPy, since semantics have to match exactly Implementations of functions may change, which may break downstream library compatibility There may be time taken in merging this everywhere, so why not take time to define proper protocols? Hide this entire interface behind an environment variable, possibly to be removed later. Stephan: Semantics don?t have to match exactly, that isn?t the intent of most duck-arrays. This won?t happen given NumPy?s conservativeness. The protocols will just be copies of __array_function__, but less capable Provide an interface that only end-users may turn on. My viewpoints: I don?t think any Duck array implementers intend to copy semantics on that level. Dask, which is the most complete one, doesn?t have views, only copies. Many other semantics simply don?t match. The intent is to allow for code that expresses, well, intent (no pun intended) instead of relying heavily on semantics, but that can use arbitrary duck-array implementations instead of just ndarray. Most of the implementations in NumPy are pretty stable, and the only thing that?s likely to happen here is bug fixes. And we are free to fix bugs those; I doubt implementation-specific bugs will be copied. However, these first two points are for/against duck arrays in general, and not specific to this protocol, so IMO this discussion is completely orthogonal to this one. I agree with Stephan here: Defining a minimum API for NumPy that will complete duck arrays will produce a lot of functions in every case that cannot be overridden, as they simply cannot be expressed in terms of the protocols we have added so far. This will lead to more protocols being produced, and so on ad infinitum. We have to consider the burden that such a design would place on the maintainers of NumPy as well? I personally feel that the amount of such protocols we?ll so need are large enough that this line of action is more burdensome, rather than less. I prefer an approach with __array_function__ + mailing list ping before adding a function. May I propose an alternative that was already discussed, and one that I think everyone will be okay with: We put all overridable functions inside a new submodule, numpy.api, that will initially be a shallow-ish copy of the numpy module. I say ish because all modules inside NumPy will need to be shallow-copied as well. If we need to add __array_function__, we can always do that there. Normal users are using ?regular? NumPy unless they know they?re using the API, but it is separately accessible. As for hiding it completely goes: We have to realise, the Python computation landscape is fragmenting. The slower we are, the more fragmented it will become. NumPy already isn?t ?the standard? for machine learning. Regards, Hameer Abbasi > On 22. Aug 2018, at 03:46, Nathaniel Smith wrote: > > On Tue, Aug 21, 2018 at 9:39 AM, Stephan Hoyer > wrote: >> On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: >>> >>> On Wed, Aug 15, 2018 at 9:45 AM, Stephan Hoyer wrote: >>>> This avoids a classic subclassing problem that has plagued NumPy for >>>> years, >>>> where overriding the behavior of method A causes apparently unrelated >>>> method >>>> B to break, because it relied on method A internally. In NumPy, this >>>> constrained our implementation of np.median(), because it needed to call >>>> np.mean() in order for subclasses implementing units to work properly. >>> >>> I don't think I follow... if B uses A internally, then overriding A >>> shouldn't cause B to break, unless the overridden A is buggy. >> >> >> Let me try another example with arrays with units. My understanding of the >> contract provided by unit implementations is their behavior should never >> deviate from NumPy unless an operation raises an error. (This is more >> explicit for arrays with units because they raise errors for operations with >> incompatible units, but practically speaking almost all duck arrays will >> have at least some unsupported operations in NumPy's giant API.) >> >> It is quite possible that NumPy functions could be (re)written in a way that >> is incompatible with some unit implementations but is perfectly valid for >> "full" duck arrays. We actually see this even within NumPy already -- for >> example, see this recent PR adding support for the datetime64 dtype to >> percentile: >> https://github.com/numpy/numpy/pull/11627 > > I clicked the link, but I don't see anything about units? > > Of course units are a tricky example to make inferences from, because > they aren't a good fit for the duck array concept in general. (In > terms of numpy's core semantics, data-with-units is a special dtype, > not a special container type.) > > From your mention of "full" duck arrays I guess you're thinking of > this distinction?: > http://www.numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html#principle-1-focus-on-full-duck-arrays-but-dont-rule-out-partial-duck-arrays > > You're right: if numpy changes the implementation of some high-level > function to use protocol-A instead of protocol-B, and there's some > partial-duck-array that only implements protocol-B, then it gets > broken. Of course, in general __array_function__ has the same problem: > if sklearn changes their implementation of some function to call numpy > function A instead of numpy function B, and there's a > partial-duck-array that only implements numpy function B, then sklearn > is broken. I think that as duck arrays roll out, we're just going to > have to get used to dealing with breakage like this sometimes. > > The advantage of __array_function__ is that we get to ignore these > issues within numpy itself. The advantage of having focused-protocols > is that they make it easier to implement full duck arrays, and they > give us a vocabulary for talking about degrees of partiality. For > example, with __array_concatenate__, a duck array either supports all > the concatenation/stacking operations or none of them ? so sklearn > never has to worry that switching between np.row_stack and np.stack > will cause issues. > >> A lesser case of this are changes in NumPy causing performance issues for >> users of duck arrays, which is basically inevitable if we share >> implementations. > > NumPy (and Python in general) is never going to make everything 100% > optimized all the time. Over and over we choose to accept small > inefficiencies in order to improve maintainability. How big are these > inefficiencies ? 1% overhead, 10% overhead, 10x overhead...? Do they > show up everywhere, or just for a few key functions? What's the > maintenance cost of making NumPy's whole API overrideable, in terms of > making it harder for us to evolve numpy? What about for users dealing > with a proliferation of subtly incompatible implementations? > > You may be right that the tradeoffs work out so that every API needs > to be individually overridable and the benefits are worth it, but we > at least need to be asking these questions. > >>> And when we fix a bug in row_stack, this means we also have to fix it >>> in all the copy-paste versions, which won't happen, so np.row_stack >>> has different semantics on different objects, even if they started out >>> matching. The NDArrayOperatorsMixin reduces the number of duplicate >>> copies of the same code that need to be updated, but 2 copies is still >>> a lot worse than 1 copy :-). >> >> >> I see your point, but in all seriousness if encounter a bug in np.row_stack >> at this point we might just call it a feature instead. > > Yeah, you're right, row_stack is a bad example :-). But of course the > point is that it's literally any bug-fix or added feature in numpy's > public API. > > Here's a better, more concrete example: back in 2015, you added > np.stack (PR #5605), which was a great new feature. Its implementation > was entirely in terms of np.concatenate and other basic APIs like > .ndim, asanyarray, etc. > > In the smallish-set-of-designed-protocols world, as soon as that's > merged into numpy, you're done: it works on sparse arrays, dask > arrays, tensorflow tensors, etc. People can use it as soon as they > upgrade their numpy. > > In the __array_function__ world, merging into numpy is only the > beginning: now you have to go make new PRs to sparse, dask, > tensorflow, etc., get them merged, released, etc. Downstream projects > may refuse to use it until it's supported in multiple projects that > have their own release cycles, etc. > > Or another example: at a workshop a few years ago, Matti put up some > of the source code to numpypy to demonstrate what it looked like. I > immediately spotted a subtle bug, because I happened to know that it > was one we'd found and fixed recently. (IIRC it was the thing where > arr[...] should return a view of arr, not arr itself.) Of course > indexing for duck arrays is its own mess that's somewhat orthogonal to > __array_function__, but the basic point is that numpy has a lot of > complex error-prone semantics, and we are still actively finding and > fixing issues in numpy's own implementations. > >>> >>>> 1. The details of how NumPy implements a high-level function in terms of >>>> overloaded functions now becomes an implicit part of NumPy?s public API. For >>>> example, refactoring stack to use np.block() instead of np.concatenate() >>>> internally would now become a breaking change. >>> >>> The way I'm imagining this would work is, we guarantee not to take a >>> function that used to be implemented in terms of overridable >>> operations, and refactor it so it's implemented in terms of >>> overridable operations. So long as people have correct implementations >>> of __array_concatenate__ and __array_block__, they shouldn't care >>> which one we use. In the interim period where we have >>> __array_concatenate__ but there's no such thing as __array_block__, >>> then that refactoring would indeed break things, so we shouldn't do >>> that :-). But we could fix that by adding __array_block__. >> >> >> ""we guarantee not to take a function that used to be implemented in terms >> of overridable operations, and refactor it so it's implemented in terms of >> overridable operations" >> Did you miss a "not" in here somewhere, e.g., "refactor it so it's NOT >> implemented"? > > Yeah, sorry. > >> If we ever tried to do something like this, I'm pretty sure that it just >> wouldn't happen -- unless we also change NumPy's extremely conservative >> approach to breaking third-party code. np.block() is much more complex to >> implement than np.concatenate(), and users would resist being forced to >> handle that complexity if they don't need it. (Example: TensorFlow has a >> concatenate function, but not block.) > > I agree, we probably wouldn't do this particular change. > >>>> 2. Array libraries may prefer to implement high level functions >>>> differently than NumPy. For example, a library might prefer to implement a >>>> fundamental operations like mean() directly rather than relying on sum() >>>> followed by division. More generally, it?s not clear yet what exactly >>>> qualifies as core functionality, and figuring this out could be a large >>>> project. >>> >>> True. And this is a very general problem... for example, the >>> appropriate way to implement logistic regression is very different >>> in-core versus out-of-core. You're never going to be able to take code >>> written for ndarray, drop in an arbitrary new array object, and get >>> optimal results in all cases -- that's just way too ambitious to hope >>> for. There will be cases where reducing to operations like sum() and >>> division is fine. There will be cases where you have a high-level >>> operation like logistic regression, where reducing to sum() and >>> division doesn't work, but reducing to slightly-higher-level >>> operations like np.mean also doesn't work, because you need to redo >>> the whole high-level operation. And then there will be cases where >>> sum() and division are too low-level, but mean() is high-level enough >>> to make the critical difference. It's that last one where it's >>> important to be able to override mean() directly. Are there a lot of >>> cases like this? >> >> >> mean() is not entirely hypothetical. TensorFlow and Eigen actually do >> implement mean separately from sum, though to be honest it's not entirely >> clear to me why: >> https://github.com/tensorflow/tensorflow/blob/1c1dad105a57bb13711492a8ba5ab9d10c91b5df/tensorflow/core/kernels/reduction_ops_mean.cc >> https://eigen.tuxfamily.org/dox/unsupported/TensorFunctors_8h_source.html >> >> I do think this probably will come up with some frequency for other >> operations, but the bigger answer here really is consistency -- it allows >> projects and their users to have very clearly defined dependencies on >> NumPy's API. They don't need to worry about any implementation details from >> NumPy leaking into their override of a function. > > When you say "consistency" here that means: "they can be sure that > when they disagree with the numpy devs about the > semantics/implementation of a numpy API, then the numpy API will > consistently act the way they want, not the way the numpy devs want". > Right? > > This is a very double-edged sword :-). > >>>> 3. We don?t yet have an overloading system for attributes and methods on >>>> array objects, e.g., for accessing .dtype and .shape. This should be the >>>> subject of a future NEP, but until then we should be reluctant to rely on >>>> these properties. >>> >>> This one I don't understand. If you have a duck-array object, and you >>> want to access its .dtype or .shape attributes, you just... write >>> myobj.dtype or myobj.shape? That doesn't need a NEP though so I must >>> be missing something :-). >> >> >> We don't have np.asduckarray() yet or whatever we'll end up calling our >> proposed casting function from NEP 22, so we don't have a fully fleshed out >> mechanism for NumPy to declare "this object needs to support .shape and >> .dtype, or I'm going to cast it into something that does". > > That's true, but it's just as big a problem for NEP 18, because > __array_function__ is never going to do much if you've already coerced > the thing to an ndarray. Some kind of asduckarray solution is > basically a prerequisite to any other duck array features. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Aug 22 11:15:38 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 22 Aug 2018 08:15:38 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: <195B08D2-F6E8-48F9-9FD6-605EF5CE85DD@gmail.com> References: <195B08D2-F6E8-48F9-9FD6-605EF5CE85DD@gmail.com> Message-ID: On Wed, Aug 22, 2018 at 4:22 AM Hameer Abbasi wrote: > May I propose an alternative that was already discussed, and one that I > think everyone will be okay with: > That's a dangerous assumption on this list:) We put all overridable functions inside a new submodule, numpy.api, that > will initially be a shallow-ish copy of the numpy module. > This is not desirable. There are projects (e.g. statsmodels) that have added a .api submodule before. It's generally considered not a good idea, it's not very Pythonic. Everything one can import that doesn't have an underscore is normally part of the API of a package. In this particular case, I definitely prefer an envvar and relying on what is documented as part of __array_function__ rather than a new namespace. Cheers, Ralf I say ish because all modules inside NumPy will need to be shallow-copied > as well. If we need to add __array_function__, we can always do that there. > Normal users are using ?regular? NumPy unless they know they?re using the > API, but it is separately accessible. As for hiding it completely goes: We > have to realise, the Python computation landscape is fragmenting. The > slower we are, the more fragmented it will become. NumPy already isn?t ?the > standard? for machine learning. > > Regards, > Hameer Abbasi > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Wed Aug 22 12:26:58 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Wed, 22 Aug 2018 09:26:58 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Tue, Aug 21, 2018 at 6:47 PM Nathaniel Smith wrote: > On Tue, Aug 21, 2018 at 9:39 AM, Stephan Hoyer wrote: > > It is quite possible that NumPy functions could be (re)written in a way > that > > is incompatible with some unit implementations but is perfectly valid for > > "full" duck arrays. We actually see this even within NumPy already -- for > > example, see this recent PR adding support for the datetime64 dtype to > > percentile: > > https://github.com/numpy/numpy/pull/11627 > > I clicked the link, but I don't see anything about units? > To clarify: np.datetime64 arrays can be considered a variant of NumPy arrays that support units. Namely, they use a dtype for representing time units. I expect that the issues we've encountered with datetime64 will be indicative of some of the sort of issues that authors of unit-aware arrays will encounter. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Thu Aug 23 11:35:04 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Thu, 23 Aug 2018 08:35:04 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Tue, Aug 21, 2018 at 6:57 PM Nathaniel Smith wrote: > I mean, the idea of the envvar is to be a temporary measure enable > devs to experiment with a provisional feature, while being awkward > enough that people don't build lots of stuff assuming its there. It > doesn't have to 100% supported in every environment. > My understanding of the idea of the envvar is to obtain informed consent from NumPy users, e.g., "I understand that this is a unsupported experimental feature that may be removed in the future without warning." It's pretty important for me personally that's it's possible to use this in a flexible set of environments, and in particular to have something that works in my preferred notebook environment. How else are we going to test this? Every limitation that we put into the experimental version of this feature decreases the likelihood that it gets used enough to know if it's even a viable solution. If it's too awkward, nobody's even going to bother testing it, and this whole effort will fall flat on its face. > > I'm in complete agreement that only authors of end-user applications > should > > invoke this option, but just because something is technically possible > > doesn't mean that people will actually do it or that we need to support > that > > use case :). > > I didn't say "authors of end-user applications", I said "end-users" :-). > These are mostly the same for NumPy, but I do disagree with you here. Ultimately we have to trust application developers to make the right choices for their tools. If they are willing to accept that maintenance burden of either (1) potentially being stuck on NumPy 1.16 forever or (2) needing to rewrite their code, that's their tradeoff to make. It's a little preposterous to force this decision onto end-users, who may not even know a tool is written in NumPy. > That said, I dunno. My intuition is that if we have a function call > like this then libraries that define __array_function__ will merrily > call it in their package __init__ and it accomplishes nothing, but > maybe I'm being too cynical and untrusting. People can do lots of dumb things in Python (e.g., monkeypatching) -- the language doesn't stop them. Fortunately this mostly isn't a problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Thu Aug 23 12:02:26 2018 From: einstein.edison at gmail.com (einstein.edison at gmail.com) Date: Thu, 23 Aug 2018 18:02:26 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Hi everyone, > On 23. Aug 2018, at 17:35, Stephan Hoyer wrote: > >> On Tue, Aug 21, 2018 at 6:57 PM Nathaniel Smith wrote: >> I mean, the idea of the envvar is to be a temporary measure enable >> devs to experiment with a provisional feature, while being awkward >> enough that people don't build lots of stuff assuming its there. It >> doesn't have to 100% supported in every environment. > > My understanding of the idea of the envvar is to obtain informed consent from NumPy users, e.g., "I understand that this is a unsupported experimental feature that may be removed in the future without warning." > > It's pretty important for me personally that's it's possible to use this in a flexible set of environments, and in particular to have something that works in my preferred notebook environment. How else are we going to test this? > > Every limitation that we put into the experimental version of this feature decreases the likelihood that it gets used enough to know if it's even a viable solution. If it's too awkward, nobody's even going to bother testing it, and this whole effort will fall flat on its face. > >> > I'm in complete agreement that only authors of end-user applications should >> > invoke this option, but just because something is technically possible >> > doesn't mean that people will actually do it or that we need to support that >> > use case :). >> >> I didn't say "authors of end-user applications", I said "end-users" :-). > > These are mostly the same for NumPy, but I do disagree with you here. Ultimately we have to trust application developers to make the right choices for their tools. If they are willing to accept that maintenance burden of either (1) potentially being stuck on NumPy 1.16 forever or (2) needing to rewrite their code, that's their tradeoff to make. It's a little preposterous to force this decision onto end-users, who may not even know a tool is written in NumPy. > >> That said, I dunno. My intuition is that if we have a function call >> like this then libraries that define __array_function__ will merrily >> call it in their package __init__ and it accomplishes nothing, but >> maybe I'm being too cynical and untrusting. > > People can do lots of dumb things in Python (e.g., monkeypatching) -- the language doesn't stop them. Fortunately this mostly isn't a problem. I might add that most duck array authors are highly unlikely to be newcomers to the Python space. We should just put a big warning there while enabling and that?ll be enough to scare away most devs from doing it by default. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Best Regards, Hameer Abbasi Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Thu Aug 23 12:37:03 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Thu, 23 Aug 2018 09:37:03 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: RE: the types argument On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: > This is much more of a detail as compared to the rest of the > discussion, so I don't want to quibble too much about it. (Especially > since if we keep things really-provisional, we can change our mind > about the argument later :-).) Mostly I'm just confused, because there > are lots of __dunder__ functions in Python (and NumPy), and none of > them take a special 'types' argument... so what's special about > __array_function__ that makes it necessary/worthwhile? > What's special about __array_function__ is that it's a hook that lets you override an entire API through a single interface. Unlike protocols like __add__, implementers of __array_function__ don't know exactly which arguments could have implemented the operation. > Any implementation of, say, concatenate-via-array_function is going to > involve iterating through all the arguments and looking at each of > them to figure out what kind of object it is and how to handle it, > right? That's true whether or not they've done a "pre-check" using the > types set, so in theory it's just as easy to return NotImplemented at > that point. But I guess your point in the last paragraph is that this > means there will be lots of chances to mess up the > NotImplemented-returning code in particular, especially since it's > less likely to be tested than the happy path, which seems plausible. > So basically the point of the types set is to let people factor out > that little bit of lots of functions into one common place? It's also a pragmatic choice: libraries like dask.array and autograd.numpy have already implemented NumPy's API without overrides. These projects follow the current numpy convention: non-native array objects are coerced into native arrays (i.e., dask or autograd arrays). They don't do any type checking. I doubt there would be much appetite for writing alternative versions of these APIs that return NotImplemented instead -- especially while this feature remains experimental. > I guess some careful devs might be unhappy with paying extra so that other > lazier devs can get away with being lazy, but maybe it's a good > tradeoff for us (esp. since as numpy devs, we'll be getting the bug > reports regardless :-)). > The only extra amount we pay extra is the price of converting these types into a Python data structure and passing them into the __array_function__ method call. We already had to collect them for __array_function__ itself to identify unique types to call -- so this is a pretty minimal extra cost. > If that's the goal, then it does make me wonder if there might be a > more direct way to accomplish it -- like, should we let classes define > an __array_function_types__ attribute that numpy would check before > even trying to dispatch to __array_function__? > This could potentially work, but now the __array_function__ protocol itself becomes more complex and out of sync with __array_ufunc__. It's a much smaller amount of additional complexity to add an additional passed argument. -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Thu Aug 23 16:05:03 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Thu, 23 Aug 2018 22:05:03 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: > On 23. Aug 2018, at 18:37, Stephan Hoyer wrote: > > RE: the types argument > > On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith > wrote: > This is much more of a detail as compared to the rest of the > discussion, so I don't want to quibble too much about it. (Especially > since if we keep things really-provisional, we can change our mind > about the argument later :-).) Mostly I'm just confused, because there > are lots of __dunder__ functions in Python (and NumPy), and none of > them take a special 'types' argument... so what's special about > __array_function__ that makes it necessary/worthwhile? > > What's special about __array_function__ is that it's a hook that lets you override an entire API through a single interface. Unlike protocols like __add__, implementers of __array_function__ don't know exactly which arguments could have implemented the operation. > > Any implementation of, say, concatenate-via-array_function is going to > involve iterating through all the arguments and looking at each of > them to figure out what kind of object it is and how to handle it, > right? That's true whether or not they've done a "pre-check" using the > types set, so in theory it's just as easy to return NotImplemented at > that point. But I guess your point in the last paragraph is that this > means there will be lots of chances to mess up the > NotImplemented-returning code in particular, especially since it's > less likely to be tested than the happy path, which seems plausible. > So basically the point of the types set is to let people factor out > that little bit of lots of functions into one common place? > > It's also a pragmatic choice: libraries like dask.array and autograd.numpy have already implemented NumPy's API without overrides. These projects follow the current numpy convention: non-native array objects are coerced into native arrays (i.e., dask or autograd arrays). They don't do any type checking. > > I doubt there would be much appetite for writing alternative versions of these APIs that return NotImplemented instead -- especially while this feature remains experimental. > > I guess some careful devs might be unhappy with paying extra so that other > lazier devs can get away with being lazy, but maybe it's a good > tradeoff for us (esp. since as numpy devs, we'll be getting the bug > reports regardless :-)). > > The only extra amount we pay extra is the price of converting these types into a Python data structure and passing them into the __array_function__ method call. We already had to collect them for __array_function__ itself to identify unique types to call -- so this is a pretty minimal extra cost. > > If that's the goal, then it does make me wonder if there might be a > more direct way to accomplish it -- like, should we let classes define > an __array_function_types__ attribute that numpy would check before > even trying to dispatch to __array_function__? > > This could potentially work, but now the __array_function__ protocol itself becomes more complex and out of sync with __array_ufunc__. It's a much smaller amount of additional complexity to add an additional passed argument. I might add that if it?s a mandatory part of the protocol, then not all things will work. For example, if XArray and Dask want to support sparse arrays, they?ll need to add an explicit dependency. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffrey.yancey at gmail.com Thu Aug 23 16:25:19 2018 From: jeffrey.yancey at gmail.com (Jeffrey Yancey) Date: Thu, 23 Aug 2018 13:25:19 -0700 Subject: [Numpy-discussion] polynomial TODOs and enhancements Message-ID: Hi everyone, I spent some time going through the polynomial directory and found several locations where there were comments left suggestion potential improvements down the road, TODOs, and a line of commented out code. I wanted to share these here to gather a consensus on which, if any, of these tasks should be done. If some of these should not be done, do we want to remove the comments implying changes should be made, or kick the can down the road? In `numpy/polynomial/_polybase.py, there is the following class method: @staticmethod def _repr_latex_scalar(x): # TODO: we're stuck with disabling math formatting until we handle # exponents in this function return r'\text{{{}}}'.format(x) Further down, in `_repr_latex_` there is this comprehension, where the leading comment is no longer true, and the old conditional has been commented out: # filter out uninteresting coefficients filtered_coeffs = [ (i, c) for i, c in enumerate(self.coef) # if not (c == 0) # handle NaN ] If this behavior is correct, shouldn?t we remove the leading comment and the commented out line, or at least leave an explanation as to why it is commented out and we wish it to remain. Further down there is also a comment in the `__div__` function implying there may have been the intention of future modification: def __div__(self, other): # set to __floordiv__, /, for now. return self.__floordiv__(other) There is also a comment, ?made more efficient by using powers of two in the usual way? in the various `pow` functions specified below: chebyshev.py, line 862 hermite_e.py, line 679 hermite.py, line 630 laguerre.py, line 627 legendre.py, line 661 polynomial.py, line 471 I am happy to make any or all of the enhancements, modify the documentation, or review PRs. I appreciate advice or direction on any of these issues. Jeffrey Yancey -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.bell.ia at gmail.com Thu Aug 23 17:38:02 2018 From: andrew.bell.ia at gmail.com (Andrew Bell) Date: Thu, 23 Aug 2018 17:38:02 -0400 Subject: [Numpy-discussion] Copies with MaskedArray? Message-ID: Hi, I'm creating a structured numpy array in C, wrapping it with a MaskedArray and passing this to a Python script: ... PyObject *arr = PyArray_NewFromDescr(&PyArray_Type, descr, 1, &longCount, &pointSize, data, NPY_ARRAY_WRITEABLE, nullptr); // Now wrap the created array in a MaskedArray. PyObject *module = PyImport_ImportModule("numpy.ma"); PyObject *dict = PyModule_GetDict(module); PyObject *keys = PyDict_Keys(dict); PyObject *o = PyDict_GetItemString(dict, "MaskedArray"); PyObject *args = PyTuple_New(1); PyTuple_SetItem(args, 0, arr); m_directArray = PyObject_CallObject(o, args); This works fine and the data is seen in python as it was in C. What I'd like to happen is to have changes made to the array in Python be reflected in the memory that was passed in. This seems to work in some cases. Given the dtype: [('X', ' From njs at pobox.com Fri Aug 24 03:38:41 2018 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 24 Aug 2018 00:38:41 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Message-ID: On Thu, Aug 23, 2018 at 9:02 AM, wrote: > I might add that most duck array authors are highly unlikely to be newcomers > to the Python space. We should just put a big warning there while enabling > and that?ll be enough to scare away most devs from doing it by default. That's a reasonable idea... a Big Obnoxious Warning(tm) when it's enabled, or on first use, would achieve a lot of the same purpose. E.g. if this_is_the_first_array_function_usage(): sys.stderr.write( "WARNING: this program uses NumPy's experimental '__array_function__' feature.\n" "It may change or be removed without warning, which might break this program.\n" "For details see http://www.numpy.org/neps/nep-0018-array-function-protocol.html\n" ) -n -- Nathaniel J. Smith -- https://vorpus.org From einstein.edison at gmail.com Fri Aug 24 04:34:07 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Fri, 24 Aug 2018 10:34:07 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Message-ID: Hi everyone, On Fri, Aug 24, 2018 at 9:38 AM Nathaniel Smith wrote: > On Thu, Aug 23, 2018 at 9:02 AM, wrote: > > I might add that most duck array authors are highly unlikely to be > newcomers > > to the Python space. We should just put a big warning there while > enabling > > and that?ll be enough to scare away most devs from doing it by default. > > That's a reasonable idea... a Big Obnoxious Warning(tm) when it's > enabled, or on first use, would achieve a lot of the same purpose. > E.g. > > if this_is_the_first_array_function_usage(): > sys.stderr.write( > "WARNING: this program uses NumPy's experimental > '__array_function__' feature.\n" > "It may change or be removed without warning, which might > break this program.\n" > "For details see > http://www.numpy.org/neps/nep-0018-array-function-protocol.html\n" > ) > > -n > > I was thinking of a FutureWarning... That's essentially what it's for. Writing to stderr looks un-pythonic to me. > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Best Regards, Hameer Abbasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Fri Aug 24 11:54:54 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 24 Aug 2018 08:54:54 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Thu, Aug 23, 2018 at 1:06 PM Hameer Abbasi wrote: > I might add that if it?s a mandatory part of the protocol, then not all > things will work. For example, if XArray and Dask want to support sparse > arrays, they?ll need to add an explicit dependency. > I don't follow -- can you please elaborate? If you don't want to do anything with the 'types' argument, you can simply ignore it. The problem of identifying whether arguments have valid types or not remains unchanged from the situation with __add__ or __array_ufunc__. 'types' just gives you another optional tool to help solve it. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Fri Aug 24 12:00:09 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Fri, 24 Aug 2018 18:00:09 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Fri, Aug 24, 2018 at 5:55 PM Stephan Hoyer wrote: > On Thu, Aug 23, 2018 at 1:06 PM Hameer Abbasi > wrote: > >> I might add that if it?s a mandatory part of the protocol, then not all >> things will work. For example, if XArray and Dask want to support sparse >> arrays, they?ll need to add an explicit dependency. >> > > I don't follow -- can you please elaborate? > If we make specifying __array_function_types__ a mandatory part -- And such that it is a whitelist, the XArray or Dask would need to import sparse in order to specify that they accept mixing sparse arrays with native arrays (i.e. for adding sparse.SparseArray to __array_function_types__). Which is basically what I mean. It might be a 'soft' dependency, but there will be a dependency nonetheless. > > If you don't want to do anything with the 'types' argument, you can simply > ignore it. > > The problem of identifying whether arguments have valid types or not > remains unchanged from the situation with __add__ or __array_ufunc__. > 'types' just gives you another optional tool to help solve it. > >> _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Aug 24 16:24:53 2018 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 24 Aug 2018 13:24:53 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: Message-ID: On Fri, Aug 24, 2018, 09:07 Hameer Abbasi wrote: > > > On Fri, Aug 24, 2018 at 5:55 PM Stephan Hoyer wrote: > >> On Thu, Aug 23, 2018 at 1:06 PM Hameer Abbasi >> wrote: >> >>> I might add that if it?s a mandatory part of the protocol, then not all >>> things will work. For example, if XArray and Dask want to support sparse >>> arrays, they?ll need to add an explicit dependency. >>> >> >> I don't follow -- can you please elaborate? >> > > If we make specifying __array_function_types__ a mandatory part -- And > such that it is a whitelist, the XArray or Dask would need to import sparse > in order to specify that they accept mixing sparse arrays with native > arrays (i.e. for adding sparse.SparseArray to __array_function_types__). > Which is basically what I mean. It might be a 'soft' dependency, but there > will be a dependency nonetheless. > Oh yeah, if we did this then we definitely wouldn't want to make it mandatory. Some `__array_function__` implementations might want to do checking another way, or support different types in different overloaded functions, or be able to handle arbitrary types. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Fri Aug 24 16:46:33 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 24 Aug 2018 13:46:33 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Message-ID: On Fri, Aug 24, 2018 at 1:36 AM Hameer Abbasi wrote: > > On Fri, Aug 24, 2018 at 9:38 AM Nathaniel Smith wrote: > >> On Thu, Aug 23, 2018 at 9:02 AM, wrote: >> > I might add that most duck array authors are highly unlikely to be >> newcomers >> > to the Python space. We should just put a big warning there while >> enabling >> > and that?ll be enough to scare away most devs from doing it by default. >> >> That's a reasonable idea... a Big Obnoxious Warning(tm) when it's >> enabled, or on first use, would achieve a lot of the same purpose. >> E.g. >> >> if this_is_the_first_array_function_usage(): >> sys.stderr.write( >> "WARNING: this program uses NumPy's experimental >> '__array_function__' feature.\n" >> "It may change or be removed without warning, which might >> break this program.\n" >> "For details see >> http://www.numpy.org/neps/nep-0018-array-function-protocol.html\n" >> ) >> >> -n >> >> > I was thinking of a FutureWarning... That's essentially what it's for. > Writing to stderr looks un-pythonic to me. > Issuing a FutureWarning seems roughly appropriate here. The Python 3.7 docs write: "Base category for warnings about deprecated features when those warnings are intended for end users of applications that are written in Python." Writing to sys.stderr directly is generally considered poor practice for a Python libraries. In my experience FutureWarning does a good job of satisfying the goals of being a "Big Obnoxious Warning" while still being silence-able and testable with standard tools. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Aug 24 18:13:56 2018 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 24 Aug 2018 15:13:56 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Message-ID: On Fri, Aug 24, 2018 at 1:46 PM, Stephan Hoyer wrote: > On Fri, Aug 24, 2018 at 1:36 AM Hameer Abbasi > wrote: >> >> >> On Fri, Aug 24, 2018 at 9:38 AM Nathaniel Smith wrote: >>> >>> On Thu, Aug 23, 2018 at 9:02 AM, wrote: >>> > I might add that most duck array authors are highly unlikely to be >>> > newcomers >>> > to the Python space. We should just put a big warning there while >>> > enabling >>> > and that?ll be enough to scare away most devs from doing it by default. >>> >>> That's a reasonable idea... a Big Obnoxious Warning(tm) when it's >>> enabled, or on first use, would achieve a lot of the same purpose. >>> E.g. >>> >>> if this_is_the_first_array_function_usage(): >>> sys.stderr.write( >>> "WARNING: this program uses NumPy's experimental >>> '__array_function__' feature.\n" >>> "It may change or be removed without warning, which might >>> break this program.\n" >>> "For details see >>> http://www.numpy.org/neps/nep-0018-array-function-protocol.html\n" >>> ) >>> >>> -n >>> >> >> I was thinking of a FutureWarning... That's essentially what it's for. >> Writing to stderr looks un-pythonic to me. > > > Issuing a FutureWarning seems roughly appropriate here. The Python 3.7 docs > write: > "Base category for warnings about deprecated features when those warnings > are intended for end users of applications that are written in Python." > > Writing to sys.stderr directly is generally considered poor practice for a > Python libraries. > > In my experience FutureWarning does a good job of satisfying the goals of > being a "Big Obnoxious Warning" while still being silence-able and testable > with standard tools. Yeah, the reason warnings are normally recommended is because normally, you want to make it easy to silence. But this is the rare case where I didn't want to make it easy to silence, so I didn't suggest using a warning :-). Calling warnings.warn (or the C equivalent) is also very expensive, even if the warning ultimately isn't displayed. I guess we could do our own tracking of whether we've displayed the warning yet, and only even attempt to issue it once, but that partially defeats the purpose of using warnings in the first place. -n -- Nathaniel J. Smith -- https://vorpus.org From einstein.edison at gmail.com Fri Aug 24 18:54:55 2018 From: einstein.edison at gmail.com (einstein.edison at gmail.com) Date: Sat, 25 Aug 2018 00:54:55 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Message-ID: <3D34CF11-E3C6-4198-BAFE-7A3E8B673478@gmail.com> > I?m On 25. Aug 2018, at 00:13, Nathaniel Smith wrote: > >> On Fri, Aug 24, 2018 at 1:46 PM, Stephan Hoyer wrote: >> On Fri, Aug 24, 2018 at 1:36 AM Hameer Abbasi >> wrote: >>> >>> >>>> On Fri, Aug 24, 2018 at 9:38 AM Nathaniel Smith wrote: >>>> >>>>> On Thu, Aug 23, 2018 at 9:02 AM, wrote: >>>>> I might add that most duck array authors are highly unlikely to be >>>>> newcomers >>>>> to the Python space. We should just put a big warning there while >>>>> enabling >>>>> and that?ll be enough to scare away most devs from doing it by default. >>>> >>>> That's a reasonable idea... a Big Obnoxious Warning(tm) when it's >>>> enabled, or on first use, would achieve a lot of the same purpose. >>>> E.g. >>>> >>>> if this_is_the_first_array_function_usage(): >>>> sys.stderr.write( >>>> "WARNING: this program uses NumPy's experimental >>>> '__array_function__' feature.\n" >>>> "It may change or be removed without warning, which might >>>> break this program.\n" >>>> "For details see >>>> http://www.numpy.org/neps/nep-0018-array-function-protocol.html\n" >>>> ) >>>> >>>> -n >>>> >>> >>> I was thinking of a FutureWarning... That's essentially what it's for. >>> Writing to stderr looks un-pythonic to me. >> >> >> Issuing a FutureWarning seems roughly appropriate here. The Python 3.7 docs >> write: >> "Base category for warnings about deprecated features when those warnings >> are intended for end users of applications that are written in Python." >> >> Writing to sys.stderr directly is generally considered poor practice for a >> Python libraries. >> >> In my experience FutureWarning does a good job of satisfying the goals of >> being a "Big Obnoxious Warning" while still being silence-able and testable >> with standard tools. > > Yeah, the reason warnings are normally recommended is because > normally, you want to make it easy to silence. But this is the rare > case where I didn't want to make it easy to silence, so I didn't > suggest using a warning :-). I really doubt anyone is going to silence a FutureWarning and then come complaining that a feature was removed. > > Calling warnings.warn (or the C equivalent) is also very expensive, > even if the warning ultimately isn't displayed. I guess we could do > our own tracking of whether we've displayed the warning yet, and only > even attempt to issue it once, but that partially defeats the purpose > of using warnings in the first place. How about calling it at enable-time once? That?s why I suggested that in the first place. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Best regards, Hameer Abbasi Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Fri Aug 24 19:00:27 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 24 Aug 2018 16:00:27 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Message-ID: On Fri, Aug 24, 2018 at 3:14 PM Nathaniel Smith wrote: > Yeah, the reason warnings are normally recommended is because > normally, you want to make it easy to silence. But this is the rare > case where I didn't want to make it easy to silence, so I didn't > suggest using a warning :-). > > Calling warnings.warn (or the C equivalent) is also very expensive, > even if the warning ultimately isn't displayed. I guess we could do > our own tracking of whether we've displayed the warning yet, and only > even attempt to issue it once, but that partially defeats the purpose > of using warnings in the first place. I thought the suggestion was to issue a warning when np.enable_experimental_array_function() is called. I agree that it's a non-starter to issue it every time an __array_function__ method is called -- warnings are way too slow for that. People can redirect stderr, so we're really not stopping anyone from silencing things by doing it in a non-standard way. We're just making it annoying and non-standard. Developers could even run Python in a subprocess and filter out all the warnings -- there's really nothing we can do to stop determined abusers of this feature. I get that you want to make this annoying and non-standard, but this is too extreme for me. Do you seriously imagine that we'll consider ourselves beholden in the future to users who didn't take us at our word? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Mon Aug 27 13:28:46 2018 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Mon, 27 Aug 2018 10:28:46 -0700 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d Message-ID: Chuck suggested ( https://github.com/numpy/numpy/pull/11805#issuecomment-416069436 ) that we may want to consider deprecating np.ediff1d, which is perhaps not much more useful than np.diff, apart from having some arguably strange prepend / append behavior added in. Related discussion on SO: https://stackoverflow.com/questions/39014324/difference-between-numpy-ediff1d-and-diff Thoughts? Best wishes, Tyler -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Aug 27 13:36:58 2018 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 27 Aug 2018 10:36:58 -0700 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Mon, Aug 27, 2018 at 10:30 AM Tyler Reddy wrote: > Chuck suggested ( > https://github.com/numpy/numpy/pull/11805#issuecomment-416069436 ) that > we may want to consider deprecating np.ediff1d, which is perhaps not much > more useful than np.diff, apart from having some arguably strange prepend / > append behavior added in. > > Related discussion on SO: > https://stackoverflow.com/questions/39014324/difference-between-numpy-ediff1d-and-diff > > Thoughts? > Huh. Never knew this existed. I'd say about 50% of the time I use np.diff(), I'm doing that prepend/append behavior manually (and less readably, possibly inefficiently, but most importantly annoyingly). -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Mon Aug 27 13:38:19 2018 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 27 Aug 2018 19:38:19 +0200 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: <873e95c9-ba2a-41e3-687a-9ccd522e6f6b@ntc.zcu.cz> Hi, it was originally used in other functions of arraysetops (culprit yours truly) which used to use the padding functionality. Nowadays the only use I can see git-grepping the numpy sources is in the tests, so +1 for deprecating. Cheers, r. On 08/27/2018 07:28 PM, Tyler Reddy wrote: > Chuck suggested ( > https://github.com/numpy/numpy/pull/11805#issuecomment-416069436 ) that we > may want to consider deprecating np.ediff1d, which is perhaps not much more > useful than np.diff, apart from having some arguably strange prepend / > append behavior added in. > > Related discussion on SO: > https://stackoverflow.com/questions/39014324/difference-between-numpy-ediff1d-and-diff > > Thoughts? > > Best wishes, > Tyler > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Mon Aug 27 13:46:33 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 27 Aug 2018 11:46:33 -0600 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Mon, Aug 27, 2018 at 11:37 AM Robert Kern wrote: > On Mon, Aug 27, 2018 at 10:30 AM Tyler Reddy > wrote: > >> Chuck suggested ( >> https://github.com/numpy/numpy/pull/11805#issuecomment-416069436 ) that >> we may want to consider deprecating np.ediff1d, which is perhaps not much >> more useful than np.diff, apart from having some arguably strange prepend / >> append behavior added in. >> >> Related discussion on SO: >> https://stackoverflow.com/questions/39014324/difference-between-numpy-ediff1d-and-diff >> >> Thoughts? >> > > Huh. Never knew this existed. I'd say about 50% of the time I use > np.diff(), I'm doing that prepend/append behavior manually (and less > readably, possibly inefficiently, but most importantly annoyingly). > I was thinking we might want to add something to `np.diff`, maybe using `np.pad`. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wieser.eric+numpy at gmail.com Mon Aug 27 13:58:55 2018 From: wieser.eric+numpy at gmail.com (Eric Wieser) Date: Mon, 27 Aug 2018 10:58:55 -0700 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: There is already a patch to add such a feature to np.diff at https://github.com/numpy/numpy/pull/8206 ? On Mon, 27 Aug 2018 at 10:47 Charles R Harris wrote: > On Mon, Aug 27, 2018 at 11:37 AM Robert Kern > wrote: > >> On Mon, Aug 27, 2018 at 10:30 AM Tyler Reddy >> wrote: >> >>> Chuck suggested ( >>> https://github.com/numpy/numpy/pull/11805#issuecomment-416069436 ) that >>> we may want to consider deprecating np.ediff1d, which is perhaps not much >>> more useful than np.diff, apart from having some arguably strange prepend / >>> append behavior added in. >>> >>> Related discussion on SO: >>> https://stackoverflow.com/questions/39014324/difference-between-numpy-ediff1d-and-diff >>> >>> Thoughts? >>> >> >> Huh. Never knew this existed. I'd say about 50% of the time I use >> np.diff(), I'm doing that prepend/append behavior manually (and less >> readably, possibly inefficiently, but most importantly annoyingly). >> > > I was thinking we might want to add something to `np.diff`, maybe using > `np.pad`. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Aug 27 15:28:15 2018 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 27 Aug 2018 15:28:15 -0400 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Mon, Aug 27, 2018 at 1:46 PM, Charles R Harris wrote: > > > On Mon, Aug 27, 2018 at 11:37 AM Robert Kern > wrote: > >> On Mon, Aug 27, 2018 at 10:30 AM Tyler Reddy >> wrote: >> >>> Chuck suggested ( https://github.com/numpy/ >>> numpy/pull/11805#issuecomment-416069436 ) that we may want to consider >>> deprecating np.ediff1d, which is perhaps not much more useful than np.diff, >>> apart from having some arguably strange prepend / append behavior added in. >>> >>> Related discussion on SO: https://stackoverflow.com/ >>> questions/39014324/difference-between-numpy-ediff1d-and-diff >>> >>> Thoughts? >>> >> >> Huh. Never knew this existed. I'd say about 50% of the time I use >> np.diff(), I'm doing that prepend/append behavior manually (and less >> readably, possibly inefficiently, but most importantly annoyingly). >> > same for me never seen, never used > > I was thinking we might want to add something to `np.diff`, maybe using > `np.pad`. > ediff1d avoids one extra temp array compared to simple padding. However, ediff1d pads the final, difference array instead of the original data array. AFAIR, I almost always need padding of the original array. eg as in the standard example of round tripping between cumsum and diff Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Mon Aug 27 22:04:26 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Mon, 27 Aug 2018 19:04:26 -0700 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Mon, Aug 27, 2018 at 10:30 AM Tyler Reddy wrote: > Chuck suggested ( > https://github.com/numpy/numpy/pull/11805#issuecomment-416069436 ) that > we may want to consider deprecating np.ediff1d, which is perhaps not much > more useful than np.diff, apart from having some arguably strange prepend / > append behavior added in. > > Related discussion on SO: > https://stackoverflow.com/questions/39014324/difference-between-numpy-ediff1d-and-diff > > Thoughts? > > Best wishes, > Tyler > I don't think there's much to be gained by dropping edit1d from NumPy. It's really not a maintenance burden to keep it around unchanged. My preference, in keeping with our tradition of not unnecessarily causing disruption, would be to keep this function around but mention that np.diff should be preferred for almost all use cases in the docs. This is "Official discouragement" strategy that came up in the recent discussion about our deprecation policy: https://mail.python.org/pipermail/numpy-discussion/2018-July/078474.html I did a search in Google's codebase and turned up only a handful of uses (~20 uses total) but in a variety of different projects: - It appears in astropy, dask, pandas, pint, scipy and TensorFlow. - It used in six different internal projects Cheers, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Aug 28 11:03:21 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 28 Aug 2018 09:03:21 -0600 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Mon, Aug 27, 2018 at 8:05 PM Stephan Hoyer wrote: > On Mon, Aug 27, 2018 at 10:30 AM Tyler Reddy > wrote: > >> Chuck suggested ( >> https://github.com/numpy/numpy/pull/11805#issuecomment-416069436 ) that >> we may want to consider deprecating np.ediff1d, which is perhaps not much >> more useful than np.diff, apart from having some arguably strange prepend / >> append behavior added in. >> >> Related discussion on SO: >> https://stackoverflow.com/questions/39014324/difference-between-numpy-ediff1d-and-diff >> >> Thoughts? >> >> Best wishes, >> Tyler >> > > I don't think there's much to be gained by dropping edit1d from NumPy. > It's really not a maintenance burden to keep it around unchanged. > > My preference, in keeping with our tradition of not unnecessarily causing > disruption, would be to keep this function around but mention that np.diff > should be preferred for almost all use cases in the docs. This is "Official > discouragement" strategy that came up in the recent discussion about our > deprecation policy: > https://mail.python.org/pipermail/numpy-discussion/2018-July/078474.html > > I did a search in Google's codebase and turned up only a handful of uses > (~20 uses total) but in a variety of different projects: > - It appears in astropy, dask, pandas, pint, scipy and TensorFlow. > - It used in six different internal projects > Maybe we need a "NumpyObsoleteWarning" :) At the least, we should probably have a list of obsolete functions in the documentation somewhere. My main concern is that as we go forward we might end up supporting a bunch of functions that are seldom used and have better replacements. We need some method of pruning. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Aug 28 12:03:24 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 28 Aug 2018 09:03:24 -0700 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Tue, Aug 28, 2018 at 8:04 AM Charles R Harris wrote: > > > On Mon, Aug 27, 2018 at 8:05 PM Stephan Hoyer wrote: > >> On Mon, Aug 27, 2018 at 10:30 AM Tyler Reddy >> wrote: >> >>> Chuck suggested ( >>> https://github.com/numpy/numpy/pull/11805#issuecomment-416069436 ) that >>> we may want to consider deprecating np.ediff1d, which is perhaps not much >>> more useful than np.diff, apart from having some arguably strange prepend / >>> append behavior added in. >>> >>> Related discussion on SO: >>> https://stackoverflow.com/questions/39014324/difference-between-numpy-ediff1d-and-diff >>> >>> Thoughts? >>> >>> Best wishes, >>> Tyler >>> >> >> I don't think there's much to be gained by dropping edit1d from NumPy. >> It's really not a maintenance burden to keep it around unchanged. >> >> My preference, in keeping with our tradition of not unnecessarily causing >> disruption, would be to keep this function around but mention that np.diff >> should be preferred for almost all use cases in the docs. This is "Official >> discouragement" strategy that came up in the recent discussion about our >> deprecation policy: >> https://mail.python.org/pipermail/numpy-discussion/2018-July/078474.html >> >> I did a search in Google's codebase and turned up only a handful of uses >> (~20 uses total) but in a variety of different projects: >> - It appears in astropy, dask, pandas, pint, scipy and TensorFlow. >> - It used in six different internal projects >> > > Maybe we need a "NumpyObsoleteWarning" :) At the least, we should probably > have a list of obsolete functions in the documentation somewhere. My main > concern is that as we go forward we might end up supporting a bunch of > functions that are seldom used and have better replacements. We need some > method of pruning. > Given the list of uses Stephan turned up and Robert saying it's a useful function, I'm -1 on any warning. If np.diff gets the same padding behavior, we can document ediff1d in its document as being superceded with a recommendation to use np.diff instead. In such a docstring warning we could include an easily searchable phrase that we start using for all such functions, but I don't think there's much value in that. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Aug 28 13:38:49 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 28 Aug 2018 10:38:49 -0700 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Tue, Aug 28, 2018 at 9:03 AM Ralf Gommers wrote: > Maybe we need a "NumpyObsoleteWarning" :) At the least, we should probably >> have a list of obsolete functions in the documentation somewhere. My main >> concern is that as we go forward we might end up supporting a bunch of >> functions that are seldom used and have better replacements. We need some >> method of pruning. >> > > Given the list of uses Stephan turned up and Robert saying it's a useful > function, I'm -1 on any warning. If np.diff gets the same padding behavior, > we can document ediff1d in its document as being superceded with a > recommendation to use np.diff instead. > To be clear, I don't think np.ediff1d is particularly useful or necessary, despite these uses. Most of these uses don't even use the optional arguments, so the author was probably simply ignorant of np.diff. This is more or less inevitable for most corners of NumPy's API, given how many users we have. "PendingDeprecationWarning" is Python's built-in warning for signaling that something is obsolete but not deprecated yet. It might be appropriate to use in these cases. The default warning filters silence it for users, so it doesn't show up unless you're very aggressive about enabling all warnings. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Tue Aug 28 14:24:26 2018 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Tue, 28 Aug 2018 20:24:26 +0200 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: In the meantime I'll make a PR to get rid of it from SciPy. We can also signal other libraries to do so. Anything frees up the already-very-crowded namespace of NumPy dot is worth it in my opinion. On Tue, Aug 28, 2018 at 7:40 PM Stephan Hoyer wrote: > > > On Tue, Aug 28, 2018 at 9:03 AM Ralf Gommers > wrote: > >> Maybe we need a "NumpyObsoleteWarning" :) At the least, we should >>> probably have a list of obsolete functions in the documentation somewhere. >>> My main concern is that as we go forward we might end up supporting a bunch >>> of functions that are seldom used and have better replacements. We need >>> some method of pruning. >>> >> >> Given the list of uses Stephan turned up and Robert saying it's a useful >> function, I'm -1 on any warning. If np.diff gets the same padding behavior, >> we can document ediff1d in its document as being superceded with a >> recommendation to use np.diff instead. >> > > To be clear, I don't think np.ediff1d is particularly useful or necessary, > despite these uses. Most of these uses don't even use the optional > arguments, so the author was probably simply ignorant of np.diff. This is > more or less inevitable for most corners of NumPy's API, given how many > users we have. > > "PendingDeprecationWarning" is Python's built-in warning for signaling > that something is obsolete but not deprecated yet. It might be appropriate > to use in these cases. The default warning filters silence it for users, so > it doesn't show up unless you're very aggressive about enabling all > warnings. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Tue Aug 28 15:20:38 2018 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Tue, 28 Aug 2018 15:20:38 -0400 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Mon, Aug 27, 2018 at 10:05 PM Stephan Hoyer wrote: - It appears in astropy, dask, pandas, pint, scipy and TensorFlow. > The only reason it appears in astropy is because of tests that Quantity works correctly with it; we do not actually use it... So that's at least a few hits that do not count as arguments to keep it! I'm in favour of a PendingDeprecationWarning. -- Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Aug 28 15:27:03 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 28 Aug 2018 12:27:03 -0700 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Tue, Aug 28, 2018 at 12:21 PM Marten van Kerkwijk < m.h.vankerkwijk at gmail.com> wrote: > > > On Mon, Aug 27, 2018 at 10:05 PM Stephan Hoyer wrote: > > - It appears in astropy, dask, pandas, pint, scipy and TensorFlow. >> > > The only reason it appears in astropy is because of tests that Quantity > works correctly with it; we do not actually use it... > > So that's at least a few hits that do not count as arguments to keep it! > I'm in favour of a PendingDeprecationWarning. > We should at least first merge the PR that adds the same padding behavior to np.diff before doing this, then such a warning could say to just use that and get unchanged behavior. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Aug 28 17:05:46 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 28 Aug 2018 14:05:46 -0700 Subject: [Numpy-discussion] Possible Deprecation of np.ediff1d In-Reply-To: References: Message-ID: On Tue, Aug 28, 2018 at 12:27 PM Ralf Gommers wrote: > On Tue, Aug 28, 2018 at 12:21 PM Marten van Kerkwijk < > m.h.vankerkwijk at gmail.com> wrote: > >> >> >> On Mon, Aug 27, 2018 at 10:05 PM Stephan Hoyer wrote: >> >> - It appears in astropy, dask, pandas, pint, scipy and TensorFlow. >>> >> >> The only reason it appears in astropy is because of tests that Quantity >> works correctly with it; we do not actually use it... >> >> So that's at least a few hits that do not count as arguments to keep it! >> I'm in favour of a PendingDeprecationWarning. >> > > We should at least first merge the PR that adds the same padding behavior > to np.diff before doing this, then such a warning could say to just use > that and get unchanged behavior. > The proposed behavior for np.diff() is different, but it should solve the same use-cases. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Aug 29 04:37:47 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 29 Aug 2018 01:37:47 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Message-ID: On Fri, Aug 24, 2018 at 4:00 PM, Stephan Hoyer wrote: > On Fri, Aug 24, 2018 at 3:14 PM Nathaniel Smith wrote: >> >> Yeah, the reason warnings are normally recommended is because >> normally, you want to make it easy to silence. But this is the rare >> case where I didn't want to make it easy to silence, so I didn't >> suggest using a warning :-). >> >> Calling warnings.warn (or the C equivalent) is also very expensive, >> even if the warning ultimately isn't displayed. I guess we could do >> our own tracking of whether we've displayed the warning yet, and only >> even attempt to issue it once, but that partially defeats the purpose >> of using warnings in the first place. > > > I thought the suggestion was to issue a warning when > np.enable_experimental_array_function() is called. I agree that it's a > non-starter to issue it every time an __array_function__ method is called -- > warnings are way too slow for that. If our protection against uninformed usage is a Big Obnoxious Warning(tm), then I was imagining that we could simplify by dropping enable_experimental_array_function entirely. Doesn't make a big difference either way though. > People can redirect stderr, so we're really not stopping anyone from > silencing things by doing it in a non-standard way. We're just making it > annoying and non-standard. Developers could even run Python in a subprocess > and filter out all the warnings -- there's really nothing we can do to stop > determined abusers of this feature. > > I get that you want to make this annoying and non-standard, but this is too > extreme for me. Do you seriously imagine that we'll consider ourselves > beholden in the future to users who didn't take us at our word? Let's break that question down into two parts: 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them? 2. is it plausible that we'll find ourselves in that situation? For the first question, I think the answer is... yes? We constantly bend over backwards to try to avoid breaking users. Our deprecation policy explicitly says that it doesn't matter what we say in the docs, the only thing that matters is whether a change breaks users. And to make things more complicated, it's easy to imagine scenarios where the people being broken aren't the ones who had a chance to read the docs ? e.g. if a major package starts relying on __array_function__, then it's all *their* users who we'd be breaking, even though they had nothing to do with it. If any of {tensorflow/astropy/dask/sparse/sklearn/...} did start relying on __array_function__ for normal functionality, then *of course* that would come up in future discussions about changing __array_function__, and *of course* it would make us reluctant to do that. As it should, because breaking users is bad, we should try to avoid ending up in situations where that's what we have to do, even if we have a NEP to point to to justify it. But... maybe it's fine anyway, because this situation will never come up? Obviously I hope that our downstreams are all conscientious, and friendly, and take good care of their users, and would never create a situation like that. I'm sure XArray won't :-). But... people are busy, and distracted, and have pressures to get something shipped, and corners get cut. Companies *care* about what's right, but they mostly only *do* the minimum they have to. (Ask anyone who's tried to get funding for OSS...) Academics *care* about what's right, but they just don't have time to care much. So yeah... if there's a quick way to shut up the warning and make things work (or seem to work, temporarily), versus doing things right by talking to us, then I do think people might take the quick hack. The official Tensorflow wheels flat out lie about being manylinux compatible, and the Tensorflow team has never talked to anyone about how to fix this, they just upload them to PyPI and leave others get to deal with the fallout [1]. That may well be what's best for their users, I don't know. But stuff like this is normal, it happens all the time, and if someone does it with __array_function__ then we have no leverage. -n [1] https://github.com/tensorflow/tensorflow/issues/8802#issuecomment-401703703 -- Nathaniel J. Smith -- https://vorpus.org From matti.picus at gmail.com Wed Aug 29 05:44:30 2018 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 29 Aug 2018 11:44:30 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Message-ID: <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> On 29/08/18 10:37, Nathaniel Smith wrote: > it's easy to imagine scenarios where the > people being broken aren't the ones who had a chance to read the docs > ? e.g. if a major package starts relying on __array_function__, then > it's all*their* users who we'd be breaking, even though they had > nothing to do with it. This is a packaging problem. This proposal is intended for use by other "major packages", not so much for end-users. We would have much more trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages. Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP? Matti From einstein.edison at gmail.com Wed Aug 29 07:00:29 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Wed, 29 Aug 2018 13:00:29 +0200 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> Message-ID: > On 29. Aug 2018, at 11:44, Matti Picus wrote: > > On 29/08/18 10:37, Nathaniel Smith wrote: >> it's easy to imagine scenarios where the >> people being broken aren't the ones who had a chance to read the docs >> ? e.g. if a major package starts relying on __array_function__, then >> it's all*their* users who we'd be breaking, even though they had >> nothing to do with it. > This is a packaging problem. This proposal is intended for use by other "major packages", not so much for end-users. We would have much more trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages. One thing that might help here is nightly or continuous CI builds of the NumPy wheels. This would be good, as we could test it in CI, and fix it when it comes up. But I guess that?s another discussion. Personally, for as long as this protocol is experimental, I?ll add a warning in the docs of sparse as well; saying this might disappear anytime. > > Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP? > Matti > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From mrocklin at gmail.com Wed Aug 29 08:31:04 2018 From: mrocklin at gmail.com (Matthew Rocklin) Date: Wed, 29 Aug 2018 08:31:04 -0400 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> Message-ID: >> 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them? I think that it would be useful for Numpy's continued evolution to develop the ability to include code on a provisional basis. Other projects do this and they just have big bold "Experimental" notes everywhere that a user might go to learn about the functionality (docs, docstrings, blogposts). Some users will definitely get burned, yes, but the alternative is to burn all other users a little bit by moving too slowly and not trying things out. This is different from how Numpy has operated in the past, but that might be ok. >> 2. is it plausible that we'll find ourselves in that situation? > Personally, for as long as this protocol is experimental, I?ll add a warning in the docs of sparse as well; saying this might disappear anytime Yup. Same for Dask. We're pretty accustomed to this. On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi wrote: > > On 29. Aug 2018, at 11:44, Matti Picus wrote: > > > > On 29/08/18 10:37, Nathaniel Smith wrote: > >> it's easy to imagine scenarios where the > >> people being broken aren't the ones who had a chance to read the docs > >> ? e.g. if a major package starts relying on __array_function__, then > >> it's all*their* users who we'd be breaking, even though they had > >> nothing to do with it. > > This is a packaging problem. This proposal is intended for use by other > "major packages", not so much for end-users. We would have much more > trouble if we were proposing a broad change to something like indexing or > the random number module (see those NEPs). If we break one of those major > packages, it is on them to pin the version of NumPy they can work with. In > my opinion very few end users will be implementing their own ndarray > classes with `__array_function__`. While we will get issue reports, we can > handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem > and urge users to complain to those packages. > > One thing that might help here is nightly or continuous CI builds of the > NumPy wheels. This would be good, as we could test it in CI, and fix it > when it comes up. But I guess that?s another discussion. > > Personally, for as long as this protocol is experimental, I?ll add a > warning in the docs of sparse as well; saying this might disappear anytime. > > > > > Other than adding a warning, I am not sure what the concrete proposal is > here. To not accept the NEP? > > Matti > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Wed Aug 29 09:44:23 2018 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Wed, 29 Aug 2018 09:44:23 -0400 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> Message-ID: HI All, On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, i.e., that it will continue to be possible to override, say, concatenate. It is not a big deal if the actual implementation changes (say, `__array_concatenate__` is introduced) and we need to do some version-dependent magic to ensure that our users do not notice the change; we did the same with `__array_ufunc__` - the wrap/prepare machinery will be gone only in the next version of astropy, when our minimum version for numpy reaches 1.13. One thing perhaps worth remembering that what really matters is not so much `__array_function__` but the set of functions that actually implement the override. It would seem unlikely this would be the complete numpy API on the first go; perhaps it is an idea to be somewhat specific about which part we start with? My vote would go towards array manipulation routines like concatenate. I would also suggest to avoid those functions for which ufunc implementations would seem quite possible (i.e., avoid things like median, etc.) All the best, Marten On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin wrote: > >> 1. if we do find ourselves in a situation where changing this would > break lots of users, will we consider ourselves beholden to them? > > I think that it would be useful for Numpy's continued evolution to develop > the ability to include code on a provisional basis. Other projects do this > and they just have big bold "Experimental" notes everywhere that a user > might go to learn about the functionality (docs, docstrings, blogposts). > Some users will definitely get burned, yes, but the alternative is to burn > all other users a little bit by moving too slowly and not trying things out. > > This is different from how Numpy has operated in the past, but that might > be ok. > > >> 2. is it plausible that we'll find ourselves in that situation? > > > Personally, for as long as this protocol is experimental, I?ll add a > warning in the docs of sparse as well; saying this might disappear anytime > > Yup. Same for Dask. We're pretty accustomed to this. > > On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi > wrote: > >> > On 29. Aug 2018, at 11:44, Matti Picus wrote: >> > >> > On 29/08/18 10:37, Nathaniel Smith wrote: >> >> it's easy to imagine scenarios where the >> >> people being broken aren't the ones who had a chance to read the docs >> >> ? e.g. if a major package starts relying on __array_function__, then >> >> it's all*their* users who we'd be breaking, even though they had >> >> nothing to do with it. >> > This is a packaging problem. This proposal is intended for use by other >> "major packages", not so much for end-users. We would have much more >> trouble if we were proposing a broad change to something like indexing or >> the random number module (see those NEPs). If we break one of those major >> packages, it is on them to pin the version of NumPy they can work with. In >> my opinion very few end users will be implementing their own ndarray >> classes with `__array_function__`. While we will get issue reports, we can >> handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem >> and urge users to complain to those packages. >> >> One thing that might help here is nightly or continuous CI builds of the >> NumPy wheels. This would be good, as we could test it in CI, and fix it >> when it comes up. But I guess that?s another discussion. >> >> Personally, for as long as this protocol is experimental, I?ll add a >> warning in the docs of sparse as well; saying this might disappear anytime. >> >> > >> > Other than adding a warning, I am not sure what the concrete proposal >> is here. To not accept the NEP? >> > Matti >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at python.org >> > https://mail.python.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrocklin at gmail.com Wed Aug 29 09:52:59 2018 From: mrocklin at gmail.com (Matthew Rocklin) Date: Wed, 29 Aug 2018 09:52:59 -0400 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> Message-ID: > On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings attached to it. In that case astropy might just wait before adopting it until that label was dropped. Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely. This would give the ecosystem an opportunity to try something and modify it after having some experience without promising to support it in a fully backwards compatible way forever. On Wed, Aug 29, 2018 at 9:46 AM Marten van Kerkwijk < m.h.vankerkwijk at gmail.com> wrote: > HI All, > > On the backwards compatibility: from an astropy perspective, I would > expect that the introduction of `__array_function__` implies a guarantee > that the *functionality* it provides will remain, i.e., that it will > continue to be possible to override, say, concatenate. It is not a big deal > if the actual implementation changes (say, `__array_concatenate__` is > introduced) and we need to do some version-dependent magic to ensure that > our users do not notice the change; we did the same with `__array_ufunc__` > - the wrap/prepare machinery will be gone only in the next version of > astropy, when our minimum version for numpy reaches 1.13. > > One thing perhaps worth remembering that what really matters is not so > much `__array_function__` but the set of functions that actually implement > the override. It would seem unlikely this would be the complete numpy API > on the first go; perhaps it is an idea to be somewhat specific about which > part we start with? My vote would go towards array manipulation routines > like concatenate. I would also suggest to avoid those functions for which > ufunc implementations would seem quite possible (i.e., avoid things like > median, etc.) > > All the best, > > Marten > > On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin > wrote: > >> >> 1. if we do find ourselves in a situation where changing this would >> break lots of users, will we consider ourselves beholden to them? >> >> I think that it would be useful for Numpy's continued evolution to >> develop the ability to include code on a provisional basis. Other projects >> do this and they just have big bold "Experimental" notes everywhere that a >> user might go to learn about the functionality (docs, docstrings, >> blogposts). Some users will definitely get burned, yes, but the >> alternative is to burn all other users a little bit by moving too slowly >> and not trying things out. >> >> This is different from how Numpy has operated in the past, but that might >> be ok. >> >> >> 2. is it plausible that we'll find ourselves in that situation? >> >> > Personally, for as long as this protocol is experimental, I?ll add a >> warning in the docs of sparse as well; saying this might disappear anytime >> >> Yup. Same for Dask. We're pretty accustomed to this. >> >> On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi >> wrote: >> >>> > On 29. Aug 2018, at 11:44, Matti Picus wrote: >>> > >>> > On 29/08/18 10:37, Nathaniel Smith wrote: >>> >> it's easy to imagine scenarios where the >>> >> people being broken aren't the ones who had a chance to read the docs >>> >> ? e.g. if a major package starts relying on __array_function__, then >>> >> it's all*their* users who we'd be breaking, even though they had >>> >> nothing to do with it. >>> > This is a packaging problem. This proposal is intended for use by >>> other "major packages", not so much for end-users. We would have much more >>> trouble if we were proposing a broad change to something like indexing or >>> the random number module (see those NEPs). If we break one of those major >>> packages, it is on them to pin the version of NumPy they can work with. In >>> my opinion very few end users will be implementing their own ndarray >>> classes with `__array_function__`. While we will get issue reports, we can >>> handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem >>> and urge users to complain to those packages. >>> >>> One thing that might help here is nightly or continuous CI builds of the >>> NumPy wheels. This would be good, as we could test it in CI, and fix it >>> when it comes up. But I guess that?s another discussion. >>> >>> Personally, for as long as this protocol is experimental, I?ll add a >>> warning in the docs of sparse as well; saying this might disappear anytime. >>> >>> > >>> > Other than adding a warning, I am not sure what the concrete proposal >>> is here. To not accept the NEP? >>> > Matti >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at python.org >>> > https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Wed Aug 29 10:43:55 2018 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Wed, 29 Aug 2018 10:43:55 -0400 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> Message-ID: On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin wrote: > > On the backwards compatibility: from an astropy perspective, I would > expect that the introduction of `__array_function__` implies a guarantee > that the *functionality* it provides will remain, > > My guess is that you wouldn't have this expectation if Numpy released this > feature with explicit "Experimental" warnings attached to it. In that > case astropy might just wait before adopting it until that label was > dropped. Projects that were more risk-friendly would then take on the > burden of trying it out, working out some kinks, and then hopefully in a > version or two the "Experimental" label would be dropped and astropy would > step in and adopt it more safely. > > Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain. I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that - the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.) -- Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From dallan at bnl.gov Wed Aug 29 11:02:42 2018 From: dallan at bnl.gov (Allan, Daniel) Date: Wed, 29 Aug 2018 15:02:42 +0000 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> , Message-ID: > Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain. That's fair enough, but I think Matt's point still stands. Any given project can weigh the benefits of early adoption against the risks, considering its need for a new feature and its resources available to deal with possible future changes. An "Experimental" designation gives numpy a little space to innovate and lets individual projects opt in. Without that space, numpy has to take on all the burden of continuity instead of sharing that burden with the community of projects that opt in to early adoption. Best, Dan Daniel B. Allan, Ph.D Associate Computational Scientist, Brookhaven National Lab (631) 344-3281 (no voicemail set up) ________________________________ From: NumPy-Discussion on behalf of Marten van Kerkwijk Sent: Wednesday, August 29, 2018 10:43:55 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin > wrote: > On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings attached to it. In that case astropy might just wait before adopting it until that label was dropped. Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely. Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain. I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that - the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.) -- Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Wed Aug 29 12:34:18 2018 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Wed, 29 Aug 2018 12:34:18 -0400 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> Message-ID: Absolutely fine to have to deal with future chances - my main point is that by accepting the NEP, I think numpy is committing to provide some way to override whatever functions __array_function__ is introduced for, i.e., we cannot reasonably go back to not providing any way to override such a function (but by marking the feature experimental, we *can* change the way the override of any given function is implemented). -- Marten p.s. And, yes, do count me in for helping to back up that commitment! -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Aug 29 15:04:10 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 29 Aug 2018 12:04:10 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> Message-ID: On Wed, Aug 29, 2018, 02:44 Matti Picus wrote: > On 29/08/18 10:37, Nathaniel Smith wrote: > > it's easy to imagine scenarios where the > > people being broken aren't the ones who had a chance to read the docs > > ? e.g. if a major package starts relying on __array_function__, then > > it's all*their* users who we'd be breaking, even though they had > > nothing to do with it. > This is a packaging problem. This proposal is intended for use by other > "major packages", not so much for end-users. We would have much more > trouble if we were proposing a broad change to something like indexing > or the random number module (see those NEPs). If we break one of those > major packages, it is on them to pin the version of NumPy they can work > with. In my opinion very few end users will be implementing their own > ndarray classes with `__array_function__`. While we will get issue > reports, we can handle them much as we do the MKL or OpenBLAS ones - > pinpoint the problem and urge users to complain to those packages. > > Other than adding a warning, I am not sure what the concrete proposal is > here. To not accept the NEP? > The proposal is just that while the NEP is considered experimental and provisional, we should use some kind of technical measures to discourage use in a non-experimental settings. We want to stay in control of when it escapes the lab, and docs alone, or trivially disableable messages, aren't a very effective way to do that. -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Wed Aug 29 17:07:58 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Wed, 29 Aug 2018 14:07:58 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> <18371e7e-6ae2-7e11-ec6d-eb862f9c0b58@gmail.com> Message-ID: On Wed, Aug 29, 2018 at 9:34 AM Marten van Kerkwijk < m.h.vankerkwijk at gmail.com> wrote: > Absolutely fine to have to deal with future chances - my main point is > that by accepting the NEP, I think numpy is committing to provide some way > to override whatever functions __array_function__ is introduced for, i.e., > we cannot reasonably go back to not providing any way to override such a > function (but by marking the feature experimental, we *can* change the way > the override of any given function is implemented). > I'm not entirely sure that we're ready to make this commitment yet, but I think this ties into our decision about what sort of user opt-ins to require. I'd like to structure this decision in terms of the types of breaking changes we contemplate possibly making in this future with regards to this protocol: 1. Abandoning this approach of allowing overrides for NumPy's high level API entirely, in favor of requiring the use of a dedicated namespace like numpy.api. This might arise if we decide that the overhead of checking for __array_function__ attributes is too much, or if it's simply too confusing to allow NumPy functions to be overloaded in arbitrary ways. 2. Changing the general __array_function__ protocol in a breaking way, e.g., to eliminate the "types" argument. 3. Replacing __array_function__ with another override mechanism, either (a) in general (e.g., __array_function_v2__ without the "types" argument) or (b) for specific functions (e.g., switching to a more specific protocol like __array_ufunc__). 4. Removing the possibility of overriding specific functions altogether, because we want to require using a more generic interface (e.g., to require overriding np.stack implicitly via np.expand_dims/np.concatenate instead). Possible changes (1) or (2) would be good reasons for requiring end-users to make an intentional choice to enable __array_function__. Either of these changes would break everyone who uses the protocol. Concern about change (1) would be a particularly good reason to require an opt-in, because it's concerned about impact on users who don't use any new functionality. Possible changes (3) or (4) would be annoying to users, but in my mind would not justify requiring an explicit opt-in. Both could be achieved without immediately breaking users by making use of a standard deprecation cycle, i.e., by still calling __array_function__ but issuing a FutureWarning. The main reason for requiring the explicit opt-in would be if we don't want to need to bother with a deprecation cycle. To be clear, with the exception of change (3)(b), I don't think any of these are particularly likely or desirable outcomes. So in my personal opinion, I don't think we need the explicit opt-in. But I'm willing to make to go ahead with requiring the opt-in if that's what it takes to get everyone on board -- as long as we don't make the opt-in too onerous in a way that imposes hard limits on this protocol can be used. (I would be OK with an explicit Python API for enabling this feature that raises a FutureWarning when it's called, without any API for turning it off.) Best, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Wed Aug 29 17:34:00 2018 From: shoyer at gmail.com (Stephan Hoyer) Date: Wed, 29 Aug 2018 14:34:00 -0700 Subject: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol In-Reply-To: References: <56500FA5-94B5-4DCB-84AA-35843D4F5F20@gmail.com> Message-ID: On Wed, Aug 29, 2018 at 1:38 AM Nathaniel Smith wrote: > The official Tensorflow wheels flat out lie about being manylinux > compatible, and the Tensorflow team has never talked to anyone about > how to fix this, they just upload them to PyPI and leave others get to > deal with the fallout [1]. That may well be what's best for their > users, I don't know. But stuff like this is normal, it happens all the > time, and if someone does it with __array_function__ then we have no > leverage. Yes, this is a good example of how library authors sometimes break rules imposed by their dependencies when not enforced by technical means. I'd like to think the TensorFlow team was making an intentional choice here and is willing to live with the consequence of their decision. Unfortunately right now many of those consequences are imposed on others, so this was certainly an inconsiderate choice. I don't know if they were aware of these costs that they are imposing on the rest of the Python ecosystem. I don't see any mention in PEP-513 about the consequences of lying about manylinux compatibility -- even though arguably such warnings should not be necessary. One lesson we might take from this is documentation is essential: we should have more prominent warnings in NEP-18 (given that it's also likely to serve as a form of user-facing docs) about how experimental this protocol is, and about the specific types of changes we anticipate in the future (e.g., see my other email). I also hope that Python packaging maintainers would not hesitate to break this abuse of the manylinux1 tag, e.g., if they wanted to add compatibility checks on wheels when uploaded to PyPI. -------------- next part -------------- An HTML attachment was scrubbed... URL: